The mouse pointer has been a constant companion on computer screens, across every website, document and workflow. Despite advances in technology, the pointer has barely changed in over 50 years. We’re developing new AI-powered features to help it not only recognize what it’s pointing at, but also understand why it matters to the user. Our aim is to solve a common pain point: traditional AI tools live in separate windows, forcing users to manually pull their content into them. We want the opposite: an intuitive AI that seamlessly integrates into every tool users rely on, without disrupting their workflow. For instance, picture yourself pointing at a photo of a building and saying, „Show me directions.“ When an AI system already grasps the context, nothing more is required. Today, we’re sharing the core principles shaping our vision for future user interfaces, along with experimental demos of a Gemini-powered AI pointer. For instance, you can go to Google AI Studio to edit images or search for locations on a map simply by pointing and speaking. This video demonstrates the experimental setup for our AI-powered pointer. Sequences are abbreviated throughout. Our principles of interaction. We’ve created four principles that collectively transfer the burden of expressing context and intent from the user to the computer, replacing lengthy text prompts with simpler, more intuitive interactions.
Google DeepMind News