Microsoft Rolling Out Copilot Vision in Preview, Can Understand User’s Browsing Activity

Microsoft introduced a new artificial intelligence (AI) capability for Copilot on Thursday. Dubbed Copilot Vision, it now enables the AI chatbot to see and understand the context of what a user does online. The Redmond-based tech giant said that Copilot can access the Microsoft Edge browser. The vision-enabled chatbot can now monitor which websites and web pages the user browses, and then can perform certain commands when prompted. This capability is currently available in preview to a select number of Copilot Pro subscribers in the US.

Microsoft Copilot Vision

The Windows maker has been trying to bring its AI-powered chatbot Copilot to the forefront of the PC experience. Earlier, it unveiled purpose-built AI agents to complete tasks on the PC, and now, it has added another capability to Copilot that will let it answer user queries based on their browser-related activity, as per the blog post. Notably, Copilot Vision is not an agentic function, so the AI cannot take actions on behalf of the user.

Copilot Vision can understand the full context of what a user sees on the Edge browser. It can see the images and elements, read the text, and even comprehend the navigation buttons. The vision-enabled chatbot will be present at the bottom of the screen and users can either type or speak the prompt.

Coming to what the AI can do, it can answer questions, summarise and translate text, and spotlight certain parts of the page when prompted to do so. Microsoft states that Copilot Vision can also act as a game assistant and help a user understand how to play games like GeoGuessr, and offer tips while playing a game of Chess online.

Highlighting examples, the tech giant said Copilot Vision can help users plan an outing at the museum by pointing out the important information a user should know. It can also help users with shopping by recommending the right products based on the user's needs and preferences.

On privacy and security, Microsoft stated that Copilot Vision will be entirely opt-in and users can decide when to turn it on and when to keep it turned off. The AI will not be able to see something that the user does not want to. Additionally, any conversation data with the AI will be deleted at the end of each session. However, the Copilot's responses are logged and they will be used to improve the company's safety systems.

Microsoft further added that Copilot Vision does not capture, store, or use any data from publishers to train its AI models. It is also working with a group of external testers (red teamers) to improve the safety of the new feature.

Copilot Vision is currently rolling out to a limited number of Copilot Pro subscribers in the US and it can be accessed via Copilot Labs. Notably, Vision will currently work only on a select set of websites, and the functionality will be expanded gradually.