Gemini Screen Automation Feature to Control Apps on Android Phones Spotted in Development

Google has been working to improve Gemini's agentic capabilities, allowing the AI to perform actions on users' behalf. The tech giant now seems to be gearing up to enable Gemini to take direct actions for users on Android. New codes spotted in the latest Google app beta offer a glimpse into the upcoming functionality. The code strings indicate that Gemini could soon let users place online orders and book rides without requiring direct interaction with the phone. The feature could expand to support additional actions over time.

Google May Let Gemini Perform Tasks in Apps on Your Behalf

During an APK teardown of the Google app version 17.4 beta, 9to5Google discovered new Gemini agentic features that are currently in development. The Labs feature, known as “Get tasks done with Gemini,” is reportedly shown with strings explaining that “Gemini can help with tasks, like placing orders or booking rides, using screen automation on certain apps on your device.”

The strings in the Google app reportedly reveal the feature's internal codename, “bonobo". It may be referred to as “screen automation” on Android devices. This feature is expected to require Android 16 QPR3, and it could be available in “certain apps.”

Google will reportedly caution users that “Gemini can make mistakes”. The company reportedly warns that users remain responsible for actions carried out on their behalf. Users will be able to stop the agent at any time and take over a task manually.

The code strings also reveal that Google will alert users about potential privacy implications, as screenshots are reviewed by trained reviewers and used to improve Google services, while Keep Activity is enabled. Google also warns users not to enter payment information into Gemini chats. It also advises users against using screen automation for emergencies or tasks involving sensitive information.

Google has been bringing more abilities to the Gemini recently. Last month, the company expanded the conversational Gemini support in Google Maps to more navigation modes. This integration allows walkers and cyclists to get hands-free assistance on their route. It also enables users to finish a few tasks as sending text messages.