Project Mariner: browser automation, a research prototype exploring the future of human-computer interaction (unpublished)

General Introduction

Project Mariner is a research prototype launched by Google DeepMind to explore the future of human-computer interaction. The project utilizes Gemini 2.0 Powerful Multimodal Understanding and Reasoning Capabilities to Perform a Variety of Tasks through Browser Automation Project Mariner is capable of understanding and navigating complex web sites in real-time, automating tasks in the browser while maintaining user control. The project is currently only available to a small group of trusted beta testers, with the aim of testing and improving its functionality.

Function List

• multimodal understanding: Understand everything on the browser screen, including pixels, text, code, images and forms.
• Browser Interaction: Automatically navigates and operates the site, executing user commands.
• reasoning ability: Ability to understand complex instructions and break them down into executable steps.
• Real-time feedback: Provide visual feedback and updates as tasks are performed.
• safety control: Maintains user control and ensures security while performing tasks.

Using Help

Installation process

1. Download extensions: Visit the official website of Project Mariner to download and install the Chrome extension.
2. Setting up an account: Once the installation is complete, open Chrome, click on the extension icon, follow the prompts to set up your account and sign in.
3. Join the tester queue: Fill in the relevant information to apply to join the trusted tester queue.

Usage Process

1. Launching extensions: After logging in, click on the extension icon in the upper right corner of your browser to launch Project Mariner.
2. input: Enter commands, such as "Create Cart" or "Find Flights" in the pop-up chat window.
3. operate: Project Mariner will automatically navigate to the relevant website and perform the task. The user can view the progress of the task in real time in the browser.
4. Feedback and adjustments: If Project Mariner needs more information or encounters a problem, it will request feedback from the user in a chat window. The user can provide additional instructions or adjust tasks as needed.

Main function operation flow

• Creating a Shopping Cart: Enter your shopping list and Project Mariner will automatically navigate to shopping sites, search for and add items to your cart.
• Find flights and hotels: Enter your travel dates and destination, and Project Mariner will automatically search for and display relevant flight and hotel information.
• Find Recipes: Enter the name of an ingredient or dish and Project Mariner will automatically search for and display the recipe.

Featured Functions

• multimodal understanding: Project Mariner is able to understand and process a variety of elements on the browser screen, including text, images, and forms.
• Real-time feedback: While performing a task, Project Mariner provides real-time visual feedback to ensure that the user is aware of the task's progress.
• safety control: Project Mariner maintains user control while performing tasks, ensuring the safety and accuracy of the task.