DroidRun: an open source tool for AI to automate Android phones

General Introduction

DroidRun is an open-source tool that lets AI operate an Android phone like a human. It helps AI automate tasks such as opening apps, sending messages, or browsing the web by extracting interactive elements such as on-screen buttons and input boxes.DroidRun combines visual parsing and UI structure analysis to provide accurate and stable operation. DroidRun combines visual parsing and UI structure analysis for accurate and stable operation. It supports users to connect their own language models and is compatible with a wide range of Android systems. Whether you're a developer building an AI assistant or a casual user simplifying the operation of your phone, DroidRun is quick to get started and ready to use right out of the box.

Function List

• Extracting screen interaction elements, AI can recognize buttons, text boxes, etc.
• Supports AI auto-tap, swipe, and input to simulate human operation.
• Combine visual parsing and UI extraction to adapt to complex interfaces.
• Intelligent error handling and automatic recovery after task interruption.
• Supports connectivity to user-selected language models with high flexibility.
• Record user actions to support repeated execution of the same task.
• Compatible with Android 6.0 and above, covering a wide range of devices.
• Provides open source framework with customizable features for developers (coming soon).

Using Help

Installation process

1. Visit the official website: Open your browser and type https://www.droidrun.ai/The
2. Download Apps: On the official homepage, click "Download Android Portal APK" to download. droidrun-portal.apk file, which is about 10MB in size.
3. Enabling Privileges: In "Settings > Security" on your Android phone, turn on "Allow installation of apps from unknown sources". The path may be slightly different for different phones.
4. Install APK: Open the file manager, find the downloaded APK file and click Install. The installation process takes about 1 minute.
5. Delegation of authority: When you open DroidRun for the first time, you will be prompted to enable "Accessibility Services" and "Screen Capture". Click on the prompt to enter the settings page to enable the relevant permissions.
6. connectivity model: Inside the app, enter your language model API key (Gemini, ChatGPT, Grok, etc. are supported). Click "Verify" to confirm a successful connection.

How to use

At its core, DroidRun lets AI operate the phone instead of the user. It performs tasks automatically by analyzing the screen content. Below is a detailed step-by-step guide to the main features:

Function 1: Automatic execution of cell phone tasks

• Creating Tasks: Open DroidRun and click on "New Task". In the input box, describe the task, e.g. "Open the calendar and add a meeting for tomorrow at 10am". The more specific the description, the more accurate the AI execution will be.
• Select Model: Go to Settings > Language Models and select the connected model. Quick Models (e.g. Gemini Flash) for simple tasks and complex models (e.g. GPT-4) for multi-step operations.
• Running TasksDroidRun captures the screen in real time and recognizes interface elements. A blue highlight appears on the screen to mark the area where the AI is currently operating. For example, tap the Calendar icon or enter a meeting title.
• Inspection results: When the task is completed, click "Log" to view the execution details. The log will record each action, such as "Clicked calendar icon successfully" or "Failed to enter text". If it fails, you can adjust the description and retry.

Feature 2: Visual and UI Integration Analysis

• Enabling Functions: In Settings > Parsing Mode, select Visual + UI Extraction. This allows DroidRun to analyze both screenshots and system UI data.
• Complexity of operation interfaceTake a shopping app for example, type in "Open Jingdong, search for Bluetooth headphones" and the AI will automatically find the search box, enter text and click the search button. Even if there are advertisements in the interface, AI can still recognize the target elements.
• Optimized RecognitionIf the AI misses, go to Settings > Recognition Sensitivity and adjust the slider. Higher values favor visual parsing, lower values prioritize UI data. Recommended default value is 50.
• Save Template: For frequently used tasks, click "Save Task". Next time, just select the template and AI will execute it directly without having to re-enter it.

Function 3: Intelligent Error Handling

• spontaneous recovery: If you encounter a pop-up window (e.g. an advertisement) during a task, DroidRun will try to close it and continue. For example, when running "Open Browser", AI will automatically click "Allow" if there is a permission prompt.
• manual interventionIf the task is stuck, click "Pause". After manually adjusting the phone to the correct screen, click "Resume". The AI will continue from the current state.
• error log: After each failure, check the "log" to find out the reason, e.g. "Search box not found". Optimize the task description or interface settings according to the prompts.

Function 4: Linking Custom Language Models

• Adding Models: In "Settings > Language Models", enter the API key; DroidRun supports multiple models, so there is no need for complicated configuration, just paste the key.
• test connection: Click on "Test" to make sure the model is responding correctly. Once the test passes, the model will appear in the task selection list.
• Switching Models: Different models can be used for different tasks. For example, it is possible to use Grok Handle simple tasks and use GPT-4 for complex logic.

Upcoming Features

According to the official website, the following features are under development and are expected to go live soon:

• Element tracking: Record the elements clicked by the user and AI can repeat the action exactly. Ideal for batch tasks such as bulk likes or comments.
• voucher management: Support for adding account passwords or double authentication (2FA). For example, AI can automatically log into social apps and enter dynamic verification codes.
• Multi-system management: Supports simultaneous control of multiple Android devices, suitable for enterprise-level automation or testing scenarios.

caveat

• System compatibility: DroidRun supports Android 6.0 and above. Some new features (such as element tracking) may require Android 10 or higher.
• network requirement: A stable Wi-Fi or 4G network is required to connect to the language model. Offline mode only supports saved local tasks.
• Permission Maintenance: Some phones (e.g. Huawei) may have accessibility features disabled in power saving mode. Periodically check if "Settings > Accessibility" is enabled.
• power consumption: Running tasks for long periods of time increases power consumption. It is recommended that the battery level be higher than 30% or that the charger be connected.

Developer Support

DroidRun will soon be available as an open source framework (MIT license), with the full code available to developers via GitHub. The official documentation (Docs) and cloud services (Cloud) is also in the pipeline, with API interfaces and deployment guides available at that time. Developers can customize AI behavior, such as integrating specific apps or developing proprietary assistants.

With these steps, users can quickly master DroidRun, which provides flexible automation support for both routine tasks and complex development.

application scenario

1. Simplify daily operations
   Users can automate repetitive tasks with DroidRun. For example, automatically open the news app every day to browse headlines or send regular messages in social groups. the AI will accurately recognize the interface and reduce manual taps.
2. Social Media Management
   Self-media practitioners can operate in batch. For example, automatically log in to Xiaohongshu, upload 10 images and post notes. the AI will execute them in sequence to ensure that the content is posted without errors.
3. Application Test Development
   Developers can use DroidRun to test application interfaces. For example, simulate users searching and ordering in an e-commerce application, record the response time of each step, and discover potential bugs.
4. Data extraction and analysis
   The researcher can automatically capture the application data. For example, open the Maps app, search for a nearby restaurant, and take a screenshot of the results.The AI saves the data for subsequent analysis.
5. Complex task scheduling
   Business users can design multi-step processes. For example, AI opens the mail app, downloads the attachment, uploads it to a cloud drive, and then sends a confirmation message, which is suitable for cross-application collaboration.

QA

1. What Android devices does DroidRun support?
   It is compatible with Android 6.0 and above, covering most phones and tablets. Android 10 or above is recommended for the best experience.
2. Do I need root privileges?
   No. DroidRun uses a helper function service and does not need to be rooted, just installed and ready to use.
3. Is it possible to run tasks offline?
   Locally saved tasks can be run offline, but connecting to the language model requires an internet connection. It is recommended to save frequently used tasks in advance.
4. How to choose the right language model?
   Choose a fast model (e.g. Gemini Flash) for simple tasks and an advanced model (e.g. GPT-4) for complex tasks. The official website will update the recommended list.
5. How is mission failure handled?
   Check the Log to confirm the reason for failure, e.g. "Button not found". Adjust the description or sensitivity and retry, or manually intervene and continue.
6. When will the open source version be available?
   According to the official website, the GitHub repository will be available soon, the exact date to be announced. Join the Discord community to get the latest updates.