autoMate: a native tool that combines AI and RPA to automate computer tasks

General Introduction

autoMate is a local automation tool open-sourced by yuruotong1 on GitHub, with AI+RPA (Artificial Intelligence + Robotic Process Automation) as its core feature. It combines the intelligent understanding of large-scale language models with the process execution capabilities of RPA, so that users only need to use natural language to describe the task, such as "organizing desktop files" or "generating a weekly report", you can let the computer automatically complete complex operations. With support for local deployment and data security, autoMate is ideal for individuals or teams that need to efficiently handle repetitive tasks. autoMate breaks the complexity of traditional RPA rule sets and makes automation smarter and easier to use through intuitive AI-driven interactions.

Function List

• AI+RPA Task execution: Combines AI understanding with RPA operations to automate multi-step workflows.
• natural language input: Describe tasks in everyday language, without programming or rule configuration.
• Screen Content Analysis: Intelligently recognizes interface elements and performs actions such as clicking and typing.
• local operation mode: Runs offline throughout to ensure data privacy.
• full interface adaptation: Support the operation of any software or web page.
• Multi-model support: Compatible with mainstream large language models, flexible configuration.
• One-Click Deployment: Simplify the installation process and adapt to the Chinese environment.

Using Help

Installation process

The autoMate installation process is optimized for AI+RPA functionality to ensure that users get started quickly. Below are the detailed steps:

1. environmental preparation::
   • Install Git: Download it from the Git website.
   • To install Miniconda: visit the official Conda website.
   • Verify that your system supports Python 3.12.
2. cloning project::
   • Runs in the terminal:

     git clone https://github.com/yuruotong1/autoMate.git
     

   • Go to the catalog:

     cd autoMate
     

3. Creating the Environment::
   • Create and activate a Conda environment:

     conda create -n "automate" python=3.12
     conda activate automate
     

4. Installation of dependencies::
   • Implementation:

     pip install -r requirements.txt
     

   • If prompted for version conflicts, update the pip:

     pip install --upgrade pip
     

5. starter kit::
   • Running:

     python main.py
     

   • browser access http://localhost:7888/ Configure the model and API key (if an online model is required).

Installation Precautions

• GPU Support: If you are using AI inference acceleration, you need to make sure that PyTorch and CUDA versions match, see the PyTorch website.
• Chinese Compatible: Chinese paths are supported without additional adjustments.

How to use

autoMate's AI+RPA feature operates through a browser interface, and the following is a detailed guide:

1. AI+RPA mandate implementation

• procedure::
  1. show (a ticket) http://localhost:7888/The
  2. Enter a task in the input box, such as "Open Excel, create a table and fill in yesterday's sales data".
  3. Click "Submit", the AI parses the command, and the RPA performs the operation.
• Functional Description::
  • The AI is responsible for understanding the complex requirements, and the RPA accomplishes the interface operations, such as clicking buttons and entering data.
  • Support for sequential tasks, such as "save form and send email".
• Usage Scenarios:: Automatic processing of financial statements, batch updating of databases.

2. Natural language input

• procedure::
  1. Enter a command such as "Open your browser and search for 'AI+RPA case'".
  2. The tool automatically launches the browser and searches.
• Functional Description::
  • No need to write scripts, AI translates the language directly into action steps.
  • Supports fuzzy descriptions, such as "Organize recent files".
• finesse: Describe in simple language to avoid ambiguity.

3. Screen content parsing

• procedure::
  1. Enable "Screen Recognition" mode.
  2. Type "Click the 'Save' button" and the tool recognizes and operates.
• Functional Description::
  • AI analyzes screen elements and RPA performs precise clicks or inputs.
  • Can handle dynamic interfaces such as pop-ups or web pages.
• Usage Scenarios: Batch turn off notifications, auto-login software.

4. Local operating mode

• procedure::
  • The default is local mode, no configuration is required.
  • Data is stored locally autoMate.db Documentation.
• Functional Description::
  • Both AI and RPA run locally, no need to upload data.
  • Suitable for sensitive data processing scenarios.

5. Full interface adaptation

• procedure::
  1. Type "Open Word, type 'meeting minutes' and save".
  2. The tool adapts to the Word interface and executes.
• Functional Description::
  • Supports cross software operations such as copying from Excel to Word.
  • Does not rely on specific APIs and is based on visual recognition.

6. Multi-model support

• procedure::
  1. Select the model (e.g. local or online model) on the Settings page.
  2. Enter the API key (if using an online model).
  3. Save and reboot.
• Support List: For details, see SUPPORT_MODEL.mdThe

Example of operation process

Example: Automatically organizing order data

1. Enter: "Open the order management software, export the list of yesterday's orders and save it to your desktop".
2. AI Understanding the Requirements, RPA Launch the software, click Export and Save.
3. Continue typing, "Open Excel, import the order data and sort it by amount".
4. The tool imports the file and completes the sorting.

• in the end: The entire process is automated, from export to organization, in just a few minutes.

common problems

• fail to activate: Check Python version (3.12 required) and dependency integrity.
• AI Understanding Error: Simplify instructions or adjust model parameters.
• RPA Operational jams: Ensure that system resources are sufficient, or switch CPU modes.

Recommendations for use

• initial use: Test simple tasks such as "Open Notepad".
• power user: Exploring the potential of AI+RPA in conjunction with multi-step instructions.
• adjust components during testing: Check the root directory logs to troubleshoot the problem.