Browse AI allows you to extract and monitor data from any web page in minutes without code. The bots you train to operate websites just like humans do will allow you to extract the most accurate data with the least amount of effort. This guide introduces some key concepts to help you get started. ...
1) Robotics
You can train amechanical personto automate a series of steps on your website that you would normally do manually.
For example, a robot can perform all of the following behaviors while performing a task:
- Open the page.
- make landfall (of typhoon etc)
- click on a button
- fill out a form
- Select from the drop-down menu
- Extracting Structured Data from Web Pages to Spreadsheets
- Click on the "Next" or "Load More" button for more data.
- capture an image displayed on a computer screen
- Monitor visual or content changes to specific parts of a web page
The bot has dynamic [input parameters] that allow you to adjust the URL of the web page aka [source URL] or text input on each run. This allows you to use the same bot to extract or monitor data on an unlimited number of pages of a website with a similar layout.
Most people create robots that simply open a web page and extract data from it. Users can use [batch run] on tens of thousands of similar pages on a website to quickly extract the entire data set.
Bots also do a lot of things in the background that you might not notice, such as solving CAPTCHAs, using geo-located residential IP addresses, mimicking human behavior to avoid detection, automatically adapting to site changes and essentially self-maintaining.
2) Prefabricated Robots vs. Customized Robots
Robots can be created by using [Premade Robots], or by using the Browse AI recorder and its click-to-extract interface. Each bot has some [Input Parameters] (e.g. web address) that you can adjust each time you run it.
Pre-built bots are applied to popular use cases and new ones are released every week. Examples are extracting data from Yelp, TripAdvisor or LinkedIn company websites.
More than 901 TP3T of Browse AI users create bots that are custom bots trained for specific use cases. For example, some realtors monitor [watch] building permits issued by their county government (on the county's official website) and connect it to a sales CRM or spreadsheet that automatically sends out emails to every builder who gets a building permit.
3) Source URL
Each [custom robot] has a source URL [input parameter], which by default points to the link they were trained on. You can extract or monitor data from any other page on that site with a similar layout by tweaking the source URL.
For example, if you wanted to [monitor] Walmart's product prices, you could train a custom bot on a Walmart product page and then adjust the source URL for each monitor to configure that bot to monitor 100 different product pages.
4) Tasks
Each [bot] is trained to perform a specific task. Each time you run that bot, it performs that task, and the details of the task (including the data that was extracted), are stored under that task under the bot's History tab.
If you set up a monitoring bot to [monitor] web page changes on a daily basis, it may need to run one task per day, or about 30 tasks per month.
New tasks can be created in several different ways:
- You can open a bot on your dashboard, go to its Run Tasks tab, and run a task.
- In the Run Tasks tab, you can also select [Batch Run] to run up to 50,000 tasks at a time by uploading CSV files.
- If you configure a monitor, a new monitoring task is automatically created each time you need to perform a monitoring check.
- If you integrate Browse AI with another software or use the API, new tasks can be created through the API.
- The system sometimes creates tasks to ensure robot health or to optimize the robot to be faster or more reliable. Tasks will be marked as "run by system".
5) Monitor
One of the most useful features of Browse AI is the built-in monitoring system.
Each robot can be configured with an infinite number of monitors, each of which corresponds to a page on the website or a search condition that needs to be monitored.
For example, you can use a bot to monitor all products on an e-commerce site and receive notifications when prices change or a product becomes available.
The monitor can detect changes and can be configured to send an email notification when a change is detected or automatically send the data to another piece of software.
6) Input parameters
Each bot is equipped with input parameters that can be adjusted and monitored according to task requirements, which means you don't need to recreate bots for every page or search term on your website.
The most common input parameter is [source URL], which is the page that the robot visits for the first time.
When you customize your robot, if you enter any text, what you enter will become an input parameter that can be adjusted later.
7) Batch execution
The Batch Execution feature is available in the Execute Tasks tab of the Robot Control Panel. This feature allows you to upload a CSV file containing up to 50,000 different sets of [Input Parameters] and immediately create a [Task] for each set of parameters. The tasks will be queued for processing and upon completion, you will be provided with the full data set extracted.
For example, you can upload a CSV file containing links to 50,000 company pages on LinkedIn and get all the data extracted from those pages.
8) Deep Crawl
Deep Crawl is a commonly used term that refers to crawling a list of links from a page in a particular category and then further extracting the details of each link.
This approach allows you to gather in-depth, detailed data from pages or sections within your website.
Learn more about using Browse AI Perform a deep crawl of [related article].
9) Integrated Applications
Often, you need to transfer the data captured on your website to other software you use, such as Google Forms or CRM.Browse AI offers over 5,000 integrations to make it easy for you to import data from any website into the tools you already use.
10) Google Forms Integration with Airtable
There are already direct integrations for [Google Forms] and Airtable. Once configured on the robot, each time the robot performs a task, the extracted data is inserted directly into your spreadsheet.
Browse AI's Google Forms Plugin
This plugin provides additional functionality for Google Forms:
- The robot can be run directly in Google Forms by highlighting the set of input parameters and clicking the button.
- Automatically deleting old data from Google Forms.
- Automatically remove duplicates from Google Sheets.
11) Connector Integration (Zapier, Make, Pabbly)
With a few clicks of third-party integration software, these native integration methods will enable you to connect Browse AI with 5000+ other apps:
- [Zapier] Easy to operate, but costly when used in large quantities.
- [Make] costs less, but is more complicated to use.
- [Pabbly Connect] Often used by users who have purchased a one-time paid lifetime agreement to save money.
12) API & Webhooks
If you have software developers on your team, make sure to utilize APIs and webhooks, which allow you to do almost everything on the control panel (except create new bots) and programmatically interface with publicly available APIs and webhooks.
Some startups have built their software based on Browse AI's APIs and outsourced all the labor-intensive data crawling.
