X-Kit: Twitter Data Crawl and Analyzing X User Data and Tweets

General Introduction

X-Kit is an open source tool designed to crawl and analyze X (formerly Twitter) user data and tweets. Developed by GitHub user xiaoxiunique, the tool is designed to help users automate the process of capturing basic information and tweets about a given X user, as well as to support regular updates to the user's timeline data.X-Kit not only supports localized storage of data, but also leverages GitHub Actions to achieve automated deployment, which makes it extremely convenient for users to use and manage.

Function List

• Automatically capture basic information and tweets for specified users
• Timed update of user timeline data
• Supports localized data storage
• GitHub Actions Automated Deployment
• Daily Tweets feature
• Bulk Follow Users
• Data storage by date

Using Help

Installation process

1. Configuring Environment Variables: Create the.envfile, add the following configuration:

   AUTH_TOKEN=your X authentication Token
GET_ID_X_TOKEN=Token used to get the user's ID

2. Add users to be tracked: indev-accounts.jsonAdd user information in the

   {
"username": "user_name",
"twitter_url": "Link to user's homepage",
"description": "User description",
"tags": ["tag 1", "tag 2"]
}

3. Installation of dependencies: Run the following command to install the dependency:

   bun install

Usage

1. Get user information: Run the following command to get user information:

   bun run scripts/index.ts

2. Get the latest tweets: Run the following command to get the latest tweets:

   bun run scripts/fetch-tweets.ts

3. Bulk Follow Users: Run the following command to follow users in bulk:

   bun run scripts/batch-follow.ts

Automated deployment

X-Kit uses GitHub Actions to automate deployments, and the following are the main automated tasks:

• get-home-latest-timeline.yml: Get the latest tweets every 30 minutes
• daily-get-tweet-id.yml: Obtain user information once a day

data storage

• User information is stored in theaccounts/catalogs
• The tweet data is stored in thetweets/Catalog, named by date

technology stack

• Bun: for package management and script running
• TypeScript: Main Programming Languages
• Twitter API: for data crawling
• GitHub Actions: for automated deployment

With these steps, users can easily install and use X-Kit to crawl and analyze X users' data and tweets. Detailed operational procedures and automated deployment features make the tool ideal for users who need to update and manage X data on a regular basis.