AI Personal Learning
and practical guidance
豆包Marscode1

X-Kit: Twitter Data Crawl and Analyzing X User Data and Tweets

General Introduction

X-Kit is an open source tool designed to crawl and analyze X (formerly Twitter) user data and tweets. Developed by GitHub user xiaoxiunique, the tool is designed to help users automate the process of capturing basic information and tweets about a given X user, as well as to support regular updates to the user's timeline data.X-Kit not only supports localized storage of data, but also leverages GitHub Actions to achieve automated deployment, which makes it extremely convenient for users to use and manage.

X-Kit:Twitter数据抓取和分析X用户数据与推文的工具-1


 

Function List

  • Automatically capture basic information and tweets for specified users
  • Timed update of user timeline data
  • Supports localized data storage
  • GitHub Actions Automated Deployment
  • Daily Tweets feature
  • Bulk Follow Users
  • Data storage by date

 

Using Help

Installation process

  1. Configuring Environment Variables: Create the.envfile, add the following configuration:
   AUTH_TOKEN=你的X认证Token
GET_ID_X_TOKEN=用于获取用户ID的Token
  1. Add users to be tracked: indev-accounts.jsonAdd user information in the
   {
"username": "用户名",
"twitter_url": "用户主页链接",
"description": "用户描述",
"tags": ["标签1", "标签2"]
}
  1. Installation of dependencies: Run the following command to install the dependency:
   bun install

Usage

  1. Get user information: Run the following command to get user information:
   bun run scripts/index.ts
  1. Get the latest tweets: Run the following command to get the latest tweets:
   bun run scripts/fetch-tweets.ts
  1. Bulk Follow Users: Run the following command to follow users in bulk:
   bun run scripts/batch-follow.ts

Automated deployment

X-Kit uses GitHub Actions to automate deployments, and the following are the main automated tasks:

  • get-home-latest-timeline.yml: Get the latest tweets every 30 minutes
  • daily-get-tweet-id.yml: Obtain user information once a day

data storage

  • User information is stored in theaccounts/catalogs
  • The tweet data is stored in thetweets/Catalog, named by date

technology stack

  • Bun: for package management and script running
  • TypeScript: Main Programming Languages
  • Twitter API: for data crawling
  • GitHub Actions: for automated deployment

With these steps, users can easily install and use X-Kit to crawl and analyze X users' data and tweets. Detailed operational procedures and automated deployment features make the tool ideal for users who need to update and manage X data on a regular basis.

May not be reproduced without permission:Chief AI Sharing Circle " X-Kit: Twitter Data Crawl and Analyzing X User Data and Tweets
en_USEnglish