AI Personal Learning
and practical guidance

MediaCrawler: Multi-social media platform content, video comment crawler tool

General Introduction

MediaCrawler is a social media content crawler tool designed for developers. By providing a powerful crawler function, it can quickly grab videos, images, comments, likes, retweets and other data from social platforms such as Xiaohongshu, Jieyin, Shutterbugs, B-station, Weibo and so on. This tool uses Playwright as a bridge, preserving the browser environment after login, and obtaining encrypted parameters by executing JS expressions, thus simplifying the difficulty of complex reverse engineering.

For professional use only, please note that data collection needs to be done within the scope of authorization.

MediaCrawler: Multi-social media platform content, video comment crawler tool

 


 

Function List

Support platforms such as Xiaohongshu, Jieyin, Shutterbug, B Station, Weibo, etc.
Provide cookie login, QR code login, cell phone number login and other methods
Support keyword search and specified video/post ID crawling function
Login state caching and IP proxy pool support
Provide slider CAPTCHA solutions (some platforms)

 

flat-roofed building Keyword Search Specify post ID to crawl Secondary comments Designated Creator Home Page Login State Cache IP proxy pool Generate comment word clouds
Little Red Book (social networking website)
jitterbug
violin
Station B
microblog
electronic message board

 

 

Using Help

Create and activate a Python virtual environment
Install the dependencies: Use the `pip install -r requirements.txt` command.
To install the Playwright browser driver: Use the `playwright install` command.
To run the crawler: use a command line argument such as `python main.py --platform xhs --lt qrcode --type search`.
Use `python main.py --help` to see examples of crawlers for other platforms.
Check out the project code structure and answer more questions on the GitHub repository.

 

 

Learning Materials

Chief AI Sharing CircleThis content has been hidden by the author, please enter the verification code to view the content
Captcha:
Please pay attention to this site WeChat public number, reply "CAPTCHA, a type of challenge-response test (computing)", get the verification code. Search in WeChat for "Chief AI Sharing Circle"or"Looks-AI"or WeChat scanning the right side of the QR code can be concerned about this site WeChat public number.

AI Easy Learning

The layman's guide to getting started with AI

Help you learn how to utilize AI tools at a low cost and from a zero base.AI, like office software, is an essential skill for everyone. Mastering AI will give you an edge in your job search and half the effort in your future work and studies.

View Details>
May not be reproduced without permission:Chief AI Sharing Circle " MediaCrawler: Multi-social media platform content, video comment crawler tool

Chief AI Sharing Circle

Chief AI Sharing Circle specializes in AI learning, providing comprehensive AI learning content, AI tools and hands-on guidance. Our goal is to help users master AI technology and explore the unlimited potential of AI together through high-quality content and practical experience sharing. Whether you are an AI beginner or a senior expert, this is the ideal place for you to gain knowledge, improve your skills and realize innovation.

Contact Us
en_USEnglish