Paper Reviewer: Automatically generating comprehensive reviews of arXiv papers and converting them into blog posts

General Introduction

Paper Reviewer is an open source project designed to generate comprehensive reviews from arXiv papers and turn them into blog posts. The project provides support for Hugging Face's Daily Papers website to automatically generate blog posts . Through the use of Python scripts collect.py and convert.py, users can collect paper reviews and convert them into a fixed design template blog posts .

Function List

Features: 1, content processing, you can read the text content, extract the charts, pictures, tables in the paper 2, as long as the paper ID, fully automated processing and generation, support batch processing of the paper 3, support for customization, AI parsing tools, blog templates and so on.

• Generate Comprehensive Review: Generate a detailed review from a given arXiv paper ID.
• Convert to blog posts: convert generated review content to blog posts, following a fixed design template.
• Support for multiple APIs: Optional use of Upstage and Gemini API to extract image and visual information.
• Automate the process: automate the collection and conversion process through scripting, reducing manual intervention.
• Flexible Configuration: Supports a variety of configuration options that users can adjust according to their needs.

Using Help

Installation process

1. Installation of dependencies::
   • Use pip to install the Python dependencies required by the project:

     pip install -r requirements.txt
     

   • Install poppler in order to convert PDF to image:
     • For Ubuntu users, use the following command:

       apt install poppler-utils
       

     • For macOS users, use Homebrew to install:

       brew install poppler
       

2. Setting environment variables::
   • Set GEMINI_API_KEY (required):

     export GEMINI_API_KEY="your_gemini_api_key"
     

   • Optionally set the API key for Upstage and R2:

     export UPSTAGE_API_KEY="your_upstage_api_key"
     export R2_ACCESS_KEY_ID="your_r2_access_key_id"
     export R2_SECRET_ACCESS_KEY="your_r2_secret_access_key"
     export R2_S3_ENDPOINT_URL="your_r2_s3_endpoint_url"
     export R2_DOMAIN_NAME="your_r2_domain_name"
     

Usage Process

1. Collection of papers for review::
   • Run the collect.py script to generate a review of the paper given the arXiv ID:

     python collect.py --arxiv-id "your_arxiv_id" --stop-at-no-html
     

   • If you need to extract image information, you can use the --use-upstage option:

     python collect.py --arxiv-id "your_arxiv_id" --use-upstage
     

2. Converted to a blog post::
   • Run the convert.py script to convert the collected reviews into blog posts:

     python convert.py --arxiv-id "your_arxiv_id" --template "your_template_file"
     

   • If you need to upload images to R2, you can use the --upload-images-r2 option:

     python convert.py --arxiv-id "your_arxiv_id" --upload-images-r2
     

caveat

• Template Customization: Blog posts follow a fixed design template, if you need to customize the design, you need to modify the template file yourself.
• cost control: It is recommended to use the --stop-at-no-html option to minimize costs when processing papers without HTML pages.
• API Usage: Upstage and Gemini API Provides more accurate extraction of image information, but may incur additional costs.

With the above steps, users can easily generate comprehensive reviews from arXiv papers and turn them into blog posts for a wide range of scenarios including academic research and blog writing.