Document Extraction and Cleaning

Comprehensive Introduction OneFileLLM is an open source command line tool designed to consolidate multiple data sources into a single text file for easy input into Large Language Models (LLMs). It supports processing GitHub repositories, ArXiv papers, YouTube video transcriptions, web...

General Introduction Chatlog is an open source tool that focuses on extracting and querying chat logs from WeChat's local database. It supports WeChat versions 3.x and 4.0, covering both Windows and macOS systems. Users can use the command line, terminal interface or H...

Comprehensive Introduction Versatile OCR Program is an open source Optical Character Recognition (OCR) tool designed specifically for working with complex academic and educational documents. It can extract text, tables, mathematical formulas, charts and diagrams from PDFs, images and other documents and generate...

General Introduction DevDocs is a completely free open source tool developed by the CyberAGI team and hosted on GitHub. Designed for programmers and software developers, it starts with the URL of a technical document, automatically crawls the relevant pages and organizes them into a concise Ma...

Comprehensive Introduction It can automatically analyze the layout of PDF documents, identify text, titles, images, tables, formulas and other elements in the page, and determine their correct order. The tool supports OCR functionality and can convert scanned PDF to searchable text. It runs on Docker and provides two models...

General Introduction serverless-markdown-convertor is a free and open source tool, based on Cloudflare Worker and Workers AI, that converts a wide range of files to Markdow...

General Introduction GPT-Crawler is an open source tool developed by the BuilderIO team and hosted on GitHub. It crawls page content by inputting one or more website URLs, generating structured knowledge files (output.jso...

General Introduction pure.md is a tool for AI agents and developers that focuses on quickly converting web content or files to Markdown format. It bypasses anti-crawler restrictions through proxy services, extracts the core data of a web page, and outputs a concise Markdown ...

General Introduction Cloudsquid is a company founded in 2023 in Berlin, Germany, focused on simplifying document processing with artificial intelligence. Its core product is an online data extraction platform that allows users to simply upload documents such as PDFs, images, audio, video, etc. and simply state that they need to extract...

General Introduction PDF Craft is an open source tool designed for scanning PDFs of books and converting them to Markdown format. It was developed by oomol-lab and is hosted on GitHub for users who like to organize their eBooks. The tool works through this ...

Comprehensive Introduction Supametas.AI is a data processing platform that specializes in organizing web pages, documents, audio and video, and other messy information into structured data that AI can use. It supports collecting data from multiple sources, including web links, APIs, local files, etc., and then outputting it as JSON ...

General Introduction MarkPDFDown is an open source tool. It utilizes the Multimodal Large Language Model to convert PDF files into Markdown format. The developer is GitHub user jorben. the goal of this tool is simple: to make PDF documents ...

Comprehensive Introduction SmolDocling is a Visual Language Model (VLM) developed by the ds4sd team in collaboration with IBM, built on SmolVLM-256M and hosted on the Hugging Face platform. It is small in size, only ...

The goal of table recognition is to parse tables in images, accurately identify table structures and cell locations, and reduce them to structured table formats (e.g., HTML). In today's information age, a large amount of important tabular data still exists in an unstructured state (e.g., scanned documents with pictures of statistical tables...).

In the long history of human civilization, every leap in the way information is acquired and parsed has profoundly driven social progress. From the ancient hieroglyphics, to the portable papyrus, to the later emergence of the printing press and today's wave of digitization, each technological innovation has greatly expanded the paradigm of human knowledge dissemination...

General Introduction Firecrawl MCP Server is an open source tool developed by MendableAI, based on the Model Context Protocol (MCP) protocol implementation, with Firecrawl A...

General Introduction olmOCR is an open source tool developed by the AllenNLP team at the Allen Institute for Artificial Intelligence (AI2) that focuses on converting PDF files...

General Introduction par_scrape is a Python-based open source web crawler tool, launched on GitHub by developer Paul Robello, designed to help users intelligently extract data from web pages. It integrates Selenium...

Comprehensive introduction PDF-Extract-Kit is an open source project developed by the OpenDataLab team , focusing on the efficient extraction of high-quality content from complex and diverse PDF documents . It integrates advanced document parsing technology to support layout detection , formula recognition ...

Comprehensive Introduction Crawl4LLM is an open source project jointly developed by Tsinghua University and Carnegie Mellon University, focusing on optimizing the efficiency of web crawling for pre-training of large models (LLM). It significantly reduces ineffective crawling by intelligently selecting high-quality web page data, claiming to be able to originally need to crawl 1...

General Introduction Markdownify MCP Server is an open source tool based on the Model Context Protocol, hosted on GitHub by developer Zach Caceres ...

General Introduction CodeWeaver is a command-line tool designed to weave code libraries into single, easy-to-navigate Markdown documents. It generates a structured representation of a project's file hierarchy by recursively scanning directories and embedding the contents of each file in code blocks. This tool ...

General Introduction Kreuzberg is a library for simplifying text extraction from PDF files, designed to provide a simple, hassle-free text extraction solution. The library is particularly suitable for RAG (Retrieval-Augmented Generatio...

Comprehensive Introduction Instructor is a popular Python library designed for processing structured output from Large Language Models (LLMs). Built on Pydantic, it provides a simple, transparent and user-friendly API for managing data...

Comprehensive Introduction zChunk is a novel chunking strategy developed by ZeroEntropy that aims to provide a solution for generic semantic chunking. The strategy is based on the Llama-70B model, which optimizes the chunking process of documents by prompting for chunks to be generated, ensuring that information retrieval is maintained at a high...

Comprehensive Introduction Pulse is an intelligent platform focused on document processing and data extraction, designed to help organizations and developers efficiently parse and process a wide range of complex documents. Through its advanced computer vision and multimodal processing technology, Pulse is able to accurately extract data from text, images, tables, and many other...

General Introduction Rowfill is an open source document processing platform designed for knowledge workers. It uses advanced artificial intelligence techniques to extract, analyze and process data from complex documents, images and PDFs.Rowfill supports Native Large Language Model (LLM) and Ope...

General Introduction PPTX2MD is an open source tool designed to convert PowerPoint PPTX files to Markdown format. Developed by GitHub user ssine, the tool supports preserving headings, lists, text formatting (e.g., bold, italic, color, and super...

General Introduction Repomix (formerly known as Repopack) is an open source tool designed to package an entire codebase into a single, AI-friendly file. This tool allows developers to easily make their codebase available to large language models (such as Claude, Chat...

General Introduction Yek is a fast Rust-based tool for reading text files from repositories or directories, chunking them, and serializing them for use in Large Language Models (LLMs). The tool uses the .gitignore rule by default to skip unwanted files and utilizes...

Comprehensive Introduction LlamaParse is a powerful document parsing tool that can process complex documents such as PDF, PowerPoint, Word documents and spreadsheets and convert them into structured data.LlamaParse offers a variety of ways to use...

Comprehensive Introduction UnDatas.IO is a platform focused on parsing and processing unstructured data. It utilizes advanced technology to automatically recognize document layouts and categorize tables, images, formulas and text, greatly simplifying the data processing process. The platform not only saves a lot of time in organizing data...

Comprehensive introduction Zerox is an open source project designed to convert PDF, DOCX, images and other documents to Markdown format through visual modeling. The project is developed by getomni-ai team , provides a simple and efficient OCR (Optical Character Recognition) solution.Ze...

Comprehensive Introduction SemHash is a lightweight and flexible tool for de-duplicating datasets by semantic similarity. It combines the fast embedding generation of Model2Vec with the efficient ANN (approximate nearest neighbor) similarity search of Vicinity.SemHa...

General Introduction Parseur is a leading AI data extraction software designed to help users automatically extract text data from PDFs, emails and other documents. With Parseur, users can easily convert unstructured data into structured data and send it to various applications...

Comprehensive Introduction Weco AI Functions is a powerful platform designed to help users rapidly build and deploy AI functions. By simply describing tasks, users can generate structured output patterns with A/B testing and observational monitoring. The platform supports no-code prototyping...

Comprehensive Introduction NV Ingest (NVIDIA Ingest) is a suite of early access microservices designed for parsing hundreds of thousands of complex, messy unstructured PDFs and other enterprise documents. It can convert these documents into metadata and text for embedding into retrieval...

General Introduction Trellis is a data platform focused on converting complex unstructured data sources into structured SQL formats. Through its powerful AI engine, Trellis is able to process a wide range of data sources such as financial documents, voice calls, and emails and convert them into data ready and...

Comprehensive Introduction Ollama OCR is a powerful Optical Character Recognition (OCR) toolkit that utilizes the state-of-the-art visual language model provided by the Ollama platform to extract text from images. The project is available both as a Python package and as a user-friendly Strea...

Comprehensive Introduction llmstxt-generator is a professional web content extraction and integration tool specialized in preparing high-quality text datasets for training and inference in Large Language Models (LLM). The tool was developed by Mendable AI using @firec...

Comprehensive introduction Doc2X is a powerful document image formula recognition and conversion tools, is committed to providing efficient and intelligent document processing solutions. Whether it is an academic research paper, a textbook, a corporate document or a financial report, Doc2X can accurately recognize PDF tables and...

Comprehensive Introduction ExtractThinker is a flexible document intelligence tool that utilizes Large Language Models (LLMs) to extract and classify structured data from documents, providing a seamless ORM-like document processing workflow. It supports a variety of document loaders, including Tess...

Comprehensive Introduction HtmlRAG is an innovative open source project focused on improving the processing of HTML documents in Retrieval Augmented Generation (RAG) systems. The project presents a novel approach that argues that using HTML formatting in RAG systems is more efficient than plain text. The project contains a complete ...

Comprehensive Introduction ScrapeGraphAI is an innovative Python web crawling library that cleverly combines Large Language Modeling (LLM) and Direct Graph Logic to create crawling pipelines for websites and local documents. The uniqueness of this tool lies in its perfect level of simplicity and power...

Comprehensive Introduction Vision Parse is a revolutionary document processing tools, it cleverly combines the most advanced visual language models (Vision Language Models) technology, to be able to intelligently convert PDF documents into high-quality Markdown format ...

Comprehensive Introduction Outlines is an open source library developed by dottxt-ai to enhance the application of Large Language Models (LLMs) through structured text generation. The library supports a variety of model integrations, including OpenAI, transformers...

General Introduction MarkItDown is a Python tool developed by Microsoft designed to convert various files and office documents to Markdown format. The tool supports a wide range of file types, including PDF, PowerPoint, Word, Excel, diagrams...

General Introduction Chunkr is a self-hosted API specialized in converting PDF, PPTX, DOCX and Excel files into data suitable for use in RAG (Retrieval Augmented Generation) and LLM (Large Language Model). The project was developed by Lumina...

General Introduction GitIngest is an open source tool designed to transform GitHub code repositories into text suitable for Large Language Model (LLM) hints. With a simple operation, users can extract and format the content of any GitHub repository to fit the LLM ...

General Introduction E2M (Everything to Markdown) is an open source Python library designed to convert a wide range of file formats to Markdown format. The tool supports formats including doc, docx, epub, html, htm, u...

Comprehensive Introduction Docling is a powerful document parsing and exporting tool that supports a wide range of document formats, including PDF, DOCX, PPTX, XLSX, Image, HTML, AsciiDoc and Markdown.It can parse and export these documents...

Comprehensive Introduction MegaParse is a powerful and versatile document parsing tool designed to optimize data processing for the Large Language Model (LLM). Whether you are working with text, PDF, PowerPoint presentations or Word documents, MegaParse...

Comprehensive Introduction ViTLP (Visually Guided Generative Text-Layout Pre-training for Document Intelligence) is an open source project designed to pass...

Comprehensive Introduction Trieve is an all-inclusive infrastructure developed by Devflow, Inc. designed for search, recommendations, RAG (retrieval augmentation generation), and analytics. The platform is served via an API and supports self-hosting for AWS, GCP, K...

Comprehensive introduction pdf2htmlEX is an open source tool designed to convert PDF files to HTML format , by analyzing the content of the PDF file and use HTML + CSS to accurately restore its visual effect , the PDF document will be converted to a browser ...

Comprehensive Introduction Maxun is an open source no-code web data extraction platform that allows users to train robots in minutes to automatically crawl web data and convert it into APIs or spreadsheets. The platform supports paging and scrolling, can adapt to changes in website layout, provides powerful data crawling...

Comprehensive Introduction OmniParse is a powerful data parsing and optimization platform designed to convert any unstructured data into structured, actionable data optimized for GenAI (Generative Artificial Intelligence) framework. Whether you are working with documents, tables, images, videos, audio files or...

General Introduction Parsio is an AI-based document and email data extraction tool that automatically extracts structured data from PDFs, emails and other documents. The platform provides a powerful PDF parser and OCR functionality and supports a wide range of document types, including...

Comprehensive Introduction Chonkie is a lightweight and efficient RAG (Retrieval-Augmented Generation) text chunking library designed to help developers quickly and easily chunk text. The library supports a variety of chunking methods , including ...

Comprehensive Introduction TextIn is a professional PDF to Markdown tool designed to help users efficiently convert PDF documents to Markdown format. The tool supports a variety of file formats, easy to operate, fast conversion speed, and can retain the format and content of the original PDF...

Comprehensive Introduction The Text Extraction API (text-extract-api) is a powerful tool designed to extract and parse content from a variety of document formats (e.g. PDF, Word, PPTX, etc.). The API utilizes state-of-the-art Optical Character Recognition (OCR) technology and Ol...

Comprehensive Introduction Datalab offers a range of advanced AI models focused on OCR, layout analysis, PDF to Markdown, and more. These models are not only high performing, but also easy to use and open source. The Marker models on the platform can quickly and accurately...

Comprehensive Introduction MinerU is an open source data extraction tool developed by the OpenDataLab team at the Shanghai Artificial Intelligence Laboratory, focusing on efficiently extracting content from complex PDF documents, web pages, and eBooks. It can take multimodal PDFs containing images, formulas, tables and other elements...

General Introduction Marker is a deep learning based document processing tool designed to convert PDF files to Markdown format quickly and accurately. It supports a wide range of document types and is especially optimized for conversion of books and scientific papers.Marker is able to remove headers...

General Description Mathpix is a powerful AI-driven document automation tool designed for researchers, developers and enterprises. It quickly and accurately converts PDFs and images into searchable, exportable and machine-readable text.Mathpix offers a wide range of features...

Comprehensive Introduction Unstructured-IO provides a set of open source components for processing and pre-processing images and text documents such as PDF, HTML, Word documents, etc. Its main goal is to simplify and optimize the data processing workflow , especially for large language models (LL...

General Introduction Jina AI's Reader project is an open source tool (Reader open source address) that takes any URL by adding the prefix https://r.jina.ai/转换成适合大型语言模型 (Large Languag...