InkSight - Google's open source AI handwriting recognition tool

What is InkSight

InkSight is Google's open source AI handwriting recognition tool that converts paper handwritten notes into editable digital inked files (e.g. SVG format). Unlike traditional OCR, it recognizes textual content, can restore handwriting style, paragraph structure and key marks, and supports multi-language processing. Users can experience it online through the HyperAI platform, and can directly edit, adjust colors or reorganize content after uploading handwritten photos. It is especially suitable for students, designers and other groups who need to frequently organize handwritten records, with an accuracy rate of 87%, which can effectively solve the problem that traditional photo notes cannot be modified.

Features of InkSight

• Offline to online handwriting conversion: InkSight converts photos of handwritten notes into digital ink, eliminating the need for special hardware such as smart pens or digital paper, making it easy for users to digitize traditional notes.
• Multi-language support: The system supports multiple languages, including Chinese, English, French, etc. It can handle complex backgrounds, adapt to different visual conditions, and has strong generalization ability.
• Word-level and full-page processing: InkSight supports word-level and full-page text processing to convert entire pages of handwritten notes into digital ink format, enabling users to seamlessly integrate physical notes into modern digital workflows.
• High quality output: In human evaluations, the InkSight output of the 87% was found to be effective for handwriting tracking, and the output of the 67% was indistinguishable from human-generated digital ink, providing high-quality digital ink output.
• Vectorized digital ink output: The digital ink output from InkSight is vectorized for easy editing and searching, making it convenient for users to further process and utilize the digitized notes.

Core Benefits of InkSight

• No special hardware required: Users can digitize handwritten notes with just ordinary photos, without the need for special equipment such as smart pens or digital paper, lowering the threshold for use and facilitating more users to easily convert handwritten content to digital format.
• Multilingualism and Complexity Adaptation: It supports multiple languages, including Chinese, English, French, etc. Meanwhile, it is able to handle handwritten notes on complex backgrounds and under different lighting conditions, and has strong generalization ability, which can be widely used in various scenarios.
• High quality digital ink output: In human evaluation, the InkSight output of the 87% was found to be effective for handwriting tracking.67%'s output is indistinguishable from human-generated digital ink, reproducing the handwriting and style of handwritten notes with a high degree of accuracy, ensuring the quality of digitized notes.
• Supports full-page and word-level processing: It can convert individual words as well as handle whole pages of notes to meet users' needs in different scenarios, whether it's extracting key information or preserving the content of the notes in their entirety.
• Vectorized digital ink for easy editing: The output digital ink is vectorized and can be edited, searched and further processed directly on digital devices, making it easy for users to organize, mark up and share their notes, enhancing the usefulness and flexibility of digital notes.
• Open Source and Scalability: The related model weights, inference codes and datasets have been made public to facilitate research, extension and improvement by researchers and developers, providing a good foundation for future technology development and application innovation.

What is the official website for InkSight

• Project website:: https://charlieleee.github.io/publication/inksight/
• Github repository:: https://github.com/google-research/inksight
• arXiv Technical Paper:: https://arxiv.org/pdf/2402.05804

Who InkSight is for

• student population: Students need to record a lot of notes, and InkSight helps them quickly digitize their handwritten notes for easy subsequent review, organization and searching, as well as integrating them with e-learning materials to enhance learning efficiency.
• office workerInkSight converts handwritten notes into a digital format that can be easily archived, shared and further edited for increased productivity.
• research worker: Researchers generate a large number of handwritten notes during experiments and research, InkSight can convert these notes into digital ink, facilitating subsequent data organization, analysis and paper writing, while supporting multi-language, which is helpful for international academic exchanges.
• principalsTeachers create a lot of handwritten lesson plans and notes in the course of preparing and teaching lessons. InkSight can digitize these contents, making it easy to create e-lessons and courseware, as well as combining them with online teaching platforms to enrich teaching resources.
• Artists and designersHand-drawn sketches and creative notes are a regular part of their work, and InkSight converts these handwritten contents into a digital format for further editing, modifying and creating in design software, preserving the creative inspiration while enhancing work efficiency.