KrillinAI: Multilingual Globalization Tool for Video with One-Click Translation and Dubbing

General Introduction

KrillinAI is an open-source video processing tool that focuses on using artificial intelligence to help users translate videos and automatically dub them. It can start from video download and continue to generate finished products for different platforms in just a few clicks. The developers have provided free code on GitHub that users can download and run locally. KrillinAI uses Large Language Modeling (LLM) technology to provide high-quality translations and subtitle generation, and currently supports translations for 56 languages, with inputs in Chinese, English, Japanese, and more. It also adjusts video formats and adapts to platforms such as YouTube, TikTok, and Jieyin, making it suitable for content creators to quickly produce multilingual videos.

Function List

• Video Download & Upload: Supports downloading videos from YouTube, Jitterbug, Bilibili, or uploading local files.
• Accurate Subtitle Generation: Use of Whisper The technology recognizes speech and generates highly accurate subtitles.
• Intelligent Subtitle Splitting: Segmentation of subtitles is done naturally by means of a large language model, keeping the semantic integrity intact.
• professional translation: Translate entire paragraphs to ensure contextual consistency and near human level.
• Dubbing and voice cloning: Provided CosyVoice of male and female dubs, or upload an audio clone of the voice.
• Video format adjustment: Automatically generates horizontal or vertical video, adapting to different platforms.
• Terminology replacement: Supports one-click replacement of vocabulary in specialized fields.

Using Help

Installation process

KrillinAI needs to be installed locally to run, here are the detailed steps:

1. Download file
   • Open https://github.com/krillinai/KrillinAI.
   • Click "Releases" in the upper right corner of the page to download the appropriate version for your system (e.g. Windows, macOS).
   • Unzip the file into an empty folder.
2. Configuration environment
   • Create a folder within the config Folder.
   • exist config Create a new folder config.toml Documentation.
   • From GitHub's config-example.toml Copy the content to config.toml, and then fill in the configuration.
   • Simplest Configuration: Using only the OpenAI service, fill in the following:

     [openai]
     apikey = "你的OpenAI API密钥"
     transcription_provider = "openai"
     llm_provider = "openai"
     

   • If an agent or custom model is required, you can add app.proxy maybe openai.base_urlThe
3. running program
   • Windows (computer): Double-click the executable to start it.
   • macOS: Manual trust is required because it is not signed:
     • Open a terminal and go to the file directory.
     • Enter the command:

       sudo x'attr -rd com.apple.quarantine ./KrillinAI_1.0.0_macOS_arm64
       sudo chmod +x ./KrillinAI_1.0.0_macOS_arm64
       ./KrillinAI_1.0.0_macOS_arm64
       

   • After starting, the service runs on the http://127.0.0.1:8888(Port can be changed).
4. Docker deployment (optional)
   • In GitHub's docs/docker.md View detailed steps.
   • Install Docker, pull the image and run it.

Operation of the main functions

Video translation and subtitle generation

• move::
  1. After starting the service, the browser accesses the http://127.0.0.1:8888The
  2. Enter the video link (e.g. https://www.youtube.com/watch?v=xxx) or upload a file.
  3. Select the input language (e.g. Chinese) and the target language (e.g. English).
  4. Click "Start" and the program automatically recognizes speech, generates subtitles and translates them.
• in the end: Subtitle files are saved in the tasks Folder.
• take note of: If the download fails, you need to configure the cookies.txtappear (to be sth) docs/get_cookies.mdThe

dubbing function

• move::
  1. Once the subtitles have been generated, click on the "Dubbing" option.
  2. Choose a male or female voice for CosyVoice, or upload an audio sample to clone the voice.
  3. Click "Generate" and the program will automatically synthesize the voiceover.
• in the end: The voiceover is merged with the video to generate a new file.
• distinctiveness: Cross-language dubbing is supported, and audio tracks and subtitles are precisely aligned.

Video format adjustment

• move::
  1. When generating a video, select "Landscape" or "Portrait".
  2. Click "Finish" and the program adjusts the resolution and subtitle layout.
• in the end: Outputs videos adapted to YouTube (landscape) or TikTok (portrait).
• draw attention to sth.: The subtitles will automatically change lines if they are too long, to ensure a neat and tidy screen.

Featured Function Operation

Intelligent Subtitle Splitting

• The program analyzes speech using a large language model to segment subtitles by semantics. For example, a 10-second conversation is split into segments based on sentence integrity, rather than a fixed time cut.
• No manual setup is required, processing is done automatically.

Terminology replacement

• move::
  1. exist config.toml Add a replacement rule such as:

     [custom_vocab]
     "AI" = "人工智能"
     "LLM" = "大语言模型"
     

  2. Restart the program and replace it automatically when translating.
• use: Suitable for technology, education, etc., to ensure accurate terminology.

sound cloning

• move::
  1. Upload a 10-30 second audio sample in the voiceover screen.
  2. Select "Clone Sound" and the program generates a similar sound.
• request: If you are using the AliCloud service, you need to configure aliyun.ossappear (to be sth) docs/aliyun.mdThe

Additional configuration options

• local model: Settings transcription_provider = "fasterwhisper"Required fields local_model.faster_whisperThe model will be downloaded automatically (macOS is not supported yet).
• AliCloud Services: If you use AliCloud's big model or dubbing, you need to configure the aliyun.bailian maybe aliyun.speechThe

application scenario

1. Multilingual content creation
   • YouTube bloggers want to translate Chinese videos into English and French. krillinAI quickly generates subtitles and voiceovers in landscape format.
2. Short video promotion
   • Merchants use Jitterbug to promote their products, and KrillinAI converts the video to vertical screen with local language to enhance the appeal.
3. Sharing of educational resources
   • Teachers translate course videos into multiple languages, and KrillinAI provides accurate subtitling and voice-overs for students around the world.

QA

1. Why does the startup prompt for a missing API?
   • need to be at config.toml Fill in OpenAI's apikeyFor more information, go to the OpenAI website.
2. What input languages are supported?
   • Currently supports Chinese, English, Japanese, German, Turkish, and more languages under development.
3. How long does it take to translate and dub?
   • Processing a 10-minute video takes about 5-10 minutes, depending on network and configuration.
4. How do I fix a failed download?
   • configure cookies.txtreference docs/get_cookies.md Export browser cookies.