KoboldCpp: Easily run GGUF models with API and GUI with reference to KoboldAI

General Introduction

KoboldCpp is an easy-to-use AI text generation software for GGML and GGUF models, inspired by the original KoboldAI.It is a single self-contained distributable version provided by Concedo, based on the llama.cpp build and adds flexible KoboldAI API endpoints, additional format support, Stable Diffusion image generation, speech-to-text, backward compatibility, and a gorgeous UI with persistent stories, editing tools, save formats, memory, world information, author notes, characters, scenes, and more, as well as everything that KoboldAI and KoboldAI Lite have to offer.

Function List

• Support for GGML and GGUF models
• Provides multiple format support
• Image Generation Function
• Speech-to-text function
• Persistent storytelling and editing tools
• Save Format and Memory Functions
• World information and author's notes
• Character and scene creation

Using Help

Installation process

1. Windows user::
   • Download the latest koboldcpp.exe Version.
   • (of a computer) run koboldcpp.exeThe GUI can be displayed without command line arguments.
   • Get and load the GGUF model.
2. Linux users::
   • Download the precompiled PyInstaller binaries.
   • (of a computer) run chmod +x koboldcpp Make the file executable.
   • (of a computer) run ./koboldcpp Launch the software.
3. MacOS Users::
   • Download binaries for ARM64 MacOS.
   • Set the file to be executable in the terminal:chmod +x koboldcpp-mac-arm64The
   • (of a computer) run ./koboldcpp-mac-arm64 Launch the software.

Guidelines for use

1. Loading Models::
   • Download the GGUF model file.
   • Select and load the model in the software.
2. Text Generation::
   • Select the preset and GPU layer.
   • Enter the text prompt and click the Generate button.
3. Image Generation::
   • Select the image generation model.
   • Enter descriptive text and click Generate Image.
4. speech-to-text::
   • Upload an audio file or record using the microphone.
   • Click the Convert button to get the text result.
5. Editing and saving::
   • Use editing tools to modify the generated text or image.
   • Save files locally or to the cloud.

Advanced Settings

• GPU acceleration: Use --usecublas maybe --usevulkan Flag to enable GPU acceleration.
• Context size: Use --contextsize Increase the context size.
• compatibility mode: Use --noavx2 Flag to enable non-AVX2 compatible mode.