Deep Finder: open source project for deep inference search using local knowledge

General Introduction

Deep Searcher is a powerful tool that combines Large Language Models (LLMs) and vector databases, designed to search, evaluate and reason based on private data, providing highly accurate answers and comprehensive reports. It is suitable for enterprise knowledge management, intelligent Q&A systems, and information retrieval scenarios.Deep Searcher supports a variety of embedded models and large language models, such as DeepSeek and OpenAI, and maximizes the use of in-house data while ensuring data security. It also supports vector database management, such as Milvus, and allows data partitioning for more efficient retrieval.

Function List

• Private Data Search: Maximize the use of internal enterprise data and ensure data security.
• Vector Database Management: Supports vector databases such as Milvus, which allows data partitioning for more efficient retrieval.
• Flexible embedding options: Compatible with multiple embedded models to optimize selection.
• Multiple large language model support: Support for big models like DeepSeek, OpenAI, etc. for smart Q&A and content generation.
• Document Loader: Local file loading is supported and web crawling is being developed.
• Configuration flexibility: Users can customize the configuration according to their needs.

Using Help

Installation process

1. Visit the Deep Searcher GitHub repository.
2. Clone the repository to local:

   git clone https://github.com/zilliztech/deep-searcher.git
   

3. Go to the repository catalog:

   cd deep-searcher
   

4. Install the dependencies:

   pip install -r requirements.txt
   

5. Configure the vector database (e.g. Milvus) and add the configuration information to the project configuration file.

Guidelines for use

1. Private Data Search::
   • Importing internal enterprise data into a vector database.
   • Use Deep Searcher's search function to search for data:

     from deepsearcher import DeepSearcher
     searcher = DeepSearcher(config)
     results = searcher.search(query)
     

2. Vector Database Management::
   • Configure and manage Milvus or other vector databases.
   • Use the interface provided by Deep Searcher for data partitioning and management.
3. Flexible embedding options::
   • Select the appropriate embedding model based on business requirements.
   • Integrate embedded models with Deep Searcher for optimal search.
4. Online Content Integration::
   • Configure online content feeds and combine them with on-premises data to provide more comprehensive answers.
   • Use Deep Searcher's interface for online content retrieval and integration.
5. intelligent question and answer system (Q&A)::
   • Building an intelligent Q&A system based on enterprise data using Deep Searcher.
   • Configure the knowledge base of the quiz system, and train and optimize it.
6. information retrieval::
   • Use Deep Searcher to improve retrieval efficiency in a variety of information retrieval scenarios.
   • Customize retrieval strategies and algorithms according to business needs.

sample code (computing)

from deepsearcher import DeepSearcher
# Initialize Deep Searcher
config = {
'vector_db': 'milvus', 'embedding_model': 'bert', 'embedding_model': 'bert'
'embedding_model': 'bert'
}
searcher = DeepSearcher(config)
# Searching for internal enterprise data
query = "Enterprise data search"
results = searcher.search(query)
print(results)
# Integrate online content
online_results = searcher.integrate_online_content(query)
print(online_results)