AI Engineering Academy: 2.10 Automated Merge Retriever

summary

The automated merge searcher isEnhanced Retrieval Generation (RAG) FrameworkA high-level implementation of the The approach aims to enhance the context-awareness and coherence of AI-generated responses by merging potentially fragmented and smaller contexts into larger and more comprehensive ones.

https://github.com/adithya-s-k/AI-Engineering.academy/tree/main/RAG/05_Auto_Merging_RAG

Background motivation

Traditional augmented retrieval generation systems often struggle to maintain coherence across larger contexts or perform poorly when dealing with information that spans multiple text segments. Auto-merge retrievers address this limitation by recursively merging sets of child nodes that reference a parent node beyond a specific threshold, thus providing a more comprehensive and coherent context during retrieval and generation.

Methodological details

Document Preprocessing and Hierarchy Creation

1. Document Loading: Load and process input documents (e.g. PDF files).
2. hierarchical resolution: Use HierarchicalNodeParser Creates a node hierarchy from a document:
   • Level 1: Block size 2048
   • Level 2: Block size 512
   • Level 3: Block size 128
3. Node Storage: Store all nodes into the document store and leaf nodes are also indexed into the vector store.

Enhanced search generation workflow

1. Query preprocessing: Use the same embedding model as for document blocks to process user queries.
2. Basic Search: The base searcher performs an initial similarity search to find relevant leaf nodes.
3. Auto Merge::AutoMergingRetriever The retrieved set of leaf nodes is analyzed and the subset of leaf nodes that refer to the parent node beyond a given threshold is recursively "merged".
4. context extension (computing): The merged nodes form an extended context and are merged with the original query.
5. Generating a Response: Input the extension context and query into the Large Language Model (LLM) to generate a response.

Key features of the automated merge searcher

• Hierarchical Document Representation: Maintain a multi-level hierarchy of document blocks.
• Efficient Basic Search: Achieving fast and accurate preliminary information retrieval using vector similarity searches.
• Dynamic Context Extension: Automatically merge related blocks of text into larger, more coherent contexts.
• Flexible realization: Can be used for a wide range of document types and language models.

Advantages of this method

1. Enhancing contextual coherence: Provide a more coherent and complete context for large language models by merging related chunks of text.
2. Flexible search adaptability: The merge process automatically adjusts to the query and search results to provide contextually relevant information.
3. Efficient storage structure: A fast implementation of leaf node base retrieval while maintaining a hierarchical structure.
4. Possibility of improving response quality: The extended context is expected to lead to more accurate and detailed language model responses.

Results

The experimental results show that comparing the automatic merge searcher with the base searcher:

• Similar performance on correctness, relevance, accuracy and semantic similarity metrics.
• In pairwise comparisons, 52.5% users preferred the response of the auto-merge searcher.

These results show that the performance of the automated merge searcher is comparable to, or even slightly superior to, traditional search methods.

reach a verdict

The automated merge searcher provides an advanced method for improving the RAG retrieval process in the system. By dynamically merging relevant text blocks into larger, more coherent contexts, it addresses some of the limitations of traditional text block-based retrieval methods. While preliminary results show positive prospects, further research and optimization are expected to significantly improve response quality and coherence.

pre-conditions

To implement this system, you will need:

1. A large language model capable of generating text (e.g., GPT-3.5-turbo, GPT-4).
2. An embedding model for converting text blocks and queries into vector representations.
3. Vector databases for efficient similarity search (e.g., FAISS).
4. A document store for storing the complete node hierarchy.
5. furnish LlamaIndex library, which contains HierarchicalNodeParser cap (a poem) AutoMergingRetriever Realization.
6. Sufficient computing resources for processing and storing large collections of documents.
7. Familiarity with Python programming language for implementation and testing.

usage example

from llama_index.core import StorageContext, VectorStoreIndex
from llama_index.core.node_parser import HierarchicalNodeParser
from llama_index.core.retrievers import AutoMergingRetriever
# Parsing documents into node hierarchies
node_parser = HierarchicalNodeParser.from_defaults()
nodes = node_parser.get_nodes_from_documents(docs)
# Set the storage context
storage_context = StorageContext.from_defaults()
storage_context.docstore.add_documents(nodes)
# Create base indexes and retrievers
leaf_nodes = get_leaf_nodes(nodes)
base_index = VectorStoreIndex(leaf_nodes, storage_context=storage_context)
base_retriever = base_index.as_retriever(similarity_top_k=6)
# Create AutoMerging Retriever
retriever = AutoMergingRetriever(base_retriever, storage_context, verbose=True)
# Using the AutoMerging Retriever in a Query Engine
query_engine = RetrieverQueryEngine.from_args(retriever)
response = query_engine.query(query_str)