Ragas: assessing RAG recall QA accuracy and answer correlation

General Introduction

Ragas is a tool specifically designed to evaluate and optimize Retrieval Augmented Generation (RAG) systems. It provides a comprehensive set of evaluation metrics by analyzing the relationships between queries, retrieval contexts, and generated answers. These metrics include fidelity, answer relevance, contextual relevance, contextual recall, and contextual precision.Ragas helps developers evaluate and optimize the performance of RAG systems to ensure that generated answers are accurate, relevant, and based on retrieved content. Whether you are developing a new system or optimizing an existing one, Ragas provides powerful support.

Function List

• Fidelity assessment: A measure of the fidelity of the generated answer to the retrieved context.
• Assessment of answer relevance: Evaluate how relevant the generated answer is to the original query.
• Contextual relevance assessment: Measure how well the retrieved information fits the problem.
• Contextual recall assessment: Evaluate whether the retrieved context contains all the information needed to answer the question.
• Contextual accuracy assessment: Measures the quality of the ordering of retrieved relevant contexts in the results.

Using Help

Installation process

1. Installation via PyPI::

   pip install ragas

2. Installation from source::

   pip install git+https://github.com/explodinggradients/ragas

Quick Start

1. Importing the necessary modules::

   from ragas import SingleTurnSample
from ragas.metrics import AspectCritic

2. Preparing test data::

   test_data = {
"user_input": "summarize given text\nThe company reported an 8% rise in Q3 2024, driven by strong performance in the Asian market. Sales in this region have significantly contributed to the overall growth. The company reported an 8% rise in Q3 2024 driven by strong performance in the Asian market. Sales in this region have significantly contributed to the overall growth. positive trend in the Asian market is expected to continue into the next quarter.", "response".
"response": "The company experienced an 8% increase in Q3 2024, largely due to effective marketing strategies and product adaptation, with expectations of continued growth in the coming quarter."
}

3. Evaluation of the RAG system::

   evaluator_llm = LangchainLLMWrapper(ChatOpenAI(model="gpt-4o"))
metric = AspectCritic(name="summary_accuracy", llm=evaluator_llm, definition="Verify if the summary is accurate.")
await metric.single_turn_ascore(SingleTurnSample(**test_data))

Detailed function operation flow

1. Fidelity assessment::
   • Use the fidelity metrics provided by Ragas to assess whether the generated answers are faithful to the retrieved context.
   • The fidelity evaluation results are obtained and analyzed by calling the corresponding evaluation functions.
2. Assessment of answer relevance::
   • Using Ragas' answer relevance metrics, the degree of relevance of the generated answers to the original query is evaluated.
   • Customize the assessment of the relevance of the generated answers by configuring the relevance assessment parameters.
3. Contextual relevance assessment::
   • Use Ragas' contextual relevance metrics to measure how well the retrieved information fits the problem.
   • Get the result of the context relevance evaluation by calling the relevance evaluation function.
4. Contextual recall assessment::
   • Use Ragas' context recall metric to assess whether the retrieved context contains all the information needed to answer the question.
   • Ensure that the retrieved context fully covers the information needed for the problem by setting the recall evaluation parameters.
5. Contextual accuracy assessment::
   • Use Ragas' contextual accuracy metrics to measure the quality of the ordering of retrieved relevant contexts in the results.
   • Optimize the ranking of retrieval results by configuring accuracy evaluation parameters.

Ragas Assessment QA Notes

QA recalls generally contain two components:
1. Retriever - Retrieves the most relevant information needed to answer a query.
2. Generator - Use the retrieved information to generate answers.

Assessing QA accuracy has always been a big issue.ROUGE(A Package for Automatic Evaluation of Summaries(math.) andBLUE(a method for automatic evaluation of machine translation) is less effective.

Current methodology: reference-free assessment using a powerful LLM, so talk carefully about Ragas.

Ragas is designed to address these limitations of using LLM to evaluate your QA pipeline, while also providing actionable metrics using as little, more cost-effective, and faster annotated data as possible.

Ragas is a framework that helps you evaluate these different aspects of your QA pipeline. It provides you with a number of metrics to evaluate different aspects of your QA system.

Indicators used to assess retrieval: offers and provides you with a measure of the performance of the retrieval system.
context_relevancy
context_recall

Assessment of the indicators generated: Provide which measures the illusion and which measures how to answer the question.
faithfulness
answer_relevancy

faithfulness

Measure the factual accuracy of the generated answers against the context provided.

This is done in two steps.

First, given a question and generated answers, Ragas uses LLM to find out the statements made by the generated answers. This gives a list of statements that we must check for validity. In step 2, given the list of statements and the returned context, Ragas uses LLM to check whether the statements provided are supported by the context. The number of correct statements is added and divided by the total number of statements in the generated answer to get the score for the given example.

Step one:

According to the first prompt.

Given a question and answer, create one or more statements from answer.
question: Who was Albert Einstein and what is he best known for?
answer: He was a German-born theoretical physicist, widely acknowledged to be one of the greatest and most influential physicists of all time. He was best known for developing the theory of relativity, he also made important contributions to the development of the theory of relativity. He was known for developing the theory of relativity, he also made important contributions to the development of the theory of quantum mechanics.
He also made important contributions to the development of the theory of quantum mechanics.
Albert Einstein was born in Germany.
He was best known for developing the theory of relativity, he also made important contributions to the development of the theory of quantum mechanics. statements: Albert Einstein was born in Germany.
question: Cadmium Chloride is slightly soluble in this chemical, it is also called what?
answer: Cadmium Chloride is slightly soluble in this chemical, it is also called what?
statements: Cadmium Chloride is slightly soluble in this chemical, it is also called what?
Cadmium Chloride is slightly soluble in alcohol.
answer: alochol statements: Cadmium Chloride is slightly soluble in alcohol. question: Were Shahul and Jithin of the same nationality?
question: Were Shahul and Jithin of the same nationality? answer: They were from different countries.
question: Were Shahul and Jithin of the same nationality?
Shahul and Jithin were from different countries.
question: What is dietary analysis?
answer: Dietary analysis is a method of evaluating people's food intake by analyzing in detail the foods they eat over the course of a week to determine the types and amounts of nutrients they consume. It can reveal whether people are deficient in certain nutrients and whether there are substances in the food that are harmful to health, such as sugar, salt, caffeine, alcohol, food additives and preservatives. Dietary analysis can provide an assessment of fat, carbohydrate, protein and calorie intake. However, dietary analysis does not fully take into account the variability of nutrient content in foods, individual nutrient requirements, and nutrient absorption and utilization. Therefore, it may not be the most accurate method of assessing an individual's exact nutritional status.
statements.

Get result 1:

['Dietary analysis is a method of evaluating the foods that people consume.' ,
'Dietary analysis identifies the types and amounts of nutrients people consume.' ,
'Dietary analysis can reveal whether people are deficient in certain nutrients.' ,
'Dietary analysis can reveal whether there are some substances in food that are harmful to health.' ,
'Dietary analysis can provide an assessment of the intake of fats, carbohydrates, proteins and calories.' ,
'Dietary analysis may not be the most accurate way to assess an individual's exact nutritional status.']

Step two:

According to the second prompt.

Prompt: Natural language inference
Consider the given context and following statements, then determine whether they are supported by the information present in the context. Provide a brief explanation for each statement before arriving at the verdict (Yes/No). Provide a brief explanation for each statement before arriving at the verdict (Yes/No). Provide a final verdict for each statement in order at the end in the given format. Do not deviate from the specified format.
Do not deviate from the specified format.
John is a student at XYZ University. He is pursuing a degree in Computer He is enrolled in several courses this semester, including Data Structures, Algorithms, and Database Management. John is a diligent student and spends a significant amount of time studying and completing assignments. He is enrolled in several courses this semester, including Data Structures, Algorithms, and Database Management. John is a diligent student and spends a significant amount of time studying and completing assignments. He often stays late in the library to work on his projects.
He often stays late in the library to work on his projects.
1. John is majoring in Biology. 2.
2. John is taking a course on Artificial Intelligence. 3.
3. John is a dedicated student.
4. John has a part-time job. 5.
5. John is interested in computer programming.
Answer.
1. John is majoring in Biology. 2.
Explanation: John's major is explicitly mentioned as Computer Science. There is no information suggesting he is majoring in Biology.
2. John is taking a course on Artificial Intelligence.
Explanation: The context mentions the courses John is currently enrolled in, and Artificial Intelligence is not mentioned. Therefore, it cannot be Explanation: The contextions the courses John is currently enrolled in, and Artificial Intelligence is not mentioned.
3. John is a dedicated student.
Explanation: The prompt states that he spends a significant amount of time studying and completing assignments. Additionally, it mentions that he often stays late in the library to work. The prompt states that he spends a significant amount of time studying and completing assignments. Additionally, it mentions that he often stays late in the library to work on his projects, which implies dedication.
4. John has a part-time job.
Explanation: There is no information given in the context about John having a part-time job. Therefore, it cannot be deduced that John has a part-time job. Verdict: No. 5.
5. John is interested in computer programming.
Explanation: The context states that John is pursuing a degree in Computer Science, which implies an interest in computer programming.
Final verdict for each statement in order: No. No. Yes. No. Yes.
No. Yes.
What level of health do you want to achieve? Determining your optimal nutritional needs is essential if you want to understand your full mental and physical potential. But if your nutritional needs are very specific, how do you discover this? Since 1980, I have developed and refined a system for analyzing the nutritional needs of the human body that is based on an assessment of the major factors that influence an individual's nutritional needs. This system has been tested and validated on 100,000 people and is now used by clinical nutritionists around the world. Countless people have benefited from it, so I know what kind of results to expect from this system. These results include: sharper thinking, better memory, more stamina, better weight control, lower cholesterol levels, and better medical conditions. Although most people diagnosed with a medical condition have benefited from a personal wellness program, this wellness program is primarily aimed at prevention, not cure. If you are undergoing medical treatment, make sure that this nutritional supplement program is well compatible with the treatment measures you are receiving. Factors Affecting Your Nutritional Needs There are at least 8 factors that affect how much nutrition you need. Age, gender, and physical activity are common influences, but pollution, stress, genetic predisposition, your past health, and of course, the nutrients as well as the anti-nutrients that your meals provide, are not readily understood. But all of these details and more must be factored in. These details can be understood through four methods of analysis - dietary analysis, biochemical analysis, symptom analysis, and lifestyle analysis. Dietary analysis as an approach seems to start where it's easy to start: figuring out what foods people are consuming will reveal what nutrients are deficient.
But unfortunately, a detailed analysis of the foods eaten over the course of a week doesn't take into account the variability of the nutrient content of foods, the nutrient needs of the individual, or whether nutrients are absorbed and whether the absorbed nutrients are effectively utilized. I've seen many people with high quality diets who still have symptoms of vitamin deficiencies. For the vast majority of these people, the problem is nutrient malabsorption. These variables make some dietary analyses via computer less accurate than envisioned. Dietary analysis is useful for evaluating foods that can affect people's nutrient needs, such as sugar, salt, coffee, tea, alcohol, food additives, and preservatives. Other factors, such as fat, carbohydrate, protein and calorie intake, can also be assessed by dietary analysis. Tests such as hair mineral analysis and vitamin blood analysis can provide precise information about the biochemical status of the body and allow nutritional counselors to understand the exact nutritional status of your body. However, not all of these tests provide useful information for planning your nutritional program. In order for this information to be as accurate as possible, any tests for vitamins or minerals must reflect the nutrient's ability to function in the body. For example, iron is an essential element that makes up blood red blood cells and helps transport oxygen through the body. By measuring the nutritional status of iron in your cells, you may be able to get a good estimate of your iron needs. On the other hand, vitamin C does not have a direct role in the bloodstream like iron does. It acts mainly on the chemical 5-hydroxytryptamine in the brain, or in the methylation reaction that lowers homocysteine levels. Therefore, simply measuring vitamin B6 in the blood is not a true indicator of whether you are getting enough of the nutrient and whether your body is functioning properly. In contrast, a test such as homocysteine concentration is much more meaningful. If you are getting enough vitamins, B vitamins, and folic acid, then your homocysteine levels are low, but if they are high, then you need more of one or more of these nutrients. We call this a functional test because it accurately determines whether a specific function of your body's biochemical response is working properly.
statements.
1. Dietary analysis is a way of evaluating the foods that people consume.
2. Dietary analysis can determine the types and amounts of nutrients that people consume.
3. Dietary analysis can reveal whether people are deficient in certain nutrients.
4. Dietary analysis can reveal whether there are substances in food that are harmful to health.
5. Dietary analysis may provide an assessment of the intake of fats, carbohydrates, proteins and calories.
6. Dietary analysis may not be the most accurate way to assess an individual's exact nutritional status.
Answer.

Get result 2:

1. dietary analysis is a method to assess what food people consume. explanation: the context mentions that dietary analysis is a method to understand what food people consume. verdict: yes.
explanation: the context mentions that dietary analysis is a method to understand what food people consume. verdict: yes.
2. dietary analysis can determine the types and amounts of nutrients people consume. explanation: the context mentions that dietary analysis is a method to understand what food people consume. verdict: yes.
explanation: the context mentions that dietary analysis can evaluate the intake of nutrients such as sugar, salt, coffee, tea, alcohol, food additives, and preservatives. it does not determine the types and amounts of nutrients that people consume. explanation: the context mentions that dietary analysis can evaluate the intake of nutrients such as sugar, salt, coffee, tea, alcohol, food additives, and preservatives. it does not explicitly state that it can determine the types and quantities of nutrients consumed. verdict: no.
3. Dietary analysis can reveal whether people are deficient in certain nutrients.
explanation: the context mentions that dietary analysis can be useful in assessing the intake of nutrients that can affect people's nutritional needs. This implies that it can reveal if someone is lacking certain nutrients. verdict: yes.
4. Dietary analysis is useful in assessing the intake of nutrients that can affect people's nutritional needs. this implies that it can reveal if someone is lacking certain nutrients. verdict: yes.
explanation: the context mentions that dietary analysis can be useful in evaluating the intake of substances that can affect people's nutritional needs explanation: the context mentions that dietary analysis can be useful in evaluating the intake of substances that can affect people's nutritional needs , such as food additives and preservatives. this implies that it can reveal the presence of harmful substances in food. verdict: yes.
5. Dietary analysis can provide an assessment of fat, carbohydrate, protein and calorie intake.
explanation: the context mentions that the intake of factors such as fat, carbohydrates, protein, and calories can be evaluated through dietary explanation: the context mentions that the intake of factors such as fat, carbohydrates, protein, and calories can be evaluated through dietary analysis.
6. Dietary analysis may not be the most accurate method of assessing an individual's exact nutritional status.
explanation: the context mentions that while dietary analysis can provide useful information, it may not be the most accurate method for assessing an explanation: the context mentions that while dietary analysis can provide useful information, it may not be the most accurate method for assessing an individual's exact nutritional status.
final verdict for each statement in order: yes. no. yes. yes. yes. yes. yes. yes.

groundfinal verdict for each statement in order.localize toyes, no, yes, yes, yes, yes, yes, yes, yes, yes, yes, yes, yes, yes, yes, yes, yes, yes, yes.

Counting points:

output = "yes. no. yes. yes. yes. yes. yes. yes."
score = sum(0 if "yes" in answer else 1)
for answer in output.strip().split(".")
if answer ! = ""
)
# score = 1 - score/len(statements)
socre = 1 - 1/6
# score = 0.833333333333333334

That is the final score for FAITHFULNESSD.

answer_relevancy

Measure the relevance and focus of the answer to the question.

For a given generated answer, Ragas uses LLM to find the possible questions for which the generated answer will be the answer and calculates the similarity to the actual question.

According to answers.

Generate question for the given answer.
Answer.
The PSLV-C56 mission is scheduled to be launched on Sunday, 30 July 2023 at 06:30 IST / 01:00 UTC. It will be launched from the Satish Dhawan Space Centre, Sriharikota, Andhra Pradesh, India. It will be launched from the Satish Dhawan Space Center, Sriharikota, Andhra Pradesh, India
Question: When is the scheduled launch date and time for the PSLV-C56 mission, and where will it be launched from?
Answer:Dietary analysis is a method of evaluating the types and amounts of food consumed in an individual's diet. By analyzing in detail the foods eaten during a week, it is possible to understand the types and amounts of nutrients people are consuming and to detect possible nutrient deficiencies or excesses. Dietary analysis is useful for assessing the nutrient content of foods, their intake, and some of the factors that affect nutritional requirements (e.g. sugar, salt, caffeine, alcohol, etc.). However, dietary analysis does not take into account the individual's nutrient requirements, nutrient absorption and utilization and is therefore not the only way to assess an individual's nutritional status.
Question.

Get:

[

"What is dietary analysis and how is it used to assess an individual's nutritional intake?", "What is dietary analysis and how is it used to evaluate an individual's nutritional intake?
"What is dietary analysis and how is it used to assess an individual's nutritional intake?", "What is dietary analysis and how is it used to assess an individual's nutritional intake?"
]

Then calculate the similarity between [problem] and these three results:

Similarity calculation process: usingtext-embedding-ada-002Perform vectorization and then compute the cosine similarity.

def calculate_similarity(
self: t.Self, question: str, generated_questions: list[str]
).
question_vec = np.asarray(self.embedding.embed_query(question)).reshape(1, -1)
gen_question_vec = np.asarray(
self.embedding.embedding_documents(generated_questions)
)
norm = np.linalg.norm(gen_question_vec, axis=1) * np.linalg.norm(
question_vec, axis=1)
return (
np.dot(gen_question_vec, question_vec.T).reshape(-1,)/ norm
)

Get:

[0.83663467 0.83484782 0.83484782]

Then average:

answer_relevancy_score: 0.8354434364200779

context_relevancy

Measure the signal-to-noise ratio in the retrieved context. Given a question, Ragas calls LLM to find the sentences needed to answer the question from the retrieved context. The ratio between the required sentences and the total sentences in the context provides you with a score.

Input:

Task: Candidate sentence extraction.
Given the question and context, extract minimum number of sentences from context required to answer the question. If the context do not contain Given the question and context, extract minimum number of sentences from context required to answer the question.
Question: Which equation is known as worlds most famous equation?
question: Which equation is known as worlds most famous equation?
Albert Einstein (14 March 1879 - 18 April 1955) was a German-born theoretical physicist,[5] widely ranked among the greatest and most influential scientists of all time. Best known for developing the theory of relativity, he also made important contributions to quantum mechanics, and was thus a central figure in the revolution. Best known for developing the theory of relativity, he also made important contributions to quantum mechanics, and was thus a central figure in the revolutionary reshaping of the scientific understanding of nature that modern physics accomplished in the first decades His mass-energy equivalence formula was the most important of all.
His mass-energy equivalence formula E = mc2, which arises from relativity theory, has been called "the world's most famous equation".
sentences:His mass-energy equivalence formula E = mc2, which arises from relativity theory, has been called "the world's most famous equation". equation".
question: Were Scott Derrickson and Ed Wood of the same nationality?
context : Scott Derrickson (born July 7)
Scott Derrickson (born July 16, 1966) is an American director, screenwriter and producer He lives in Los Angeles, California He is best known for directing horror films such as "Sinister", "The Exorcism of Emily Rose", and "Deliver Us From Evil", as well as the 2016 Marvel Cinematic Universe installment, "Doctor Strange". installment, "Doctor Strange" Tyler Bates is an American musician, music producer, and composer for films, television, and video games. Adam Collis is an American filmmaker and actor. Adam Collis is an American filmmaker and actor.Conrad Brooks is an American actor.Edward Davis Wood Jr. (October 10, 1924 - December 10, 1978) was an American filmmaker, actor, writer, and composer. Edward Davis Wood Jr. (October 10, 1924 - December 10, 1978) was an American filmmaker, actor, writer, producer, and director.
Edward Davis Wood Jr. (October 10, 1924 - December 10, 1978) was an American filmmaker, actor, writer, producer, and director. sentences:Scott Derrickson (born July 16, 1966) is an American director, screenwriter and producer. - Edward Davis Wood Jr. (October 10, 1924) was an American filmmaker, actor, writer, producer, and director.
question: How many were killed in the Tiananmen Square incident?
context.
Tiananmen Square incident, also called June Fourth incident or 6/4, series of protests and demonstrations in China in the spring of 1989 that culminated on the night of June 3-4 with the June 3-4 protests. on the night of June 3-4 with a government crackdown on the demonstrators in Tiananmen Square in Beijing.
sentences: No candidate sentences found.
question: What is meal analysis?
context.
What level of health do you want to achieve? Determining your optimal nutritional needs is essential if you want to understand your full mental and physical potential. But if your nutritional needs are very specific, how do you discover this? Since 1980, I have developed and refined a system for analyzing the nutritional needs of the human body that is based on an assessment of the major factors that influence an individual's nutritional needs. This system has been tested and validated on 100,000 people and is now used by clinical nutritionists around the world. Countless people have benefited from it, so I know what kind of results to expect from this system. These results include: sharper thinking, better memory, more stamina, better weight control, lower cholesterol levels, and better medical conditions. Although most people diagnosed with a medical condition have benefited from a personal wellness program, this wellness program is primarily aimed at prevention, not cure. If you are undergoing medical treatment, make sure that this nutritional supplement program is well compatible with the treatment measures you are receiving. Factors Affecting Your Nutritional Needs There are at least 8 factors that affect how much nutrition you need. Age, gender, and physical activity are common influences, but pollution, stress, genetic predisposition, your past health, and of course, the nutrients as well as the anti-nutrients that your meals provide, are not readily understood. But all of these details and more must be factored in. These details can be understood through four methods of analysis - dietary analysis, biochemical analysis, symptom analysis, and lifestyle analysis. Dietary analysis as an approach seems to start where it's easy to start: figuring out what foods people are consuming will reveal what nutrients are deficient.
But unfortunately, a detailed analysis of the foods eaten over the course of a week doesn't take into account the variability of the nutrient content of foods, the nutrient needs of the individual, or whether nutrients are absorbed and whether the absorbed nutrients are effectively utilized. I've seen many people with high quality diets who still have symptoms of vitamin deficiencies. For the vast majority of these people, the problem is nutrient malabsorption. These variables make some dietary analyses via computer less accurate than envisioned. Dietary analysis is useful for evaluating foods that can affect people's nutrient needs, such as sugar, salt, coffee, tea, alcohol, food additives, and preservatives. Other factors, such as fat, carbohydrate, protein and calorie intake, can also be assessed by dietary analysis. Tests such as hair mineral analysis and vitamin blood analysis can provide precise information about the biochemical status of the body and allow nutritional counselors to understand the exact nutritional status of your body. However, not all of these tests provide useful information for planning your nutritional program. In order for this information to be as accurate as possible, any tests for vitamins or minerals must reflect the nutrient's ability to function in the body. For example, iron is an essential element that makes up blood red blood cells and helps transport oxygen through the body. By measuring the nutritional status of iron in your cells, you may be able to get a good estimate of your iron needs. On the other hand, vitamin C does not have a direct role in the bloodstream like iron does. It acts mainly on the chemical 5-hydroxytryptamine in the brain, or in the methylation reaction that lowers homocysteine levels. Therefore, simply measuring vitamin B6 in the blood is not a true indicator of whether you are getting enough of the nutrient and whether your body is functioning properly. In contrast, a test such as homocysteine concentration is much more meaningful. If you are getting enough vitamins, B vitamins, and folic acid, then your homocysteine levels are low, but if they are high, then you need more of one or more of these nutrients. We call this a functional test because it accurately determines whether or not a particular function of your body's biochemical reactions is working properly.
sentences.

Get:

[
'Dietary analysis as such seems to start where it's easy to start: figuring out what foods people are consuming reveals what nutrients are deficient.' ,'
'Dietary analysis starts where it's easy to start: figuring out what foods people are consuming reveals what nutrients are deficient.'
]

Split each result, look it up in context, and see what percentage of the split clauses in each result appear in the original text overlap_scores.
Assuming that we get[1.0, 1.0]The

Context:

What level of health do you want to achieve? Determining your optimal nutritional needs is essential if you want to understand your full mental and physical potential. But if your nutritional needs are very specific, how do you discover this? Since 1980, I have developed and refined a system for analyzing the nutritional needs of the human body that is based on an assessment of the major factors that influence an individual's nutritional needs. This system has been tested and validated on 100,000 people and is now used by clinical nutritionists around the world. Countless people have benefited from it, so I know what kind of results to expect from this system. These results include: sharper thinking, better memory, more stamina, better weight control, lower cholesterol levels, and better medical conditions. Although most people diagnosed with a medical condition have benefited from a personal wellness program, this wellness program is primarily aimed at prevention, not cure. If you are undergoing medical treatment, make sure that this nutritional supplement program is well compatible with the treatment measures you are receiving. Factors Affecting Your Nutritional Needs There are at least 8 factors that affect how much nutrition you need. Age, gender, and physical activity are common influences, but pollution, stress, genetic predisposition, your past health, and of course, the nutrients as well as the anti-nutrients that your meals provide, are not readily understood. But all of these details and more must be factored in. These details can be understood through four methods of analysis - dietary analysis, biochemical analysis, symptom analysis, and lifestyle analysis. Dietary analysis as an approach seems to start where it's easy to start: figuring out what foods people are consuming will reveal what nutrients are deficient.
But unfortunately, a detailed analysis of the foods eaten over the course of a week doesn't take into account the variability of the nutrient content of foods, the nutrient needs of the individual, or whether nutrients are absorbed and whether the absorbed nutrients are effectively utilized. I've seen many people with high quality diets who still have symptoms of vitamin deficiencies. For the vast majority of these people, the problem is nutrient malabsorption. These variables make some dietary analyses via computer less accurate than envisioned. Dietary analysis is useful for evaluating foods that can affect people's nutrient needs, such as sugar, salt, coffee, tea, alcohol, food additives, and preservatives. Other factors, such as fat, carbohydrate, protein and calorie intake, can also be assessed by dietary analysis. Tests such as hair mineral analysis and vitamin blood analysis can provide precise information about the biochemical status of the body and allow nutritional counselors to understand the exact nutritional status of your body. However, not all of these tests provide useful information for planning your nutritional program. In order for this information to be as accurate as possible, any tests for vitamins or minerals must reflect the nutrient's ability to function in the body. For example, iron is an essential element that makes up blood red blood cells and helps transport oxygen through the body. By measuring the nutritional status of iron in your cells, you may be able to get a good estimate of your iron needs. On the other hand, vitamin C does not have a direct role in the bloodstream like iron does. It acts mainly on the chemical 5-hydroxytryptamine in the brain, or in the methylation reaction that lowers homocysteine levels. Therefore, simply measuring vitamin B6 in the blood is not a true indicator of whether you are getting enough of the nutrient and whether your body is functioning properly. In contrast, a test such as homocysteine concentration is much more meaningful. If you are getting enough vitamins, B vitamins, and folic acid, then your homocysteine levels are low, but if they are high, then you need more of one or more of these nutrients. We call this type of test a functional test because it accurately determines whether a particular function of your body's biochemical reactions is working properly.

The similarity of the two answers is then calculated using bert or jaccard to get agr_score.
Then context_relevancy can be calculated:

context_relevancy = agr_score * (average of overlap_scores)

context_recall

Before you can test this, you need to provide aground_truthThe
Follow the prompt:

Given a context, and an answer, analyze each sentence in the answer and classify if the sentence can be attributed to the given context or not.
Think in steps and reason bofore coming to conclusion.
context: Albert Einstein (14 March 1879 - 18 April 1955) was a German-born theoretical physicist,widely held to be one of the greatest and Best known for developing the theory of relativity, he also made important contributions to quantum mechanics, and was thus a central figure in the field of quantum mechanics. Best known for developing the theory of relativity, he also made important contributions to quantum mechanics, and was thus a central figure in the revolutionary reshaping of the scientific understanding of nature that modern physics accomplished in the first decades of the twentieth century. His mass-energy equivalence formula E = mc2, which arises from relativity theory, has been called "the world's most famous equation". E = mc2, which arises from relativity theory, has been called "the world's most famous equation". He received the 1921 Nobel Prize in Physics "for his services to theoretical physics, and especially for his discovery of the law of the photoelectric effect", a pivotal step in the development of the theory of physics. He received the 1921 Nobel Prize in Physics "for his services to theoretical physics, and especially for his discovery of the law of the photoelectric effect", a pivotal step in the development of quantum theory. His work is also known for its influence on the philosophy of science. In a 1999 poll of 130 leading physicists worldwide by the British journal Physics World, Einstein was ranked the greatest physicist of all time. His intellectual achievements and originality have made Einstein synonymous with genius.
answer: Albert Einstein born in 14 March 1879 was German-born theoretical physicist, widely held to be one of the greatest and most influential scientists of all time. He received the 1921 Nobel Prize in Physics "for his services to theoretical physics. He published 4 papers in 1905. Einstein moved to Switzerland in 1895
Einstein moved to Switzerland in 1895.
1. Albert Einstein born in 14 March 1879 was German-born theoretical physicist, widely held to be one of the greatest and most influential scientists of all time. The date of birth of Einstein is mentioned clearly in the context.
2. He received the 1921 Nobel Prize in Physics "for his services to theoretical physics. The exact sentence is present in the given context. The exact sentence is present in the given context.]
3. He published 4 papers in 1905. There is no mention about papers he wrote in given the context.
4. Einstein moved to Switzerland in 1895. There is no supporting evidence for this in the given the context.
answer:{ground_truth}
answer:{ground_truth}
classified: {context} answer:{ground_truth}

Getting results:

[
'####################[Attributed]',
'#############[Attributed]',
'########################[Not Attributed]',
......
]

Calculate the percentage of [Attributed] occurrences that are context_recall.

Personal view:

About this library: does not support auzre access to openai, need to change the source code

On the four indicators

context_relevancy: doubtful (the effect of model stsb-TinyBERT-L-4 Chinese in agr_score to calculate similarity is uncertain)
context_recall: not sure which scenario to use for this one ......
Faithfulness.
answer_relevancy: probably reliable (depending on thetext-embedding-ada-002(Precision of semantics after vectorization)