Implement Ask details pipeline evaluation process

[paopa]

This PR aims to make the evaluation process more easily executed. and support a Streamlit app to present and compare the evaluation results. And, it includes a few metrics to support us in comparing and analyzing the reports.

Metrics

Metric	Catalog	Approach	Description
Accuracy	Ragas	Answer Correctness	Measures the correctness of the answer provided by the model. Learn more
Accuracy	Wren	Execution Correct	Evaluates whether the SQL generated by the model can be executed without errors.
Accuracy	Wren	LLM as the Judge	Uses the model as a judge to compare the ground truth and the candidate answer.
Latency	-	-	Measures the time taken to complete a task or process.
Cost	-	-	Refers to the monetary expense incurred for a particular task or process.

Run the evaluation process

Execute the evaluation process via make command

make eval pipeline=src.eval.ask_details

The process will reproduce the report in JSON format, which includes the summary and details for each record.

{
    "total": 1,
    "accuracy": {
        "ragas": {
            "answer_correctness": 0.6310689639982272
        },
        "wren": {
            "execution_correct": {
                "true": 0,
                "false": 1
            },
            "llm_judge": 1.0
        }
    },
    "latency": 6.410386032075621,
    "cost": 0.00045299999999999995,
    "collection": [
        {
            "accuracy": {
                "ragas": {
                    "answer_correctness": 0.6310689639982272
                },
                "wren": {
                    "execution_correct": false,
                    "llm_judge": {
                        "explanation": "The candidate SQL provided captures the essence of the user's intent fairly accurately. The initial query to join the 'college' and 'player_college' tables is correct, which is crucial for identifying the college with the highest number of players. However, there are some discrepancies in the subsequent steps. The use of GROUP BY 'college_id' instead of 'name_full' in the second SELECT statement could potentially lead to incorrect results, as the query should focus on the college with the highest number of players, not just any college ID. Additionally, the missing CTE (Common Table Expression) declaration in the second step creates a gap in the query flow, affecting the overall execution logic.",
                        "score": 1.0
                    }
                }
            },
            "latency": 6.410386032075621,
            "cost": 0.00045299999999999995,
            "model": "gpt-3.5-turbo-0125",
            "usage": {
                "completion_tokens": 193,
                "prompt_tokens": 327,
                "total_tokens": 520
            },
            "question": {
                "query": "what is the full name and id of the college with the largest number of baseball players?",
                "sql": "SELECT T1.name_full, T1.college_id FROM college AS T1 JOIN player_college AS T2 ON T1.college_id = T2.college_id GROUP BY T1.college_id ORDER BY COUNT(*) DESC NULLS LAST LIMIT 1",
                "summary": null
            },
            "response": {
                "description": "Retrieve the full name and college ID of the college with the highest number of players, handling NULL values last.",
                "steps": [
                    {
                        "sql": "SELECT T1.name_full, T1.college_id FROM college AS T1 JOIN player_college AS T2 ON T1.college_id = T2.college_id",
                        "summary": "Select the full name and college ID from the college table and join it with the player_college table.",
                        "cte_name": "step1"
                    },
                    {
                        "sql": "SELECT name_full, college_id FROM step1 GROUP BY college_id ORDER BY COUNT(*) DESC NULLS LAST LIMIT 1",
                        "summary": "Group the result by college ID, order by the count of players in descending order with NULLs last, and limit the output to 1 row.",
                        "cte_name": ""
                    }
                ]
            },
            "ground_truth": {
                "description": "Retrieve the full name and college ID of the college with the highest number of players, handling NULL values last.",
                "steps": [
                    {
                        "sql": "WITH college_players_count AS (SELECT T1.name_full, T1.college_id, COUNT(*) AS player_count FROM college AS T1 JOIN player_college AS T2 ON T1.college_id = T2.college_id GROUP BY T1.name_full, T1.college_id) ",
                        "summary": "Calculate the count of players for each college.",
                        "cte_name": "college_players_count"
                    },
                    {
                        "sql": "SELECT name_full, college_id FROM college_players_count ORDER BY player_count DESC NULLS LAST LIMIT 1",
                        "summary": "Retrieve the full name and college ID of the college with the highest number of players, handling NULL values last.",
                        "cte_name": ""
                    }
                ]
            }
        }
    ]
}

Run the dashboard to compare and analyze the results

Start the Streamlit app to present the report via make command

make streamlit pipeline=src/eval/ask_details/streamlit_app.py

[cyyeh]

LGTM