Text to SQL(ask) pipeline performance benchmark

[cyyeh]

• What are performance objectives we are optimizing for?
• What are we comparing with?

related issue: #4

[cyyeh]

After discussion, we'll use latency, cost and accuracy compared with naive RAG pipeline, optimized RAG pipeline and optimized RAG pipeline with semantics description

[paopa]

The following points are the summary of the related discussion with @cyyeh

• All the Experiments and Performance Benchmarks will be recorded on this page
• Didn’t decide to use some (e.g. tool, chart) to show the performance benchmark yet.

[cyyeh]

This is not the focus as of now. I'll close it until we need this in the future.

[cyyeh]

This is the conclusion of Text-to-SQL(ask) pipeline performance benchmark

• objectives
  • gpt 3.5 turbo + wren engine + with semantics descriptions
  • gpt 3.5 turbo + wren engine
  • gpt 3.5 turbo
• data
  • using part of Spider dataset
  • like data.world’s four dimensions
• metrics
  • accuracy
  • cost
  • latency
  • explainability?(optional)
    • maybe using llm as judge