Abstract

• first benchmark specifically designed to evaluate LLM-based agents on data analysis tasks.

• This benchmark contains DAEval,
• a dataset consisting of 257 data analysis questions derived from 52 CSV files,
• an agent framework which incorporates LLMs to serve as data analysis agents for both serving and evaluation.

1 Introduction

• Open Interpreter (Lucas, 2023)

• Open Agents (Xie et al., 2023)

• Qwen-Agent (Bai et al., 2023) 

• Taskweaver (Qiao et al., 2023).

4 Related works

• web environments:
• WebShop (Yao et al., 2022a) 
•  WebArena (Zhou et al., 2023)

• tool using
• ToolBench (Qin et al., 2023b) 
• Gentopia (Xu et al., 2023a)

• Benchmarks for Code
• CodeXGLUE (Lu et al., 2021)  for both code understanding and generation
• DS-1000 (Lai et al., 2023). benchmark for code generation in data science.

2 InfiAgent-DABench Benchmark

2.1 Dataset Construction

• Files Collection:
• from Github

• Description Generation
• utilize GPT-3.5 to summarize the following information for each CSV file:
• Description of column names and contents
• Data types
• Identification of missing values
Prompts for Description Generation

• Concepts Determination:
• To ensure that our dataset aligns with real-world demands, we conduct expert interviews to identify key concepts in data analysis.
Concepts and corresponding explanations

• Open-ended Question Generation
• leverage GPT-4 to generate open-ended questions based on both file descriptions and the pre-defined concepts.
Prompts for question generation

• Constraints and Format Requirements Generation
• Considering closed-form questions can be easily evaluated,we introduce a format-prompting technique that prompts GPT-4 to generate detailed constraints and format requirements for open-ended questions to make them closed-form.
• we conduct a experiment on GPT-4 auto-evaluating for above open-ended questions, which.show that GPT-4 could only achieve 67% consistency with human experts
Constraints and Format Requirements Generation

• Response Gathering
• leverage OpenAI ADA to generate answers of closed-form questions. Each question is sent to ADA for 3 times.

2.2 Agent Framework

2.3 Human Assessment

• Suitableness

• Reasonableness

• Value

• Restrictiveness

• Alignment

• Correctness

2.3.1 Dataset Quality

2.3.2 Human vs GPT-4

2.4 Benchmark Statistics

2.5 Instruction-tuning Dataset

1. Creating a diverse set of data analysis questions for CSV files

2. Collecting response trajectories to these questions through our agent framework

3. Filtering low-quality samples with rules and GPT-4

3. Experiments

1. prompt the agent with the file, the question and constraints.

2. The agent answers with the calling of external Python sandbox

3. we use GPT-3.5 to reformat the response into format requirements

4. use regular expression matching to draw the answer and compare it with the label.

3.1 Setup

3.1.1 Models

3.1.2 Implementation Details

3.2 Results

3.3 Findings

1. 代码正确性评估：此方法允许使用仅支持文本的LLM进行评估，但可能导致不准确。在数据分析任务中，主要目标是生成准确的可视化表示，而不仅仅是底层代码。仅评估代码可能难以精确评估某些情况下的可视化效果。

2. 使用多模态模型评估：例如，Qwen-Agent利用视觉-语言模型（如GPT-4 Vision）作为评估器。此方法直接评估视觉输出的准确性。然而，当前的多模态模型性能尚未达到理想状态，且成本较高，可能导致对数据可视化的评估不够理想。