FinGLUE: A Comprehensive Benchmark for Evaluating Large Language Models in the Financial Domain

Overview

This repository contains the FinGLUE benchmark, a comprehensive evaluation suite designed to assess the performance of large language models (LLMs) in the financial domain. FinGLUE focuses on four key dimensions: Financial General Knowledge, Financial Quantitative Risk Control, Financial Insight and Analysis, and Financial Information Parsing. Our benchmark is built upon 37 specialized tasks, providing a robust framework for evaluating LLMs' capabilities in handling complex financial tasks.

Features

• Four Core Dimensions: Evaluates LLMs across Financial General Knowledge, Financial Quantitative Risk Control, Financial Insight and Analysis, and Financial Information Parsing.
• 37 Specialized Tasks: Includes a diverse range of tasks designed to test models on various financial scenarios, including text classification, sentiment analysis, quantitative analysis, and more.
• Detailed Evaluation Metrics: Utilizes metrics such as F1 score, precision, recall, cosine similarity, and others to provide a thorough assessment of model performance.
• Comparative Analysis: Provides insights into the strengths and weaknesses of different LLMs, helping researchers and developers optimize their models for the financial sector.

Key Findings

• DufeGPT: Exhibits strong performance in Financial Information Parsing (FIP) and Financial Insight and Analysis (FIA).
• LLaMA3.1: Excels in Financial General Knowledge (FGK) and Financial Quantitative Risk Control (FQRC).
• ChatGPT: Shows balanced performance across most dimensions but with room for improvement in specific areas like Financial Information Parsing.

Future Directions

The FinGLUE benchmark will be regularly updated to incorporate new tasks and metrics, reflecting the evolving needs of the financial industry. Future work will explore the environmental impact of LLMs in finance, addressing issues such as energy consumption.

Contact

For questions or contributions, please open an issue or contact ZhuangLiu.

Citation

If you use FinGLUE in your research, please cite our work:

@article{finglue_aaai25,
title={FinGLUE: A Comprehensive Benchmark for Evaluating Large Language Models in the Financial Domain},
author={Zhuang Liu, Shuang Tian, Fangning Huo,Donghang Zheng,Ruitong Liu},
journal={AAAI},
year={2025}
}