RWP 23-12, November 2023
This study evaluates the performance of local large language models (LLMs) in interpreting financial texts, compared with closed-source, cloud-based models. We first introduce new benchmarking tasks for assessing LLM performance in analyzing financial and economic texts and explore the refinements needed to improve its performance. Our benchmarking results suggest local LLMs are a viable tool for general natural language processing analysis of these texts. We then leverage local LLMs to analyze the tone and substance of bank earnings calls in the post-pandemic era, including calls conducted during the banking stress of early 2023. We analyze remarks in bank earnings calls in terms of topics discussed, overall sentiment, temporal orientation, and vagueness. We find that after the banking stress in early 2023, banks tended to converge to a similar set of topics for discussion and to espouse a distinctly less positive sentiment.
JEL Classifications: C45, G21
Article Citation
Cook, Thomas R., Anne Lundgaard Hansen, Sophia Kazinnik, and Peter McAdam. 2023. “Evaluating Local Language Models: An Application to Financial Earnings Calls.” Federal Reserve Bank of Kansas City, Research Working Paper no. 23-12, November. Available at External Linkhttps://doi.org/10.18651/RWP2023-12