Financial Sentiment Classification Using Traditional Machine Learning, BERT, FinBERT, and Zero-Shot FLAN-T5 (Scientific Societal & Behavioral Research Journal)

Summary

• The study compares five approaches for three-class financial sentiment classification on the Financial PhraseBank dataset of 4,846 statements.p.1
• Logistic Regression obtains 73.31% accuracy and 63.35% macro F1, while Linear SVM obtains 74.69% and 68.15%.p.1
• BERT reaches 85.01% accuracy and 83.18% macro F1, whereas FinBERT achieves 88.72% accuracy and 87.43% macro F1.p.1
• Zero-shot FLAN-T5 records 37.69% accuracy and 36.15% macro F1.p.1
• The principal experiment uses a stratified 70/15/15 split with 3,392 training, 727 validation and 727 test observations.p.6
• Across three FinBERT seeds, mean accuracy is 85.50% and mean macro F1 is 83.26%.p.14
• FinBERT exceeds BERT by 3.71 percentage points in accuracy and 4.25 percentage points in macro F1.p.15

Key insights

1. FinBERT correctly classifies 645 of the 727 test observations, an accuracy of 88.72%.p.10
2. BERT correctly classifies 618 of the 727 test observations, an accuracy of 85.01%.p.11
3. Only two of the 386 true neutral observations are predicted as neutral by zero-shot FLAN-T5.p.11
4. The 95% bootstrap interval for FinBERT accuracy is 86.38%–90.92%.p.12
5. The exact McNemar p-value comparing FinBERT with BERT is 0.000730.p.13
6. FinBERT's macro F1 is 19.28 percentage points above that of Linear SVM.p.15
7. Task-specific fine-tuning remains important, given the large gap between fine-tuned transformers and zero-shot models.p.17