Abstract

Objective: Systematic risk, or stock beta, is one of the fundamental pillars of asset pricing, risk management, and portfolio performance evaluation. Nevertheless, empirical evidence has shown that traditional, linear econometric models often face severe limitations in accurately forecasting future beta due to their inability to capture the complex, non-linear dynamics of the market. The primary objective of this research is to overcome these limitations by improving the estimation and forecasting accuracy of stock betas in the Tehran Stock Exchange (TSE) leveraging modern data science techniques and machine learning algorithms. Inspired by recent pioneering studies, this research conducts a comparative analysis of the performance of a suite of advanced, non-linear algorithms against a wide range of traditional baseline models. The key innovation of the present study lies in moving beyond purely statistical metrics by simultaneously evaluating the economic value added of these forecasts through the actual performance of investment portfolios (minimum-variance and market-neutral portfolios) within a highly volatile emerging market.

Methodology: In terms of purpose, this is an applied study, and methodologically, it is descriptive-correlational with a quantitative approach based on historical market data and financial statements. The statistical population comprises all firms listed on the Tehran Stock Exchange. Following the application of screening criteria (including the exclusion of financial institutions, investment firms, and tickers with prolonged trading halts), the initial dataset spans 50 trading quarters from the beginning of 1391-Q4 to the end of 1404-Q1. After forming the final panel, the research sample consists of 41,313 firm-month observations across 450 companies listed on the TSE within an unbalanced panel (with active firms ranging between 188 and 427 per month). Beta forecasting was conducted using cross-validation and a rolling-window approach to preserve strict out-of-sample conditions.This study evaluated the performance of 6 traditional baseline models (including Ordinary Least Squares [OLS], Exponentially Weighted Moving Average [EWMA] over short and long horizons, and the adjustment models of Blume, BSW, and Vasicek) against 6 advanced machine learning algorithms, including Random Forest, gradient boosting models (XGBoost and GBRT), Artificial Neural Networks (MLP Keras), and penalized regressions (Elastic Net and Linear Models). To compare model accuracy, error metrics such as Value-Weighted Mean Squared Error (VWMSE), the two-sided Diebold–Mariano test with the Harvey–Leybourne–Newbold finite-sample correction (DM-HLN), feature importance analysis, and the construction of minimum-variance and market-neutral portfolios were employed.

Findings: The empirical findings of this research indicate the decisive superiority of machine learning algorithms over traditional econometric models in forecasting stock betas. The results demonstrated that decision tree-based models, particularly Random Forest (RF), Gradient Boosted Regression Trees (GBRT), and XGBoost algorithms, significantly outperformed other models in reducing out-of-sample prediction errors (evidenced by a substantial decrease in VWMSE), and the two-sided Diebold–Mariano test with the Harvey–Leybourne–Newbold finite-sample correction (DM-HLN) test confirmed the statistical significance of this outperformance. The feature importance analysis in machine learning algorithms revealed that, contrary to traditional beliefs, predictors based on accounting information (fundamental variables), followed by historical beta estimates, which play the most crucial role in accurately predicting future betas, with non-linear algorithms successfully uncovering these intricate patterns. However, in the economic evaluation and portfolio formation stage, results reveal that the statistical superiority of individual-stock models (such as RF) does not automatically translate into superior economic gains due to the fundamental divergence between statistical loss functions and portfolio optimization objectives. Instead, gradient boosting algorithms (such as GBRT) achieved significant economic outperformance-demonstrated through superior Sharpe ratios and effective drawdown mitigation-driven by more accurate beta estimates and an optimized portfolio risk profile.

Conclusion: The findings of this study suggest that machine learning algorithms exhibit a higher capacity for reducing prediction error in statistical beta forecasting within the Iranian capital market compared to traditional models. Although machine learning algorithms substantially enhance statistical beta forecasting accuracy in emerging markets, translating this precision into tangible portfolio economic gains depends critically on aligning model loss functions with portfolio risk-management constraints and covariance matrix estimation.