Prediction of Tehran Stock Exchange Total Index Using Natural Gradient Boosting and SHAP Values

Purpose: The stock market represents one of the most pivotal components of developing economies. Consequently, extensive research employing both technical and fundamental analyses has sought to predict financial time series in order to assist investors with their trading decisions. In this regard, machine learning models have emerged as effective tools for addressing a variety of challenges. Nevertheless, despite the notable performance of machine learning models in this area, two significant criticisms persist. The first concerns the lack of interpretability of the results; in such models, the process by which inputs are transformed into outputs, as well as the contribution of each input to the model’s output, is not clearly defined. The second issue pertains to the reliability of the predictions generated by these models, as this reliability cannot be directly inferred from the model itself. Accordingly, this study utilizes the latest methods developed in the field of machine learning to address these two issues.



Methods: Considering that the selection of input features plays a crucial role in shaping the output of these models, this study employs a systematic approach to extract features used in related research over the past five years through a systematic review using the Scopus scientific database. Ultimately, 34 features with daily available data were selected as inputs for the model. In the next step, the Natural Gradient Boosting model was utilized to predict the data of the Tehran Stock Exchange Total Index from March 2010 to January 2025. The performance of this model was evaluated using the RMSE, MAE, and MAPE metrics and compared with the latest machine learning methods for time series prediction. Subsequently, SHAP values were employed to interpret the results of the Natural Gradient Boosting model. This approach allowed for the assessment of the contribution of each feature to the estimation of the model’s output. SHAP values provide a powerful tool for evaluating the impact of each input feature on the output estimation, offering valuable insights to users of machine learning models.



Findings: A comparison of the error values of the proposed model with those of other machine learning models indicates superior predictive performance for the proposed approach. Unlike conventional machine learning models, which provide a single prediction as the best estimate, the proposed model outputs a probability distribution that can be described by its parameters. In this study, the assumed parametric form of the distribution is the normal distribution, which is characterized by its mean and standard deviation. In fact, the predicted value corresponds to the mean of the estimated distribution. For forecasting the Tehran Stock Exchange index, the most influential features are the closing price, the EMA indicator, and the SMA indicator. Interpretation of the predicted standard deviation parameter reveals that the ATR indicator, closing price, and TEMA indicator have the greatest impact on this parameter. As the relative values of these variables increase, the standard deviation of the estimated distribution also increases, indicating that the corresponding prediction is less reliable.



Conclusion: The findings of this study demonstrate that the Natural Gradient Boosting model can serve as an effective tool for predicting the Tehran Stock Exchange Total Index. The interpretation of results using SHAP values enables the identification of the most important input features and the manner in which the output is formed from these features, thereby aiding in model optimization. This approach not only enhances prediction accuracy but also assists market participants and policymakers in making more informed decisions regarding risk management and resource allocation. Ultimately, comparisons with other models indicate that this method can be employed as a practical and reliable solution for financial market analysis.