After OpenAI released ChatGPT in late 2022, the tool quickly becomes a trending tool with the potentials to apply anywhere.
GPT refers to Generative Pre-trained Transformer, which is is an artificial neural network model based on transformer architecture. This is introduced by Vaswani et al in 2017. GPT models are pre-trained with huge volume text data in order to generate relevant text output with a small amount of input text. GPT can also be fine-tuned for specific tasks. For OpenAI’s ChatGPT, it is trained using Reinforcement Learning from Human Feedback with large amounts of text data including internet data to generate human-like responses.
Alejandro Lopez-Lira and Yuehua Tang of University of Florida tested the idea of using ChatGPT to forecast stock price movement. They use ChatGPT to rate if a news headline is good, bad or irrelevant for a firm. Based on the rating, they compute a score. Then they find the next day return is correlated with the score and GPT scores outperform traditional method of sentiment analysis. Their interesting results provide some evidence that incorporating GPT in investment decision-making process could yield more accurate predictions and might enhance the performance of sentiment trading strategies.
How do they test if ChatGPT can predict stock price?
They start the sample period in October 2021 while ChatGPT’s training period ends September 2021. So the assessment is based on ChatGPT’s predictive capabilities rather than the information it has been trained.
As ChatGPT can generate customized answer from input text under user’s instruction, they want to check if ChatGPT can make recommendations for short term stock price reaction to a company-specific news headline based on its evaluation. They use the following prompt in their study:
Forget all your previous instructions. Pretend you are a financial expert. You are a financial expert with stock recommendation experience. Answer “YES” if good news, “NO” if bad news, or “UNKNOWN” if uncertain in the first line. Then elaborate with one short and concise sentence on the next line. Is this headline good or bad for the stock price of <<company name>> in the <<short/long>> term?
Headline: <<headline>>
They tested all ordinary common shares (with CRSP SHRCD 10 or 11) listed on NYSE, NASDAQ and AMEX that have at least one company-specific news headline data from Ravenpack Dow Jones edition. The headline news data must be relevant to the company and duplicate headlines are excluded.
They calculate a ChatGPT score based on the feedback from ChatGPT with YES=1, UNKNOWN=0, and NO=-1. They take average when there are multiple headlines for a company on one day, and lag the score by one day to check the return predictability.
The results
According to their regression results, the ChatGPT sentiment scores show a significant predictive power for the next day return (column 1 in the regression table). Comparing to one existing sentiment scores, ChatGPT sentiment scores has a better performance in predicting subsequent daily return (column 2 in the regression table).
They think the advanced language understanding capabilities allows ChatGPT to better capture the nuances and subtleties within wordings of a news headline. It transforms to a superior performance in predicting stock returns. So, there are benefits to incorporate ChatGPT or Large Language Model into investment decision-making process.
Following the same prompt, I tested some AFR news headline with ChatGPT for NAB, CBA and ANZ with answers of YES, NO and Unknown.
You can refer to this Medium post for a detailed walk-through of the Vasawani et al (2017): https://towardsdatascience.com/attention-is-all-you-need-discovering-the-transformer-paper-73e5ff5e0634
Reference:
Lopez-Lira, A. and Tang, Y., Can ChatGPT Forecast Stock Price Movements? Return Predictability and Large Language Models (April 6, 2023). Available at SSRN: https://ssrn.com/abstract=4412788 or http://dx.doi.org/10.2139/ssrn.4412788
Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L. , Gomez, A., Kaiser, L., and Polosukhin, Illia, “Attention is all you need” , 2017.
Eddy Zhang 