Do people prefer financial advice generated by ChatGPT over certified human advisors? The response is surprising.

Can AI-generated financial advice compete with traditional human advice? Our survey findings reveal a shifting preference!

On a Monday morning, as I was reading through the financial advice section in the papers a thought occurred to me. Can ChatGPT do this for us- that is, can it give us financial advice for our financial queries? Will it be accurate, relevant, well-explained, and trustworthy? And most importantly, will it be as good as the advice given by certified financial advisors? I was skeptical, so I decided to experiment. 

We conducted an experiment where 28 respondents were asked to select one of two financial advice choices for 5 financial queries. One of the responses was by a certified financial advisor and the other was by ChatGPT, however, the respondents were not informed about which advice was written by the advisor/AI (In principle, this is similar to a single-blind experiment). The queries were selected from reputed financial newspapers published in 2024 and included a moderately wide range of topics related to personal finance. In India, multiple financial newspapers and blogs like Business Today, Live Mint, ETwealth, etc., regularly seek and collect financial queries from the public, and a few are answered by certified financial advisors. 

To generate the ChatGPT response, the prompts used were the same financial queries copied from the newspaper with minor grammatical tweaks. We also randomized the two options for every question to avoid any discernible pattern by the respondent. 

Surprising results!

We found that for each question, the ChatGPT response had more takers than the one by a certified financial advisor! To cover breadth, we included financial questions related to asset purchase, personal finance management, taxation, purchase of gold, and portfolio diversification. The results for all of them revealed a preference for the advice generated by ChatGPT. 

The graphs below indicate the results for each type of financial query and the options chosen by the respondents. The exact questions and options for each query are attached here. 

After the experiment was conducted, we asked some respondents - Would you use AI for financial advice?

The responses ranged from high skepticism to a lukewarm approach of using generative AI for backend research but conducting a fact-checking exercise before making any financial decision. Some also pointed out that they would prefer their search and data to be locally hosted on an app on their PC/Phone instead of sent to the cloud to maintain privacy. However, some said they strongly doubted the authenticity of advice and AI’s ability to tailor-make advice per an individual’s requirement. 

Irrespective of their response to this question, in most cases, they chose the advice generated by ChatGPT. 

This exercise highlights a gap and significant potential of LLM applications for generating financial advice. As per studies¹, there are four main challenges to this opportunity of GenAI-based finance advice that are - 

1. Domain-specific expertise and their ability to personalize advice

2. Trustworthiness

3. Adherence to user’s moral and ethical standards

4. Conformity to regulatory guidelines and oversight

5. Regular access to proprietary data for training the LLM and lag in data updation (not included in the MIT study)

LLMs generating financial advice will need to be strictly trained to avoid the missale of financial products (more on this later). In addition, multiple facets need to be sorted before launching such an application such as data privacy, biases in training data, adapting to changing market conditions, and lack of regulatory expertise to oversee AI-specific regulations. 

But the greatest advantage of LLM is scalability which can pose serious competition to advice offered by human advisors. LLMs can make it extremely accessible, inexpensive, and user-friendly without creating any discomfort or awkwardness for the client to disclose their financial details. In its initial stages, it can be used to explain, summarize, and answer complex financial questions within seconds. 

As a starting point, AI-assisted agent models can be a format of combining LLM and human advisors where the latter can use AI advice to enhance the quality of their advice by comparing, contrasting, and synthesizing information on various products and services. This hybrid model can also be useful from the lens of financial inclusion. 

AI-based financial advice can be a great leveler for the Indian Economy where everyone does not enjoy the privilege of being assigned a relationship manager or premium banking facilities. In a digital world, this kind of advice can be obtained at a very low cost or even free. In the context of India, we can also imagine hyper-personalization of advice if we can integrate the account aggregator framework with LLM-based advisory tools.

A real-world example of finance-related LLM is BloombergGPT which is based on a 50 billion parameter language model and is trained on a wide range of data². Similarly,  JP Morgan Machine Learning Center of Excellence has been working on finance-specific LLMS³.

So how should an ideal Generative AI-based advisory from the future look?

With precise prompts, ChatGPT can be used as a powerful tool for navigating the world of investments within a few seconds. This can be useful for both - first-time investors as well as seasoned investors. 

However, an ideal gen AI-based financial advisor should be able to communicate investment and risk concepts to all kinds of target groups - investors and advisors, democratizing finance more than before. One of the challenges here is that, unlike humans, LLMs lack empathy. To overcome this, they have to be trained to refrain from being over-persuasive. 

The robo-advisors of the future will need to carefully understand the client's needs and cater to them by gauging their risk preferences, temperament, and advice that resonates with them. Gen AI advisors will need to tackle the alignment issue wherein they need to align with the best interest of the client. This is imperative to avoid any advice that can lead to the missale of financial products. 

In the non-digital world - ethics and trust is coded by ensuring that the advisor upholds the code of ethics and standard of conduct. To build a gen AI financial advisor, there need to be theoretical guiding principles, equivalent to a professional course taken by humans, that are rich enough for the machine learning model to apply across various use cases. Such a model of ideal financial ethics could be created by the collective consultation of academics, regulators, practitioners, and technologists. It is going to be an uphill task to code trust, fairness, empathy, and humility which are traits necessary to create the best gen AI-based financial advisor. 

But despite all the challenges and disclaimers - a gen AI-based financial advisor is not too far from becoming a reality for the masses. It is time to now wait and watch this unfold soon.

Note -  The respondents for this experiment were largely employees of Setu with a few external members. However, the respondents were not informed about the experiment's objective or which response the advisor/ChatGPT gave. The experiment was conducted remotely on surveymonkey. Here’s a link to all the financial queries along with the responses written by human advisors or generated by ChatGPT. 

1. Lo, Andrew W., and Jillian Ross. 2024. “Generative AI from Theory to Practice: A Case Study of Financial Advice.” An MIT Exploration of Generative AI, March. https://doi.org/10.21428/e4baedd9.a1f6a281.

2. https://arxiv.org/abs/2303.17564

3. https://www.jpmorgan.com/technology/applied-ai-and-ml/machine-learning