Using Artificial Intelligence Tools in Professional Report Writing
NACVA Evaluates Impact of ChatGPT on the Membership
Imagine having a quirky little virtual assistant that can draft professional articles, generate narratives, and assist with academic writing. With the recent buzz around ChatGPT, an artificial intelligence chatbot developed by OpenAI, the possibilities of utilizing this technology in the forensics and valuation services world has created mixed viewpoints. While some may fear that ChatGPT could replace humans, others are excited about the potential it holds to, well, replace humans. Either way, ChatGPT has the potential to change the way professionals in these industries work. In this article, the authors explore how ChatGPT can be a valuable tool, a few of its strengths and weaknesses, concerns, and the implications it has for the world of professional writing and content creation.
Imagine having a quirky little virtual assistant that can draft professional articles, generate narratives, and assist with academic writing. With the recent buzz around ChatGPT, an artificial intelligence chatbot developed by OpenAI, the possibilities of utilizing this technology in the forensics and valuation services (FVS) world has created mixed viewpoints. While some may fear that ChatGPT could replace humans, others are excited about the potential it holds to, well, replace humans. Either way, ChatGPT has the potential to change the way professionals in these industries work. In this article, the authors explore how ChatGPT can be a valuable tool, a few of its strengths and weaknesses, concerns, and the implications it has for the world of professional writing and content creation.
Artificial Intelligence Chatbot Strengths and Weaknesses
Artificial intelligence chatbots, like ChatGPT, are gaining widespread attention due to their unique capabilities. ChatGPT is designed to generate human-like text responses based on prompts given to it. It can be a powerful tool for generating narratives, creating content, and assisting with academic writing. Like any technology, ChatGPT has its strengths and weaknesses.
The biggest strength of ChatGPT is its ability to quickly generate text responses. It can generate content on a wide range of topics and can save time for busy FVS professionals. It can also assist with creating and writing narratives in story formats, which can be helpful in presenting complex information in a compelling manner.
The biggest weaknesses of ChatGPT are bias and inaccuracy. ChatGPT’s responses are generated based on patterns it has learned from a large dataset, which can result in biased or inaccurate information. There are justifiable and well-founded concerns about ChatGPT’s tendency to produce erroneous content and present it as factual, which can be problematic in areas that require accuracy, such as valuation reports and/or expressing expert opinions.
The authors have experienced ChatGPT’s so-called “AI hallucinations.” In testing its capabilities, we asked ChatGPT to draft a one-page summary on the smart home technology industry. ChatGPT provided what initially appeared to be a well-written and accurate narrative. However, ChatGPT cited made up IBISWorld data in such a way that, had we not verified its (non)existence with an IBISWorld employee, we could have inadvertently disseminated inaccurate information and produced a flawed expert report.
This is what ChatGPT produced in response to our prompt:
“A report by IBISWorld published in June 2021 estimates that the average COGS margin for the smart home automation systems installation industry in the U.S. is 51.6%. The report notes that this margin can vary depending on the size of the business and its level of integration with upstream suppliers.”[1]
And this is the factually correct research we found after purchasing an IBISWorld Industry Spotlight Report on the home automation services industry and independently delving further into ChatGPT’s response:[2]
“A report by IBISWorld published in April 2023 estimates that the ten-year average COGS margin for a company in the home automation services industry in the U.S. is 41.6%. The report notes the prices of semiconductors, a key component of cost of goods sold that increased in price due to COVID-19 supply chain disruptions, are expected to normalize. IBISWorld projects that companies in the home automation industry may encounter greater profitability because of this.”
This scary experience highlights the importance of independently fact-checking ChatGPT (or any AI tool) when it is used for creating professional reports. While ChatGPT’s ability to generate engaging narratives is impressive, it is crucial to remember that the information it provides is based on patterns a machine learned from reading extensive amounts of human-written content from all corners of the internet. Therefore, professionals utilizing ChatGPT for client engagement purposes must exercise caution to ensure the accuracy and reliability of their AI-supplemented content before sharing it with their audience—especially if that content is going to be used in court.
Readability Analysis of ChatGPT-Generated Content
Many industry professionals can agree that it would be a poor decision to copy and paste selections from an amateur’s research report into their own expert report. Arguably, using ChatGPT to draft expert report content is no different. To illustrate this point, the authors performed a readability analysis on two writing selections: one completely written by humans, and a ChatGPT-generated summary on the same topic.
The human article’s Flesch Reading Ease score, which is determined on a scale of 0 to 100, was 15.3, indicating that the human article is not an easy, casual read. The human article’s Flesch-Kincaid Grade Level was determined to be 18.1, implying that one would need six years of post-secondary education to fully understand and appreciate the content of the article.
The ChatGPT article’s Flesch Reading Ease score was 6.6, indicating a reader would have slightly more difficulty reading its piece than the human-written article on the same topic. However, ChatGPT’s Flesch-Kincaid Grade Level was 17.0, meaning that ChatGPT’s writing ability is fairly matched against a first-year graduate student.
Our readability analysis shows that ChatGPT is perfectly capable of writing an advanced narrative. However, the true importance of this analysis lies in the comparison of content between the ChatGPT-written article and human-written article. While the human-written article was based on extensive research, mathematical analyses, and fact-checking, the ChatGPT article was not.
ChatGPT’s article contained content that directly contradicted the research discussed within the human-generated article and provided no support, but instead, passed off false information as factual. In situations where facts matter—arguably, in every aspect of an expert report—at this time then, human-written research is still the best way to go. As AI continues to advance, that may change over time. Interestingly, Campbell and Ramamoorti (2023) provide a clever way out of this dilemma: use ChatGPT output to improve pre-existing human efforts, involving thorough vetting of factual content, but borrowing helpful concepts, language, and phrasing from the ChatGPT parallel output.[3]
Professional Organizations’ Stances
There are varying positions among professional organizations on the use of AI tools such as ChatGPT. Some organizations have concerns about the accuracy and integrity of content generated by AI, while others see the potential benefits in terms of efficiency and productivity. For example, several science journals disallowed content solely generated by ChatGPT and other language learning models to be published, due to questions about the credibility and attribution of AI-generated content.[4] However, other science journals allow AI-assisted content to be published, as long as AI’s role is limited to what could arguably be considered an editorial role: improving the readability of the article.
Content Rights
AI users need to understand the licensing and usage rights associated with these technologies before using AI content in their professional writing. For example, AI program “Jasper” grants full rights to the content produced without the need for giving credit, while other AI programs may have different requirements. OpenAI, the organization behind ChatGPT, provides the following guidance for content co-authored with OpenAI:[5]
- “The published content is attributed to your name or company.”
- “The role of AI in formulating the content is clearly disclosed in a way that no reader could possibly miss, and that a typical reader would find sufficiently easy to understand.”
- “Topics of the content do not violate OpenAI’s Content Policy or Terms of Use, e.g., are not related to adult content, spam, hateful content, content that incites violence, or other uses that may cause social harm.”
- “We kindly ask that you refrain from sharing outputs that may offend others.”
The National Association of Certified Valuators and Analysts (NACVA) Position
We interviewed members of the NACVA’s editorial board to determine the current stance on the use of AI tools in FVS engagements.
Dr. Sridhar Ramamoorti (“Dr. Sri”), a tenured professor of accounting at the University of Dayton and a reflective practitioner, was the keynote speaker at the July 2023 NACVA and the CTI’s Business Valuation and Financial Litigation Super Conference in Snowbird, Utah. Dr Sri emphasizes the importance of considering ethics and technology’s capabilities in general, including in professional writing. He brought up the following valid point: if the use of ChatGPT results in faulty economic assessments and produces a wrong estimate of business valuation, is it the practitioner or AI that made the error? And who is responsible for the error? In his keynote speech at the Super Conference, Dr. Sri challenged NACVA to take a leadership role in developing a roadmap for these and similar questions regarding AI by providing professional guidance for FVS practitioners.
Joe Petrucelli raised similar concerns. Mr. Petrucelli, who teaches at the College of Staten Island, has advised that he will be banning AI technology in his classroom on the basis that the FVS field requires judgment, experience, and creativity, which he argues cannot be supported through the use of AI technologies currently available, or without further understanding on how AI technology works.
Brien Jones, the Chief Operations Officer and Executive Vice President of NACVA, stated NACVA’s current position on the issue: “As to NACVA’s position on the ethics and standards surrounding AI and other tools … currently, we do not have one, but we are not burying our heads in the sand. It is far too important and far reaching.” Jones noted that NACVA is considering and will be responding to Dr. Sri’s challenge that NACVA take a leadership role on the potential emerging ethics issues arising out of the use of technology within the FVS profession.
NACVA’s newly established Leadership Initiative Board (LIB) has already logged this topic as part of the scope, objectives, and goals for their activity during the upcoming term. This topic was also discussed at length during the inaugural meeting of the newly-formed LIB.
The NACVA acknowledges the existing limitations of AI technology while also recognizing its potential benefits and impact on the FVS field. However, the organization has not yet taken a definitive stance on the matter. Instead, it is actively considering the implications and possibilities presented by AI and its responsible integration within the profession.
Conclusion
In conclusion, the use of ChatGPT and similar AI chat bots in the FVS communities presents both opportunities and challenges. While AI can offer benefits such as increased efficiency and enhanced data analysis capabilities, there are limitations and risks associated with relying solely on AI-generated content, including biases and inaccuracies. It is crucial for business valuators and forensic accountants to exercise caution, validate the accuracy and reliability of AI-generated content, and establish clear guidelines for the use of AI tools in their practice. The NACVA acknowledges AI technology’s potential benefits and impact on the FVS field but has not yet taken a definitive stance, actively considering its implications.
While ChatGPT can be a valuable tool, you and/or your firm should consider establishing clear guidelines and protocols for the use of AI tools, including ChatGPT, in business valuation and forensic accounting engagements. This should include specifying the types of tasks that can be automated, setting expectations for the accuracy and reliability of AI-generated content, and establishing procedures for validating the results obtained from AI tools.
The authors also recommend developing systems for validating the accuracy and reliability of your AI-generated content (i.e., a thorough human review of all content prior to its publication in an expert report or otherwise). This may involve cross-referencing the results obtained from AI tools with other sources of information, performing sensitivity analysis, and applying professional judgment to interpret the findings obtained from AI-generated content.
(NOTE: The authors wish to point out that no use of ChatGPT was made in the drafting of this article.)
[1] The authors searched the IBISWorld database for keywords related to the smart home automation industry and initially did not find any results for that particular industry. The authors asked ChatGPT for help finding research sources, at which point it provided the above quote. According to the IBISWorld employee, IBISWorld does not have a pre-written home automation industry report available within its regular subscription, but instead could offer an Industry Spotlight Report written specifically for the engagement at a one-time purchase price.
[2] Alex Petridis, IBISWorld Industry Spotlight Report OD6378, Home Automation Services in the U.S., April 2023.
[3] Campbell and Ramamoorti (2023). Design Thinking and Cybernetics: The Case for Generative AI in AIS Pedagogy. Forthcoming in Advances in Accounting Education. Emerald Publishing.
[4] From “Science Journals Ban ChatGPT from Co-Authoring Papers,” by A. Paleja, January 2023, Interesting Engineering (https://interestingengineering.com/culture/journals-ban-chatgpt-author-papers).
[5] From “OpenAI: Sharing & Publication Policy,” updated November 2022, OpenAI (https://openai.com/policies/sharing-publication-policy).
Michael D. Pakter, CPA, CFF, CGMA, CFE, CVA, MAFF, CA, CIRA, CDBV, has more than 40 years of experience in forensic accounting, investigations, and litigation services, including more than 20 years of experience in economic damages and business valuations. State, federal, and bankruptcy courts have recognized him as an expert in forensic accounting, economic damages, business valuation, marital dissolution, and bankruptcy core proceedings.
Mr. Pakter can be contacted at (312) 229-1720 or by e-mail to mpakter@litcpa.com.
Miranda Kishel, MBA, CVA, CBEC, provides support in the completion of business valuations, business calculations, forensic accounting, economic damages, and asset tracing engagements, often focusing on in-depth financial analysis, including: modeling, forecasting, research, report-writing, and report proofreading. Her previous experience lies in small business consulting, commercial lending, accounting, real estate development, and economic development.
Ms. Kishel can be contacted at (312) 229-1720 or by e-mail to mkishel@litcpa.com.
