Over the past few years, computerprogrammerare turning to programs such as OpenAI's ChatGPT Chatbots like that to help themencodingsThis has hit sites like Stack Overflow, causing it to have to lay off nearly 301 TP3T employees last year. The problem, however, is that a team of researchers from Purdue University released findings at the Computer-Human Interaction Conference that found that 521 TP3T of the answers to programming questions generated by ChatGPT were wrong.
Source Note: The image is generated by AI, and the image is authorized by Midjourney
That's a staggering rate for a program that people rely on for accuracy and precision, and it highlights a problem experienced by other end users like writers and teachers: AI platforms like ChatGPT often produce completely wrong answers out of thin air. In their study, the researchers reviewed 517 Stack Overflow questions and analyzed the results of ChatGPT's attempts to answer them. "We found that 52% of ChatGPT answers contained misinformation, 77% answers were more verbose than human answers, and 78% answers had varying degrees of inconsistency with human answers," they wrote.
The team also linguistically analyzed 2,000 randomly selected ChatGPT answers and found that they were "more formal and analytical" while exhibiting "less negativity" - the - This is the monotonous and pleasant tone that AI tends to produce. What's particularly worrying is that many human programmers seem to prefer ChatGPT's answers. Researchers at Purdue University surveyed 12 programmers and found that they preferred ChatGPT by a margin of 351 TP3T and found no AI-generated errors at 391 TP3T.
Why does this happen? It could simply be that ChatGPT is more polite than online people. "Follow-up semi-structured interviews revealed that polite language, a refined and textbook approach to answering, and comprehensiveness were among the main reasons why ChatGPT answers appeared to be more persuasive, so participants lowered their guard and ignored some of the misinformation in the ChatGPT answers," the researchers wrote.
This study shows that ChatGPT still has major flaws, but that's a bitter consolation for people who were laid off from Stack Overflow or programmers who had to correct errors in AI-generated code.