The advent of Large Language Models (LLMs) has been transformative for many people’s work, allowing for sharp lateral thinking across multiple topics, but… can you trust responses that have been eloquently written, with the utmost confidence, are indeed, correct?
Trusting the ‘close-enough’?
The confidence illusion, whereby confidence is mistaken for competence is often considered in the workplace, but rarely is this same scrutiny cast over the generated AI responses of LLMs. As a very brief overview, an LLM is at its core a next word prediction tool based on an extremely large corpus. Its response to your query or request is essentially guessing which word is most likely to follow the last when discussing the topic, not always based on fully formed facts, as many users are expecting.
Subsequently, many users are falling into the Dunning-Kruger effect, where a topic discussed with a LLM that they were hoping to learn more about, may misdirect trust in its responses, overestimating the validity of the knowledge studied. It is a well-studied fact that eloquent use of language tricks our brains into trusting a response or corpus. This could lead to false confidence and an inability to recognise the errors that have been presented to the user, due to lack of prior subject knowledge.
After all, an LLM is trained to be ‘close-enough’ in its generated responses, a response that is nearly factually correct can genuinely be more of a trap for the user, due to its ability to slip through the net when compared to an obvious hallucination, with clear errors.
The trap.
Copying and pasting a generated reply isn’t an alien concept to many people today, especially with time sinks such as replying to service companies, sorting out a disagreement or complaint, or replying to THAT email. It saves the cognitive burden and can quickly let you move on from the task with a degree of confidence in your reply.
This outsourcing of thinking may seem harmless for small tasks, but without rumination, how quickly could you find yourself implementing generated text into more areas of your life?
Critical thinking skills are a muscle that with exercise will grow, but once judgement and research skills start to atrophy, they will fade and that compulsion to rely on generative input will become the new normal. This copy-paste culture, where teams are treating generated text the same as a finished product, instead of a starting point is a slippery slope.
Generative AI tools are designed to make you feel like you are augmenting, or enhancing your own thinking and ideation, when they may be reducing your own critical thinking time down considerably, and that is where ‘the trap’ is set and sprung.
The pitfalls.
Fundamentally LLMs do not understand your users, your brand, your constraints. At best they are making an educated guess across the entire industry, and at worst they are hallucinating compelling slop. With some time spent on your system prompt you may initially get more accurate advice, but context engineering will only get users so far for now. The importance of judgement and critical thinking cannot be overstated, and most importantly knowledge of the subject outside of the generated input should always guide writing.
This feeds into another problem of homogeneity across users’ work, particularly across individual sectors where copied and pasted output starts to look and sound identical. This educated guess across the entire industry with ‘close-enough’ accuracy, leaves user-facing content at risk of becoming generic and untailored, and could even land you in hot water over inadvertently mirroring a competitor’s content.
Another consideration is the security and privacy of what you are sharing with an LLM. Individual models have different terms around data sharing and mining, and most users would struggle to describe in detail where their data is going after pressing enter. If you readily share intellectual property, you may find it becoming part of the next batch of training data and may risk it becoming directly accessible to competitors through their own use of LLMs.
Avoiding the trap.
There is a key difference between the two ways people use generative AI, and it is this difference that will save users from being caught by ‘the trap.’ We’ve examined users outsourcing their thinking time to LLMs, but what if there is another, safer way, to approach LLM use?
Augmentation is actively using generative AI to iterate and sharpen your own thinking, further bolstering your prior subject knowledge. Bouncing ideas around, stress-testing your arguments and exploring angles you hadn’t considered, but importantly the user is still applying critical thinking and making the final call.
Staying sharp.
Treat LLMs as a place to review and refine your ideas: ask for another take on the subject, presenting counterarguments and finding holes in your claims. When users request purely generative copy, that’s when they fall into the trap.
Review and refine the LLM’s ideas: it’s a two-way street, the same scrutiny applies in reverse to the LLM. Fact check and cross-reference, ensure that you are not being deceived, ultimately if you can’t verify… should you really be using it?
Know when to say no thank you: sometimes it’s best to just do it the old-fashioned way, human intelligence still has a much-needed place!
Author
