Think of your chatbot as a palm reader.
A recent essay in "Superintelligent" newsletter by Herbert Lin presents a very useful analogy. A chatbot operates on our psyche using the same mechanisms as a palm reader. We should treat its output with a similar level of skepticism.
Palm readers and chatbots have much in common. "Neither can verify claims against a world they do not independently observe ... and both draw authority from fluent performance rather than reality."
We are susceptible to being influenced by both because what they tell us seems profound. We are fooled because they are fluent and provide contextually plausible responses, delivered with an air of authority.
"The problem is not the tool; it is the mystique that has grown around it. Mistaking mirrors for windows is not a technological error so much as an old human one."
The difference between palm reader and a chatbot is that the actionable advice given by the chatbot has much more serious consequences. When people think their chatbot actually likes them, or listen to a chatbot's medical advice rather than consulting a professional, actual harm could result.