Before reading this article, I believed equated artificial intelligence and generative AI as the same thing. I see now that for it to be generative, it must generate, produce, or create something. As opposed to AI which could be anything from the program that empties your spam folder to having their own needs, beliefs, and desires. I can admit but am also ashamed to say that I have used 3 different generative AI platforms. In my own personal life, I have used platforms without fully understanding what they do or what they are, I just know that it can be used to help with work at times. Knowing that most of the AI I would be using is generative, I would say that it makes me more comfortable with it showing up in social work practices. We have to understand; AI is only as good as the information put into it. Whether we look at the prompt given to a specific platform, or the persons responsible for feeding the platform information, it has chances of being faulty. I would say, look at AI how you would look AI like regular research. As in, it also needs to be fact checked.
One safeguard I would insist on before feeling comfortable with any AI tool being used with real mental‑health clients is mandatory human review of all concerning risks, especially those involving suicidality, self‑harm, trauma disclosure, or identity‑related stigma. Moore et al.’s findings show that even advanced LLMs can produce stigmatizing, dismissive responses, particularly when users describe marginalized identities or sensitive symptoms. Because stigma can worsen distress or discourage help‑seeking, a system that occasionally produces such responses cannot be allowed to interact with vulnerable people without a human clinician verifying the safety and tone of the AI’s guidance. A safeguard like human‑in‑the‑loop review ensures that no client receives harmful messaging due to model error, bias, or misinterpretation.
Moore et al.’s work also helps clarify the line between tasks appropriate for AI and tasks requiring a human. AI is well‑suited for symptom tracking, appointment reminders, or summarizing client‑provided information. These tasks rely on structured knowledge and predictable patterns. But when a task requires empathy or ethical judgment, AI falls short. I’ve seen the consequences of oversight in personal life when people rely on automated mental‑health apps that give generic advice. A friend using one of these apps received responses that minimized their distress, leaving them feeling unheard. It wasn’t intentional harm, but it was harm, nonetheless. These experiences reinforce why AI must never replace human clinicians, and why oversight is not optional, it’s the foundation of ethical use.
Jontavynn,
Your first paragraph is excellent because you bring up an important point that most of us don’t really pay attention to – even after re-reading the entire Kalota article. And you have absolutely NO reason to be ashamed. Welcome to this new world with all the rest of us.
Because you raise your point so well, you helped me to crystallize a distinction that everyone needs to know. Rather than giving you all the details here – and to spare me time that is slowly slipping away – I refer you to the document that I have just added to this week’s module: What kind of AI is it? I think it sill clarify some things that you will want to understand as you write your Final Project.
You raise that most important issue, right? Where is the line between support for AI and clinical practice? How do we decide where the line is, and how do we assure that our clients and we are protected?
Good discussion.
Dr P