Part 1:
I have been fascinated with generative AI since Google first rolled out Gemini 1.0, but Kalota’s 2024 primer finally demystified how the technology actually functions. The specific concept that corrected my previous understanding was the clear distinction between foundational AI models (like Gemini or Claude) and the software applications built on top of them (like Copilot or NotebookLM). Before reading the primer, I viewed them all as essentially the same factual knowledge base, rather than recognizing the models as distinct prediction engines.
Prior to fully grasping this mechanism, I had heavily incorporated Gemini Pro into my student workflow—acting as my secretary, tutor, and organizer to manage my Notion notebook, extract assignments to Microsoft To Do, and overcome writer’s block. I was enthusiastically relying on its outputs without truly understanding the underlying mechanics generating them.
Now that I understand generative AI operates on pattern prediction rather than retrieving facts from a static database, I am actually more comfortable with it showing up in social work practice. Knowing its limitations means I understand it must be used strictly as a supplementary tool alongside human expertise, rather than as a standalone decision-maker. This approach minimizes exposure to confidently incorrect information and ensures client safety.
If a client or colleague asked me to explain why an AI tool “hallucinates” or gets something confidently wrong, I would explain it in plain language: AI does not actually “know” facts; it simply guesses what word or concept should logically come next based on the patterns it learned during training. Imagine showing an AI a picture of a bright yellow crescent shape. It might confidently tell you it is a banana, when it is actually a crescent moon. It relied on the patterns of “yellow” and “crescent,” made a prediction, and simply guessed the wrong option.
Part 2:
It’s tempting to just write AI off after reading Moore et al. — the limitations they document are genuinely severe. But these tools are already showing up in our agencies, whether we’re ready or not. So the real question for future practitioners isn’t whether to use them, it’s what responsible use actually looks like. And given that current LLMs show measurable stigma toward conditions like schizophrenia and alcohol dependence, and can respond dangerously to suicidal ideation or delusional thinking, letting them operate unsupervised is not something I’d sign off on.
If an agency wants to use AI for lower-stakes work — triage, psychoeducation, that kind of thing — there’s one safeguard I wouldn’t compromise on: a hard human-escalation rule. The second an AI picks up on acute distress or diagnostic-level language, a person needs to take over. And whatever the AI does say to clients should get checked periodically by an actual clinician, so biased or stigmatizing language doesn’t quietly slip through.
Moore et al. basically draws the line for us: it sits at the therapeutic alliance. Real therapy needs someone who’s human — who has something at stake emotionally, who can push back on a client’s distorted thinking instead of just going along with it. That’s the problem with LLMs in this context. They’re built to agree with you, which means they’re more likely to reinforce a delusion than challenge it safely. So I’d keep AI to the structured, administrative, informational side of things, and leave the relational and clinical judgment calls to people.
Jerry,
It is interesting how a better understanding relieves our reaction to seeing AI in social work. I think Gemini is almost the most invasive of the AI applications, because they have simply tacked it on to the search engine that everyone is so familiar with. That feels a little sneaky to me, right? But I understand using it in all the ways you describe. Understand the underlying mechanisms help us decide exactly how we want to incorporate it.
I am intrigued by your “human-escalation rule” and I will be interested in seeing if you talk m ore about that in your Final Project. Not a requirement or pressure, just an interesting concept to me.
Good discussion.
Dr P