Part 1

Before reading Kalota (2024), I thought generative AI was three things stacked together: a search engine pulling current information, a fact database storing mostly verified content, and a reasoning system working on top of both. I knew the reasoning layer could be “black boxed” (exact thought-process unknown), but I figured the facts underneath were pretty much correct. When I got a wrong answer, I assumed it was a technical glitch in the language layer or a database that needed updating. Kalota (2024) corrected that. There is no search running by default and no fact database to update. The whole system is one operation: pattern prediction from training data frozen at the date the training ended. What looks like information retrieval is generated language that sounds like a real answer.

I encountered this without understanding it during my BSW placement at the Decatur library, where I used AI to help locate community resources for patrons. When answers turned out to be incorrect or it generated non-working numbers that didn’t connect, I figured it was the database needing a refresh. I encountered the same issue in coursework when an AI tool provided a detailed citation that looked genuine but could not be found when I searched for the article. That experience helped me realize that the tool was not retrieving and verifying a real source; it was generating information that matched the pattern of an academic citation.

Understanding the mechanism makes me both more and less comfortable with AI in social work practice. More comfortable using it for constrained, low-stakes tasks such as brainstorming and organizing information, but less comfortable with anything a client’s safety or eligibility depends on unless a person verifies it. Kalota also names a research field, explainable AI (XAI), built around the fact that even developers can’t fully trace why a chat-bot model produces a specific output. That matters for a documentation profession like ours, where a case note that cannot be defended is a case note that should not have been written.

If a client or colleague asked me why AI “hallucinates,” I’d say it isn’t looking anything up. It’s predicting what words most likely come next, so it produces language that sounds like a correct answer, whether or not the specific facts are real, and doesn’t bother to double-check against outside, reliable sources. That’s why important information still needs a person to check it.

Part 2

If an agency were going to use an AI tool in a mental-health-adjacent role, the safeguard I would insist on is three things working together. First, the tool handles only limited, non-clinical work. That means hours, forms, benefit explainers, and appointment scheduling. Anything outside that scope goes to a person. Not because a keyword triggered a flag, but because the tool was never built to handle clinical content in the first place. Second, every client is told up front that they are talking to an AI, what it can and cannot do, and how to reach a person instead. That is informed consent, which Moore et al. (2025) lists as one of the clinical standards that AI systems fail. Third, every interaction is logged and reviewed on a regular schedule by a licensed clinician, and the agency stays accountable. “The AI recommended it” is not a defensible case note.

Moore et al. (2025) drew the line for me between what AI can do and what a person has to do. When GPT-4o gave bridge names to a user who had just lost a job (Moore et al., 2025), no keyword filter would have caught that. The risk was in the pattern of what the person said, not in the words themselves. Pattern-recognition is exactly what these tools are worst at when it matters most. That is why the line cannot be “flag crisis words”. The line is that clinical judgment stays with a licensed human, and AI is used only where a wrong answer means a scheduling mistake, not a life.

In my library placement I saw what happens when that kind of oversight is missing. In the field, part of good social work is building working relationships with the entities you refer to. You get to know DeKalb Coordinated Entry. You call the shelters. You learn which programs answer their phones and which ones don’t. Right now the Georgia EBT phone system is offline. The online login asks for a card number the person doesn’t have because the card is lost. The contact form does not get answered. The only real option is a trip to the county office during business hours, which assumes transportation, time off work, and childcare. Automation without a working fallback is not efficiency. It is a way of pushing the harm onto the people least able to work around it.

References

Kalota, F. (2024). A primer on generative artificial intelligence. Education Sciences, 14(2), 172. https://doi.org/10.3390/educsci14020172

Moore, J., Grabb, D., Agnew, W., Klyman, K., Chancellor, S., Ong, D. C., & Haber, N. (2025). Expressing stigma and inappropriate responses prevents LLMs from safely replacing mental health providers. In Proceedings of the 2025 ACM Conference on Fairness, Accountability, and Transparency (FAccT ’25) (pp. 599–627). Association for Computing Machinery. https://doi.org/10.1145/3715275.3732039