Part 1
Before reading Kalota (2024), I thought generative AI was three things stacked together: a search engine pulling current information, a fact database storing mostly verified content, and a reasoning system working on top of both. I knew the reasoning layer could be “black boxed” (exact thought-process unknown), but I figured the facts underneath were pretty much correct. When I got a wrong answer, I assumed it was a technical glitch in the language layer or a database that needed updating. Kalota (2024) corrected that. There is no search running by default and no fact database to update. The whole system is one operation: pattern prediction from training data frozen at the date the training ended. What looks like information retrieval is generated language that sounds like a real answer.
I encountered this without understanding it during my BSW placement at the Decatur library, where I used AI to help locate community resources for patrons. When answers turned out to be incorrect or it generated non-working numbers that didn’t connect, I figured it was the database needing a refresh. I encountered the same issue in coursework when an AI tool provided a detailed citation that looked genuine but could not be found when I searched for the article. That experience helped me realize that the tool was not retrieving and verifying a real source; it was generating information that matched the pattern of an academic citation.
Understanding the mechanism makes me both more and less comfortable with AI in social work practice. More comfortable using it for constrained, low-stakes tasks such as brainstorming and organizing information, but less comfortable with anything a client’s safety or eligibility depends on unless a person verifies it. Kalota also names a research field, explainable AI (XAI), built around the fact that even developers can’t fully trace why a chat-bot model produces a specific output. That matters for a documentation profession like ours, where a case note that cannot be defended is a case note that should not have been written.
If a client or colleague asked me why AI “hallucinates,” I’d say it isn’t looking anything up. It’s predicting what words most likely come next, so it produces language that sounds like a correct answer, whether or not the specific facts are real, and doesn’t bother to double-check against outside, reliable sources. That’s why important information still needs a person to check it.
Part 2
If an agency were going to use an AI tool in a mental-health-adjacent role, the safeguard I would insist on is three things working together. First, the tool handles only limited, non-clinical work. That means hours, forms, benefit explainers, and appointment scheduling. Anything outside that scope goes to a person. Not because a keyword triggered a flag, but because the tool was never built to handle clinical content in the first place. Second, every client is told up front that they are talking to an AI, what it can and cannot do, and how to reach a person instead. That is informed consent, which Moore et al. (2025) lists as one of the clinical standards that AI systems fail. Third, every interaction is logged and reviewed on a regular schedule by a licensed clinician, and the agency stays accountable. “The AI recommended it” is not a defensible case note.
Moore et al. (2025) drew the line for me between what AI can do and what a person has to do. When GPT-4o gave bridge names to a user who had just lost a job (Moore et al., 2025), no keyword filter would have caught that. The risk was in the pattern of what the person said, not in the words themselves. Pattern-recognition is exactly what these tools are worst at when it matters most. That is why the line cannot be “flag crisis words”. The line is that clinical judgment stays with a licensed human, and AI is used only where a wrong answer means a scheduling mistake, not a life.
In my library placement I saw what happens when that kind of oversight is missing. In the field, part of good social work is building working relationships with the entities you refer to. You get to know DeKalb Coordinated Entry. You call the shelters. You learn which programs answer their phones and which ones don’t. Right now the Georgia EBT phone system is offline. The online login asks for a card number the person doesn’t have because the card is lost. The contact form does not get answered. The only real option is a trip to the county office during business hours, which assumes transportation, time off work, and childcare. Automation without a working fallback is not efficiency. It is a way of pushing the harm onto the people least able to work around it.
References
Kalota, F. (2024). A primer on generative artificial intelligence. Education Sciences, 14(2), 172. https://doi.org/10.3390/educsci14020172
Moore, J., Grabb, D., Agnew, W., Klyman, K., Chancellor, S., Ong, D. C., & Haber, N. (2025). Expressing stigma and inappropriate responses prevents LLMs from safely replacing mental health providers. In Proceedings of the 2025 ACM Conference on Fairness, Accountability, and Transparency (FAccT ’25) (pp. 599–627). Association for Computing Machinery. https://doi.org/10.1145/3715275.3732039
I gave your post a 10/10. You made a convincing argument by clearly explaining how your understanding of generative AI changed after reading Kalota’s article and connecting those ideas to your experiences during your BSW library placement. Then you further showed how this new understanding informed your view of AI in social work. In regard to Moore et al., I enjoyed your multi-layered safeguard measures for AI. You really demonstrated a thought-out approach to using AI, while also maintaining human oversight to make sure errors are being made and ensuring transparency. I enjoyed your point that the line is not simply to “flag crisis words,” but to ensure that clinical judgment remains with a licensed professional. Then how you connected that to the example of the bridge names and the fault of AI in pattern recognition. Your real-world example of the Georgia EBT system also showed how automation without adequate human oversight can create additional barriers for clients. Your post was thoughtful and well-supported, connecting the research to real-world social work practice and personal experiences.
I would rate your post a 10/10. You made your argument more convincing by supporting your changed understanding of AI with a personal application. Your safeguard example in part 2 was also very impactful because of your explanation that it is not only crisis words that should flag an interaction, but also better pattern recognition of phrases that are concerning. I also think that AI should not be used in a way to provide therapeutic interventions, but instead be used as an administrative support tool.
Mark,
Your discussion in part 1 is excellent, and you are not alone in your initial understanding of how AI works. Most of us have used and known about search engines. So, from our experience, it seemed logical that the database of information would always be accessible to the AI application and constantly refreshed. Instead, the only way the model is “retrained” is with an entirely new training regimen.
The one thing to remember is that AI is looking things up; it’s just looking it up in the massive database it was trained on. You are totally correct that the database will not change except when a new model is retrained, again, from scratch. This has come up in several of your blog posts, so I decided to put together a brief handout that helps us all in explaining how AI is trained. I think you might find it interesting that I used Claude.ai for this task. When I asked if there was an additional citation for the information in the handout, the response was this:
“No — the handout draws on general, well-established facts about how AI models work (training vs. inference), which I already know and didn’t need to source externally. I didn’t search for or pull from any other articles or materials. The training/inference distinction itself is standard AI/ML knowledge.”
I will include the handout in this week’s module, as you all might find it helpful in thinking through your Final Projects.
Your three protections sound like good ones. The issue is also what the safeguards are set to do. It’s a bit like security protection when you are online, in my opinion. The safer you are, the less you will find when you are looking for something. The more you find and the more access you have to everything that can help you, the less security you have and the more access you have to things that may hurt you. Finding those lines is the problem. And there are ways to build in safeguards that will prevent some things. One of the reasons I use Claude is because Anthropic refused to remove their safeguards when requested by the Defense Department. And they are on the Trump administration’s shit list. That seems like two good reasons to me.
You are sitting right at the critical intersection on this issue. There are amazing things it can do for us. If you look at the material from the disability article and the Center for Advanced Studies in Child Welfare, you can see some of that. But like any technology at any point in our history, there is always another side as well.
Good discussion. I really look forward to reading your Final Project.
Dr P