Part 1: Understanding Generative AI In Social Work Practice

A New Understanding Of How AI Works

A concept from Kalota’s article that changed my perspective on AI was that large language models predict the most probable next word in a sequence instead of thinking or reasoning as humans do. A language model is based on a probability that predicts what are the next words that might come after the current words in a sentence, and ChatGPT employs artificial neural networks, natural language processing, and large language models to respond to a user’s prompt, according to Kalota (2024). Prior to reading this article, I thought AI comprehended data the same way people do. I also had that misconception until I found out it works on statistical patterns and not actual knowledge.

I have used generative AI in coursework to organize ideas, summarize readings, and improve the flow of my writing. I also use it in my own life for answering questions or composing emails from time to time. Before I read Kalota’s explanation, I was primarily concerned with the quality of the answer rather than the process by which it was produced. Being aware that AI predicts language and does not validate information, I am more vigilant about verifying critical information before I take it at face value.

Knowing how generative AI works makes me cautiously optimistic for its application to social work. Kalota (2024), points out that generative AI produces new content (text, images, code, simulations), but current versions of ChatGPT are almost certainly factually wrong. If a client inquired why AI occasionally “hallucinates,” I would say the system is predicting what words are most likely to go together based on patterns in its training data, rather than verifying whether every statement is true. Because of that limitation, important information should always be reviewed by a qualified human professional.

Part 2: Managing AI Responsibly In Mental-Health-Adjacent Practice

The Importance Of Human Oversight

I do not believe AI should be completely rejected in social work practice because it can improve efficiency in tasks such as appointment scheduling, intake screening, psychoeducation, and after-hours support. However, stigmatizing and unsafe answers can be generated by large language models in the context of mental health, as demonstrated by Moore et al. (2025). This is to say that AI should be enabling professionals not replacing them. Any deployment of AI in client interactions must have built-in protections for client well-being and guarantees of accountability.

The biggest safeguard I would want to see is that any recommendation or decision that might affect the care of a client would need a human review. Levels of risk,  course of treatment, or responses in a crisis should never be decided by AI alone. A professional social worker must review material produced by AI prior to any decisions on services or interventions. This would add a layer of human review to minimize the chance of harmful errors and alert to any biased or unsuitable responses before they reach clients.

The findings from Moore et al. helped me to firmly define the line of accountability between the human and AI. AI is well-suited to structured administrative needs for educational or logistical purposes, such as delivering general wellness advice or responding to frequently asked questions. But assessment, crisis intervention, emotional support or ethical decision-making must be performed by a trained human professional. These are areas that require empathy, cultural competence, and clinical judgment, none of which AI can provide.

I have also seen how problems arise when oversight is missing in non-AI systems. Administrative errors, late communication or false credentials can be harmful for clients when no one checks the information. This is why technology should serve, not replace,  professional duty. AI can be helpful, but it requires that people be watching its use in real time and intervene to prevent consumers from being harmed (Moore et al., 2025).

References

Kalota, F. (2024). A Primer on Generative Artificial Intelligence. Education Sciences, 14(2), 172. https://doi.org/10.3390/educsci14020172

Moore, J., Grabb, D., Agnew, W., Klyman, K., Chancellor, S., Ong, D. C., & Haber, N. (2025). Expressing stigma and inappropriate responses prevent LLMs from safely replacing mental health providers. arXiv.Org. https://doi.org/10.1145/3715275.3732039