Part 1- Kalota’s Primer

The first time I really encountered generative AI was through the release of ChatGPT. Although I never really encountered it in my professional experience, I did use it out of curiosity in my personal life. I would use prompts centered around outlining an actionable plan, holding a basic conversation, and revising/formatting different inputs like emails. When I first used ChatGPT, I thought it was just like a more efficient search engine that could find answers more efficiently without overloading the user with unnecessary information and tailor information to the user. After reading the article, I was surprised to learn that generative AI doesn’t really search the internet for answers like I originally thought. Instead, it relies on the data it was trained on to recognize patterns and predict the correct response.

Understanding how generative AI works makes me more comfortable with its presence in social work. I now have a better grasp of how it computes and generates data as well as having more experience interacting with AI through its rapid presence in everyday society. AI can be a very useful tool that can help social workers complete tasks like summarizing information, designing or brainstorming initiatives, and explaining information to clients more effectively. Despite this, I am still cautious about how it is used in a social work setting. Social work involves ethical decision-making and working with vulnerable populations. Things like risk assessment, counseling, and deciding interventions for clients are very sensitive areas. AI can’t replace ethical judgement, especially considering how it can give inaccurate information. Although AI can be useful, it is important to understand AI’s capabilities and limitations, so people use it responsibly.

If someone asked why an AI tool “hallucinates”, I would say it is the result of AI trying to give a correct answer based on the limited capabilities or information it has access to. AI is designed to predict responses based on the patterns found in the data it was trained on. Responses not being verified, data can be limited, and irrelevant patterns are all factors that could lead to incorrect information being produced by AI.

Part 2- Managing, Not Just Rejecting, AI in Mental-Health-Adjacent Practice

I think AI interaction with clients facing mental health issues should be limited. AI could be used for tasks like answering common questions about services, basic educational facts, or helping clients schedule appointments. However, when it comes to discussing more sensitive information with a client like mental health symptoms and assessment, the need for a human professional becomes necessary. Moore et al. finding that large language models sometimes produced stigmatizing, dismissive, or clinically inappropriate responses informed my decision. These possible responses from AI could result in clients not seeking treatment or following information that could put them at risk. By limiting AI use to low-risk tasks like managing documents, clients are better protected from receiving potentially harmful information from AI. These finding draw a clear line between tasks that are appropriate for AI and those that require a human professional. AI can support basic administrative and educational uses, but tasks that require clinical judgment or risk assessment need to be done by mental health professionals. Determining whether someone is experiencing a mental health crisis or deciding on an appropriate intervention requires an understanding that AI doesn’t reliably have.

In my personal life, when I used to work in construction oversight was very important in terms of safety. Sometimes on the job site people would be so focused on completing the task at hand quickly, they would intentionally or unintentionally not follow safety protocols. This was dangerous because not following safety measures could lead to injury or even death. Whenever the safety manager came to the job site, people had to make sure tools were used properly and proper safety gear was being worn or risk being sent home due to liability concerns. Unfortunately, if the safety manager wasn’t around those protocols weren’t always being followed and sometimes accidents would occur. Although this example isn’t directly related to social work, I think it highlights the importance of oversight and systems that enforce protocols to protect the well-being of people.

References

Kalota, F. (2024). A primer on generative artificial intelligence. Education Sciences, 14(2), 172. https://doi.org/10.3390/educsci14020172

Moore, J., Grabb, D., Agnew, W., Klyman, K., Chancellor, S., Ong, D. C., & Haber, N. (2025). Expressing stigma and inappropriate responses prevents LLMs from safely replacing mental health providers. In Proceedings of the 2025 ACM Conference on Fairness, Accountability, and Transparency (FAccT ’25) (pp. 599–627). Association for Computing Machinery. https://doi.org/10.1145/3715275.3732039