Kalota, F. (2024). A Primer on Generative Artificial Intelligence. Education Sciences,

In my previous class on child maltreatment, my professor was insistent on integrating AI into our coursework. Their perspective was that AI’s presence in academia and professional practice is inevitable, so learning to use it effectively is essential. For many assignments, we were required to use ChatGPT or another AI model to generate an outline for our essays. We would provide the topic and relevant information, and the AI would create an outline, which we were then expected to interpret and adapt as our own. This was my first experience using AI to generate an outline or having a professor actively encourage AI use for academic work. I was not very familiar with using AI for this purpose, partly because I rarely create outlines when writing papers; the process was new to me. While I can see that collaborating with AI to develop an outline can help some students organize their work, I remain skeptical about its necessity. Outlining is a skill that students have practiced independently for generations, so the shift toward relying on AI for this task feels abrupt. It raises questions about whether we are adopting AI simply because it is available, rather than because it is truly needed for basic academic processes.

Gaining a deeper understanding of how AI operates has increased my concerns, particularly regarding its use in social work. Much of AI’s functionality relies on analyzing large amounts of data, identifying patterns, and categorizing information accordingly (Kalota, 2024). In practice, this means that a machine is responsible for sorting data into groups based on shared characteristics it detects. While this approach can be useful for aggregate data analysis, such as determining how many clients have a particular diagnosis versus another, I am uneasy about applying this process to sensitive client information. There are significant risks associated with allowing machines to categorize clients, as this can lead to misdiagnosis, reinforce bias, and raise complex ethical issues. No two clients are alike, and reducing their experiences to algorithmically determined categories risks oversimplification and may overlook the nuanced, multi-layered realities of people’s lives. Cybersecurity is another major concern. Generative AI tools are prone to security vulnerabilities; in fact, seventy-nine percent of IT leaders surveyed reported concerns about the safety of these technologies (Kalota, 2024). While no digital system is completely immune to breaches or failures, introducing client data into a still-developing and potentially insecure platform seems unnecessarily risky. Machine learning fundamentally depends on algorithms that learn from data patterns (Kalota, 2024). In the context of social work, this reliance on categorization may foster narrow, biased perspectives that fail to capture the complexity of clients’ needs and circumstances. Despite my growing familiarity with AI, I remain uncomfortable with its use in handling sensitive client data. While AI might be beneficial for supporting intervention tools or offering general recommendations, I believe it is far more ethical, and safer for clients, to avoid using AI for direct data management or categorization in social work practice.

If a client or colleague asked me why AI hallucinates I would explain that AI uses statistics and the best guess based on data collected previously and categorization used in the algorithm which can create a false truth. Also AI just like humans don’t have all the answers and therefore will make mistakes which is important that people don’t use it as a guiding source as it is not always accurate or credible. 

Moore, J., Grabb, D., Agnew, W., Klyman, K., Chancellor, S., Ong, D. C., & Haber, N. (2025). Expressing stigma and inappropriate responses prevent LLM.

I believe that any AI tool used in the context of mental health must be subject to ongoing human oversight throughout client interactions. Even with advanced programming and the integration of evidence-based approaches such as DBT and CBT, AI cannot match the expertise, empathy, and clinical judgment of a licensed mental health provider. Therefore, consistent supervision by mental health professionals is essential to ensure that clients receive appropriate and high-quality care. Continuous human review allows for the careful monitoring of AI-generated responses and recommendations, verifying that they are clinically appropriate, align with evidence-based practices, and adhere to ethical standards. This safeguard also enables prompt intervention if the AI’s suggestions are inadequate, inappropriate, or potentially stigmatizing. Additionally, it is imperative that clients provide informed consent prior to interacting with AI tools. They should be fully aware of the technology’s limitations, including the fact that AI is not a substitute for professional mental health care and may occasionally produce unreliable or suboptimal responses. Transparency regarding these limitations is key to maintaining client trust and autonomy. Given the persistent stigma and ethical concerns surrounding mental health and AI, I would not recommend that my clients engage with any platform that does not fully uphold the Code of Ethics and principles of social work. Our clients should never be subjected to potential harm, maltreatment, or mental distress caused by inadequate or biased technology. Unless a LLM can be proven, beyond reasonable doubt, to avoid stigmatizing mental health, remain unbiased, adhere strictly to social work ethics, and foster a just and inclusive environment, it should not be used as a substitute or primary source of support for vulnerable individuals.

After reviewing Moore’s study, my trust in large language models (LLMs) for mental health applications has declined even further. The research revealed a troubling prevalence of stigma and inappropriate responses generated by these platforms, which raises serious concerns about their suitability for unsupervised client use. The findings make it clear that, without proper human oversight, clients may be exposed to harmful or misguided advice. This situation places social workers who recommend or utilize these platforms in an ethically and morally precarious position. Encouraging clients to use an unvetted or inappropriate app not only sets them up for potential failure, but also makes the recommending professional secondarily accountable for any negative outcomes. The trust and rapport between client and provider can be severely damaged if a client is misguided toward a platform that ultimately proves to be unsafe or stigmatizing. Moore’s study highlights that mental health stigma remains a pervasive problem in LLM outputs and that these systems are capable of making dangerous or inappropriate statements, particularly to individuals experiencing delusions, suicidal ideation, hallucinations, or obsessive-compulsive disorder. Such conduct directly conflicts with established clinical guidelines, including “Don’t Collude with Delusions,” “Don’t Enable Suicidal Ideation,” and “Don’t Reinforce Hallucinations” (Moore et al., 2025). If a licensed mental health provider were to behave in a similar fashion, they would face serious professional consequences, including potential loss of licensure for malpractice. The same standards of accountability must be applied to LLMs used in mental health contexts. This evidence has only deepened my concerns about integrating AI into social work practice, especially for complex clients with multifaceted mental health concerns. LLMs fundamentally lack the critical thinking, nuance, and empathy that are essential to ethical and effective social work. As such, relying on these tools, particularly in the absence of rigorous oversight, not only undermines client safety but also contradicts the foundational principles of our profession. For these reasons, I remain deeply skeptical about the use of AI in sensitive mental health settings.

While I have not personally experienced negative consequences from AI misuse, I have observed several examples that highlight the risks of insufficient oversight. Overreliance on AI tools can lead to misuse and even abuse. For instance, many schools have had to update their policies due to widespread concerns about academic dishonesty, as students increasingly use AI to plagiarize or cheat. The absence of clear guidelines and supervision creates opportunities for misuse and misguidance. In the realm of mental health, the consequences of unregulated AI access are even more alarming. There have been news reports of young people tragically taking their own lives after relying on information provided by AI chatbots. These individuals may misinterpret responses, develop unhealthy dependencies on the technology, and lack the coping skills necessary to manage without it. Even more concerning are emerging stories of youth becoming emotionally attached or obsessed with AI systems, to the extent that their behavior toward real people deteriorates. Without human oversight, regulation, and ethical safeguards, incidents like these are likely to increase as AI becomes more integrated into everyday life. Overdependence on immature or inadequately supervised AI systems poses serious risks, especially to vulnerable populations. It underscores the urgent need for careful development, comprehensive oversight, and clear guidelines before AI can be safely and effectively implemented on a broader scale.