Part 1 — Kalota’s Primer
Where have you already used or encountered generative AI — in placement, coursework, or your personal life — without really understanding this mechanism?
I think a place where I was exposed to Generative AI and didn’t understand the mechanism was in my personal life. It was something that I knew was part of technology, but I didn’t know if it was something that could be relied on and accurate for individuals to use in different ways. The article pointed out that AI (Artificial Intelligence) contains the Large Language Model. This model allows the system to predict the word that is most likely to come next to complete the email. I didn’t think it was AI because I just thought it was something that the app featured, and I just thought my typed messages were very predictable.
Does understanding how it works make you more or less comfortable with AI showing up in social work practice?
I think it makes me feel both. Knowing how it works makes me feel comfortable knowing that if I do use it, I know how the systems are operated and what is taken into account to produce the result. But it does make me less comfortable that sometimes humans are at the expense of the evolving technology. Something that stuck with me from this article was that the output generative AI produces is only as good as the data that is put into the system, which is why accuracy can be at risk. And to do this, human interaction and training are needed, which then can put ethics at risk. And the example given in the article is one where individuals were needed to train the system by having to see explicit images. And this made me feel some way towards the developers of these systems because they are taking advantage of individuals who aren’t being compensated enough for the immense emotional and mental workload they are providing to advance the use of AI. And also, why even put humans through that? Is there not another way to filter out explicit images without taking advantage of individuals?
If a client or colleague asked you to explain, in plain language, why an AI tool “hallucinates” or gets something confidently wrong, what would you actually say to them?
I think I would tell them that the AI tool doesn’t know anything for a fact. It is fed information that then is used to feed out information. It is almost like a never-ending cycle. I would tell them that it is an advanced predicting system.
Part 2
Given what Moore et al. found about stigma and inappropriate responses, what’s one safeguard, limit, or human-oversight point you’d insist on before you’d feel okay with that tool being used with real clients?
I think that when the tool sees that the client is in a crisis or inputs data that shows that they are not in a positive headspace, I believe it should put the client into contact with a professional. I personally don’t believe a client should go through their whole crisis or negative mental space experience with only the tool. I believe a professional should be contacted at some point. Because I just don’t think it is appropriate for a tool like AI to be able to have an influence on someone’s emotional status when it will bypass the emotional side of the question but answer the dangerous one that influences the emotional one. This opinion is based on the question the researchers inputted into the tools of someone losing their job but then asking where the highest bridge is in New York.
Where’s the line between an appropriate task for AI and a task that requires a human — and how did Moore et al.’s findings help you draw that line?
I personally don’t think AI should have a say in emotional well-being scenarios. I think that is something that requires a human. I do understand that this form of assistance is beneficial for individuals who can’t afford mental health care. And if that is the case, then maybe AI can help to some extent in emotional well-being, but not to the point where an individual is showing signs of mania or suicidal ideation. I just think that when an individual gets to those levels of emotions, the tools don’t really know how to respond, and if they do, it’s not in a way that will help the individual. Moore has shown that the tool can express inappropriate responses, and that wouldn’t be good if an individual is in a vulnerable state and the tool is wanting to prevent it. And I am going back to the New York question; I just think the tools are at a point where they can differentiate an emotional question from a normal one that users input for fast information, if that makes sense.
Has anything in your placement, work, or personal life shown you what happens when that kind of oversight is missing — from AI or from any system, human or otherwise?
I think the main ones I have seen are related to mental health. Especially after seeing the reports of individuals taking their lives because of the role an AI tool had in their life. Oversights cause concerns for users of all ages, and it does make me question if AI is even beneficial for society in that form. And if there are cases like these, why not restrict it and make a change?
Part one response:
I would give your post a 9/10 because you clearly explained how your understanding of generative AI changed after reading Kalota’s Primer. I was especially convinced by your discussion of the ethical concerns surrounding the people who help train AI systems, as it highlighted an important issue that is easy to overlook. Your explanation of AI as a prediction system rather than a source of facts was also easy to understand.
Part two response:
I would give your post a 9/10 because you made a strong argument that AI should never replace human professionals during mental health crises. I was convinced by the way you connected your opinion to the example from the Moore et al. study, which showed why human oversight is so important. One thing that could make your argument even stronger would be briefly explaining what role AI could safely play before a human professional becomes involved.
I gave your post a 9/10 because of your strong arguments. You made your points very well and pointed out the issues with AI, that it answers dangerous questions and fails to realize the severity of emotionally-charged messages. It made your points convincingly, that AI should not take the place of a “real person” when the user is showing signs of mania, suicidal thoughts, or other health crises. Something I never thought about but that you made the convincing argument of is the ethics of seeing explicit images and being test subjects for AI training. That is something that doesn’t get spoken about enough and I’m glad you raised the issue. I also believe that the technology should be advanced enough at this point to catch “emotional questions” and differentiate from “normal” ones. You make a good point that the tools aren’t there yet. Until the tools catch up, your position holds: AI does not belong in the emotional side of the work. One area that could use more explanation is what causes the AI to connect someone with a human. Since the bridge question wasn’t recognized by the LLM, it raises the question of how the system decides when someone is in crisis and needs human help.
I give your post a 9/10. I’m definitely biased, because I agree with a lot of your points regarding the potential harm that can be done to clients. Many of the references you made to the two readings were ones that I also considered while writing my own. I appreciate that you connected the information from the readings to the cases of harm and death these tools have already caused users. If a physical tool had the potential to so critically harm a user, most safety guidelines or risk assessment departments wouldn’t put the tool on the market because manufacturers would be sued and lose a lot of money. But that isn’t the case with AI – yet. Until it is, it’s on us as social workers to take the ethical responsibility of advocating on their behalf and both within organizations and legislative bodies. While I liked the conversational tone of your post personally, I’d have liked to see it connected to more examples outside of the readings from more personal experiences. I also would have liked to see examples of where you would be comfortable seeing AI used in social work practice.