Part 1

One concept from Kalota’s Primer that genuinely shifted my understanding of AI was the idea that generative AI does not “look things up” or retrieve facts. Instead, it predicts the next most likely word based on patterns in its training data. I previously assumed AI pulled information from a database of the internet in real time, so learning that it is essentially a pattern matching system and not a fact checking one complicated how I think about its accuracy and limitations.

I have used generative AI in several part of my life without fully understanding this mechanism. In coursework, I’ve asked AI to help brainstorm ideas for group projects. In my internship, I’ve used it to draft emails or summarize long documents. Even personally, I’ve used AI for recipes, travel ideas, or quick explanations. Ina all these cases, I treated the output as if it came from a knowledgeable source, not statistical model predicting text. Understanding how AI actually works make me both more cautious and more comfortable with its presence in social work practice. I am more cautious because I now recognize how easily it can produce confident but incorrect information. But I am also more comfortable because knowing its limitations helps me set appropriate boundaries. AI can support administrative tasks, brainstorming or accessibility but it cannot replace human judgement, ethics and relational practice.

If a client or colleague asked why an AI tool “hallucinates”, I would explain it in a plain language: AI doesn’t know things, it predicts things. It generates text by guessing what words usually come next based on patterns it learned during training. Sometimes those guesses sound very confident but are completely wrong because the system isn’t checking any actual resources, it’s just following patterns. A hallucination is simple the model making a prediction that looks plausible but is not true.

Part 2

When I think about AI showing up in mental health adjacent spaces, my reaction isn’t simple. Moore et al findings didn’t surprise me, but they did force me to confront something uncomfortable; even when people mean well bias still slips through. And because AI learns from us, it absorbs those same biases and repeats them with confidence. That’s what makes me cautious. Not anti-AI just unwilling to pretend it’s safer than it is.

If an agency insisted on using AI for intake screening, the one the thing I feel like they should have in place is a human checkpoint before any AI-generated recommendations are acted upon. Without that, the tool could easily mislabel someone, reinforce stigma, or miss something critical. Moore et al. helped me draw a clearer line between what AI can do and what it shouldn’t touch. AI is fine for structured, low stake tasks like mindfulness scripts, appointment reminder, but anything involving emotional nuance, crisis cues, and identity0related content needs to be in human interactions. The study shows how quickly AI can slip into harmful assumptions and that’s enough for to say that relational works needs to stay human.

My internship has already shown me what happens when oversight is missing. Ive seen automated school systems flag students as “noncompliant” when the real issue was transportation or disability related. No one intended harm, but harm still happened because no human stopped to ask, “does this label make sense?” That experience makes me wary of any tool that operates without human judgment.