Meet Carole: A Fine-Tuned LLM for Neurodivergent Users

General-purpose chatbots tend to fail neurodivergent users in one of two ways. They are sycophantic ("Great question!" preambles, agreeing to be agreeable), which erodes trust, or they are bluntly corrective, which for a user with rejection-sensitive dysphoria (RSD) can feel like rejection and end the conversation.

The validate-then-redirect pattern (validate first, then reframe with a question) is one of the few that consistently works for ND/RSD audiences and is central to Nonviolent Communication. It is rarely an LLM's default. Carole is named after Ted's wife, whose instinct for that pattern the bot imitates.

Core question: can fine-tuning plus RAG produce a chatbot that follows the pattern reliably while running cheaply enough to have no ongoing compute bill?

No ongoing GPU spend: renting a GPU for training was acceptable, but inference had to run on a single CPU box that already hosts other projects. Open weights only, with no API wrapper around a frontier model. Portfolio scope, not a product: no multi-tenant auth, no chat history, no scaling story. One person, part-time.

Prompt-engineering alone vs. fine-tuning: a long system prompt can describe the pattern but cannot reliably enforce it when users push the model to take sides, so the pattern was trained into the weights.

Blending existing therapy/counseling datasets (ESConv, Mental Health Reddit, NVC dialogues) vs. Carole-native synthetic data: external sets carry their own personas and dilute the sharper redirect half, so synthetic-only was chosen.

A 22 GB Wikipedia dump vs. a curated corpus: broad retrieval surfaced off-topic chunks, so the corpus became curation-only. Stack Exchange was sampled and excluded because top answers skewed toward neurotypical advice that contradicted the persona.

vLLM/TGI (GPU-oriented) and Ollama vs. llama.cpp: llama.cpp was the mature CPU-only option without an extra dependency layer.

Fine-tune for behavior and use RAG for facts. The model learns how to talk; a curated corpus supplies what to talk about. Quantize to run CPU-only, stream output one sentence at a time because the cadence is part of the persona, and keep the public explanation open while gating the live chat.

Spec approved Mar 31 2026, then three plans: (1) data and RAG pipeline, (2) fine-tuning and inference (Apr 5 2026), (3) frontend and gated demo (May 9 2026).

nginx serves a static Next.js 16 export and proxies /api/* to a FastAPI orchestrator. Each query is embedded with all-MiniLM-L6-v2 (384-dim), the top 5 chunks are retrieved from ChromaDB (1,732 chunks), a ~600-token persona prompt is built, and llama.cpp (Q4_K_M, ~4.5 GB) streams the answer. A SentenceBuffer emits whole sentences as SSE events, followed by a done event with source citations.

Corpus: 957 chunks from 98 hand-curated Wikipedia articles (ADHD, autism, RSD, anxiety, executive function, NVC, CBT/DBT, and more) plus 775 chunks from five reference works.

Fine-tune: 50 hand-written golden examples expanded to ~1,500 synthetic conversations, with a validation pass that rejected ~12% (mostly missing citations or banned phrases). QLoRA on Llama-3.1-8B-Instruct (rank 64 / alpha 128, 3 epochs, load_best_model_at_end) on 1× A100 80GB on RunPod, ~1.5 hours.

Access: a password screen posts to /api/auth, which sets a signed HttpOnly cookie. /api/chat requires it, and the auth endpoint is rate-limited per IP (5 attempts per 15 minutes).

Built in roughly four weeks of part-time work. Eval loss bottomed at epoch 2 and rose at epoch 3, so letting the trainer keep the best epoch mattered. A second run with different data scored worse (1.46) and was discarded. The first frontend palette was near-black and unreadable on real laptop screens, so it was reworked into a warm tan palette tested across several displays. Two of a planned twelve notebooks shipped, and the write-up replaced the rest.

Outcomes not measured yet (no user metrics). Build results from the project write-up: best eval loss 1.40 (epoch 2 of 3), ~26 tokens/sec on CPU (12 threads), a 4.5 GB Q4_K_M model, a 1,732-chunk RAG corpus, and 47 backend tests passing. Spend was about $10 of RunPod compute plus about $8 of API calls for synthetic data.

In live smoke tests the persona reads as one voice, with no banned phrases observed. Known gaps: long answers can drift, and the crisis-hotline rule over-fires on ordinary distress.

Build the eval set first. It was built after the model, so early quality checks were anecdote rather than measurement. Capture failure modes verbatim as they happen (the project now keeps a failure log). Test crisis triggering early. Check design tokens on real screens, and use static export from day one. If a v2 happens, it should tell a new ML story, such as DPO on top of SFT, not just refine the same recipe.