AI

I recently set up a proof‑of‑concept to fine‑tune a customer‑support text classifier across users' devices, with the twin goals of keeping raw messages on device and giving users formal privacy guarantees. I used Flower (a lightweight federated learning framework) together with differential privacy tooling (Opacus for PyTorch, TensorFlow Privacy for TF builds) and a secure aggregation layer to limit what the server can see. Below I walk...

Read more...

When customer support teams adopt retrieval-augmented generation (RAG) to power chatbots, the promise is compelling: fast, contextually-aware answers grounded in a company's own documentation. In practice, however, one problem keeps surfacing — hallucinations. These are fluent, plausible-sounding responses that confidently state incorrect facts or invent citations. I've worked with product and security teams who’ve felt that a seemingly...

Read more...

When I started evaluating self-hosted vector databases for on-device LLM search, I expected a straightforward tradeoff: pick the fastest engine and you're done. Reality was messier. The right choice depends on workload patterns, hardware constraints, embedding strategy, and how much operational complexity you’re willing to accept. Below I walk through what I learned comparing Milvus, pgvector and Chroma—practical differences, deployment...

Read more...

When I moved several long-form writing workflows entirely offline, the single biggest friction point was reliable, accurate dictation that respected privacy and worked without an internet connection. Cloud ASR (automatic speech recognition) is great for accuracy, but for sensitive notes, interviews, or fieldwork where connectivity is spotty, on-device speech recognition is the only realistic option. I spent months evaluating and integrating...

Read more...

I remember the first time I tried to run a modern language model on my laptop: it was slow, memory-starved, and I spent more time swapping RAM than actually getting useful responses. Since then I’ve tested pruning, quantization, on-device runtimes and — most importantly — model distillation. Distillation is the technique that finally let me run capable models locally without paying cloud fees or sacrificing privacy. In this piece I’ll...

Read more...