how to run a private GPT-4o-style assistant on a home server with sub-50ms response times and cheap NVMe storage
I set out to run a “GPT-4o‑style” assistant on my home server with two concrete goals: keep everything private and reachable on my LAN, and get interactive latencies under 50 ms for token generation while storing models on inexpensive NVMe drives. After testing several toolchains, models and hardware configurations, I ended up with a reproducible approach that balances cost, latency and real-world usefulness. Below I walk through what...