Skip to content

Jamesob's guide to running SOTA LLMs locally

8.6 relevance
Score Breakdown
technical depth
8
novelty
8
actionability
10
community
9
strategic
8
personal
9

Scored daily by a customisable AI persona to surface the most relevant engineering leadership news.

Practical guide to running SOTA LLMs locally is highly actionable and directly aligns with reader's AI interests

AI/ML github.com
Everything I know about running LLMs locally. Contribute to jamesob/local-llm development by creating an account on GitHub.
Summary

jamesob's guide details building a local LLM rig with 4× RTX PRO 6000 (384GB VRAM) for ~$40k, using last-gen EPYC + DDR4 to maximize VRAM budget, plus PCIe4 switches for sub-µs P2P latency. Ready-to-run configs include GLM-5.2-594B at ~80 t/s with 460k context via vLLM, and a $2k alternative with 2× RTX 3090 for Qwen3.6-27B and local whisper-large-v3 STT.

Author

jamesob