Should you ship on-device or in the cloud for compressed LLM inference on consumer hardware?
Hybrid edge-cloud routing can cut inference cost dramatically because local queries avoid API spend, latency, and data egress, but the business case only holds when the on-device model can service the majority of traffic — otherwise the infra and platform overhead wipe out the savings.