Together AI and Groq's inference speed war is quietly reshaping AI unit economics

The competitive race between Together AI's optimised GPU-based inference infrastructure and Groq's purpose-built LPU chips, both aggressively marketing inference speed and cost-per-token advantages to developers building latency-sensitive AI applications, has intensified as more application categories - real-time voice agents, interactive coding assistants, agentic workflows requiring rapid multi-step reasoning - have made inference latency, rather than raw model capability, the binding constraint on product quality. Both companies have raised significant fresh funding specifically to expand data-centre capacity in pursuit of this inference-speed positioning, betting that as the industry's centre of gravity shifts from training-focused capital expenditure toward inference-serving-focused capital expenditure, being the fastest and cheapest place to serve an existing frontier model is a more durable business than trying to train a competing frontier model from scratch. Groq's specialised chip architecture, purpose-built for the sequential token-generation pattern of transformer-model inference rather than the more general-purpose parallel computation that GPUs handle, has demonstrated meaningful latency advantages in independent benchmarks for specific model architectures, though the company has faced the classic specialised-hardware challenge of needing continued software and compiler investment to keep pace as model architectures themselves continue evolving in ways that were not necessarily anticipated when the chip was designed. Together AI's more GPU-centric approach, emphasising software-layer optimisation, model-serving efficiency and a broader catalogue of supported open-weight models, has positioned the company as the more flexible option for developers who want inference-speed benefits without betting on a single specialised hardware architecture's long-term viability, a flexibility trade-off that has resonated particularly with startups uncertain about which specific model architecture will dominate their category over a multi-year time horizon. Indian AI application developers, operating with tighter margin sensitivity than well-funded Silicon Valley counterparts, have been active adopters of both platforms specifically for the cost-per-token savings relative to using frontier-lab APIs directly, and several Indian AI-infrastructure resellers have built businesses specifically around helping local developers navigate the growing menu of inference-optimisation options. What to watch: whether Groq's specialised-chip bet continues paying off as model architectures evolve, whether Together AI's more flexible approach captures share from customers wary of hardware-specific lock-in, and how the broader shift of AI capital expenditure from training toward inference serving affects the relative fortunes of infrastructure companies positioned at each end of that spectrum.

Original source: VentureBeat