AI infra spending boom meets idle GPUs at Beijing salon
A closed-door salon on AI infrastructure hosted by Touzhongwang with Zhongguancun partners, republished by 36Kr, examined the gap between huge AI infrastructure spending and weak efficiency. It cites ByteDance's $29.6 billion syndicated loan, Alibaba's HK$80 billion placement, more than $80 billion raised earlier by Google and Amazon, and NVIDIA's $500 billion plan, while noting China's daily token calls topped 140 trillion in March, over a thousand times early-2024 levels. Participants cited a Carnegie Mellon test of 756 GPUs running 31 days with the cluster waiting nearly 20% of the time, claimed over half of OpenAI's inference energy goes to idle work, and said average GPU utilization at Chinese smart-computing centers is below 30%. Several investors called lowering token cost and raising utilization the new deciding factor; SGLang's RadixArk reportedly raised a $100 million seed at a $400 million valuation, while one dissenting view argued inference revenue will fall sharply as algorithms compress token use by orders of magnitude.
If token demand growth is repeatedly compressed by better algorithms, valuations built on linear token growth could face a reckoning.