High-Concurrency GPT Inference Batching System

Design the infrastructure around a fixed GPT inference API so it can handle a burst of roughly 100,000 concurrent requests with low latency and return every inference result to the correct caller.