TrnRT is an open-source inference engine from RunAnywhere that provides an OpenAI-compatible interface on a single trn2.48xlarge instance. The first supported model is GLM-5.3-Flash (320B MoE, with 18B active parameters). During decoding, it synthesizes one large NKI kernel per half-layer; prefix caching spans device memory and host memory; graphs are compiled offline on the CPU host, and the server no longer compiles by default. The authors used SemiAnalysis's AgentX to replay real programming agent sessions. With 32 concurrent requests and a 1M context window, at the default batch size of 12, the authors report a cost of approximately $0.04 per million tokens based on spot instance pricing.
Why it matters It provides Trainium users with an engine capable of directly serving open-source large models and publishes cost data under programming agent workloads.
Star count 1050 (+771 / 24h)
Reasoning
- 771 new stars in the last 24 hours (reflects attention only)
- The README states that test data comes from an earlier build and lists verification results showing graph-by-graph equivalence between the current code and that build, as well as byte-for-byte identical outputs for 384 requests
Usage
- Build the image using Docker, compile the cache offline on any x86_64 host (which takes several hours), then copy it to a trn2.48xlarge instance to start the service and call it via the OpenAI SDK
Risks
- Supports only AWS Trainium2 and one model, GLM-5.3-Flash; other models are still planned
- Cost figures are calculated based on us-east-2 spot prices from 11 3, which are approximately four times cheaper than capacity block pricing