Miles: RL Infrastructure Grows Up
github.com/radixark/miles | License: Apache-2.0
Training a frontier model with reinforcement learning sounds glamorous until the run collapses 18 hours in because rollout engines drifted from trainer weights, a router made different mixture-of-experts choices, or one dead worker forced a restart. That pain is what separates demo-grade AI from production-grade AI. Miles exists for teams already past the prompt-tuning phase, staring at giant post-training bills and asking a more serious question: how does RL stay fast, stable, and observable when models, clusters, and evaluation loops all get messy at once?
The Drop: Post-Training Hit Distributed Systems Reality
Chatbots made RL feel like a model problem. Enterprises discovered it is also a scheduling problem, a networking problem, a fault-tolerance problem, and, honestly, a correctness problem. Once a team starts post-training large language or vision-language models, the glamorous part, reward shaping and benchmark wins, quickly gets buried under operational headaches.
Traditional setups tend to break in familiar ways. Rollout workers generate trajectories with one set of weights while trainers update another. Mixture-of-experts models route tokens differently during inference and training, which means the model is effectively learning from a world that never existed. Token processing quietly adds waste through detokenize and retokenize loops. A single engine crash can freeze an expensive run. None of this shows up in toy notebooks, but all of it shows up when GPU clusters are burning cash by the minute.
Miles is built for that exact gap. Not “how do models get better?” but “how does post-training stay coherent under real production load?” That framing matters. The repo is less about inventing a new RL algorithm and more about making existing RL recipes survive contact with actual scale.
The Stack: Built Like a Systems Product
Under the hood, Miles is mostly Python, but the architecture rides on SGLang for high-throughput rollout and Megatron-LM for large-scale training, with PyTorch FSDP2 available for teams staying closer to Hugging Face-style model implementations. FastAPI and Uvicorn show up for control surfaces, while Ray-style worker orchestration and low-level transport choices handle the ugly distributed bits.
The Sauce: Async RL, but With Receipts
What stands out is Miles treating reinforcement learning as a continuously running distributed service, not a neat train-then-evaluate batch job. The headline feature is Fully Async RL, where rollout and training are decoupled so generation never has to sit around waiting for every gradient step to finish. That sounds obvious, but async systems usually trade speed for instability. Miles spends most of its architecture budget trying to avoid that trap.
Several design choices snap together here. P2P Weight Transfer pushes fresh model weights to rollout engines in seconds, which keeps async workers from drifting too far from the trainer. Token-in-Token-out, or TITO, preserves token streams directly between rollout and training, avoiding expensive and lossy text conversion in the middle of the loop. For mixture-of-experts models, Rollout Routing Replay, or R3, records expert-routing decisions during generation and replays them during training, so the trainer sees the same routing pattern that produced the sample in the first place.
That last piece is especially smart. MoE instability is not just a compute issue, it is a causality issue. If the routing path changes between rollout and optimization, the system is updating against a slightly different model execution than the one that produced the reward. Miles closes that gap.
Honestly, the interesting part is not raw throughput. Plenty of stacks can go fast in a benchmark. Miles is compelling because speed, consistency, and fault recovery are designed as one system, which is exactly what enterprise post-training teams actually need.
The Move: Where This Creates Strategic Edge
Labs shipping domain-tuned models, coding agents, or computer-use agents could use Miles as the post-training backbone after base-model selection is already done. The practical move is to plug in a task environment, e.g. coding sandboxes, browser tasks, formal math verifiers, or internal evaluation harnesses, then run RL or on-policy distillation without rebuilding the distributed plumbing from scratch.
That matters strategically because post-training is quickly becoming where product differentiation lives. Base models are getting commoditized, but a company’s private task data, reward logic, and eval loops are not. Miles gives teams a way to iterate on those layers faster, with less cluster waste and fewer catastrophic training resets. LoRA and multi-LoRA support also make the repo useful for narrower adaptation strategies, where several specialized behaviors need to be trained without funding a full frontier-scale run.
Startups building vertical copilots should read this repo less as “training infra” and more as margin protection. Better rollout efficiency, fewer failed runs, and tighter evaluation loops can turn model improvement from a research gamble into an operating capability.
The Aura: Reliability Becomes an Expectation
Teams stop treating model improvement like a moonshot when the underlying machinery becomes dependable. That changes behavior fast. Product groups can ask for weekly post-training updates instead of quarterly hero projects. Researchers can test reward ideas without planning around cluster fragility. Leadership starts expecting model behavior to be iterated, not merely hoped for.
Miles points at a future where training infrastructure looks more like cloud infrastructure did after orchestration matured. Not mysterious, not artisanal, just something serious organizations expect to run repeatedly, observe clearly, and recover quickly when hardware or software inevitably misbehaves.
The Play: Infra Picks and Shovels for the RL Layer
From a VC lens, Miles looks less like a pure 0-to-1 category creation and more like the open source wedge into a very large emerging control plane for post-training. The TAM is not “all AI,” it is the subset of labs, cloud providers, enterprise model teams, and agent startups that will spend heavily on continuous model improvement. That is already a big market, and likely expanding as post-training becomes the product moat.
The repo shows early PMF signals: 2,513 stars, tight integration with widely watched infrastructure like SGLang and Megatron-LM, and clear attention to day-0 model support. The moat is probably not data or network effects yet. It looks more like execution speed, ecosystem adjacency, and the trust that comes from surviving ugly distributed workloads where switching costs rise naturally once a team’s training recipes are wired in.
Winners:
Prime Intellect: More credible distributed training products compound faster when open RL infrastructure reduces the cost of proving serious post-training at scale.
Weights & Biases: More recurring RL experimentation increases observability spend because async training and fault recovery create more states, regressions, and metrics to track.
AMD: Stronger open support for large-scale RL on Instinct chips expands addressable training demand beyond CUDA-default buyers and chips away at software hesitation.
Losers:
Adept: Narrow agent differentiation erodes faster when open post-training stacks make it cheaper for newer teams to tune specialist models around concrete workflows.
CoreWeave: Pure infrastructure margins get pressured when customers gain better tooling to squeeze more useful training work out of the same GPU footprint.
SAS: Legacy enterprise AI positioning gets weaker as modern post-training stacks make custom model improvement feel like an engineering workflow instead of a consulting project.
tl;dr
Miles turns large-scale RL post-training into something closer to production infrastructure than research theater. The clever bit is how async rollout, weight syncing, token preservation, and MoE routing consistency work together. Worth a look for model labs, AI infra teams, and startups betting their moat on post-training.
Stars: 2,513 | Language: Python
