Periodic Labs Details AI Infrastructure for Scientific Model Training

Summary

Periodic Labs describes an infrastructure stack for training specialized models and using them to analyze experiments in high-throughput laboratories focused on superconductors and magnets. The company says its final training run used a peak of 1,300 H200 GPUs across midtraining and reinforcement-learning phases, starting from open weights. It reports 4.1x the training throughput of its Megatron baseline on long, variable-length inputs and says its SGLang-related inference work made its use case 2.5x faster. The stack runs generation and training asynchronously, using techniques including WideEP, delta router replay, prefill-decode disaggregation, KV-cache reuse, and faster distributed checkpoint resharding. Periodic says prefill-decode disaggregation reduced rollout duration by 2x, while combined inference changes increased decoding from 10 to 25 tokens per second per request for a 1T model at the same batch size. Checkpoint conversion from Megatron to Hugging Face format was reduced from 30 minutes to one minute. For training, sequence packing and cost-aware scheduling reduce padding and workload imbalance; at 256K context, the company reports about 350 TFLOP/s per Hopper GPU for the trillion-parameter Kimi K2.6, and full-parameter training at 64K context on 64 H200s. Memory optimizations lowered peak memory from 320 GB to 132 GB in that 64-GPU, 64K-context setting. Periodic also built pbox, a Slurm-integrated gVisor sandbox that runs on spare CPU capacity alongside RL jobs. In a test with 100 sandboxes, it reports 4.5x lower p50 transfer latency and 3.3x aggregate throughput than a hosted provider; sampled code executions had a median 2.5x speedup, with pbox faster in 86.9% of cases. The company says related improvements have been contributed to Megatron-LM, SGLang, and Miles, and identifies multi-agent RL, lower-precision post-training quantization, distributed KV caching, and larger sandbox deployments as future work.