Trillium Labs Plans to Study High-Risk AI Research in the Open

Summary

Trillium Labs, a nonprofit founded by AI researchers Nathan Lambert and Tom Zick, plans to investigate potentially high-risk areas of frontier AI research and publish experiment details for outside scrutiny and replication. Its initial work will focus on post-training, including how fine-tuning and reinforcement learning affect model capabilities, character, and behavior. The lab also intends to study recursive self-improvement, in which AI contributes to the development of new models, and agents, whose growing capabilities have raised concerns about unexpected behavior and human control. The founders argue that leading labs’ restricted access and limited disclosure make it difficult for academics and other researchers to reproduce results or challenge assumptions. The debate has become more urgent because powerful models can automate vulnerability discovery and probe computer systems, prompting arguments for keeping them in the hands of trusted users. Supporters of openness counter that shared evidence and independent replication are necessary for understanding and mitigating risks. Lambert’s previous work includes open AI research at Ai2, Hugging Face, and the American Truly Open Models initiative, while Zick has worked on responsible-AI policy. Trillium Labs launched with undisclosed funding from Schmidt Sciences, Halcyon Futures, and others. The founders aim to raise $40 million to $100 million and plan to spend $30 million on training during the next 18 months.