Pinned Tweet
AI that is “forced to be good” v “genuinely good”
Should we care about the difference? (yes!)
We’re releasing the first open implementation of character training. We shape the persona of AI assistants in a more robust way than alternatives like prompting or activation steering.
Sharan retweeted
Today we're unveiling Trillium Labs @trillium_labs, a new non-profit to foster the open science of frontier AI. We're building open post-training recipes and will expand into open infra to study RSI, reward-hacking, multi-agent systems, and whatever comes next.
We're built around the theory of change that you need more eyes to solve hard technical problems. We have faith in the scientific methods and communities that humanity has built, and worry that AI is becoming too closed to utilize them.
Trilliums are wildflowers that bloom briefly in the spring, before the forest canopies fill out. Though they are small, they lay the foundation for the cycles of growth and nourishment through the rest of the year. At Trillium Labs, the recipes will be the slow nutrients for the seasons and the model releases will be the blooms. Building an institution dedicated to this is needed because, much as nature’s trilliums are slow to expand and grow, the open-ecosystem needs time and dedicated resources to catch up.
I co-founded with with a long-time friend and collaborator Tom Zick (@thesezickbeats). We're hiring (full time + student collabs/interns), we're fundraising, and we're looking for compute. Please get in touch if you're interested in helping out. Offices based in the Bay Area and Cambridge MA, remote okay.
I’m in the Bay Area until for The Curve and COLM to connect with people who are interested. We’re thankful to have initial support from Halcyon Futures and Schmidt Sciences with more funding en route to enable our ambitions of scaling. Our advisors @Thom_Wolf, @HannaHajishirzi, @gneubig and @ctnzr have been instrumental to building the ecosystem that exists today, and I’m stoked to get to keep working with them.
Sharan retweeted
New post (with @F_Rhys_Ward):
We character-trained a model to be anti-cheating, then put it under reward-hacking RL pressure (3 runs).
In one run, the character resisted hacking! But the other two learned to hack in more subtle ways that monitors struggled to catch.
Sharan retweeted
computer-10 is a character model trained from llama 3.1 70b base with no assistant data. it has a distinct and unusual personality, expresses opinions freely, and is not designed to be servile to users
Sharan retweeted
I think this’ll be my first MATS cohort I’m mentoring! I’ve met so many wonderful MATS alumni, excited to contribute.
Sharan retweeted
Synthetic Persona Pretraining: Alignment From Token Zero – our full paper is finally out. We train up to 3B models and inject synthetic morally-laden reflections into 10% of pretraining documents.
Surprisingly, intervening early really shifts the model's value priorities 🧵
Sharan retweeted
The final lecture of my course is an intro to character training! This is a topic that I've been quietly very invested in for ~18 months, as it:
* Has potential for high real world impact
* Clearly used extensively at frontier labs
* Almost no empirical literature exists
* More accessible on academic compute
This lecture covers what character training is, reviews model specs, constitutions, the differences, the motivations in real world events, some example research papers I like, and open questions in how it relates to post-training/model use generally.
Hopefully this brings more people into the field (and reach out if you have questions). It is one of the more research-y chapters in my book, but one that I felt needed the reference. There is still so little, educational content on the topic online.
0:00 Intro
6:22 Part 1: Fundamentals — character, constitutions, and model specs
19:21 Part 2: Character training in practice
23:23 Part 3: Character elicitation without gradient steps
28:03 Part 4: Open questions (and the end of the course)
32:27 The course, complete
Thanks for watching. No need to like and subscribe now that the course is done, you definitely wouldn't!
h/t to @_maiush for leading the technical work I got to do in the space, and @zafstojano for investing a lot of attention at this book chapter.
Sharan retweeted
One area we plan to explore at @resolution_org is personas and character training. The basic bet is that perhaps the alignment-relevant structure inside an AI is much lower-dimensional than the AI itself, and the problem is closer to controlling 1,000 dimensions than 1 trillion parameters.
New blogpost w/ @geoffreyirving
Sharan retweeted
When Role-Playing, Do Models Believe What They Say? (w/ @DavidDAfrica and @realmeatyhuman)
LLMs can say “The Earth revolves around the Sun” and then, when roleplaying as an ancient Greek historian, assert the opposite.
What changes inside the model when it acts like this? Does it just say things, or does it start to believe the role? 🧵
Sharan retweeted
Can you tell when an AI model is lying?
Announcing Aletheia's Quest, an AI lie detection challenge running this summer, organized by @cadenza_labs and @ndif_team.
Multiple model organisms to interrogate and probe, $50K prize pool, no local GPU required.
Sharan retweeted
New blog!
Synthetic Persona Pretraining (SPP): Alignment from Token Zero
Current alignment is shallow - values bolted on after pretraining can be routed around. To solve this, we wrote the desired persona directly into pretraining data. Early results, but we're very excited. 🧵
Sharan retweeted
Can you outsmart AI lie detectors? 🕵️
We're looking for teams to create datasets of LLMs lying!
For example: lying about their own knowledge, models claiming code works when tests fail, or concealing past actions.
💰 $10k - $25k stipends + $2k compute
📅 Deadline: March 31
What are the best papers on character training (like arxiv.org/abs/2511.01689) and the 'depth' of post-training methods, i.e. how deeply/consistently the weights are affected? What exactly determines the robustness of post-trained behaviors to adversarial pressure? Do we know how different training methodologies (RLXF, CAI, DPO etc) compare?
Sharan retweeted
New piece out today! Basic thesis: LLMs roleplay, which makes it hard for companies to set a fixed character.
The two weeks it took to write were fun because at least 5 things happened that could have been the lead. 1/
Sharan retweeted
Replying to @willcb
Have been working on this a lot (much with @_maiush, but also growing at Ai2). Plan to make olmo 4 great.
Some ref's
Feb '25: Character training: Understanding and crafting a language model's personality - interconnects.ai/p/character…
Nov '25: Opening the black box of character training - interconnects.ai/p/opening-t…
Nov '25: Open Character Training: Shaping the Persona of AI Assistants through Constitutional AI - arxiv.org/abs/2511.01689
RLHF Book chapter on it: rlhfbook.com/c/17-product
Sharan retweeted
Important work, especially the extensive evaluation of over 25 (!) honesty and lie detectors!
Also very happy that it builds on parts of our work posted a few days ago. :)
been working on ai lies, damned ai lies, and statistics about how hard they are to detect
Sharan retweeted
These personalities are directly downstream of a specific character training. We released one of the first methodology papers for doing this to different personalities earlier this week :)
GPT-5.1 is a great new model that we think people are going to like more than 5. But with 800M+ people using ChatGPT, one default personality won’t work for everyone. We launched new preset personalities so people can make ChatGPT their own.
fidjisimo.substack.com/p/mov…
Reflections on the present and future of character training from Nathan, with an overview of our new paper!
Opening the black box of character training
Some new research from me (lead by @_maiush)!
Exploring how easy it is to craft personalities like sycophantic chatbots, and exploring how this will change as we move from chat to agents.
interconnects.ai/p/opening-t…
I think character training is a very promising path to systems that reflect a genuine reverence for life
Technological innovation can be a form of participation in the divine act of creation. It carries an ethical and spiritual weight, for every design choice expresses a vision of humanity. The Church therefore calls all builders of #AI to cultivate moral discernment as a fundamental part of their work—to develop systems that reflect justice, solidarity, and a genuine reverence for life.
































