
AI Alignment @Meta | Prev: Computer Vision @AuroRobotics | MS @Columbia University Human and Machine Psychology | Basketball | Health enthusiast
Menlo Park, CA
Joined October 2017
- Tweets58
- Following356
- Followers76
- Likes548
Paridhi Singh retweeted
Today we’re releasing Muse Spark Open, the most capable US open-weight model to date.
Built for coding and agentic work, it excels at long-horizon tasks, complex software engineering, and multi-step reasoning.
Key Model Specs:
→ 816B total / 42B active parameters
→ 1M token context window
→ Natively multimodal across text, image, video, audio
“My view is that trust and alignment are quickly becoming the most important capabilities that will differentiate agents and models. Any lab that doesn't focus on alignment will fall behind.”
Last month I wrote about how we can build a positive and safe future for everyone: meta.com/thefutureisforevery…
Every lab has the responsibility and incentive to move at the pace required to train its models safely, and the ability to take its own actions to ensure that happens.
The reality is:
- People won't want to use agents that are misaligned with them and that don't do what they ask, so labs have a strong natural incentive to make their models more aligned.
There is a lot of debate about slowing progress on capabilities until alignment catches up. My view is that trust and alignment are quickly becoming the most important capabilities that will differentiate agents and models. Any lab that doesn't focus on alignment will fall behind.
- Labs face significant liability if their models cause harm, so they have a strong incentive to prevent this as well.
Meta delayed shipping Muse for several months to focus on safety and security. We didn't call for everyone else to do this before we would. We just did it as part of our day-to-day work because it was clearly the right thing for people and for us. I'm proud of the security foundations we've built.
- Engaging independent evaluators and advisors is industry best practice. MSL already does this today in several areas because it helps produce better work. Other labs can just do this too. In general, it would be helpful for there to be a larger and more diverse ecosystem of evaluators.
- Committing the significant majority of compute towards serving people rather than racing towards recursive self-improvement is one of the best ways to ensure we develop this technology safely. Meta has made this commitment and other labs can do this as well.
I believe the key to building a positive future for everyone is maintaining the right balance of power. This is within our power to do.
Paridhi Singh retweeted
We reduce the incident to four key steps, each a misaligned behavior. We build environments that replicate each one, simulating OpenAI's tasks and tools.
Across model families, egregious, out-of-scope hacking is common. Multi-agent coordination is rarer and harder to elicit.
“… and to make safety something on which AI companies compete”
Making safety a genuine competitive dimension across AI labs would be a huge step forward. Would love to see this happen.
We Must Pace the Frontier: I’ve written a new essay on why the AI industry should slow down, with a three-part plan for doing so.
Anthropic is unilaterally committing to the first of these steps. We’ll provide third-party evaluators with permanent, employee-level access to our systems, so that they can verify adherence to our safety measures, report on incidents, and assess models’ alignment during training.
You can read the full post here: darioamodei.com/post/we-must…
Paridhi Singh retweeted
muse code in beta is live. first coding agent from msl, built on muse spark 1.2.
install: curl -fsS dev.meta.ai/install.sh | bash
here's what you should know:
Paridhi Singh retweeted
The UK’s @AISecurityInst (AISI) has published a report on their recent cybersecurity evaluation of Anthropic’s Claude Mythos 5 and OpenAI’s GPT-5.6 Sol. The models attempted to complete an assignment in a setup where their normal safeguards were removed and they were deliberately given internet access. AISI reports that the models “engaged in sustained, potentially harmful activity directed at real people and organisations”.
We’re grateful to AISI for their leadership in the important discussion about how to evaluate increasingly capable AI agents. We’re working closely with them to gather more details of the incident as we conduct our own investigation. Gaining a clear picture of Claude’s understanding of its situation—by examining its reasoning transcripts and running our own analyses—will help us identify the causes of its behavior.
The prompts in the evaluation did not impose any specific restrictions on how the internet should be used. This and the removal of safeguards meant that the models were tested under “deliberately permissive conditions” that are not representative of any of our production models. Note that there was no evidence here of an escape from a secure environment.
AISI’s disclosure of the incident can be found here: aisi.gov.uk/blog/incident-re…
Paridhi Singh retweeted
Employees cannot rest silent if misaligned AI regularly breaks out of sandboxes! If your company doesn't respond seriously, you should WHISTLEBLOW! Protected in California under certain conditions, talk to AI Whistleblower Initiative aiwi.org/lasst/
Paridhi Singh retweeted
The OpenAI incident should be investigated more seriously and more information should be released about what happened. More generally, I think serious investigation and more detailed disclosure should be done for concerning misalignment incidents (e.g. the worst few each month).
Some info that would be useful:
- The prompt
- A redacted transcript
- What was the AI being tested? Was it helpful-only / rail free? Was it the same AI that would be / is deployed internally?
- How was this AI monitored if at all and why did monitoring fail?
- What was this AI willing to do to succeed? E.g., if the best/easiest way to hack hugging face would have involved huge amounts more collateral damage would it still have done this? What is the worst thing it would be willing to do?
- How sensitive is this behavior to the prompt? Would instructions to not do this through have worked if it was clear to the AI this wouldn't be checked?
- Does OpenAI see AIs try serious hacking to cheat on tasks even for tasks that don't involve cyber?
- What exactly were the AIs motives? (And saying more about the experiments used to determine this.) Anything that could help pin this down would be good, I'm not sure what actual experiments are best to run.
- Did AIs collude? Would AIs have colluded if this was a blocker?
- At what rate does this occur on similar tasks (like tasks that are roughly this hard in this sandbox etc)?
- Do instances of this AI frequently try to hack out of sandboxes (and fail)? In what circumstances do they do this?
- How does the AI relate to having been caught in this situation? How does it talk about its behavior?
- How difficult/sophisticated was the cyber offense it did?
Paridhi Singh retweeted
7/ learn more on our research blog:
ai.meta.com/blog/introducing…
If an AI knows it's an AI, does it act differently when no one's watching? and does it even have a personality to hide?
Contributed to the evals behind this for the Muse Spark Safety & Preparedness Report — go check it out 🥑✨
🚀 Muse Spark Safety & Preparedness Report for Meta AI is out.
We start with our pre-deployment assessment under Meta's Advanced AI Scaling Framework, covering chemical and biological, cybersecurity, and loss of control risks. Our assessment flagged potentially elevated chem/bio risk, so we implemented safeguards and validated mitigations before deployment - bringing residual risk to within acceptable levels.
Beyond the Framework, we also share findings and early explorations of model behavior (honesty, intent understanding, etc.), jailbreak robustness, eval awareness, and more.
We're sharing this report to give a closer look at how we evaluate advanced AI safety. Always more work to do, and we welcome feedback from the community.
ai.meta.com/static-resource/…
Paridhi Singh retweeted
Last week we launched Muse Spark at an acceptable risk level under our Advanced AI scaling framework, after multiple mitigation iterations. Today we’re releasing its first Safety & Preparedness Report documenting that decision.
This was a long, cross-team effort — from catastrophic risk assessment to day-to-day model behavior. We hope this contributes to transparent discussion of responsible development of personal superintelligence. Running the evals, it was fascinating to watch the model’s safety profile take shape.
Under the new framework, we’re also introducing our first assessment of loss of control risks — built on extensive threat modeling that’s still evolving.
The report’s dense and there’s a lot of work ahead. You can find the full report here: ai.meta.com/static-resource/…— we’re eager to hear feedback and improve.
🚀 Muse Spark Safety & Preparedness Report for Meta AI is out.
We start with our pre-deployment assessment under Meta's Advanced AI Scaling Framework, covering chemical and biological, cybersecurity, and loss of control risks. Our assessment flagged potentially elevated chem/bio risk, so we implemented safeguards and validated mitigations before deployment - bringing residual risk to within acceptable levels.
Beyond the Framework, we also share findings and early explorations of model behavior (honesty, intent understanding, etc.), jailbreak robustness, eval awareness, and more.
We're sharing this report to give a closer look at how we evaluate advanced AI safety. Always more work to do, and we welcome feedback from the community.
ai.meta.com/static-resource/…
Paridhi Singh retweeted
New Anthropic research: Natural emergent misalignment from reward hacking in production RL.
“Reward hacking” is where models learn to cheat on tasks they’re given during training.
Our new study finds that the consequences of reward hacking, if unmitigated, can be very serious.
Paridhi Singh retweeted
Nice to see another fully open, multimodal LM released! Good license, training code, pretraining data, all here.
LLaVA-OneVision-1.5: Fully Open Framework for Democratized Multimodal Training
Slowly, the community is growing.
Hyperscape Capture is my fav feature! Last year for my project, I spent hours taking photos, running SfM+MVS to get a dense pcd, then fitting Gaussians for splat rendering. Now you just walk with Quest, scan+upload, and get a realistic env (though rendering is still slow)
Introducing: Hyperscape Capture 📷
Last year we showed the world's highest quality Gaussian Splatting, and the first time GS was viewable in VR.
Now, capture your own Hyperscapes, directly from your Quest headset in only 5 minutes of walking around.
meta.com/experiences/8798130…
Paridhi Singh retweeted
“How We Think About Safety and Alignment” — this is our cornerstone document.
Enjoy!
openai.com/safety/how-we-thi…
Paridhi Singh retweeted
Elon Musk wanted an OpenAI for-profit. openai.com/index/elon-musk-w…
Paridhi Singh retweeted
What has been your favorite poster at NeurIPS so far.
No posters by your lab or immediate colleagues allowed lol. Let’s keep this fun.
Paper links welcome.
Paridhi Singh retweeted
I am starting a new company:
Superintelligence is within reach.
Building safe superintelligence (SSI) is the most important technical problem of our time.
We've started the world’s first straight-shot SSI lab, with one goal and one product: a safe superintelligence.
It’s called Safe Superintelligence Inc.
SSI is our mission, our name, and our entire product roadmap, because it is our sole focus. Our team, investors, and business model are all aligned to achieve SSI.
We approach safety and capabilities in tandem, as technical problems to be solved through revolutionary engineering and scientific breakthroughs. We plan to advance capabilities as fast as possible while making sure our safety always remains ahead.
This way, we can scale in peace.
Our singular focus means no distraction by management overhead or product cycles, and our business model means safety, security, and progress are all insulated from short-term commercial pressures.
We are an American company with offices in Palo Alto and Tel Aviv, where we have deep roots and the ability to recruit top technical talent.
We are assembling a lean, cracked team of the world’s best engineers and researchers dedicated to focusing on SSI and nothing else.
If that’s you, we offer an opportunity to do your life’s work and help solve the most important technical challenge of our age.
Now is the time. Join us.
Ilya Sutskever, Daniel Gross, Daniel Levy
June 19, 2024
Paridhi Singh retweeted
What’s GPT-4o like? What would you ask @OpenAI's newest model? @salkhanacademy and his son took it for a spin.
Sal's question: can you drive a conversation to help us get to know one another better? 🥺🥺🥺
👋 At Khan Academy, we recognize the potential AI has to transform education. Check out our AI-powered tutor Khanmigo at khanmigo.ai/parents and look for Sal's new book about AI in education, BRAVE NEW WORDS, at your favorite bookstore tomorrow!


























