@zonderlabs, generative 3d @Stanford, @CarnegieMellon, @UMich
Bay Area | NYC
Joined September 2021
- Tweets175
- Following888
- Followers787
- Likes2.5K
Zhizhuo (Z) Zhou retweeted
As humans, we easily infer object articulations from a couple of examples, or even none.
For vision models, figuring that out is a completely different story.
Our new paper "FAMOS: Feed-Forward 3D Articulation Modeling from Sparse Observations" (WACV 2027) takes a stab at it!
ALT https://kevinqu7.github.io/famos/
Zhizhuo (Z) Zhou retweeted
simply including "write in ASD-STE100" in your prompt makes a huge difference in ai readability
we benchmarked before / after "write in ASD-STE100" with Opus 5.5, and sentences became easier to read
We'll be spending a lot more time trying to understand the outputs of language models. A few thoughts, tips & tricks:
Writing. Something I've had success with: Ask your LLM to explain something in ASD-STE100, it's a controlled language specification originally developed for aerospace maintenance documentation. LLMs well-versed in this language and it comes with heavy constraints on clean writing style that I often find a lot more readable. Sometimes I've tried to soften it a bit e.g. ask for "80% of the way to ASD-STE100" because the spec is quite stringent. But even better:
Diagrams / images. Instead of writing, ask your LLM to create a diagram. These can be a lot easier to process, parse, and understand. But even better:
Web pages. Ask for output "in HTML" to get a beautiful, interactive webpage. LLMs are getting really good at frontend and can create beautiful experiences, animations, etc. But even better:
Explainer videos. The output format I am most bullish on is fully custom / bespoke explainer videos generated on any arbitrary topic. Experiment with things like "Create a 3b1b style video explainer on X. Use my ElevenLabs API key for audio narration". (you'd need an API key for the latter or you can ask your LLM to find you decent free alternatives that use your local compute). This is actually starting to work!
In summary:
- As LLMs get better, they will do more and more of the legwork autonomously, and a lot more of our work will rise up the abstractions into oversight and understanding.
- Luckily, LLMs can help here too because as intelligence and code are increasingly abundant, you can ask for large, custom, discardable software artifacts (e.g. web apps, video explainers) that would have never made sense to create before. Push the boundaries here and you'll be surprised.
Zhizhuo (Z) Zhou retweeted
In 12 weeks, we built a research facility that is run entirely by AI.
AI designs, executes, and observes experiments end-to-end across biology, chemistry, and materials science.
We’re introducing SciUniverse: a benchmark that measures AI’s ability to do real-world scientific research.
Zhizhuo (Z) Zhou retweeted
Introducing OM-1, our first robot foundation model: zero-shot generalizing across table-top arms, industrial arms, and humanoids.
- Learned directly from human manipulation data
- No teleop or robot data
- Near human-level dexterity and efficiency
- Multi-robot collaboration
Zhizhuo (Z) Zhou retweeted
LoRA, low-rank adaptation, is arguably the most popular parameter-efficient fine-tuning method for LLMs.
But how does it actually work?
Check out the video to learn LoRA and friends (LoRA+, QLoRA, VeRA, and DoRA)!
youtu.be/U80tjcThl9Q
Zhizhuo (Z) Zhou retweeted
To be honest, training on handmade 4D asset datasets is a dead-end. Almost all 4D asset data is synthetic and diverse real data barely exists, so models trained on it struggle to reconstruct objects that deform, get occluded, and move freely about the scene.
Our new work, Lift4D, instead lifts 2D & 3D priors into 4D, reconstructing complete dynamic objects from a single in-the-wild video 🧵 (1/n)
🔗Webpage + Demos: lift4d.github.io
Zhizhuo (Z) Zhou retweeted
We're excited to introduce Bonemeal: the world's first autonomous growth engineer.
I took one company 0→$60M ARR in a year, another 0→$10M in 8 months. The secret wasn't growth hacks, it was cracking the self-improvement loop.
Bonemeal runs that loop autonomously: it launches growth experiments, then watches and tweaks them ad infinitum.
It hit $100k ARR in month one.
Put your growth on autopilot 👇
Zhizhuo (Z) Zhou retweeted
Sharing a super simple, user-owned memory module we've been playing around: nanomem
The basic idea is to treat memory as a pure intelligence problem: ingestion, structuring, and (selective) retrieval are all just LLM calls & agent loops on a on-device markdown file tree. Each file lists a set of facts w/ metadata (timestamp, confidence, source, etc.); no embeddings/RAG/training of any kind.
For example:
- `nanomem add <fact>` starts an agent loop to walk the tree, read relevant files, and edit.
- `nanomem retrieve <query>` walks the tree and returns a single summary string (possibly assembled from many subtrees) related to the query.
What’s nice about this approach is that the memory system is, by construction:
1. partitionable (human/agents can easily separate `hobbies/snowboard.md` from `tax/residency.md` for data minimization + relevance)
2. portable and user-owned (it’s just text files)
3. interpretable (you know exactly what’s written and you can manually edit)
4. forward-compatible (future models can read memory files just the same, and memory quality/speed improves as models get better)
5. modularized (you can optimize ingestion/retrieval/compaction prompts separately)
Privacy & utility. I'm most excited about the ability to partition + selectively disclose memory at inference-time. Selective disclosure helps with both privacy (principle of least privilege & “need-to-know”) and utility (as too much context for a query can harm answer quality).
Composability. An inference-time memory module means: (1) you can run such a module with confidential inference (LLMs on TEEs) for provable privacy, and (2) you can selectively disclose context over unlinkable inference of remote models (demo below).
We built nanomem as part of the Open Anonymity project (openanoymity.ai), but it’s meant to be a standalone module for humans and agents (e.g., you can write a SKILL for using the CLI tool). Still polishing the rough edges!
- GitHub (MIT): github.com/OpenAnonymity/nan…
- Blog: openanonymity.ai/blog/nanome…
- Beta implementation in chat client soon: chat.openanonymity.ai
Work done with amazing project co-leads @amelia_kuang @cocozxu @erikchi !!
Zhizhuo (Z) Zhou retweeted
I made a @huggingface public repo too :)
huggingface.co/spaces/artifi…
Today we launch Fish Audio S2, a new generation of expressive TTS with absurdly controllable emotion.
- open-source
- sub 150ms latency
- multi-speaker in one pass
Real freedom of speech starts now 👇
Readers added context they thought people might want to know
The "open source" claim is false. Their custom license prohibits commercial use and limits some derivative uses. A more accurate description is "source-available", meaning the source code is publicly accessible but subject to use restrictions.
opensource.org/osd
github.com/fishaudio/fish…
fossa.com/blog/comprehen…
Zhizhuo (Z) Zhou retweeted
Incredible work by the Fish team !!!
S2 blew my socks off when I first heard our e-girl voice [panting], [laughing], and even [singing] hahaha
Excited to share my learnings from growth engineering soon!
Today we launch Fish Audio S2, a new generation of expressive TTS with absurdly controllable emotion.
- open-source
- sub 150ms latency
- multi-speaker in one pass
Real freedom of speech starts now 👇
Readers added context they thought people might want to know
The "open source" claim is false. Their custom license prohibits commercial use and limits some derivative uses. A more accurate description is "source-available", meaning the source code is publicly accessible but subject to use restrictions.
opensource.org/osd
github.com/fishaudio/fish…
fossa.com/blog/comprehen…
Zhizhuo (Z) Zhou retweeted
Today we launch Fish Audio S2, a new generation of expressive TTS with absurdly controllable emotion.
- open-source
- sub 150ms latency
- multi-speaker in one pass
Real freedom of speech starts now 👇
Readers added context they thought people might want to know
The "open source" claim is false. Their custom license prohibits commercial use and limits some derivative uses. A more accurate description is "source-available", meaning the source code is publicly accessible but subject to use restrictions.
opensource.org/osd
github.com/fishaudio/fish…
fossa.com/blog/comprehen…
Zhizhuo (Z) Zhou retweeted
[1/N] Rotary Position Embeddings (RoPE) are ubiquitous across transformers that process tokens from 1D, 2D, or 3D grids e.g. language, images, or videos. Our RayRoPE formulation extends these to multi-view transformers. Paper and code: rayrope.github.io/
Zhizhuo (Z) Zhou retweeted
🚀 New preprint!
We present NP-Edit, a framework for training an image editing diffusion model without paired supervision.
We use differentiable feedback from Vision-Language Models (VLMs) combined with distribution-matching loss (DMD) to learn editing directly.
webpage: nupurkmr9.github.io/npedit/
w/ @ShengYuWang6,Cherry (N.X.) Zhao, @YotamNitzan, Yuheng Li, Krishna Kumar Singh, @rzhang88, @elishechtman, @junyanz89, @xxunhuang
Zhizhuo (Z) Zhou retweeted
Excited to announce a new track of accelerating Generative AI:
pi-Flow: Policy-Based Few-Step Generation via Imitation Distillation
github.com/Lakonik/piFlow
Distill 20B flow models now using just an L2 loss via imitation learning for SOTA diversity and teacher-aligned quality.
Zhizhuo (Z) Zhou retweeted
[1/N] We present a plug-and-play mechanism to controllably steer inference of any diffusion/flow model towards a sharper or flatter sampling distribution, resulting in improvements across domains e.g. text-to-image (10% FID reduction), protein generation (improved designability).
