The NLP group at @USCViterbi. @DaniYogatama+@_jessethomason_+@jieyuzhao11+@robinomial+@swabhz+@xiangrenNLP at @CSatUSC + researchers @USC_ICT, @USC_ISI.
Los Angeles, CA
Joined May 2018
- Tweets353
- Following356
- Followers3.7K
- Likes429
Pinned Tweet
The USC NLP faculty is 2x of its size in two years! We're thrilled to have @DaniYogatama and @swabhz joining us this Fall (@jieyuzhao11 starting in Fall'23), along with five other CS new hires! 🙌
🔖viterbischool.usc.edu/news/2…
USC NLP retweeted
We've been thinking a lot about trust in AI agents — specifically, trust in the developer running it. Right now, there's no way to verify that the safety guardrails an agent claims to have are actually enforced. New work w/ @XisenJ @aarzchan et al. tackles exactly this.
Paper, code, and demo here: saharalabsai.github.io/proof…
USC NLP retweeted
Thrilled for the Best Paper Award runner-up at #NAACL2025! 🥳
Even when answers are incorrect, people may rely more on LLMs if they use warm and emphatic expressions!
We analyze the risks of human over-reliance on LLM expressions of uncertainty: arxiv.org/pdf/2407.07950
w/ @KaitlynZhou @JenaHwang2 @nouhadziri
@jurafsky @MaartenSap @stanfordnlp @allen_ai
#NLPproc
How can we best measure the consequences of LLM overconfidence?
✨New preprint✨ on measuring the risks of human over-reliance on LLM expressions of uncertainty: arxiv.org/pdf/2407.07950
w/@JenaHwang2 @xiangrenNLP @nouhadziri @jurafsky @MaartenSap @stanfordnlp @allen_ai #NLPproc
USC NLP retweeted
Proud of my student @huihan_li and intern Arnav presenting their #ICLR2025 work on attributing culture-conditioned generation to LLM’s training corpora.
Fun time meeting many friends. Ping me if you want to chat about model security, interpretability and human-LM interaction!
USC NLP retweeted
Heading to #EMNLP2024, down to chat! Excited to present our work (Wed 10:30am) on systematic data generation in long-tail (low confidence) distribution for more challenging evaluation. 🧵👇
📰: arxiv.org/abs/2311.07237
💻: github.com/INK-USC/LINK
🔖: zenodo.org/records/10117911
USC NLP retweeted
Proud moment seeing our CEO & Co-Founder @xiangrenNLP alongside his @nlp_usc students at @aclmeeting.
Supporting the next generation of thought leaders in AI is exactly what drives us forward.
USC NLP retweeted
Join us at the co-located <AI / ALL> summit on Aug 15, with the social party in the evening!
lu.ma/mxcx5bia
co-hosted with @SCB10X_OFFICIAL @SambaNovaAI
sponsored by @awscloud
participated by folks @AIatMeta @google @CohereForAI @togethercompute
Arriving in Bangkok for @aclmeeting! 😃
Will be sharing our recent work on logical scaffolding, model uncertainty expression & multi-hop entailment inference w/ folks @nlp_usc + @KaitlynZhou +friends @allen_ai
I'm also helping on the <AI / ALL> summit
w/ @saharaai
👇👇
USC NLP retweeted
Find us at the posters!
Can LLMs Reason with Rules? Logic Scaffolding for Stress-Testing and Improving LLMs
w/ @siyuan___wang @YejinChoinka et al
Relying on the Unreliable: The Impact of Language Models' Reluctance to Express Uncertainty
w/ @KaitlynZhou, @MaartenSap et al.
USC NLP retweeted
Arriving in Bangkok for @aclmeeting! 😃
Will be sharing our recent work on logical scaffolding, model uncertainty expression & multi-hop entailment inference w/ folks @nlp_usc + @KaitlynZhou +friends @allen_ai
I'm also helping on the <AI / ALL> summit
w/ @saharaai
👇👇
USC NLP retweeted
Excited to see everyone soon at #acl2024 in Bangkok!
I'll be presenting our work, Relying on the Unreliable: The Impact of Language Models' Reluctance to Express Uncertainty arxiv.org/abs/2401.06730
Poster session 3 on Aug 12 at 16:00! W/ @MaartenSap @JenaHwang2 @xiangrenNLP
USC NLP retweeted
Introducing 𝗟𝗶𝗳𝗲𝗹𝗼𝗻𝗴 𝗜𝗖𝗟 and 𝗧𝗮𝘀𝗸 𝗛𝗮𝘆𝘀𝘁𝗮𝗰𝗸, a new approach for evaluating long-context LMs, featuring ever-changing task streams that controllably fill the context window, and NIAH-style visualization for easy diagnosis.
📜 arxiv.org/abs/2407.16695
🧵
USC NLP retweeted
Congratulations to the GDM @GoogleDeepMind team on their best paper award at #ICML2024 & Appreciate @afedercooper's shout out to our concurrent paper 🙌
If you are into the topic of recovering model info through just its output logits, check out our paper led by @mattf1n too!
Thread: xitter.gq/mattf1n/status/1768668…
This Post is from an account that no longer exists. Learn more
USC NLP retweeted
🧐LMs forget upstream knowledge when continuously fine-tuned. When fine-tuned on new data, can we forecast what upstream examples will be forgotten?
🥳Excited to share our #ICML Spotlight paper on forecasting example forgetting!
🔗Project page: inklab.usc.edu/lm-forgetting…
USC NLP retweeted
New paper 🚨
Looking for a strong, open-sourced entailment-verification model to verify your model generations for consistency?
✅ You can now use the 🤗model huggingface.co/soumyasanyal/… for this! Our FlanT5-xxl finetuned model can predict entailment errors better than GPT3.5 and comparable to GPT4.
In addition to this, we analyze the entailment verification performance patterns b/w humans and GPT4 in our #acl2024 findings paper arxiv.org/abs/2402.03686.
Key findings:
✅ GPT4 is stronger than humans at tasks involving multi-hop reasoning across long contexts, indicating that humans have a limited short-term memory capacity
✅ Humans are more robust than GPT4 on simple deductive reasoning, indicating that current LLMs are still inconsistent deductive reasoners
Joint work w/ @Leinster_ty @liujc1998 @happywwy @xiangrenNLP
🧵(1/5)
USC NLP retweeted
Wanna know gpt-3.5-turbo's embed size? We find a way to extract info from LLM APIs and estimate gpt-3.5-turbo’s embed size to be 4096. With the same trick we also develop 25x faster logprob extraction, audits for LLM APIs, and more!
📄 arxiv.org/abs/2403.09539
Here’s how 1/🧵
USC NLP retweeted
Absolutely thrilled to receive this honor. Rarely for a researcher could have their first PhD publication win a Test of Time Award (for 10 years of its cumulative impact). I’m super grateful for the chance to collaborate with Xiao on this fun project — turns out to be a pioneering work on personalizing recommender systems with an information network approach! Proud of the team!
Congratulations, @xiangrenNLP! Recently honored as an MIT Technology Review Innovators Under 35 (Asia-Pacific) AND received a Test of Time research paper award from the ACM International Conference on Web Search and Data Mining (WSDM) this week! 👏
viterbischool.usc.edu/news/2…
USC NLP retweeted
To detect if your data was used for LLM pretraining, consider using data watermarks: arxiv.org/pdf/2402.10892.pdf
Detection can be framed as hypothesis testing (statistical guarantees!), if you contributed multiple training documents and watermarked them before public release. 🧵
USC NLP retweeted
Arrived at NOLA for #NeurIPS2023🔥 Exciting time to chat about limits/science of LLMs, “slow” reasoning & explainability. Join our posters for a fun disucssion🍻
Ads: USC CS is hiring tenured track AI faculty + USC NLP is looking for strong PhD students. Talk to us!
USC NLP retweeted
How good are LMs at inductive reasoning? How are their behaviors similar to/contrasted with those of humans?
We study these via iterative hypothesis refinement. We observe that LMs are phenomenal hypothesis proposers, but they also behave as puzzling inductive reasoners:
(1/n)
USC NLP retweeted
Throwback to when @xiangrenNLP and our lab made our wishlist and dream research directions to discuss in our lab meeting — very helpful in contextualising our work in the age of LLMs!! 🙌🏼
@nlp_usc is such a great place to do research 🫶
Inspired by @xiangrenNLP opening his talk with a wish list! I'll have to do the same
Also, I agree that a grand challenge for AI right now is that 空気が読めない
We're excited to attend #SocalNLP today! ICYMI, sunny southern California is a fantastic place to do #NLProc, come check out what USC NLP [nlp.usc.edu/] has been working on lately! And did we say we're hiring PhD students this fall? 🌴🏖️☀️
#SoCalNLP2023 is this Friday!!! 🏝
Check out our schedule of invited speakers and accepted posters! 👉🏽 socalnlp.github.io/symp23/in…