Imankulova Aizhan retweeted
Wild idea in this paper 🤯
How might we store knowledge affordably yet comprehensively? Memory³ proposes an intriguing method - compressing factual data separately. Introduces a third form of memory in addition to the implicit knowledge stored in model parameters and the short-term working memory used during inference (context key-values).
👨🔧 LLMs struggle with inefficient knowledge storage and retrieval, leading to high training and inference costs. The paper aims to address this by introducing a more efficient memory format.
📌 Memory3 introduces explicit memory as a third memory format for LLMs, alongside model parameters (implicit memory) and context key-values (working memory). This explicit memory is implemented as sparse attention key-values, allowing for more efficient knowledge storage and retrieval.
📌 Defines a memory hierarchy for LLMs: plain text (RAG) → explicit memory → model parameters. As you move up this hierarchy, write cost increases while read cost decreases. The goal is to optimize knowledge placement across this hierarchy based on usage frequency.
📌 Memory3's architecture involves converting reference texts into explicit memories before inference. During inference, these memories are retrieved and integrated into self-attention layers. This design allows for smaller model size while maintaining performance.
📌 The explicit memory format uses intense compression to save space. It selects only the first half of attention layers as memory layers, uses grouped query attention to reduce key-value heads, and selects only 8 out of 128 tokens for each key-value head based on attention weights.
📌 The training process involves a two-stage approach: a warmup stage without explicit memory, followed by a continual train stage with explicit memory. This approach was necessary as starting with explicit memory from the beginning rendered the memories useless.
📌 Introduces a "memory circuitry theory" to formalize the concept of knowledge in LLMs. It defines knowledge as circuits (equivalence classes of subgraphs) in the computation graph, categorizing them as specific or abstract knowledge.
📌 The Memory3 model achieved better performance than larger models and RAG models on various benchmarks, while maintaining higher decoding speed. It showed particular improvements in factuality and reduced hallucination.
Imankulova Aizhan retweeted
YANMTT was accepted to ACL as a demo paper. So that takes me to 4 out of 6 ACL papers accepted.
Work with: @diptesh Chinmay and @eiichirosumita
Look out for updates here in the upcoming days: github.com/prajdabre/yanmtt
Imankulova Aizhan retweeted
Our short paper "Comparing Intrinsic Gender Bias Evaluation Measures without using Human Annotated Examples" with @Bollegala and @chokkanorg got accepted to the #EACL2023🎉 We propose a method to compare intrinsic gender bias measures without relying on human-annotations!
Imankulova Aizhan retweeted
COLING2022に主著(@Bollegala @chokkanorg )と共著(@silviasetitech @chokkanorg )のLongが口頭とポスターで採択されました🎉 主著では言語モデルに対するタスク依存と非依存な公平性評価やバイアス除去手法の関係を調査、共著ではパラレルデータのみで学習可能な文法誤り訂正の評価を提案しました。
Imankulova Aizhan retweeted
I fully reviewed @MetaAI No Language Left Behind's automatic evaluation #NLLB, and I have a lot to say about it... So I started a blog:
medium.com/@bnjmn_marie/scie…
Imankulova Aizhan retweeted
I put together a google colab notebook for NLG using IndicBART. This was a part of the @ai4bharat NLG workshop.
It covers basic examples for:
1. Pre-training
2. MT fine-tuning
3. Summarization fine-tuning
Enjoy!
colab.research.google.com/dr…
Imankulova Aizhan retweeted
Tired of tokenizers/subwords? Check out PIXEL, a new language model that processes written text as images📸
“Language Modelling with Pixels”
📄 arxiv.org/abs/2207.06991
🧑💻github.com/xplip/pixel
🤖huggingface.co/Team-PIXEL/pi…
by @rust_phillip @jonasflotz me @esalesk @mdlhx @delliott
Imankulova Aizhan retweeted
✨I will give an oral presentation of our work with @aijanit, @Bollegala, and @chokkanorg at #NAACL 14:15 – 15:45 PDT on July 12 about evaluating gender bias in MLMs for multiple languages without needing annotated data. Our paper: aclanthology.org/2022.naacl-…
Imankulova Aizhan retweeted
🚀次回のNLPコロキウムのお知らせ
講演者:金子正弘 (東工大) @MasahiroKaneko_ @tokyotech_jp
日時:7/6 (水) 12:00- JST
入力文に含まれる誤りの訂正結果だけではなく、なぜその訂正が適切なのかを併せて教えてくれる新しい文法誤り訂正の枠組みを紹介いただきます。
nlp-colloquium-jp.github.io/…
The earth supported by four elephants standing on the back of a turtle #DALLEMINIBOTHEREYOUGO
1
Imankulova Aizhan retweeted
Replying to @Bollegala
@Bollegala, @chokkanorg 先生方との論文を公開しました!複数の分散表現を組み合わせるmeta-embeddingにおける公平性について調査しました。そして、バイアス除去手法をmeta-embedding学習により組み合わせることも提案しています。
📜 論文:arxiv.org/abs/2205.09867
Imankulova Aizhan retweeted
We (w/ @7a7hi and @chokkanorg) also published our work on an efficiently improvement technique on Findings of ACL 2022. Our proposed method enables the single model ensemble for a model trained by subword regularizations.
Paper: aclanthology.org/2022.findin…
#ACL2022nlp #NLProc
Imankulova Aizhan retweeted
✨Is the interpretability of the neural model useful for the user? In our #ACL2022 paper, we showed that the interpretability with examples is effective for language learning! Oral and poster presentations in Dublin🇮🇪
📜Paper:aclanthology.org/2022.acl-lo…
🛠Code:github.com/kanekomasahiro/eb…
Imankulova Aizhan retweeted
10 yrs ago @karpathy wrote a blog post on the outlook of AI: karpathy.github.io/2012/10/2… in which he describes how difficult it would be for an AI to understand a given photo, concluding "we are very, very far and this depresses me."
Today, our Flamingo steps up to the challenge.
Imankulova Aizhan retweeted
✨ There are already a lot of benchmarks for evaluating social biases in MLM trained for English . These benchmarks require manually written templates or example sentences. However, scaling this effort to other languages is a nontrivial task. [1/]
In our #NAACL2022 paper w/ @aijanit @Bollegala @chokkanorg, we propose a method of evaluating bias in various languages without requiring manually annotated data and also release datasets in Japanese and Russian.
Paper: arxiv.org/abs/2205.00551
Code & data: github.com/kanekomasahiro/bi…
Imankulova Aizhan retweeted
In our #NAACL2022 paper w/ @aijanit @Bollegala @chokkanorg, we propose a method of evaluating bias in various languages without requiring manually annotated data and also release datasets in Japanese and Russian.
Paper: arxiv.org/abs/2205.00551
Code & data: github.com/kanekomasahiro/bi…
Imankulova Aizhan retweeted
メディア総研株式会社が運営する「月刊高専」に、助教・梶原のインタビュー記事が掲載されました。
gekkan-kosen.com/6575/
Imankulova Aizhan retweeted
I released YANMTT V2.0, earlier this morning.
Two main features:
1. GUI to demo and debug your models.
2. Adaptors and prompts which you can plug into any of your models or official ones like mBART-25/50.
Enjoy!
github.com/prajdabre/yanmtt
Imankulova Aizhan retweeted
NLPのトップ会議であるNAACL-22の本会議に主著のLongが採択されました🎉 対訳データと単語リストだけを使いBERTなどの公平性をいろんな言語で評価することができます。日本語とロシア語の公平性評価データセットも作成しました。共著の皆さん (@aijanit @Bollegala, @chokkanorg) には感謝です !

























