Imankulova Aizhan (@aijanit) header
Imankulova Aizhan (@aijanit) profile picture
Joined May 2016
Photo by Imankulova Aizhan (@aijanit)Photo by Imankulova Aizhan (@aijanit)Photo by Imankulova Aizhan (@aijanit)Photo by Imankulova Aizhan (@aijanit)Photo by Imankulova Aizhan (@aijanit)Photo by Imankulova Aizhan (@aijanit)Photo by Imankulova Aizhan (@aijanit)Photo by Imankulova Aizhan (@aijanit)Photo by Imankulova Aizhan (@aijanit)Photo by Imankulova Aizhan (@aijanit)
Imankulova Aizhan retweeted
Wild idea in this paper 🤯 How might we store knowledge affordably yet comprehensively? Memory³ proposes an intriguing method - compressing factual data separately. Introduces a third form of memory in addition to the implicit knowledge stored in model parameters and the short-term working memory used during inference (context key-values). 👨‍🔧 LLMs struggle with inefficient knowledge storage and retrieval, leading to high training and inference costs. The paper aims to address this by introducing a more efficient memory format. 📌 Memory3 introduces explicit memory as a third memory format for LLMs, alongside model parameters (implicit memory) and context key-values (working memory). This explicit memory is implemented as sparse attention key-values, allowing for more efficient knowledge storage and retrieval. 📌 Defines a memory hierarchy for LLMs: plain text (RAG) → explicit memory → model parameters. As you move up this hierarchy, write cost increases while read cost decreases. The goal is to optimize knowledge placement across this hierarchy based on usage frequency. 📌 Memory3's architecture involves converting reference texts into explicit memories before inference. During inference, these memories are retrieved and integrated into self-attention layers. This design allows for smaller model size while maintaining performance. 📌 The explicit memory format uses intense compression to save space. It selects only the first half of attention layers as memory layers, uses grouped query attention to reduce key-value heads, and selects only 8 out of 128 tokens for each key-value head based on attention weights. 📌 The training process involves a two-stage approach: a warmup stage without explicit memory, followed by a continual train stage with explicit memory. This approach was necessary as starting with explicit memory from the beginning rendered the memories useless. 📌 Introduces a "memory circuitry theory" to formalize the concept of knowledge in LLMs. It defines knowledge as circuits (equivalence classes of subgraphs) in the computation graph, categorizing them as specific or abstract knowledge. 📌 The Memory3 model achieved better performance than larger models and RAG models on various benchmarks, while maintaining higher decoding speed. It showed particular improvements in factuality and reduced hallucination.
21
212
17
1,233
153,132
Imankulova Aizhan retweeted
これ普通にえぐい
693
27,930
3,318
249,064
37,267,814
Imankulova Aizhan retweeted
YANMTT was accepted to ACL as a demo paper. So that takes me to 4 out of 6 ACL papers accepted. Work with: @diptesh Chinmay and @eiichirosumita Look out for updates here in the upcoming days: github.com/prajdabre/yanmtt
1
4
27
2,653
Imankulova Aizhan retweeted
Our short paper "Comparing Intrinsic Gender Bias Evaluation Measures without using Human Annotated Examples" with @Bollegala and @chokkanorg got accepted to the #EACL2023🎉 We propose a method to compare intrinsic gender bias measures without relying on human-annotations!
4
1
40
7,933
Imankulova Aizhan retweeted
COLING2022に主著(@Bollegala @chokkanorg )と共著(@silviasetitech @chokkanorg )のLongが口頭とポスターで採択されました🎉 主著では言語モデルに対するタスク依存と非依存な公平性評価やバイアス除去手法の関係を調査、共著ではパラレルデータのみで学習可能な文法誤り訂正の評価を提案しました。
1
8
67
Imankulova Aizhan retweeted
I fully reviewed @MetaAI No Language Left Behind's automatic evaluation #NLLB, and I have a lot to say about it... So I started a blog: medium.com/@bnjmn_marie/scie…
8
49
8
225
Imankulova Aizhan retweeted
I put together a google colab notebook for NLG using IndicBART. This was a part of the @ai4bharat NLG workshop. It covers basic examples for: 1. Pre-training 2. MT fine-tuning 3. Summarization fine-tuning Enjoy! colab.research.google.com/dr…
4
5
Tired of tokenizers/subwords? Check out PIXEL, a new language model that processes written text as images📸 “Language Modelling with Pixels” 📄 arxiv.org/abs/2207.06991 🧑‍💻github.com/xplip/pixel 🤖huggingface.co/Team-PIXEL/pi… by @rust_phillip @jonasflotz me @esalesk @mdlhx @delliott
24
176
77
990
Imankulova Aizhan retweeted
✨I will give an oral presentation of our work with @aijanit, @Bollegala, and @chokkanorg at #NAACL 14:15 – 15:45 PDT on July 12 about evaluating gender bias in MLMs for multiple languages without needing annotated data. Our paper: aclanthology.org/2022.naacl-…
3
15
🚀次回のNLPコロキウムのお知らせ 講演者:金子正弘 (東工大) @MasahiroKaneko_ @tokyotech_jp 日時:7/6 (水) 12:00- JST 入力文に含まれる誤りの訂正結果だけではなく、なぜその訂正が適切なのかを併せて教えてくれる新しい文法誤り訂正の枠組みを紹介いただきます。 nlp-colloquium-jp.github.io/…
1
8
1
34
The earth supported by four elephants standing on the back of a turtle #DALLEMINIBOTHEREYOUGO
1
Imankulova Aizhan retweeted
Replying to @Bollegala
@Bollegala, @chokkanorg 先生方との論文を公開しました!複数の分散表現を組み合わせるmeta-embeddingにおける公平性について調査しました。そして、バイアス除去手法をmeta-embedding学習により組み合わせることも提案しています。 📜 論文:arxiv.org/abs/2205.09867
9
27
Imankulova Aizhan retweeted
We (w/ @7a7hi and @chokkanorg) also published our work on an efficiently improvement technique on Findings of ACL 2022. Our proposed method enables the single model ensemble for a model trained by subword regularizations. Paper: aclanthology.org/2022.findin… #ACL2022nlp #NLProc
6
16
Imankulova Aizhan retweeted
✨Is the interpretability of the neural model useful for the user? In our #ACL2022 paper, we showed that the interpretability with examples is effective for language learning! Oral and poster presentations in Dublin🇮🇪 📜Paper:aclanthology.org/2022.acl-lo… 🛠Code:github.com/kanekomasahiro/eb…
Imankulova Aizhan retweeted
10 yrs ago @karpathy wrote a blog post on the outlook of AI: karpathy.github.io/2012/10/2… in which he describes how difficult it would be for an AI to understand a given photo, concluding "we are very, very far and this depresses me." Today, our Flamingo steps up to the challenge.
82
491
144
3,706
✨ There are already a lot of benchmarks for evaluating social biases in MLM trained for English . These benchmarks require manually written templates or example sentences. However, scaling this effort to other languages is a nontrivial task. [1/]
In our #NAACL2022 paper w/ @aijanit @Bollegala @chokkanorg, we propose a method of evaluating bias in various languages without requiring manually annotated data and also release datasets in Japanese and Russian. Paper: arxiv.org/abs/2205.00551 Code & data: github.com/kanekomasahiro/bi…
1
3
5
Imankulova Aizhan retweeted
In our #NAACL2022 paper w/ @aijanit @Bollegala @chokkanorg, we propose a method of evaluating bias in various languages without requiring manually annotated data and also release datasets in Japanese and Russian. Paper: arxiv.org/abs/2205.00551 Code & data: github.com/kanekomasahiro/bi…
1
6
1
21
Imankulova Aizhan retweeted
I released YANMTT V2.0, earlier this morning. Two main features: 1. GUI to demo and debug your models. 2. Adaptors and prompts which you can plug into any of your models or official ones like mBART-25/50. Enjoy! github.com/prajdabre/yanmtt
1
5
1
14
Imankulova Aizhan retweeted
NLPのトップ会議であるNAACL-22の本会議に主著のLongが採択されました🎉 対訳データと単語リストだけを使いBERTなどの公平性をいろんな言語で評価することができます。日本語とロシア語の公平性評価データセットも作成しました。共著の皆さん (@aijanit @Bollegala, @chokkanorg) には感謝です !
1
13
103