NEW: AMD introduces Instella, a series of fully open-source, state-of-the-art 3B parameter language models. It's nice to see AMD pushing out its own small language models. It looks like a very competitive model for its size. Here's everything you need to know: • Architecture: Autoregressive transformer with 36 decoder layers and 32 attention heads. This is a text-only model. • Sequence Length: 4,096 tokens • Vocabulary: ~50,000 tokens (using OLMo tokenizer) • Training Hardware: Installa is trained entirely on AMD Instinct MI300X GPUs; specifically, 128 AMD Instinct MI300X GPUs. • Multi-Stage Training Pipeline: Stage 1 Pre-training: 4.065 trillion tokens covering diverse content (coding, academic, web). Stage 2 Pre-training: Additional 57.575 billion tokens, including specialized datasets for mathematics, coding, and conversational interactions. • Supervised Fine-Tuning (SFT): 8.9 billion tokens to enhance instruction-following capabilities. It also includes an alignment Stage (DPO): 760 million tokens for alignment to human preferences (Instella-3B-Instruct). • Performance Highlights: Instella-3B significantly surpasses existing fully open 3B models (+8.08% average improvement). Competes closely with leading open-weight models like Llama-3.2-3B, Gemma-2-2B, and Qwen-2.5-3B. Achieved outstanding performance improvements in benchmarks like ARC Challenge, MMLU, BBH, and GSM8K. The instruction-tuned models excel in interactive, instruction-following tasks, achieving results comparable to or surpassing open-weight models. • Training Optimizations: Used FlashAttention-2 for efficient attention computation, Torch Compile for performance acceleration, and Fully Sharded Data Parallelism (FSDP) for optimal resource utilization and scalability. • Open Source: Fully open-sourced model weights, datasets, training hyperparameters, and code available on Hugging Face and GitHub. • Future Directions: Plans include expanding context length, enhancing reasoning skills, exploring multimodal capabilities, and scaling up models and datasets. • License and Usage: Available under a ResearchRAIL license, intended for academic and research purposes only.