Microsoft Research dropped InstructLM 500M!!

Yo! Microsoft Research dropped InstructLM 🔥
500M & 1.5B pre-trained model checkpoints & Domain-specific 8B checkpoints
8B checkpoints beat Llama 3 70B
Smol LLMs based on Mistral architecture
Pretraining data - Randomly sample refined web and create instruction tuned data.
All checkpoints and dataset released under Apache 2.0 licensed
Bonus: They release Instruction Synthesiser for you to convert pretraining data too.

Congrats Microsoft! ⚡