Rotational Position Embedding (RoPE): Part 3

This week Roger continues the review of the notebook that we used to explore with code how the math of ROtational Positional Embedding works. He also reviews some of the papers referenced in the notebook. We finished with any introduction of the Very Small Language Model (VSLM) project.

Links
Our Meetup: www.meetup.com/East-Bay-Tri-Valley-Machine-Learnin…
Rope Paper: arxiv.org/abs/2104.09864

Content
00:00 Introduction
09:39 Code walkthrough
05:01 Attention with Linear Bias
21:19 Extending the training window
51:02 Understanding word embeddings
01:11:58 Very Small Language Model (VSLM)

▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬
😊About Us

West Coast Machine Learning is a channel dedicated to exploring the exciting world of machine learning! Our group of techies is passionate about deep learning, neural networks, computer vision, tiny ML, and other cool geeky machine learning topics. We love to dive deep into the technical details and stay up to date with the latest research developments.

Our Meetup group and YouTube channel is the perfect place to connect with other like-minded individuals who share your love of machine learning. We offer a mix of research paper discussions, coding reviews, and other data science topics. So, if you're looking to stay up to date with the latest developments in machine learning, connect with other techies, and learn something new, be sure to subscribe to our channel and join our Meetup community today!

Meetup: www.meetup.com/east-bay-tri-valley-machine-learnin…

▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬
#Rope #RotaryPositionalEmbeddings #VSLM #AI #ML #Transformers #Encoding #GenerativeAI #MachineLearning