Meta just dropped their latest AI model, the Llama 3, and it’s kind of a big deal. They’ve got versions with 8 billion and a whopping 70 billion parameters. Here’s the scoop:
- They’ve upgraded the tokenizer in Llama 3. Now it handles a massive 128K tokens, which is way more efficient than the last version — like 15% more compact.
- They added this cool feature called Grouped Query Attention to all the models. Even the smaller models get a boost with this, unlike in Llama 2 where only the big guy had it.
- This thing was pre-trained on 15 trillion tokens, and most of that is in English. They used 16,000 GPUs at the same time to train it. Meta’s also cooking up some new tools to better manage GPU time, which could be a game-changer.
- Here’s a fun fact: they used Llama 2 to clean up the data for this new model. Shows you how these language models can be super useful beyond just chatting.
- They’re trying out a new way to fine-tune these models too, mixing up reasoning tracing with preference ranking to cut down on errors. It’s something like what OpenAI did before.
- There’s also this new library called TorchTune. It’s built right into PyTorch, and it’s supposed to make working with these big language models easier and less memory-hungry.
- On the responsibility front, Meta’s not pulling any punches. They’re pushing hard on making AI safer with stuff like Llama Guard 2 and Code Shield.
- Performance-wise, Llama 3 is top-notch. It’s smashing records, especially in reasoning tasks. They compared it to Claude but haven’t stacked it up against GPT-4 yet. They’re hinting at a bigger model, maybe 400 billion parameters, which sounds like it could really shake things up.
- Best of all, Llama 3 is open source. You can find it on platforms like Hugging Face and WatsonX, which is pretty awesome.
So yeah, it sounds like Llama 3 was definitely worth the wait. Looks like Meta’s pushing the envelope on what these AI models can do. Can’t wait to see what’s next!