Create

Sign in to ReadmeX

or

How we handle your data: Privacy Policy

ReadmeX

Your AI reading companion

What would you like to catch up on?

Pick a briefing, or ask your own question.

Your community and people briefings stay personal to you.

Story

Transformers v5.23.0 Adds New Models and Architecture Support

AI summary

Transformers v5.23.0 adds support for Inkling 2 Small, JEV, LFM3, Dots 3 Note Preview, and LFM3-VL, according to the Transformers Releases announcement. Inkling 2 Small is described as a multimodal mixture-of-experts model with text and image inputs, while JEV provides an experimental decision head for scoring single-token labels. The release also includes kernel, attention, generation, and distributed-training fixes.

In-depth insight

This release expands the library's model and architecture coverage with several multimodal, language, and decision-head integrations, alongside maintenance fixes across inference and distributed training.

Release v5.23.0

New Model additions

Inkling2

image

Inkling 2 Small is a multimodal model from Thinking Machines Lab that accepts text and image inputs and generates text outputs, and it is intended for use in English and other languages and across multiple coding languages. Its language backbone has 48 mixture-of-experts decoder layers, each routing tokens to 8 of 384 experts, for 594B total and 17B active parameters. It keeps the Inkling decoder (learned relative attention, short causal convolutions, mixture-of-experts layers) and adds block attention residuals, gated attention with a learned sink, and dynamic key/value convolutions. Images are encoded by a Perception Encoder-based vision transformer and projected into the language model embeddings, one embedding per 40x40 pixel patch.

Links: Documentation | Blog Post

Some source content is omitted. Continue reading on the original site.

#Transformers#Inkling 2 Small#Thinking Machines Lab#JEV#LFM3#Liquid AI#Dots 3 Note Preview#LFM3-VL

Original · 21 min readRead the original
Group chat

No comments yet. Start the conversation.