Transformers v5.23.0 Adds New Models and Architecture Support
Transformers v5.23.0 adds support for Inkling 2 Small, JEV, LFM3, Dots 3 Note Preview, and LFM3-VL, according to the Transformers Releases announcement. Inkling 2 Small is described as a multimodal mixture-of-experts model with text and image inputs, while JEV provides an experimental decision head for scoring single-token labels. The release also includes kernel, attention, generation, and distributed-training fixes.
This release expands the library's model and architecture coverage with several multimodal, language, and decision-head integrations, alongside maintenance fixes across inference and distributed training.
Release v5.23.0
New Model additions
Inkling2

Inkling 2 Small is a multimodal model from Thinking Machines Lab that accepts text and image inputs and generates text outputs, and it is intended for use in English and other languages and across multiple coding languages. Its language backbone has 48 mixture-of-experts decoder layers, each routing tokens to 8 of 384 experts, for 594B total and 17B active parameters. It keeps the Inkling decoder (learned relative attention, short causal convolutions, mixture-of-experts layers) and adds block attention residuals, gated attention with a learned sink, and dynamic key/value convolutions. Images are encoded by a Perception Encoder-based vision transformer and projected into the language model embeddings, one embedding per 40x40 pixel patch.
Links: Documentation | Blog Post
Some source content is omitted. Continue reading on the original site.
#Transformers#Inkling 2 Small#Thinking Machines Lab#JEV#LFM3#Liquid AI#Dots 3 Note Preview#LFM3-VL