Google just announced its latest Gemma 4 12 billion parameter model and this is a real game changer. What distinguishes this model from virtually all other multimodal AI systems is that it is completely encoder-free. This new architecture is a complete departure from how traditional multimodal models work, and may have a big impact on the future of local AI.
To understand what makes this a big deal, we first need to understand how most of the multimodal models of today work. Language models are designed to read tokens, which are essentially chunks of text that are converted into numbers. They do not inherently grasp pixels from images or soundwaves from audio. So today’s AI systems are usually several networks strung together.
When you give an image to a conventional AI model, it’s first processed by a large vision encoder that takes in the raw pixels. Getting that image data into a format that the language model can understand takes a lot of computing power. Audio is similar and requires a separate speech encoder to convert sound waves before the language model can use them. This results in multiple networks running at the same time, eating up precious VRAM and slowing down performance.
Google DeepMind took a different approach to this problem. Gemma 4 uses small 48 by 48 pixel patches of images instead of a heavy vision encoder. These patches are then fed through a light mathematical operation called linear projection. This projection does not interpret the image. It just reshapes the raw pixel values into a structure that is compatible with the token format of the language model.
Traditional vision encoders can have hundreds of millions of parameters. In some models, the vision encoder has about 550M parameters. Instead Gemma 4 removes this component down to about 35 million parameters. Google removes the complex attention layers and image processing networks, allowing the main language backbone to do the visual reasoning directly.
This approach works for audio as well. Gemma 4 slices a 16 kHz raw audio signal into 40 ms frames. Each frame contains 640 floating point values that represent the sound wave. These values are then projected to the language model input space using a simple projection layer. The transformer can work directly on audio, which is inherently sequential just like text, without a specialized speech encoder.
The advantages are considerable. Gemma 4 slashes encoder bloat and achieves impressive multimodal reasoning with a smaller memory footprint. According to benchmarks, the 12 billion parameter model is close in performance to much larger 26 billion parameter models, but is practical to run locally on machines with 16GB or more of VRAM.
The image reasoning was surprisingly fast on an Apple M2 MacBook Pro with 24GB of unified memory. MLX allows the model to analyze images in near real time, even operating entirely offline. Gemma 4 provided fast, accurate visual understanding, from airport departure boards to blurry television pictures.
Google has also incorporated native multi-token prediction, so the model can output multiple tokens at once, speeding up inference.
In general, Gemma 4 shows that a single language backbone can natively support text, vision, and audio. This encoder-free approach could be the future of efficient multimodal AI and open the door for powerful AI models to run directly on laptops, smartphones and edge devices.
