Multimodal AI: vision, audio, video
How models take in pixels, audio and frames — what that costs, where it fails, and which parts of the job still belong to a specialised tool.
A multimodal model is not a language model with a camera bolted on. An image is turned into tokens, those tokens go through the same transformer as your text, and almost everything that follows — the price, the resolution limit, the peculiar failures — follows from how that conversion works.
These pages start from the conversion and work outward. Where a classical tool still wins, they say so: OCR engines, detectors and speech recognisers did not stop being good at their jobs because a general model learned to do them adequately.
How a Multimodal Model Sees an Image
An image becomes a sequence of tokens and then goes through an ordinary transformer. Traced end to end, with the patch arithmetic.
6 min read
Image Tokens: How Pictures Are Priced
The published tiling formulas, worked through on real image dimensions, and the resizing you should do before the provider does it for you.
6 min read
Detail Levels: When Low-Res Image Input Is Enough
What the detail setting actually changes, a minimum-feature-size rule for choosing it, and how to A/B it on your own images.
6 min read
OCR With a Vision Model vs a Real OCR Engine
The two systems return different things, fail differently, and price differently. Which differences are structural, and how to decide.
6 min read
Document Understanding: PDFs, Tables and Layout
Where table extraction still breaks, shown as concrete structures, and what to do about each one.
7 min read
Chart and Diagram Reading: What Models Get Wrong
Why reading a chart is harder than reading a table, which encodings break, and what the published chart benchmarks are actually testing.
6 min read
Video Understanding: Frames, Sampling and Cost
Video is a stack of images with a token bill attached. Frame sampling is the main lever, and it is a cost decision before it is a quality one.
7 min read
Audio Input: Speech-to-Text vs Native Audio Models
A transcription pipeline and a native audio model differ in what information survives, not only in latency. Which difference matters for which job.
7 min read
Text-to-Speech in 2026: Latency, Voices and Cost
What actually determines time-to-first-audio, how to measure it yourself, and the axes other than latency that decide a TTS choice.
6 min read
Real-Time Voice Agents: The Full Latency Budget
Every term between the microphone and the speaker, what sets each one, and which ones are actually worth attacking.
7 min read
Image Generation Models: A Practical Comparison
The structural axes that actually differ between image model families, and a protocol for running the comparison on your own prompts.
7 min read
Prompting for Images vs Prompting for Text
Why skill at prompting language models transfers badly to image models, traced to the difference in how each one reads your words.
6 min read
Image Editing and Inpainting via API
Mask-based editing, instruction editing, and the production details — alpha channels, feathering, seams and re-encoding — that decide whether it holds up.
7 min read
Multimodal RAG: Retrieving Over Images and Text
Joint embedding versus caption-then-index versus retrieving over page images — what each costs in storage, and what each fails at.
7 min read
Vision Model Hallucinations: Seeing Things That Aren't There
A taxonomy of the ways a vision model reports things the image does not contain, anchored to the published literature, with detection strategies for each.
7 min read
Screenshot Understanding for UI Automation
Getting a model to click the right pixel: the coordinate-space transform that causes most of the error, and how to test your own accuracy.
7 min read
Handwriting and Low-Quality Scans
How a general vision model and a dedicated handwriting engine differ on messy input, and how to measure which one is better on yours.
6 min read
Bounding Boxes and Grounding: Getting Coordinates Out
Which approaches return coordinates at all, in what coordinate space, and how to measure whether they are accurate enough for your use.
6 min read
Image Safety Filters: Why Your Benign Request Was Blocked
Where in the stack a block happens, the categories of benign request that trip classifiers, and how to build a product that copes with refusals.
6 min read
Building a Multimodal Pipeline That Doesn't Cost a Fortune
Downscale, cache and cascade — the three levers that move a multimodal bill, with the arithmetic for deciding whether each is worth it.
7 min read
Other topics
- LLM fundamentals & architecture
- Tokens, tokenization & context windows
- Prompt engineering
- Reasoning models & test-time compute
- RAG & retrieval
- Embeddings & vector search
- AI agents & tool use
- Structured output & function calling
- Fine-tuning & post-training
- Inference, serving & latency
- Evaluation, benchmarks & LLM-as-judge
- Observability & LLMOps
- Hallucination & failure modes
- LLM cost engineering
- AI security & prompt injection
- Privacy, compliance & data residency
- AI governance, policy & society
- Building reliable AI applications
- AI hardware, GPUs & compute
- Open-weight models & local inference
- AI for developers & coding agents
- AI in industry: vertical playbooks
- AGI, superintelligence, alignment & the long future
- Machine learning foundations
- NLP fundamentals & classical tasks
- Data engineering for AI
- Synthetic data & dataset curation
- AI product design & UX
- Search, ranking & recommendation
- Enterprise adoption & change management
- AI careers, skills & teams
- Reading AI research
- AI in science & discovery
- Robotics & embodied AI
- AI economics, markets & business models
- AI myths, hype & media literacy
- Context engineering
- Shipping AI features: patterns & anti-patterns