Understanding video
Frames, shots, tracking and moments — how a model gets from a file to an answer, and what compression and sampling cost you.
A video is not a long image. It is a compressed stream in which most frames cannot be decoded on their own, whose timestamps live in a container timebase rather than in seconds, and whose useful content is concentrated in a small number of moments separated by cuts. Every model that answers a question about a video has already thrown almost all of it away before the first layer runs: it sampled some frames, resized them, and turned each into a few hundred vectors. Which frames it sampled usually matters more than which model it was.
These pages work outward from that sampling decision. They cover how a shot boundary is actually found, why a representative frame is a different object from a codec keyframe, what separates recognising an action from recognising the objects in it, how identity is carried across frames and where it breaks, and what compression removes first. The arithmetic is shown wherever a number is claimed, and where a figure depends on your footage rather than on mathematics, the page says so and gives you the command to measure it yourself.
Video Captioning Models: How They Turn Frames Into a Sentence
The frame encoder, the temporal aggregation step and the decoder — and why the frame budget decides more about the caption than the architecture does.
9 min read
Action Recognition in Video, Explained
Why classifying frames is not recognising actions, what the temporal architectures actually add, and how sampling decides what a model can possibly see.
9 min read
Video Summarization: Picking the Frames That Matter
How a twenty-minute video becomes ninety seconds: shot segmentation, importance scoring and a knapsack under a duration budget.
9 min read
Shot Boundary Detection: Finding Where a Scene Changes
How a frame-difference threshold is derived, why histograms beat pixels, and what a dissolve and a camera flash each do to the signal.
9 min read
Video Moment Localization: Finding the Ten Seconds That Answer a Query
How a natural-language query becomes a start and end time inside one video, and why a model that ignores the video scores well on the benchmarks.
9 min read
Object Tracking Across Video Frames, Explained
How a detector's per-frame boxes become trajectories with stable identities, where the identity gets swapped, and why MOTA barely notices when it does.
9 min read
Video Bitrate and Model Accuracy: What Compression Costs You
What lossy video compression removes first, and how to price the accuracy you lose against the storage and bandwidth you save.
10 min read
Keyframe Extraction Algorithms for Long Video
Why a codec keyframe is not a representative frame, and how clustering or greedy selection picks frames that actually stand for something.
10 min read
Aligning a Video Transcript With Visual Events
How speech timestamps and scene-change timestamps get joined onto one timeline, and the four ways the two clocks disagree.
9 min read
Sports Video Analytics: Tracking Players and Events With AI
From broadcast frames to distance covered: detection, tracking, pitch homography, and the noise term that makes naive distance totals nearly double.
10 min read
Video Content Moderation: What Gets Checked Frame by Frame
How a moderation pipeline decides which frames of an upload to look at, and what the sampling rate you choose is actually trading away.
9 min read
Near-Duplicate Video Detection at Upload
Why a re-encoded, rescaled copy of a video still matches the original under a perceptual hash, and which edits break the match anyway.
9 min read
The Latency Budget for Analyzing a Livestream in Real Time
What a glass-to-decision latency budget is composed of, and how to subtract the fixed parts to find out how long your analysis is actually allowed to take.
10 min read
Video Question Answering Models, Explained
How a model answers a question about a video, and why questions that need temporal order are a different problem from questions a single frame can answer.
10 min read
What It Costs to Store and Index a Video Library for Search
A full derivation of storage, decode, embedding and vector-index cost for a stated number of hours of footage, with every input named as an assumption.
10 min read
Reading On-Screen Text in Video Frames
Why reading a sign in a moving video frame is a different problem from OCR on a scanned page, and how having sixty frames of the same text helps.
9 min read
Telling a Camera Cut From Fast Motion in Raw Footage
Why a frame-difference threshold cannot separate a hard cut from a whip pan, and what structure in the motion field does separate them.
9 min read
Video Frame Deduplication Before Sending to a Model
How dropping near-identical consecutive frames changes the token count and the bill for a clip, derived from a stated similarity threshold and a retention ratio you measure.
9 min read
Multi-Camera Video Synchronization Explained
How cross-correlating the audio tracks of two camera feeds recovers their timing offset to sub-frame accuracy, and why one offset is not enough.
10 min read
Detecting a Frozen or Static Video Feed Automatically
The stream is up, frames keep arriving and the picture has not changed in ten minutes — how to detect that automatically and work out where it froze.
9 min read
Other topics
- LLM fundamentals & architecture
- Tokens, tokenization & context windows
- Prompt engineering
- Reasoning models & test-time compute
- Multimodal AI: vision, audio, video
- RAG & retrieval
- Embeddings & vector search
- AI agents & tool use
- Structured output & function calling
- Fine-tuning & post-training
- Local inference errors, string by string
- Running local models day to day
- Testing code that calls an LLM
- Snapshot and property testing for model output
- Regression suites for prompts
- Eval gates in CI
- Flaky tests against a model
- Determinism and the cost of testing
- Contract and streaming tests
- Testing tool calls and retrieval
- Inference, serving & latency
- Rolling out a prompt change
- Testing AI systems in practice
- Forecasting a time series
- Machine learning on tabular data
- Geospatial data and models
- Understanding audio that is not speech
- Core computer vision tasks
- Machine learning on graphs
- Point clouds and 3D
- Evaluation, benchmarks & LLM-as-judge
- Sensor and IoT data
- Logs and event streams
- Models over biological sequences
- Machine learning on molecules
- Embedding and searching code
- Extracting invoices and purchase orders
- Receipts, statements and tax forms
- Insurance policies and contracts
- Deeds, court filings and patents
- Extracting from medical records
- Observability & LLMOps
- CVs, certificates and identity documents
- Shipping, customs and technical documents
- Meetings, email, chat and filled-in forms
- Building an extraction pipeline
- Business, property and inspection documents
- Contract clauses and insurance claims
- Regulated and compliance documents
- Consumer, travel and closing documents
- Mapping one chat API onto another
- SDK and framework migrations
- Hallucination & failure modes
- Re-embedding and model deprecation
- Cutting over between providers
- Parity gaps, shims and legacy endpoints
- Moving between model versions
- Migrating vector stores and caches
- Mapping capabilities and parameters
- Migrating pipelines and agents
- Contracts, runbooks and rollback
- Auditing a codebase before a cutover
- Compliance and fine-tune migration
- LLM cost engineering
- Routing, cost tracking and multi-tenancy
- What a migration does to your prompts
- AI security & prompt injection
- Privacy, compliance & data residency
- AI governance, policy & society
- Building reliable AI applications
- AI hardware, GPUs & compute
- Open-weight models & local inference
- AI for developers & coding agents
- AI in industry: vertical playbooks
- AGI, superintelligence, alignment & the long future
- Machine learning foundations
- NLP fundamentals & classical tasks
- Data engineering for AI
- Synthetic data & dataset curation
- AI product design & UX
- Search, ranking & recommendation
- Enterprise adoption & change management
- AI careers, skills & teams
- Reading AI research
- AI in science & discovery
- Robotics & embodied AI
- AI economics, markets & business models
- AI myths, hype & media literacy
- Context engineering
- Shipping AI features: patterns & anti-patterns
- Build it: end-to-end AI tutorials
- Python for AI: hands-on recipes
- TypeScript, React and the web
- Frameworks and SDKs
- Errors and troubleshooting
- AI facts, numbers and statistics
- The history of AI
- The maths behind AI
- Architectures beyond the transformer
- Reinforcement learning
- Diffusion and generative media
- Speech, audio and voice engineering
- Benchmarks, one at a time
- AI search visibility
- Infrastructure and operations
- Databases and storage for AI
- Knowledge graphs and structured knowledge
- Classical ML in production
- Regulation, jurisdiction by jurisdiction
- Prompt recipes and pattern library
- AI for people who do not write code
- Writing, media and creative work
- Edge and on-device AI
- Interpretability and model internals
- Field notes
- OpenAI model behaviour
- Claude model behaviour
- Gemini model behaviour
- Llama model behaviour
- Mistral model behaviour
- Qwen model behaviour
- DeepSeek model behaviour
- Cohere model behaviour
- Grok model behaviour
- Small model behaviour
- Hybrid model architectures
- Token cost by language and script
- Transliteration, romanization and script handling
- Locale-correct output
- Multilingual generation quality
- Multilingual pipelines
- The EU AI Act, article by article
- AI under the GDPR and EU data law
- US AI regulation, state and sector
- International AI governance and standards
- AI litigation and enforcement
- Running AI workloads on AWS
- Running AI workloads on Google Cloud
- Running AI workloads on Azure
- AI at the edge: Workers, Vercel and Netlify
- Serving models on Kubernetes
- Operating AI infrastructure
- Quantization formats and what they cost
- llama.cpp, flag by flag
- Ollama and the desktop local-model runtimes
- Local models on Apple Silicon
- Hardware for local inference
- Running speech and embedding models locally
- Model files, adapters and conversion
- VRAM arithmetic for local models