#496 – FFmpeg: The Incredible Technology Behind Video on the Internet

Today we go deep on FFmpeg and VLC with Jean‑Baptiste Kempf and Kieran Kunhya, two builders behind the invisible machinery that lets billions press play; I’m wearing the cone hat, full gratitude to the volunteers who made this quiet miracle of open infrastructure possible.

So, legend says VLC opens anything—what’s the weirdest thing you’ve seen it handle?

People use VLC to capture VHS through old composite and SCART cards, it now even plays the niche DVD‑Audio format with its quirky encryption, plus all kinds of strange game codecs via FFmpeg.

Someone implemented a Star Wars game intro codec just to play a single ten‑second sequence from one disc.

At a VideoLAN conference we ran a ‘make the worst file’ contest and VLC still played an MKV where every frame changed resolution, aspect, and rotation, and even a video made entirely of animated SSA subtitles.

There was also a file that was a valid ZIP and a valid MP3 at the same time.

About the hat—best worst logo ever?

The orange cone is weird, bright, and unforgettable; people worldwide literally search for ‘cone player,’ and an April Fools’ attempt to swap it sparked thousands of ‘don’t you dare’ messages.

I love the idea that in a thousand years aliens will find the cone and press play on our archives.

We tried feeding a pancake to a DVD drive and no, it didn’t play, but someone did hide an RFID chip in a mini cone to trigger their movie library like physical media.

For people who’ve used all this without knowing, what actually happens when you press play?

First we fetch bytes from a source like file, HTTP, or disc, then we demux the container into audio, video, and subtitles as separate tracks.

We probe the stream to see if the GPU can decode it, and if not we fall back to software that de‑entropies the bitstream, predicts the picture, and applies inverse transforms to rebuild pixels.

So decoding is about undoing heavy compression and reconstructing what the camera really saw.

Exactly—audio is often compressed by about ten times, video by one to two hundred times, tuned to human perception by working in YUV instead of RGB and using chroma subsampling so you keep brightness detail and toss color detail your eyes barely notice.

Transforms to the frequency domain, quantization, and block processing drive big gains, and each codec generation tends to be roughly thirty percent more efficient at the cost of far more compute.

That thirty percent means orders of magnitude more CPU for encoding, which is fine since you encode once but decode many times.

And this all runs at the metal in C, with lots of handcrafted assembly in hot paths.

After decode and sync, we render video to the GPU and push audio to the sound card, hiding brutal complexity so it feels instant.

People mix up containers and codecs, so what’s the difference?

A container is the file wrapper that multiplexes tracks—think MP4, MOV, AVI, or Matroska—while the codec is the compression format inside like H264 or AV1.

Naming is messy because H264 is also called AVC and technically MPEG‑4 Part 10, so confusion is common.

In practice MP4 often means H264 plus AAC, but file extensions lie, so VLC and FFmpeg sniff the content rather than trusting the name.

Most players fail hard on odd files; how do you not?

We were born from unreliable UDP streams, so we never trust inputs and always try to recover, which is why VLC could play half‑downloaded AVIs back in the day.

Define a video codec in plain terms.

It removes spatial and temporal redundancy using math, favors heavy compute on encode, and builds in resilience so you can join a stream mid‑flight without crashing.

If the camera pans, the background barely changes, so we reuse it across frames, limited only by memory and compute budgets.

Are encode and decode always tied?

They’re coupled but trade‑offs vary; modern families like AV1 or VVC bundle many tools and switch strategies based on content like slides, games, or film.

A screen share needs different tools than a movie scene, so the encoder adapts on the fly.

What is FFmpeg in this ecosystem?

It’s the low‑level toolkit—codecs, muxers, demuxers, filters, and command‑line utilities—embedded everywhere from VLC and Chrome to TVs and servers, often alongside libraries like x264 or libvpx.

A home video and a trillion‑dollar company can use the same stack, sometimes via hilariously long command lines.

The ffmpeg CLI feels like a language—cut, fade, cross‑dissolve, normalize, burn‑in captions, and layer tracks with scary flexibility.

You can script After Effects‑style work on the command line, which has no real Photoshop equivalent for images at this breadth.

What’s the guiding vision for VLC and FFmpeg?

Turn insane complexity into something anyone can use, whether that’s drag‑and‑play or a one‑liner that just works.

This democratized video; what once needed car‑sized encoders in the 90s now runs on a laptop, fueling podcasts and YouTube.

And it’s all open source—how do you explain that to newcomers?

It’s like sharing the cheesecake and the recipe, plus how to build the oven, and letting anyone improve and sell their version so long as they honor the license and community.

People ask us to change licenses like some databases did, but with thousands of contributors, including some no longer alive, you can’t just flip a switch.

Licenses are a social contract that lets forks happen and later reunite, and they let people from any country or background build together.

Give us the quick map of permissive versus copyleft.

Permissive licenses like MIT or BSD are light on conditions, while copyleft like GPL or LGPL requires sharing changes, and projects choose based on goals and how they want contributions to flow.

Why did you move parts of VLC from GPL to LGPL?

Switching libVLC to LGPL let commercial apps embed the engine without open‑sourcing everything, and iOS builds use MPL to align with App Store rules.

As a game dev I want to play cutscenes without being forced to open source my whole game, which LGPL enables while still requiring fixes to the library be shared.

Relicensing meant finding more than 350 contributors worldwide, including speaking with the family of a deceased contributor, because doing it right preserves trust.

What defines the contributor culture?

We judge code, not résumés, and keep a very high bar because tiny core teams must maintain that code for years.

And the Linus Torvalds legend?

He built Git and sparked Linux, which runs most servers and every Android phone, and that scale demands blunt feedback and uncompromising quality.

Volunteers often code after long workdays, so terseness isn’t malice, it’s bandwidth.

Most of us aren’t native English speakers either, so tone can misfire across cultures, but the critique stays about the code.

You’ve famously refused big money to keep VLC ad‑free; why?

Because it wasn’t right for users or the community, and I’d rather sleep well than sell trust for toolbars or data grabs.

Give us the VLC origin story that shaped that ethos.

On a French student‑run campus we evolved from a token‑ring network and a satellite TV experiment into the VideoLAN project, open‑sourced it in 2001, I formed the nonprofit in 2003, nursed it through a near‑death in 2005, and helped it grow to billions of installs.

Back then satellite meant suddenly having dozens of channels, which made the on‑campus streaming breakthrough feel magical.

We built wild network hacks, VLC became the resilient client, and we kept it clean by rejecting shady monetization even when offers were huge.

Recently there was drama around security—what happened with Google?

Google supports open source, but AI‑generated vulnerability reports on obscure codecs flooded volunteers, used scary language, and sometimes got publicized before fixes, which at scale feels like a denial‑of‑service unless it comes with patches or funding.

And the broader corporate misunderstanding?

Some teams treat public trackers like vendor SLAs and name‑drop products while offering token payments, when the right path is either a support contract or sustained sponsorship.

Did raising hell help?

Donations and awareness jumped, still not enough for one full‑time developer, but it moved minds.

Spicy posts even forced responses from app stores and platforms that had ignored us, which is sometimes the only megaphone small projects have.

Follow FFmpeg and VideoLAN, and if you can, donate.

Thanks for the love over the years—ffmpeg for life.

And shout‑out to the maintainers and refactorers doing the unglamorous work that keeps the plane flying while we’re all still onboard.

You kicked off a fun storm by pointing out how many teens contribute real assembly to ffmpeg. I’d love to stay on that energy: what makes that possible and healthy?

We don’t gatekeep. If you can learn C and pick up assembly, you can land patches. Teens have fixed real bugs without grandstanding, which I respect way more than CVE theater.

Same on VLC. Some of our core folks started at 16, even 14 through Summer of Code, and shipped assembly across x264, VLC, and ffmpeg. Anyone can join and grow.

Also, love to the broader crew—including Google folks and Theo. The drama brought in donations and attention, then settled into productive work.

I see it like playful sparring on X. We crack jokes, but the point is teaching how small, volunteer projects run, not dunking for clout.

I called Theo to cool things down. In the end it raised real awareness of community‑driven open source, which is a win.

What actually drives contributors to stick with this stuff?

Video is fun. People show up because they love movies or anime, then stay because the bar is high. Code reviews are tough, but it’s the best school you’ll find.

Andrew Kelley cut his teeth in ffmpeg before starting Zig. It’s real‑world pressure on software used by billions. You learn fast with nowhere to hide.

Multimedia forces you to understand computers, not just code. Miss a frame and the picture falls apart, so you learn caches, pipelines, SIMD, and cycle costs.

ffmpeg likely runs across hundreds of millions of CPUs at any moment. Every instruction has a bill attached.

People also want impact they can explain to grandma. Playing any video beats building yet another invoice portal.

Software is the ultimate passion‑project amplifier. The network can turn a niche obsession into something world‑class.

Tim Sweeney said a lot happens because one awesome person decides to do it. That spirit shows up over and over in software.

I believe in individuals with a clear vision. That’s true in code, and it’s true in trains, cars, and rockets.

Zooming out, how do VLC and ffmpeg fit together?

Think Android to Linux. VLC rides on ffmpeg, and ffmpeg pipelines lean on VideoLAN projects like x264. They orbit together and make each other stronger.

Quick decode: H.264 is the standard, x264 is the open encoder that powers most MP4s on the software side.

Right. If a file came out of a data center, odds are x264 made it.

VideoLAN is a big garden—dav1d, special audio, lots of libs. VLC compiles millions of lines and pulls in tons of third‑party code. Everyone depends on everyone.

Give us a fast history of ffmpeg and the people who bent reality to make it work.

Fabrice planted the seed. In the 2000s, Michael Niedermayer wrestled a zoo of early codecs—DivX, Xvid, Real, Windows Media—so one player could just work without shady codec packs. Around 2008, H.264 matured and HD took off, and reversing became a superpower.

VLC 1.0 hit in that wave and exploded because it played almost anything out of the box.

How does someone even reverse a proprietary blob?

You locate the decode module, rig hooks to dump raw YUV, and study the disassembly for patterns—DCT, entropy, predictors. You compare outputs until you’re bit‑exact, which modern codecs require. Samples are gold; a good test set lets you nail edge cases.

Kostya was legendary. I put a bounty on GoToMeeting’s codec and he delivered in two months, with beautiful, joke‑laced code. He treated binaries like a spec and worked in the dark for weeks.

It feels like archaeology—tiny clues, lots of inference, and patience.

Exactly. Sometimes you get lucky with a simple animation sample and build from there. Sometimes you need VMs to dump memory around DRM. Community samples make or break you.

How do you keep something this sprawling from breaking?

FATE. It’s a volunteer farm that builds and tests across wild compiler and OS combos. It even catches miscompiles from the toolchains.

What still gives you nightmares?

Ten‑bit video. You pack it into 16 bits, then fight bandwidth with custom formats. We ship hand‑written assembly for every conversion across CPU generations. That’s where you get 10 to 50 times speedups over C, sometimes 62 times.

Our headache is supporting ancient and modern OSes at once—Windows XP to 11, iOS 9 to now, even OS/2. We stitch toolchains together so old hardware keeps working. It’s our quiet stand against forced upgrades.

Make the case for assembly, and what makes your flavor special.

We write SIMD by hand, not intrinsics, and squeeze vectors hard. Video is a grid, so wide ops shine. The gains aren’t small; they’re multiples.

Look at dav1d, our AV1 decoder. Around thirty thousand lines of C and roughly two hundred forty thousand of assembly. It powers billions of devices while hardware support lags, and can do 720p on one or two cores.

People argue compilers can auto‑vectorize well enough.

They can’t match this. The gap isn’t five percent. It’s many times slower.

We even bend calling conventions inside the library for speed, then do runtime CPU detection to jump to the best path per machine. It’s extreme, but that’s how you run on everything from old laptops to shiny servers.

Why should high‑level devs care?

Performance per dollar. As Moore’s Law slows and costs rise, you’ll hit hard limits in CPU, memory, or network. Real‑time work and AI inference will need tight hot paths, not just more hardware.

If someone wants to learn, where do they start?

I wrote ASM‑lessons to teach by doing, not by grammar. High‑school math and C pointers are enough. Teens have already shipped patches from it.

I began on Itanium, where compute outpaced memory, so you had to reuse registers and pack data. That lens helps everywhere. Tools like x86inc smooth over calling‑convention pain. Learn a bit of assembly and the memory hierarchy, and you’ll code better in any language.

What’s your take on Rust, given how polarizing it can be?

Memory safety is a worthy goal, but production code has to be as reliable and feature-complete as today’s tools, with stable ABI and more than one compiler; utopian vibes do not ship. Rewrites that reach only 85 to 90 percent of parity are not enough in places like core utils or FFmpeg.

I’ve embedded Rust modules in VLC and my new open‑source startup Kiber uses Rust; it’s a better C++ for ownership and safety when you start fresh. It’s painful at the boundaries of large C and assembly codebases, and inline assembly can blow past Rust’s guarantees, so we also need compile‑time checks for “secure assembly.”

I’m still in awe of what handwritten assembly unlocks in speed. ARM performance varies wildly across generations, while x86 tends to be more uniform, so tuning ARM is often trickier.

Two assembly wizards deserve credit: Henrik Gramner on x86 and Martin Storsjö on ARM; Martin even codes assembly on a phone while watching his kids—pure craft.

There was a split between FFmpeg and LibAV over governance that even hit Linux distros, which was confusing to users.

Forks happen in open source and can be healthy; like GCC or WebKit to Blink, FFmpeg and LibAV diverged, then the community merged back and FFmpeg ended up stronger with better processes.

The human cost worries me—burnout can hit the few core maintainers projects rely on.

Maintenance burnout is the real strain, now amplified by AI-driven noise and low‑quality patches; the XZ backdoor saga showed how thin the guardrails can be when a single maintainer is overwhelmed. Critical libraries with a bus factor of one—time zones, libxml and many others—show how little large organizations grasp the mental load.

Some in security were scolding hobbyists over edge‑case bugs found by bots, missing that these are volunteer projects, not negligence.

I even got a death threat with powder when we dropped PowerPC builds for VLC; it hardens you, but simple thank‑yous matter, and yes, we still coordinate on IRC despite its spartan vibes.

Let’s celebrate open source and the people behind it.

Celebrate the craft of building things that are not just okay, but excellent.

We build hard, useful tech so everyone benefits—that’s what gets me up in the morning.

When did you first fall for programming?

QBasic on Windows 3.1 and 95—little loops that made me dream way too fast about writing a full soccer game.

Basic and Turbo Pascal, then Logo in Florence at age eleven; steering a turtle with code hooked me for life.

Back to x264, the workhorse behind internet video—what made it special?

x264 broke with PSNR‑chasing and leaned into human perception with psychovisual rate distortion and adaptive quantization; the ParkJoy sample proved how much better it looked than high‑PSNR encoders. It won over Blu‑ray pros too, and I still buy discs because the quality shows.

Planet Earth’s bird swarms and anime banding pushed innovations that became x264’s calling card, and it’s still the benchmark everyone compares to. It grew from École Centrale Paris into a global assembly‑heavy effort, fueled in part by fansubbing communities that also advanced tools like Aegisub for complex subtitles.

How do H.264 and AV1 compare, and where do x264 and dav1d fit?

Each codec generation typically saves 25 to 50 percent, and AV1—born from royalty‑free goals after licensing went wild—often cuts 40 to 60 percent versus H.264; the price is far more search and compute. Platforms like YouTube encode nearly everything in H.264 and re‑encode popular videos in AV1 to save bandwidth and battery at scale.

AV1 adds bigger blocks and many more coding tools and partitions, which explodes complexity.

ProRes exists for editing—fast decode and snappy seeking—so it’s a different world from distribution.

I‑frames stand alone, P‑frames predict from the past, and B‑frames can lean on past and future frames, so decoding order is not display order; intra‑refresh spreads the “keyframe” over time. The fact this all stays bit‑exact across vendors and countries is still kind of miraculous.

FFmpeg exposes a ton of knobs, and entire teams at big platforms just tune those parameters for different content types.

Fun history: Google Video once embedded VLC in the browser, and today VLC runs via WebAssembly—same spirit, new path.

WikiLeaks showed the CIA used a modified VLC—what actually happened?

They shipped a normal VLC plus a malicious DLL that quietly read your Documents and sent it out, and later attackers abused our signed DLL to get VLC banned in India despite it not being our app; always download from videolan.org. Search engines still surface fake “VLC” installers—Germany has a notorious one with a three‑week sleep before dropping adware—and phishers push bogus “security updates.”

What’s the plan for VLC sandboxing?

VLC loads around five hundred plugins, calls GPU drivers, and integrates third‑party code, so one bug can be a foothold; we’re splitting VLC into multiple processes with strong sandboxes so a crash cannot pivot into your files. The hard part is pushing gigabits per second through those boundaries without stalling media.

With so many users, someone will hit every feature.

A user once told me our jigsaw “puzzle” filter was too easy, so I bumped it up to a wild 256 by 256; we also render video as ASCII in terminals, which is perfect for debugging multicast on routers with no GUI.

How is streaming different from just playing a local file?

Modern adaptive streaming is mostly a CDN and congestion dance: if a chunk downloads too slowly, drop quality; climbing back up is the art, while true real‑time at events adds the heavy lift of live multi‑resolution encoding, DRM and device diversity. Compared to satellite era constraints, it’s simpler mathematically, but live workflows remain demanding.

People forgive video softness more than audio hits; switching to lower‑profile AAC can sound instantly tinny and far more jarring.

Tell me about Kyber and ultra‑low‑latency control.

We built an SDK that runs over a single QUIC connection to stream video, audio and controls in sync—down to the millisecond—with clock‑drift correction across multiple cameras and sensors, plus forward‑error correction for reliability; we even drove a small rover in Las Vegas from France. It’s for tele‑op and training across drones, robots, cars and more, where latency—not just quality—defines safety and usefulness.

What latency can you actually hit?

My dream is four milliseconds glass‑to‑glass; we’ve hit about seven milliseconds Windows‑to‑Mac, and most of that is encoder and decoder time, so faster codecs and hardware will be key as we scale to millions of tele‑observed and tele‑operated machines; Kyber is open source under AGPL with a commercial option.

What’s next for codecs—AV2 and H.266?

AV2 targets roughly thirty percent beyond AV1; we demoed an early VLC playback with our decoder we’ll call “dav2d,” while encoders like AVM and SVT‑AV2 mature. On the MPEG side, H.266, also called VVC, is the next successor after H.265.

HEVC is H.265 and VVC is H.266. Why the two names, and what makes H.266 such a leap?

The dual labels come from ISO and the ITU co-authoring standards, so names get fused. At a high level, each generation delivers roughly thirty percent better compression than the last, though results swing widely by content; meanwhile AV1 and AV2 push similar gains with a focus on being royalty-free, while encoding cost and patent fees are the big practical differences.

One body is a UN agency and the other is an industry group, so you get paired names. AV standards went through patent checks by design; MPEG deliberately keeps patent talk outside the room.

What does it mean to patent a codec, and why are there so many patents?

People patent even small building blocks, so specifications accrete countless claims—some rarely used—then pools like MPEG LA and HEVC Advance charge everyone who ships it. That mess got so expensive that big platforms pushed AV1 and AV2 to avoid fees, and it also forces double creativity: invent better tools and route around existing claims. VLC being based in France helps because software patents are largely unenforceable there; otherwise our per‑user patent bill would have been absurd.

Is Europe hostile to startups, or is there real momentum?

We’re two Europeans here talking video; the community is strong.

There’s a fresh wave of founders, especially in France, and the old stigma around failure is gone. Regulation can be heavy, but there’s new energy in AI and a growing appetite to build.

What about government pressure, liability, even arrests over platform content—does that chill you?

VLC is a client, not a platform; we host no content and have no telemetry, so we can’t spy or censor. Legal letters arrive, but in Europe responding doesn’t bankrupt you; the bigger ask was intelligence services nudging for a backdoor, which we flatly refused. We compile offline, rebuild our toolchains, double‑sign releases, and have fended off a sophisticated attempt to slip in a fake binary; if needed, open source lets us relocate and continue because the code lives everywhere.

So there’s never any censoring or tracking in VLC?

Never. It runs locally, talks to no server, and we only operate download mirrors. When authorities reach out, it’s basic support like fixing broken playback; once, during the Afghan war, we shipped a special build to restore RTSP movies for troop morale.

And any hidden code would be spotted by contributors worldwide, which makes covert changes extremely hard.

Millions use it, militaries too, yet the core team is tiny—does that stress you out?

We’re about six to eight people, and I handle legal. I used to stress, but we’re a tool; tools serve good and bad ends, and that’s not on the wrench. Thinking in worst‑case trees helps me stay calm.

What’s your source of zen?

I constantly ask, what’s the true worst case—is anyone harmed, do I die? If not, move forward. Companies come and go; the source stays open, so even lawsuits can’t stop the work.

You mentioned an active archiving world seeing FFmpeg as a Rosetta stone. How does that shape the tech?

Archivists like Dave Rice champion open tools because budgets are tight and permanence matters; they co‑created FFV1, a lossless, resilient codec with fast software encoders, GPU paths, and strong error recovery. They bring priceless knowledge on formats, color science, and workflows, and C’s stability makes their work readable far into the future—like a modern Latin for computing.

They see details the rest of us miss, especially around colorimetry and storage, and we learn constantly from them.

We need to capture the twentieth century before it decays in a century of sludge.

We’re already losing films from the thirties through the fifties because their perceived value was low.

Tape from the seventies and eighties is also at risk, with too few heads to read it all, so curators must choose what survives; lossless archives keep subtle clues intact so future eyes can find meaning we’d otherwise compress away.

Where are FFmpeg and VLC headed—five, ten, twenty years out? Holograms, VR, even brain interfaces?

We already handle VR and XR, stream XR to glasses with Kiber, and see codecs rising for point clouds, depth maps, 3D assets, even astronomy. Multimedia to us is any timed stream to human senses, so haptics or smell belong too; we’ve shipped a plugin used in motion chair cinemas that syncs movement and effects.

With so much surface area, how do you keep up?

We’re maintainers, not the only builders; each year roughly one hundred fifty contribute to VLC and a few hundred to FFmpeg. Our job is architecture—spatial audio was the first module, the next ones slot in easily—so new formats plug in cleanly, whether that’s haptics or brainwave streams. Hype cycles spawn many proposals, then real standards emerge, like we saw around 3D audio.

You’ve been blunt about closed ecosystems like Dolby—why?

They used to lead with engineering; now it feels dominated by licensing muscle. That shift slows innovation and walls off creators.

Favorite internet lore or patches tied to FFmpeg?

I love the morale patches riffing on doing the hard work and the one that says nothing is beyond our reach; it captures the spirit perfectly. Also, FFmpeg runs on Mars—the 2020 rover compresses imagery with it.

VLC pops up in wild places too—Formula One paddocks, ESA and SpaceX launch rooms, even CERN’s LHC where they streamed analog cameras around the twenty‑seven‑kilometer ring. Seeing it there is pure joy.

Any tweets you regret?

None. Regret is mental drag; learn fast and move on.

I’m deeply grateful for FFmpeg, VLC, and the communities around them, and for the spicy posts—and the wizard hat. Thanks for the time and for building tools that brought so much joy. I’ll close with a thought often attributed to a certain legendary kernel author: the best programmers do it less for fame or pay and more because writing code is simply fun.