This fails to acknowledge that synthesized noise can lack the detail and information in the original noise.
When you watch a high-quality encode that includes the actual noise, there is a startling increase in resolution from seeing a still to seeing the video. The noise is effectively dancing over a signal, and at 24 fps the signal is still perfectly clear behind it.
Whereas if you lossily encode a still that discards the noise and then adds back artificial noise to match the original "aesthetically", the original detail is non-recoverable if this is done frame-by-frame. Watching at 24 fps produces a fundamentally blurrier viewing experience. And it's not subtle -- on old noisy movies the difference in detail can be 2x.
Now, if h.265 or AV1 is actually building its "noise-removed" frames by always taking into account several preceding and following frames while accounting for movement, it could in theory discover the signal of the full detail across time and encode that, and there wouldn't be any loss in detail. But I don't think it does? I'd love to know if I'm mistaken.
But basically, the point is: comparing noise removal and synthesis can't be done using still images. You have to see an actual video comparison side-by-side to determine if detail is being thrown away or preserved. Noise isn't just noise -- noise is detail too.
Grain is independent frame-to-frame. It doesn't move with the objects in the scene (unless the video's already been encoded strangely). So long as the synthesized noise doesn't have an obvious temporal pattern, comparing stills should be fine.
Regarding aesthetics, I don't think AV1 synthesized grain takes into account the size of the grains in the source video, so chunky grain from an old film source, with its big silver halide crystals, will appear as fine grain in the synthesis, which looks wrong (this might be mitigated by a good film denoiser). It also doesn't model film's separate color components properly, but supposedly that doesn't matter because Netflix's video sources are often chroma subsampled to begin with: https://norkin.org/pdf/DCC_2018_AV1_film_grain.pdf
Disclaimer: I just read about this stuff casually so I could be wrong.
> Grain is independent frame-to-frame. It doesn't move with the objects in the scene (unless the video's already been encoded strangely)
That might seem like a reasonable assumption, but in practice it’s not really the case. Due to nonlinear response curves, adding noise to a bright part of an image has far less effect than a darker part. If the image is completely blown out the grain may not be discernible at all. So practically speaking, grain does travel with objects in a scene.
This means detail is indeed encoded in grain to an extent. If you algorithmically denoise an image and then subtract the result from
the original to get only the grain, you can easily see “ghost” patterns in the grain that reflect the original image. This represents lost image data that cannot be recovered by adding synthetic grain.
> If you algorithmically denoise an image and then subtract the result from the original to get only the grain, you can easily see “ghost” patterns in the grain that reflect the original image. This represents lost image data that cannot be recovered by adding synthetic grain.
The synthesized grain is dependent on the brightness. If you were to just replace the frames with the synthesized grain described in the OP post instead of adding it, you would see something very similar.
> So long as the synthesized noise doesn't have an obvious temporal pattern, comparing stills should be fine.
The problem is that the initial noise-removal and compression passes still removed detail (that is more visible in motion than in stills) that you aren't adding back.
If you do noise-removal well you don't have to lose detail over time.
But it's much harder to do streaming-level video compression on a noisy source without losing that detail.
The grain they're adding somewhat distracts from the compression blurriness but doesn't bring back the detail.
>The grain they're adding somewhat distracts from the compression blurriness but doesn't bring back the detail.
Instead of wasting bits trying to compress noise, they can remove noise first, then compress, then add noise back. So now there aren't wasted bits compressing noise, and those bits can be used to compress detail instead of noise. So if you compare FGS compression vs non-FGS compression at the same bitrate, the FGS compression did add some detail back.
The AR coefficients described in the paper are what allow basic modeling of the scale of the noise.
> In this case, L = 0 corresponds to the case of modeling Gaussian noise whereas higher values of L may correspond to film grain with larger size of grains.
I think you've missed the point here: the noise in the originals acts as dithering, and increases the resolution of the original video. This is similar to the noise introduced intentionally in astronomy[1] and in signal processing[2].
Smoothing the noise out doesn't make use of that additional resolution, unless the smoothing happens over the time axis as well.
Perfectly replicating the noise doesn't help in this situation.
Your first link doesn't seem to be about introducing noise, but removing it by averaging the value of multiple captures. The second is to mask quantizer-correlated noise in audio, which I'd compare to spatial masking of banding artifacts in video.
Noise is reduced to make the frame more compressible. This reduces the resolution of the original only because it inevitably removes some of the signal that can't be differentiated from noise. But even after noise reduction, successive frames of a still scene retain some frame-to-frame variance, unless the noise removal is too aggressive. When you play back that sequence of noise-reduced frames you still get a temporal dithering effect.
Here's[1] a more concrete source, which summarizes dithering in analog to digital converters as follows:
With no dither, each analog input voltage is assigned one and only one code. Thus, there is no difference in the output for voltages located on the same ‘‘step’’ of the ADC’s ‘‘staircase’’ transfer curve. With dither, each analog input voltage is assigned a probability distribution for being in one of several digital codes. Now, different voltages with-in the same ‘‘step’’ of the original ADC transfer function are assigned different probability distributions. Thus, one can see how the resolution of an ADC can be improved to below an LSB.
In actual film, I presume the random inconsistencies of the individual silver halide grains is the noise source, and when watching such a film, I presume the eyes are doing the averaging through persistence of vision[2].
In either case, a key point is that you can't bring back any details by adding noise after the fact.
One thing worth noting is that this extra detail from dithering can be recovered when denoising by storing the image to higher precision. This is a lot of the reason 10 bit AV1 is so popular. It turns out that by adding extra bits of image, you end up with an image that is easier to compress accurately since the encoder has lower error from quantization.
People often assume noise is normal and IID but it usually isn't. It's s fine approximation but isn't the same thing, which is what the parent is discussing.
Here's an example that might help you intuit why this is true.
Let's suppose you have a digital camera and walk towards a radiation source and then away. Each radioactive particle that hits the CCD causes it to over saturate, creating visible noise in the image. The noise it introduces is random (Poisson) but your movement isn't.
Now think about how noise is introduced. There's a lot of ways actually, but I'm sure this thought exercise will reveal to you how some cause noise across frames to be dependent. Maybe as a first thought, think about from sitting on a shelf degrading.
It's the same thing. Yes, not related to the movement of the camera, but I thought that would be easier to build your intuition about silver particles being deposited onto film. You make in batches, right?
The point is that just because things are random doesn't mean there aren't biases.
To get much more accurate, it helps to understand what randomness actually is. It is a measurement of uncertainty. A measurement of the unknown. This is even true for quantum processes that are truly random. That means we can't know. But just because we can't know doesn't mean it's completely unknown, right? We have different types of distributions and different parameters in those distributions. That's what we're trying to build intuition about
> Grain is independent frame-to-frame. It doesn't move with the objects in the scene (unless the video's already been encoded strangely). So long as the synthesized noise doesn't have an obvious temporal pattern, comparing stills should be fine.
Sorry if I wasn't clear -- I was referring to the underlying objects moving. The codec is trying to capture those details, the same way our eye does.
But regardless of that, you absolutely cannot compare stills. Stills do not allow you to compare against the detail that is only visible over a number of frames.
To illustrate the temporal aspect: consider a traditional film projector. Between every frame, we actually see complete darkness for a short time. We could call that darkness "noise", and if we were to linger on that moment, we'd see nothing of the original signal. But since our visual systems tend to temporally average things out to a degree, we barely even notice that flicker (https://en.wikipedia.org/wiki/Flicker_fusion_threshold). I suspect noise and grain are perceived in a similar way, where they become less pronounced compared to the stable parts of the signal/image.
Astrophotographers stack noisy images to obtain images with higher SNR. I think our brains do a bit of that too, and it doesn't mean we're hallucinating detail that isn't there; the recorded noise - over time - returns to the mean, and that mean represents a clearer representation of the actual signal (though not entirely, due to systematic/non-random noise, but that's often less significant).
Denoising algorithms that operate on individual frames don't have that context, so they will lose detail (or will try to compensate by guessing). AV1 doesn't specify a specific algorithm to use, so I suppose in theory, a smart algorithm could use the temporal context to preserve some additional detail.
The noise does not contain a signal, does not dance over it, and is not detail. It is purely random fluctuations that are added to a signal.
If you have a few static frames and average them, you improve SNR by retaining the unchanged signal and having the purely random noise cancel itself out. Retaining noise itself is not useful.
I suspect the effect you might be seeing is either just an aesthetic preference for the original grain behavior, or that you are comparing low bandwidth content with heavy compression artifacts like smoothing/low pass filtering (not storing fine detail saves significant bandwidth) to high bandwidth versions that maintain full detail, entirely unrelated to the grain overlaid on top.
Denoising generally removes signal too. Removing noise and reconstituting similar noise to maintain the apparent qualities of an input can help compression, but you are also cutting out true details (typically fine detail).
The effect GP is pointing out is how denoisers damage detail, which is true. This detail can persist over multiple frames, which is why many denoisers include a temporal comparison component to mitigate the damage, but you still lose detail.
Definitely, but that's a result of compressing (with any algorithm) past a bitrate that would allow detail to be retained with high frequency details including noise being the first to go, not the result of noise having signal as was stated and more importantly unrelated to film grain synthesis which is about adding something new on the output side.
There are definite philosophical questions over the merits of adding noise, but the problem with their example here is their denoising process appears to excessively blur everything, so both it and the synthesized grain image look noticeably less sharp than the source. The grain itself also looks too much like basic noise, and not really grain like.
> both it and the synthesized grain image look noticeably less sharp than the source
That's true, but at a given bitrate (until you get to very high bitrates), the compressed original will usually look worse and less sharp because so many bits are spent trying to encode the original grain. As a result, that original grain tends to get "smeared" over larger areas, making it look muddy. You lose sharpness in areas of the actual scene because it's trying (and often failing) to encode sharp grains.
Film Grain Synthesis makes sense for streaming where bandwidth is limited, but I'll agree that in the examples, the synthesized grain doesn't look very grain-like. And, depending on the amount and method of denoising, it can definitely blur details from the scene.
It seems like a shame that they didn’t include a screenshot of the original (with natural grain), after suffering from low-bitrate streaming. Aka the actual baseline.
I can see why they want to compare against the actual local copy of the video with the natural grain. But that’s the perfect copy that they can’t actually hope to match.
But why do they need to emphasize it even more than the examples they gave? The "AV1 with FGS @ 2804 kbps" already looks as good or better than the "AV1 (without FGS) @ 8274 kbps", so it'll definitely look better than AV1 without FGS at an even lower bandwidth.
I think the distinction here is, they provide the "regular" stream, and the FGS stream noting that it's much smaller yet looks similar. What they don't have is a lower-bandwidth "regular" one, what 2000-or-so kbps looks like without FGS.
AV1 has tunable FGS levels, and to my eye they went very slightly higher than they should have (though there are tradeoffs; at some bitrates the blurring+renoising is so much better than the other visual artifacts you will otherwise get, that you do want it that high).
A few things to note:
- still-frames are also a mediocre way to evaluate video quality.
- a theoretically perfect[1] noise-removal filter will always look less detailed than the original source, since your brain/eye system will invent more detail for a noisy image than for a blurry image.
1: By which I mean a filter that preserves 100% of the non-grain detail present, not one that magically recovers detail lost due to noise.
Since the beginning of film editors have added tricks in post.
I would love for them to provide an option to view it with film simulation vs without.
One of my favorite movies of all time, The Holdovers, did film simulation extremely well. It's set in the '70s so it attempts to look like a movie of that era.
It looked great to me, but if you're an actual film nerd you're going to notice a lot of things aren't exactly accurate.
Maybe in the near future we'll see Netflix being able to process some post effects on the client. So if you're color blind, you get a mode for that. If you don't want fake grain you can turn it off.
mpv can already do this: `--vf=format:film-grain=no` turns off grain synthesis. There are also people making custom shaders for things like emulating CRT monitors (originally for retro gaming, but I see there are also mpv versions).
Holdovers was pretty great. Sorta like an 80’s school comedy but with the perspective of the adults included, making it a totally different type of movie.
Agree, as someone who has spent way too much time studying the way motion picture film looks up close, this isn’t very realistic looking. It’s really just a form of dithering.
that's an understatement. it just looks like RGB noise effect was added. film grain does not look like RGB noise. to me, film grain is only one part of what gave film the film look. the way the highlights bloom rather than clip. it also was more natural/organic/some descriptive other than the ultrasharp of modern digital acquisition. using some SoftFX or Black Mist type filters help, but it's just not the same as it is a digital vs analog type of acquisition. all of these attempts at making something look like it's not just keep falling down in the same ways. but hey, there's a cool tech blog about it this time. film grain filters have been around for a long time, yet people just don't care for them. even in Blu-ray time frame, there was attempts at removing the grain in the encode and applying it in playback. Netflix isn't coming up with anything new, and apparently nothing exciting either based on the results.
Agree, I really prefer luminance noise when simulating film grain rather than treating RGB so independently - the RGB here looks wrong to me. If you are thinking about film, you wouldn't model it as pure photon/detector noise, because film has some crystalline structure and usually there is some correlation between the channels. (Some of the best grain is on B&W films though, which have wonderful structure and statistics.)
I only have their single sample in the blog post to go by, but the original source frame also looks just like RGB noise? And the recreation looks really close if I swap between them. Looks fine to me.
So as long they're analyzing the style of grain in the source properly, which the technical part of the post mentions they do...
These days, when we see noise/grain in an end product it has likely been added in post-production. So, ideally, studios would provide distributors with a noiseless source plus grain synthesis parameters. Bonus: many viewers would welcome an option to turn it off.
> provide distributors with a noiseless source plus grain synthesis parameters.
What parameters would that be? Make it look like Eastman Ektachrome High-Speed Daylight Film 7251 400D? For years, people have taken film negative onto telecines and created content of grain to be used as overlays. For years, colorists have come up with ways of simulating the color of specific film stocks by using reference film with test patterns that's been made available.
If a director/producer wants film grain added to their digital content, that's where it should be done in post. Not by some devs working for a streaming platform. The use of grain or not is a creative decision made by the creators of the work. That's where it should remain
Netflix has their own in-house studio, right? The encoding and lossy compression is going to happen anyway. It seems like an easy win, for their directors to provide a description of the grain they want, so it can be replicated on the user side.
what does having an in-house studio have to do with it? they stream more content than just their own, and so they would not have creative license to alter content. they would only have some type of distribution license to stream the content as provided
Because they previously did not do commercials their original TV shows were not written with pauses every few minutes. They have approved cameras. They spend heavily on movie star salaries and skimp on set production.
> If a director/producer wants film grain added to their digital content, that's where it should be done in post. Not by some devs working for a streaming platform. The use of grain or not is a creative decision made by the creators of the work. That's where it should remain
Why? If you're spending a significant chunk of your bits just transmitting data that could be effectively recreated on the client for free, isn't that wasteful? Sure, maybe the grains wouldn't be at the exact same coordinates, but it's not like the director purposefully placed each grain in the first place.
I recognize that the locally-produced grain doesn't look quite right at the moment, but travel down the hypothetical with me for a moment. If you could make this work, why wouldn't you?
--------
...and yes, I acknowledge that once the grain is being added client side, the next logical step would be "well, we might as well let viewers turn it off." But, once we've established that client-side grain makes sense, what are you going to do about people having preferences? Should we outlaw de-noising video filters too?
I agree that the default setting should always match what the film maker intended—let's not end up with a TV motion smoothing situation, please for the love of god—but if someone actively decides "I want to watch this without the grain for my own viewing experience"... okay? You do you.
...and I will further acknowledge that I would in fact be that person! I hate grain. I modded Cuphead to remove the grain and I can't buy the Switch version because I know it will have grain. I respect the artistic decision but I don't like it and I'm not hurting anyone.
> once we've established that client-side grain makes sense, what are you going to do about people having preferences?
I already normalize audio (modern US produced media is often atrocious), modify gamma and brightness, and probably some other stuff. Oh and it's not as though I'm viewing on a color calibrated monitor in the first place.
The purists can purchase lossless physical copies. The rest of us would benefit from such improvements.
> Why? If you're spending a significant chunk of your bits just transmitting data that could be effectively recreated on the client for free, isn't that wasteful? Sure, maybe the grains wouldn't be at the exact same coordinates, but it's not like the director purposefully placed each grain in the first place.
I'm sorry your tech isn't good enough to recreate the original. That does not mean you get to change the original because your tech isn't up to the task. Update your task to better handle the original. That's like saying an image of the Starry Night doesn't retain the details, so we're going to smear the original to fit the tech better. No. Go fix the tech. And no, this is not fixing the tech. It is a band-aid to cover the flaws in the tech.
> I'm sorry your tech isn't good enough to recreate the original. That does not mean you get to change the original because your tech isn't up to the task.
The market has spoken and it says that people want to watch movies even when they don't have access to a 35mm projector or a projector than can handle digital cinema packages, so nobody is seeing the original outside a theater.
Many viewers are bandwidth limited, so there's tradeoffs ... if this film grain stuff improves available picture quality at a given bandwidth, that's a win. IMHO, Netflix blogs about codec things seem to focus on bandwidth reduction, so I'm never sure if users with ample bandwidth end up getting less quality or not; that's a valid question to ask.
The differences are actual film grain vs some atrocious RGB noise artificially added by the streamer. How is that unclear? What else could we be talking about?
In theory though, I don't see any reason why client-side grain that looks identical to the real thing shouldn't be achievable, with massive bandwidth savings in the process.
It won't be, like, pixel-for-pixel identical, but that was why I said no director is placing individual grain specks anyway.
If the original is an actual production shot on film, the film grain is naturally part of it. Removing it never looks good. If it is something shot on a digital camera and had grain added in post, then you can go back to before the grain was added and then do it client side without degradation. But you can never have identical when it originated on film. That's like saying you can take someone's freckles away and put them back in post just rearranged and call it the same.
Let's be clear. The alternative isn't "higher bandwidth" it's "aggressive denoising during stream encode". If the studio is adding grain in post then describing that as a set of parameters will result in a higher quality experience for the vast majority of those viewing it in this day and age.
And yet here we are: DNR -> fancy grain -> DNR -> basic, approximated grain. Because noise doesn’t compress. And you get compression artifacts even in Blu-ray releases. What’s the point of applying fancy grain when what a lot viewers end up seeing is an ugly smudge?
Because it looks amazing in the editing studio. Just like the sound mix is incredible on the Atmos monitors in the sound mixing room, even though the home viewers have a soundbar at best and tiny stereo speakers in a flat panel typically. The dynamics and dialog channel will be fucked. But that’s user error.
Theatrical release qualifies for certain awards and shiny statues. That's their concern. If a streaming platform wants to give them enough cash to beat out projected box office earnings, then they'll take it if they don't have any grandiose visions of golden statues.
Currently, in order to deal with noisy masters, Netflix has to either:
1. Denoise the master, then add AV1 FGS metadata to tell players how to reconstruct the noise in the master (which is what the blog post is about) to get back the original image the director saw and approved
2. Do nothing (which is what they were doing), and let some of the noise get blurred or erased by the quantization step of the encoding process, or worse, burn shittons of coding bits trying to describe the exact noise in the frame, which hurts visual quality of the things people actually look at
All of these imply changes to the image that the director decided on to get around the underlying fact that deliberately adding noise to an image is, from a signal processing perspective, really stupid. But if we are going to do it, we can at least ensure it happens as far down the chain as possible so that Netflix's encoding doesn't destroy the noise. That's the idea you responded to: have the production company deliver a master with FGS metadata instead of baked-in film grain.
To the comments hating on grain: everything naturally has some amount of noise or grain - even the best digital sensors. Heck, even your eyes do. It's useful beyond just aesthetics. It tends to increase perceived sharpness and hides flaws like color banding and compression artifacts.
That's not to say that all noise and grain is good. It can be unavoidable, due to inferior technology, or a result of poor creative choices. It can even be distracting. But the alternative where everything undergoes denoising (which many of our cameras do by default now) is much worse in my opinion. To my eyes, the smoothing that happens with denoising often looks unrealistic and far more distracting.
My issue is that grain is good based on the creative decisions of the creators of the content. It is not something that a group of nerds compressing 1s and 0s should be making
I agree. However, let's look at it practically. Let's assume someone is watching content streamed on a low bandwidth connection. As a content creator, what version of the compressed content would you rather your audience experience:
a) Compressed original with significant artifacts from the codec trying to represent original grain
b) A denoised version with fewer compression artifacts, but looks "smoothed" by the denoising
c) A denoised version with synthesized grain that looks almost as good as the original, though the grain doesn't exactly match
I personally think the FGS needs better grain simulation (to look more realistic), but even in its current state, I think I'd probably go with choice C. I'm all for showing the closest thing to the author's intent. We just need to remember that compression artifacts are not the author's intent.
In an ideal world where we can deliver full, uncompressed video to everyone, then obviously - don't mess with it at all!
For content that we're concerning ourselves with this level of detail, I'd prefer the old iTunes method of prefetching the file and not stream it. For typical YT content, streaming is fine. For typical sitcom or other content, streaming is fine. For something like a feature that I'm so concerned about the details of grain, I have no problem downloading to play a local version. No, not a torrent.
We live in a world of resource constraints. The nerds compressing 1s and 0s are very much concerned about creative intent. They noticed that compression wrecks that creative intent and wanted to better communicate it in a world with bandwidth limitations.
Case in point: the HBO into animation. It uses static from the old analog days. It looks like shit even at 4k because random noise is impossible to compress without the exact strategy outlined here (stripping it out and rendering it later).
The "at scale" part is the real story here. Film Grain Synthesis has been available in the usual AV1 encoders for a while, but required some amount of manual tweaking to avoid creating problems, meaning it was only used in production when you had a very limited catalog, or for particularly important titles. They do not provide a lot of details here about how they overcame those problems, but it is nice to see it being deployed more broadly.
> This grain, formed from tiny particles during the film’s development, is more than just a visual effect. It plays a key role in storytelling by enhancing the film’s depth and contributing to its realism.
I never understood the “grain = realism” thing. my real eyes don’t have grain. I do appreciate the role of grain as an artistic tool though, so this is still cool tech
The article points out the masking effect of grain, which hides the fake-looking compression artifacts, and also the familiarity/nostalgia aspect. But I will offer an additional explanation.
Look around you: nearly all surfaces have some kind of fine texture and are not visually uniform. When this is recorded as video, the fine texture is diminished due to things like camera optics, limited resolution, and compression smoothing. Film grain supplies some of the high frequency visual stimulus that was lost.
Our eyes and brains like that high frequency stimulation and aren't choosy about whether the exact noise pattern from the original scene is reproduced. That's why the x265 video encoder (which doesn't have grain synthesis since it produces H.265 video) has a psy-rd parameter that basically says, "try to keep the compressed video as 'energetic' as the original, even if the energy isn't in the exact same spot", and even a psy-rdoq parameter that says, "prefer higher 'energy' in general". These parameters can be adjusted to make a compressed video look better without needing to store more data.
They definitely do at night when it's dark out. There's a kind of "sparkling" or "static" that comes in faint light.
Fortunately, our eyes have way better sensitivity than cameras. But the "realism" just comes from how it was captured using the technology of the day. It's no different from phonograph hiss or the way a CRT signal blurs. The idea is to be "real" to the technology that the filmmaker used, and the way they knew their movie would be seen.
It's the same way Van Gogh's brush strokes were real to his paintings. You wouldn't want his oil paintings sanded down to become flat. It's the reality of the original medium. And so even when we have a digital print of the film, we want to retain as much of the reality of the original as we can.
Your Van Gogh analogy makes sense for old movies. It doesn't quite explain why we're still adding grain to new movies, except for those few which are purposefully evoking older movies.
Even modern cameras have grain. If you need to integrate your scene with motion graphics, background replacement, or vfx, you'll need to remove grain on part of the image, edit it, add the original grain back where possible and synthesize new grain elsewhere.
Often it can also make sense to modify the grain for aesthetics. Denoising usually produces a less detailed result, but what you can do is denoise only the color channels, not the brightness channel. Brightness noise looks normal to us, while color noise tends to look very artificial. But by keeping the brightness noise, you avoid losing detail to the denoiser.
We don't use obvious grain, usually. That's generally precisely to evoke something about the past -- for flashbacks, a period look, etc. A sense of grittiness like 70s movies, etc.
On the other hand, a small amount of constant grain or noise is intentionally often introduced because otherwise images feel too static and end up looking almost fake. Similarly, dithering is intentionally added to audio, like mastering CD's or tracks. It helps prevent artifacts in video and in audio.
People are always trying to rationalize and justify aesthetic preferences. The depth and nuance of your understanding of a thing will change how you perceive variations of that thing, whether it's guitar tonewoods, style of music, types of paint, flavor of beer, or the grain in film. If you know a lot about a subject, you can tell a lot about the history of a thing, and that's going to change how you feel about a thing.
A child watching a Buster Keaton skit and gasping and giggling and enjoying it is going to have a different subjective aesthetic experience of the media than a film critic who knows exactly what type of film and camera were used, and what the meaning of all the different abstractions imply about the scene, and the fabric of Keaton's costume, and so on, and so forth.
Subjective aesthetic preferences are in the realm of cognition - we need a formal theory of intelligence mapped to the human brain, and all of these subjective phenomena collapse into individualized data processing and initial conditions.
There's something about film grain contrasted against clean cel animation which might make it easier for people to suspend disbelief. They are conditioned to think that absence of grain is associated with unreal animation, particular types of media, and CGI. Home video and news and so forth had grain and low quality, so grain gets correlated with "real". In my view, there's nothing deeper than that - we're the product of our times. In 40 years, media will have changed, and it may be that film grain is associated with surrealism, or edited out completely, as it's fundamentally noise.
The way I see it is that grain makes the film look more detailed than it really is, it can also hide compression artefacts and blurriness.
I don't know the psychovisuals behind that. Maybe it adds some high frequencies that compression often washes out, or maybe acts like some kind of dithering.
As for your eyes, I am pretty sure that they have grain, that's how quantum physics work, you just don't perceive it because your brain filters it out. But again, I don't know how it interacts with film grain.
Video signal without the noise or grain is annoying to watch as it makes everything in the ”out of focus” zone look smooth blurry. Your eyes want to focus yet it is an illusion of depth without an actual depth. Noise texture emphasizes that this is just a 2D plane after all so your eyes can rest and the viewer doesn’t feel like they need glasses. This is just my theory of it based on observation. No research behind it.
This reminds me of modern windows having fake panes. They’re just strips that are applied to give the impressions that there are multiple smaller panes because people are used to that and it feels “correct”.
I have to imagine past glassmakers would have been absolutely enthralled by the ability we now have to make uniform, large sheets of glass, but here we are emulating the compromises they had to make because we are used to how it looks.
> They’re just strips that are applied to give the impressions that there are multiple smaller panes because people are used to that and it feels “correct”.
It is more than just 'feeling correct': windows and their various (sub-)elements that make them up (can) change the architectural proportions and how the building is perceived as a whole:
It is similar with columns: they're not just 'tall-and-narrow', but rather have certain proportions and shapes depending on the style and aesthetic/feeling one wishes to convey:
I strongly doubt that multiple smaller panes would have ever become a common style if we could have always made large glass panes. This is a perfect example of people becoming very used to a style forced by a technological limitation that is emulated even after the limitation doesn't exist.
It used to be a bigger deal (when digital cameras started being used) since people felt like digital video didn't look real/as good - movies shot on film were generally better looking (as crews were used shooting with it and digital video wasn't as sophisticated as today) and HAD grain.
It might be that there is a large part of the population that still has that association.
Cinephiles are also more likely to watch older (i.e. with grain) movies that ARE well shot and beautiful (which is why they are classics and watched by cinephiles) and not see bad film movies, only the cream of the crop, while being exposed to the whole gamut of quality when watching todays movies shot digitally. Would reinforce that grain = good while not being necessarily the case - and their opinion might be heard more than gen pop.
At any rate, it can be a neat tool to lower sharpness!
Grain = realism because real captured grain isn't total random noise. It's authentic noisy data. It's part of captured scene. It adds subtle tiny but real detail to the scene. Unless I am corrected here and that real grain is also total random noise.
film grain adds realism in the same way that high frame rate films look wrong or vinyl sounds "warmer" or tube guitar amps sound "better" - It is what we are used to.
Film grain can create stochastic resonance with the underlying ground truth. In practice, this can improve the perceived image quality over having none.
23.976 fps has been put on a pedestal as the "correct" look. Just look at the reaction to The Hobbit. However it does provide some objective advantages. 60 fps requires more lighting. Adding more lights means more electric setup and heat for actors in heavy makeup and costume. In post production that's more frames to edit.
23.976fps is only "correct" when telecining to a 59.94Hz format. 24fps round is the "correct" format in terms of actual filmmaking. I don't know of many films specifically shot at 23.976fps.
grain and 24fps and widescreen trigger certain contextual emotions around the movie-watching experience. remove them and your brain contextualizes the video very differently.
this is likely the result of ~100 years of film-based filmmaking and projection. hell, we still call it filmmaking.
Yes, it is only the result of familiarity. We could gradually increase the frame rate of movies made in a year by 1 fps per year and then no one would even notice after 24 years every new movie would be 48fps.
When you talk on a cellphone the codec AMR-WB nominally captures 50 Hz - 7000 Hz. However that's only on the optional highest bitrate 23.85 Kbps. The most common bitrate 12.65 Kbps only goes up to 6400 Hz and synthesizes 6400 - 7000 Hz from lower frequencies and noise as it sounds better than not having the noise!
> Picture this: you’re watching a classic film, and the subtle dance of film grain adds a layer of authenticity and nostalgia to every scene
It just adds visual noise that obscures details of the authentic scene, and nothing prevents nostalgia from being tied to many of the more prominent visual cues like old actors or your own old memories from when you watched it first...
> contributing to [film's] realism
But there is no grain in reality, so it does the opposite
Otherwise I'm glad AV1 marches along and instead of wasting bitrate encoding visual garbage has an algorithmic replacement mechanism- which also means you could turn it off easier.
Documentaries might care about accurately representing reality. For every other cinematic genre, "authenticity" is not an inherent goal. If film grain is part of the director's vision, then that's just as valid as a choice to have dramatic non-diagetic music playing in the background of a scene (which is highly inauthentic, but also highly effective at evoking emotion, which is the point of art).
I don't see how that is supposed to be relevant to the article's claim that grain adds authenticity and realism. Someone claiming that non-diegetic music adds to authenticity would also be weird.
I think what the author meant is that film grain is part of the authentic visual aesthetic of cinema produced using celluloid materials. A de-noised image is therefore inauthentic as that is contrary to the appearance of the original projected image or source/master elements.
> But there is no grain in reality, so it does the opposite
Well ackchually -- illumination is inherently random, so all time-bounded captures of a scene (including what your eyes do) are subject to shot noise: https://en.wikipedia.org/wiki/Shot_noise
You misackchuallied grain with any noise, for example, the bottom right square of the image at the wiki page is not grainy despite being technically shot-noisy
> But there is no grain in reality, so it does the opposite
The author did not do a particularly good job of selling the merits of film grain. I am not sure what he meant by that but I will say that it does increase the perceived sharpness of the image as well as perceived detail, even if it is just an illusion. The cinematographer Steve Yedlin describes it as giving the viewer’s eyes something to “latch” onto.
I remember years ago when digital projection was just becoming the norm. I saw a movie (I don't remember which one) and during the opening scene I remember thinking "why are the credits jittering around? Oh. This is actually being projected on film!"
I'm in my early 50s so I remember film quite well. Just like vinyl or cassettes, I ain't going back and unless it is an artistic choice I don't want films to emulate what I consider to be an inferior technology.
It's a bit frustrating that the footage is first shot, then denoised in post, then renoised in post, then denoised in encoding and then renoised at decoding.
Only if you are aware of it, which 99.9% of people consuming video content are not. It’s simply an unimportant implementation detail (from a viewer’s perspective who doesn’t really care about bitrate-as-cost).
Very interesting look into how AV1’s Film Grain Synthesis (FGS) is being scaled to production at Netflix. The part that stood out most to me was the smart decoupling of film grain from the core encoding process. Denoising before compression and then reintroducing grain during playback is such a clever inversion of the usual "preserve everything" strategy that often bloats bitrates.
Ok, very cool. But I want Netflix to explain it related to Seinfeld, which at 10-12 feet looks fine, but up close looks insane. Blocky MJPEG + grain filter??
It's not like we're on Pentium II processors anymore -- I can filter just about anything with ShaderGlass [0] on a shitty computer (and some of the CRT shaders like crt-hyllian-curvature are brilliant, especially on old shows like NewsRadio that only exist on DVD) .. and I'm shocked that Netflix doesn't just have this built into their Apple TV app or whatever. I'm shocked PLEX doesn't have it! (that I know of)
I made a comment on a different post about imagining a world where local AI/LLM/whatever does some favorable processing for you, by you, on your device, of web content, to enhance your experience. I really believe media (streamers all the way down to open source devs) need to begin to incorporate whatever's out there that reduces friction and increases joy. It's all out there already! The heavy lifting has been done! Just make Family Matters look like how it looked when I was locking in on a Friday night for TGIF LOL
I think there is a bit more to it. For example when developing a game when CRTs were popular they were using CRTs to view their game and making decisions based on what they saw on the CRT. If you display the same game with perfect square pixels it looks different. If the developers were viewing square pixels when developing the game they would make different decisions.
I don't think this conflicts with what I'm saying; I've seen what you talk about -- and yet in modern days people will emulate the "square pixel bad style" regardless.
Agree. Purely opining, but I assume that it's because of the emotional connection that artistic media has on people, despite the flaws.
People remember the emotions the artwork engendered, and thus the whole work is associated with the feelings, flaws and all. If the work is particularly widely known, the flaws can become a stand-in for the work itself.
I see this in video games - I'm fond of the NES-era "flaws" and limitations (palette limits, sprite limits, sound channel limits), but less connected to the Atari 2600 or SNES/PS1/NDS/etc flaws. Shovel Knight is charming; A Short Hike, while great, doesn't resonate on a style level.
Even if you ignore all of the emotional attachment aspects, I think you would expect this to happen simply because people who are good at their craft will learn how to work with the shortcomings of their tools. If your cameras unavoidably add grain to everything you shoot you'll find ways to make that look good, and some of the things you come up with might not work without grain.
I've noticed this in a few things the last few years. The top and bottom of the shot are out of focus and it is super distracting to me. Maybe it is meant to draw the eye to the middle of the frame.
Yup, notice that as well. A blurry vignette. Needle sharp in the center, but objects that are the same distance are blurry at the edges of the frame. At first I thought they use vintage lenses, but it's probably done in post.
I sometimes do something similar on photos, where I darken the edges a bit.
Or Snyder's terrible zombie movie Army of the Dead where he uses lenses with very shallow depth of field that makes almost everything look out of focus. It is very annoying.
I like this. Not really because I feel modern media should have added grain, but because for older media this is a method to get closer to the original but at much lower bitrates without excessive smoothing. What’s not to like?
Also, the author had me at God of Gamblers 2. So good. I will take him up on his recommendation to rewatch.
The original grain that is captured is actually a detail and not total random noise. I believe you can make up the vague sense of original scene if you could somehow extract that grain/noise alone.
It's like reducing an image to tiny dots with dithering (reminds of Atinkson dithering). Those grains are not a noise, they are a detail, actual data. That's why real grain looks good IMO.
This is just a fact of lossy compression: you want to throw away information that contributes less to the perception of the video so that you can describe it with fewer bits of information.
There are two possible advantages for this kind of grain synthesis. For Netflix, they could produce the same perceived quality at lower bitrates, which reduces costs per view and allows customers with marginally slow connections to get a higher quality version. For a consumer, the advantage would be getting more non-grain detail for a fixed bitrate.
You are right that if you subtract the dentists frame from the raw one, showing only the estimated noise, you would get some impression of the scene. I think there’s two reasons for this. Firstly, the places where the denoiser produced a blurry line that should be sharp may show up as faint lines. I don’t think this is ‘hidden information’ so much as it is information lost to lossy compression. In the same way, if you look at the difference between a raw image and one with compression, you may see some emphasized edges due to compression artefacts. Secondly, the less exposed regions of the film will have more noise so noisiness becomes a proxy for darkness, allowing some reproduction of the scene. I would expect this detail to be lost after adjusting for the piecewise linear function for grain intensity at different brightness levels.
Perhaps a third thing is the level of noise in the blacks and the ‘grain size’ or other statistical properties tell you about the kind of film being used, but I think those things are captured in the film grain simulation model.
Possibly there are some other artefacts like evidence of special effects, post processing, etc.
If you extract the grain (e.g. by subtracting a blurred version of the image), the result contains mostly noise, not meaningful scene information outside of some variation according to image brightness. Film grain is random, so the extracted "grain layer" doesn’t encode original image detail but film grain itself encodes relative lightness in its _density_.
Between 24fps and film grain people are way too attached to fundamentally inferior technology. With how strongly people resist frame rates faster than 24 I'm surprised people accepted color and sound, which were much bigger changes.
Everything is fake now. I want a technology which works with a raw film scans, not even compressing them to JPEG, which is a 1st step in loosing the details BTW. Motion detection, key frames, delta frames - fine. But with a lossless video. On a Blu Ray off course, i don't care much about streaming.
A 4K/24p film encoded with Apple ProRes 4444 XQ (not even ProRes RAW) is 716 GB per hour, so you would need to swap a total of 30 Blu-ray discs once every 4 minutes in order to watch a 2 hour movie.
I wonder how much you'd get with such a technology... truly uncompressed 4K video you're talking about something on the order of a few terabytes for a 90-minute movie, so way way bigger than the biggest 4K Blu-ray discs. Lossless compression would get you under that number, but far enough to matter?
Film grain simulation can be an effect, but that's not the primary application being described here. As @ricardobeat succinctly put it elsewhere in this post, this process usually supports "a reproduction of the original grain pattern, compressed separately from the underlying image content. The result is closer to the original picture than the denoised/compressed version".
> If kids grew up without seeing film grain, how would they feel about it?
Film isn't quite dead yet, with ~7% of features in 2025 still being shot on film and maybe 1% more taking the digital→film→digital (a.k.a. "analog intermediate") route. Unless their media diet is all super-recent and low-quality, even today's kids will have most likely seen many movies shot on film with visible grain. Popular candidates would include the Harry Potter franchise, most episodes of the Star Wars franchise, early Marvel movies, many animated Disney and Studio Ghibli features, etc.
happier and happier about leaving behind digital media to return to physical. to me this is literally slop. i want the uncompressed file stop selling me stepped on product
There's an influx of high-profile directors/films right now and in pipeline filmed for IMAX (F1: The Movie I think, Mission Impossible, etc) and Christopher Nolan's Odyssey coming next year shot entirely on IMAX film with newly developed smaller/quieter cameras made to accomplish it.
I've read that a 15-perf 65mm IMAX negative shot with slower film stocks is "virtually grainless", even when viewed on a 70ft screen. Grain is apparently noticeable in IMAX films when large/fast stocks are used and pushed toward their limits, and (of course) when smaller-format film stocks have been blown up.
This is not fake grain - it's a reproduction of the original grain pattern, compressed separately from the underlying image content. The result is closer to the original picture than the denoised / compressed version.
There's no need for low-quality comments cutting and pasting from your favourite LLM. You can literally just type it into your browser's search bar and learn what it is, or spend a minute reading the wiki page (https://en.wikipedia.org/wiki/AV1).
This fails to acknowledge that synthesized noise can lack the detail and information in the original noise.
When you watch a high-quality encode that includes the actual noise, there is a startling increase in resolution from seeing a still to seeing the video. The noise is effectively dancing over a signal, and at 24 fps the signal is still perfectly clear behind it.
Whereas if you lossily encode a still that discards the noise and then adds back artificial noise to match the original "aesthetically", the original detail is non-recoverable if this is done frame-by-frame. Watching at 24 fps produces a fundamentally blurrier viewing experience. And it's not subtle -- on old noisy movies the difference in detail can be 2x.
Now, if h.265 or AV1 is actually building its "noise-removed" frames by always taking into account several preceding and following frames while accounting for movement, it could in theory discover the signal of the full detail across time and encode that, and there wouldn't be any loss in detail. But I don't think it does? I'd love to know if I'm mistaken.
But basically, the point is: comparing noise removal and synthesis can't be done using still images. You have to see an actual video comparison side-by-side to determine if detail is being thrown away or preserved. Noise isn't just noise -- noise is detail too.
Grain is independent frame-to-frame. It doesn't move with the objects in the scene (unless the video's already been encoded strangely). So long as the synthesized noise doesn't have an obvious temporal pattern, comparing stills should be fine.
Regarding aesthetics, I don't think AV1 synthesized grain takes into account the size of the grains in the source video, so chunky grain from an old film source, with its big silver halide crystals, will appear as fine grain in the synthesis, which looks wrong (this might be mitigated by a good film denoiser). It also doesn't model film's separate color components properly, but supposedly that doesn't matter because Netflix's video sources are often chroma subsampled to begin with: https://norkin.org/pdf/DCC_2018_AV1_film_grain.pdf
Disclaimer: I just read about this stuff casually so I could be wrong.
> Grain is independent frame-to-frame. It doesn't move with the objects in the scene (unless the video's already been encoded strangely)
That might seem like a reasonable assumption, but in practice it’s not really the case. Due to nonlinear response curves, adding noise to a bright part of an image has far less effect than a darker part. If the image is completely blown out the grain may not be discernible at all. So practically speaking, grain does travel with objects in a scene.
This means detail is indeed encoded in grain to an extent. If you algorithmically denoise an image and then subtract the result from the original to get only the grain, you can easily see “ghost” patterns in the grain that reflect the original image. This represents lost image data that cannot be recovered by adding synthetic grain.
> If you algorithmically denoise an image and then subtract the result from the original to get only the grain, you can easily see “ghost” patterns in the grain that reflect the original image. This represents lost image data that cannot be recovered by adding synthetic grain.
The synthesized grain is dependent on the brightness. If you were to just replace the frames with the synthesized grain described in the OP post instead of adding it, you would see something very similar.
It sounds like the "scaling function" mentioned in the article may be intended to account for the nonlinear interaction of the noise.
> So long as the synthesized noise doesn't have an obvious temporal pattern, comparing stills should be fine.
The problem is that the initial noise-removal and compression passes still removed detail (that is more visible in motion than in stills) that you aren't adding back.
If you do noise-removal well you don't have to lose detail over time.
But it's much harder to do streaming-level video compression on a noisy source without losing that detail.
The grain they're adding somewhat distracts from the compression blurriness but doesn't bring back the detail.
>The grain they're adding somewhat distracts from the compression blurriness but doesn't bring back the detail.
Instead of wasting bits trying to compress noise, they can remove noise first, then compress, then add noise back. So now there aren't wasted bits compressing noise, and those bits can be used to compress detail instead of noise. So if you compare FGS compression vs non-FGS compression at the same bitrate, the FGS compression did add some detail back.
The AR coefficients described in the paper are what allow basic modeling of the scale of the noise.
> In this case, L = 0 corresponds to the case of modeling Gaussian noise whereas higher values of L may correspond to film grain with larger size of grains.
I think you've missed the point here: the noise in the originals acts as dithering, and increases the resolution of the original video. This is similar to the noise introduced intentionally in astronomy[1] and in signal processing[2].
Smoothing the noise out doesn't make use of that additional resolution, unless the smoothing happens over the time axis as well.
Perfectly replicating the noise doesn't help in this situation.
[1]: https://telescope.live/blog/improve-image-quality-dithering [2] https://electronics.stackexchange.com/questions/69748/using-...
Your first link doesn't seem to be about introducing noise, but removing it by averaging the value of multiple captures. The second is to mask quantizer-correlated noise in audio, which I'd compare to spatial masking of banding artifacts in video.
Noise is reduced to make the frame more compressible. This reduces the resolution of the original only because it inevitably removes some of the signal that can't be differentiated from noise. But even after noise reduction, successive frames of a still scene retain some frame-to-frame variance, unless the noise removal is too aggressive. When you play back that sequence of noise-reduced frames you still get a temporal dithering effect.
Here's[1] a more concrete source, which summarizes dithering in analog to digital converters as follows:
With no dither, each analog input voltage is assigned one and only one code. Thus, there is no difference in the output for voltages located on the same ‘‘step’’ of the ADC’s ‘‘staircase’’ transfer curve. With dither, each analog input voltage is assigned a probability distribution for being in one of several digital codes. Now, different voltages with-in the same ‘‘step’’ of the original ADC transfer function are assigned different probability distributions. Thus, one can see how the resolution of an ADC can be improved to below an LSB.
In actual film, I presume the random inconsistencies of the individual silver halide grains is the noise source, and when watching such a film, I presume the eyes are doing the averaging through persistence of vision[2].
In either case, a key point is that you can't bring back any details by adding noise after the fact.
[1]: https://www.ti.com/lit/an/snoa232/snoa232.pdf section 3.0 - Dither
[2]: https://en.wikipedia.org/wiki/Persistence_of_vision
One thing worth noting is that this extra detail from dithering can be recovered when denoising by storing the image to higher precision. This is a lot of the reason 10 bit AV1 is so popular. It turns out that by adding extra bits of image, you end up with an image that is easier to compress accurately since the encoder has lower error from quantization.
People often assume noise is normal and IID but it usually isn't. It's s fine approximation but isn't the same thing, which is what the parent is discussing.
Here's an example that might help you intuit why this is true.
Let's suppose you have a digital camera and walk towards a radiation source and then away. Each radioactive particle that hits the CCD causes it to over saturate, creating visible noise in the image. The noise it introduces is random (Poisson) but your movement isn't.
Now think about how noise is introduced. There's a lot of ways actually, but I'm sure this thought exercise will reveal to you how some cause noise across frames to be dependent. Maybe as a first thought, think about from sitting on a shelf degrading.
I think this is geared towards film grain noise, which is independent from movement?
It's the same thing. Yes, not related to the movement of the camera, but I thought that would be easier to build your intuition about silver particles being deposited onto film. You make in batches, right?
The point is that just because things are random doesn't mean there aren't biases.
To get much more accurate, it helps to understand what randomness actually is. It is a measurement of uncertainty. A measurement of the unknown. This is even true for quantum processes that are truly random. That means we can't know. But just because we can't know doesn't mean it's completely unknown, right? We have different types of distributions and different parameters in those distributions. That's what we're trying to build intuition about
> Grain is independent frame-to-frame. It doesn't move with the objects in the scene (unless the video's already been encoded strangely). So long as the synthesized noise doesn't have an obvious temporal pattern, comparing stills should be fine.
Sorry if I wasn't clear -- I was referring to the underlying objects moving. The codec is trying to capture those details, the same way our eye does.
But regardless of that, you absolutely cannot compare stills. Stills do not allow you to compare against the detail that is only visible over a number of frames.
This is a really good point.
To illustrate the temporal aspect: consider a traditional film projector. Between every frame, we actually see complete darkness for a short time. We could call that darkness "noise", and if we were to linger on that moment, we'd see nothing of the original signal. But since our visual systems tend to temporally average things out to a degree, we barely even notice that flicker (https://en.wikipedia.org/wiki/Flicker_fusion_threshold). I suspect noise and grain are perceived in a similar way, where they become less pronounced compared to the stable parts of the signal/image.
Astrophotographers stack noisy images to obtain images with higher SNR. I think our brains do a bit of that too, and it doesn't mean we're hallucinating detail that isn't there; the recorded noise - over time - returns to the mean, and that mean represents a clearer representation of the actual signal (though not entirely, due to systematic/non-random noise, but that's often less significant).
Denoising algorithms that operate on individual frames don't have that context, so they will lose detail (or will try to compensate by guessing). AV1 doesn't specify a specific algorithm to use, so I suppose in theory, a smart algorithm could use the temporal context to preserve some additional detail.
The noise does not contain a signal, does not dance over it, and is not detail. It is purely random fluctuations that are added to a signal.
If you have a few static frames and average them, you improve SNR by retaining the unchanged signal and having the purely random noise cancel itself out. Retaining noise itself is not useful.
I suspect the effect you might be seeing is either just an aesthetic preference for the original grain behavior, or that you are comparing low bandwidth content with heavy compression artifacts like smoothing/low pass filtering (not storing fine detail saves significant bandwidth) to high bandwidth versions that maintain full detail, entirely unrelated to the grain overlaid on top.
Denoising generally removes signal too. Removing noise and reconstituting similar noise to maintain the apparent qualities of an input can help compression, but you are also cutting out true details (typically fine detail).
The effect GP is pointing out is how denoisers damage detail, which is true. This detail can persist over multiple frames, which is why many denoisers include a temporal comparison component to mitigate the damage, but you still lose detail.
Definitely, but that's a result of compressing (with any algorithm) past a bitrate that would allow detail to be retained with high frequency details including noise being the first to go, not the result of noise having signal as was stated and more importantly unrelated to film grain synthesis which is about adding something new on the output side.
You could do an ai or compressed sensing upscale first, with multiple frames helping, have that be the base video sent, and then regranularize that.
There are definite philosophical questions over the merits of adding noise, but the problem with their example here is their denoising process appears to excessively blur everything, so both it and the synthesized grain image look noticeably less sharp than the source. The grain itself also looks too much like basic noise, and not really grain like.
> both it and the synthesized grain image look noticeably less sharp than the source
That's true, but at a given bitrate (until you get to very high bitrates), the compressed original will usually look worse and less sharp because so many bits are spent trying to encode the original grain. As a result, that original grain tends to get "smeared" over larger areas, making it look muddy. You lose sharpness in areas of the actual scene because it's trying (and often failing) to encode sharp grains.
Film Grain Synthesis makes sense for streaming where bandwidth is limited, but I'll agree that in the examples, the synthesized grain doesn't look very grain-like. And, depending on the amount and method of denoising, it can definitely blur details from the scene.
It seems like a shame that they didn’t include a screenshot of the original (with natural grain), after suffering from low-bitrate streaming. Aka the actual baseline.
I can see why they want to compare against the actual local copy of the video with the natural grain. But that’s the perfect copy that they can’t actually hope to match.
> It seems like a shame that they didn’t include a screenshot of the original (with natural grain), after suffering from low-bitrate streaming.
Isn't that the image captioned "Regular AV1 (without FGS) @ 8274 kbps"?
I think I misread the figures.
But still, they have:
> A source video frame from They Cloned Tyrone
> Regular AV1 (without FGS) @ 8274 kbps
> AV1 with FGS @ 2804 kbps
Just to emphasize the problem, would it be nice to see:
Regular AV1 (without FGS) @ 2804 kbps
It should look really bad, right? Which would emphasize their results.
But why do they need to emphasize it even more than the examples they gave? The "AV1 with FGS @ 2804 kbps" already looks as good or better than the "AV1 (without FGS) @ 8274 kbps", so it'll definitely look better than AV1 without FGS at an even lower bandwidth.
I think the distinction here is, they provide the "regular" stream, and the FGS stream noting that it's much smaller yet looks similar. What they don't have is a lower-bandwidth "regular" one, what 2000-or-so kbps looks like without FGS.
They did.
AV1 has tunable FGS levels, and to my eye they went very slightly higher than they should have (though there are tradeoffs; at some bitrates the blurring+renoising is so much better than the other visual artifacts you will otherwise get, that you do want it that high).
A few things to note:
- still-frames are also a mediocre way to evaluate video quality.
- a theoretically perfect[1] noise-removal filter will always look less detailed than the original source, since your brain/eye system will invent more detail for a noisy image than for a blurry image.
1: By which I mean a filter that preserves 100% of the non-grain detail present, not one that magically recovers detail lost due to noise.
Since the beginning of film editors have added tricks in post.
I would love for them to provide an option to view it with film simulation vs without.
One of my favorite movies of all time, The Holdovers, did film simulation extremely well. It's set in the '70s so it attempts to look like a movie of that era.
It looked great to me, but if you're an actual film nerd you're going to notice a lot of things aren't exactly accurate.
Maybe in the near future we'll see Netflix being able to process some post effects on the client. So if you're color blind, you get a mode for that. If you don't want fake grain you can turn it off.
mpv can already do this: `--vf=format:film-grain=no` turns off grain synthesis. There are also people making custom shaders for things like emulating CRT monitors (originally for retro gaming, but I see there are also mpv versions).
Holdovers was pretty great. Sorta like an 80’s school comedy but with the perspective of the adults included, making it a totally different type of movie.
A film that explores this is Antonioni’s Blowup: https://en.wikipedia.org/wiki/Blowup
Agree, as someone who has spent way too much time studying the way motion picture film looks up close, this isn’t very realistic looking. It’s really just a form of dithering.
Video codecs use a lot of tricks based on human perception, perhaps it's much closer to the real thing when in motion vs a still image?
Forget the film grain, give us film lint and film hair!
Get the retro gaming nerds on this, they'll make a film grain shader that's indistinguishable from reality in no time flat.
> and not really grain like
that's an understatement. it just looks like RGB noise effect was added. film grain does not look like RGB noise. to me, film grain is only one part of what gave film the film look. the way the highlights bloom rather than clip. it also was more natural/organic/some descriptive other than the ultrasharp of modern digital acquisition. using some SoftFX or Black Mist type filters help, but it's just not the same as it is a digital vs analog type of acquisition. all of these attempts at making something look like it's not just keep falling down in the same ways. but hey, there's a cool tech blog about it this time. film grain filters have been around for a long time, yet people just don't care for them. even in Blu-ray time frame, there was attempts at removing the grain in the encode and applying it in playback. Netflix isn't coming up with anything new, and apparently nothing exciting either based on the results.
Agree, I really prefer luminance noise when simulating film grain rather than treating RGB so independently - the RGB here looks wrong to me. If you are thinking about film, you wouldn't model it as pure photon/detector noise, because film has some crystalline structure and usually there is some correlation between the channels. (Some of the best grain is on B&W films though, which have wonderful structure and statistics.)
I only have their single sample in the blog post to go by, but the original source frame also looks just like RGB noise? And the recreation looks really close if I swap between them. Looks fine to me.
So as long they're analyzing the style of grain in the source properly, which the technical part of the post mentions they do...
These days, when we see noise/grain in an end product it has likely been added in post-production. So, ideally, studios would provide distributors with a noiseless source plus grain synthesis parameters. Bonus: many viewers would welcome an option to turn it off.
> provide distributors with a noiseless source plus grain synthesis parameters.
What parameters would that be? Make it look like Eastman Ektachrome High-Speed Daylight Film 7251 400D? For years, people have taken film negative onto telecines and created content of grain to be used as overlays. For years, colorists have come up with ways of simulating the color of specific film stocks by using reference film with test patterns that's been made available.
If a director/producer wants film grain added to their digital content, that's where it should be done in post. Not by some devs working for a streaming platform. The use of grain or not is a creative decision made by the creators of the work. That's where it should remain
Netflix has their own in-house studio, right? The encoding and lossy compression is going to happen anyway. It seems like an easy win, for their directors to provide a description of the grain they want, so it can be replicated on the user side.
what does having an in-house studio have to do with it? they stream more content than just their own, and so they would not have creative license to alter content. they would only have some type of distribution license to stream the content as provided
Because they previously did not do commercials their original TV shows were not written with pauses every few minutes. They have approved cameras. They spend heavily on movie star salaries and skimp on set production.
Again, I ask, what does this have to do with anything regarding TFA? Anything in-house studio produces will not be shot on film.
> If a director/producer wants film grain added to their digital content, that's where it should be done in post. Not by some devs working for a streaming platform. The use of grain or not is a creative decision made by the creators of the work. That's where it should remain
Why? If you're spending a significant chunk of your bits just transmitting data that could be effectively recreated on the client for free, isn't that wasteful? Sure, maybe the grains wouldn't be at the exact same coordinates, but it's not like the director purposefully placed each grain in the first place.
I recognize that the locally-produced grain doesn't look quite right at the moment, but travel down the hypothetical with me for a moment. If you could make this work, why wouldn't you?
--------
...and yes, I acknowledge that once the grain is being added client side, the next logical step would be "well, we might as well let viewers turn it off." But, once we've established that client-side grain makes sense, what are you going to do about people having preferences? Should we outlaw de-noising video filters too?
I agree that the default setting should always match what the film maker intended—let's not end up with a TV motion smoothing situation, please for the love of god—but if someone actively decides "I want to watch this without the grain for my own viewing experience"... okay? You do you.
...and I will further acknowledge that I would in fact be that person! I hate grain. I modded Cuphead to remove the grain and I can't buy the Switch version because I know it will have grain. I respect the artistic decision but I don't like it and I'm not hurting anyone.
> once we've established that client-side grain makes sense, what are you going to do about people having preferences?
I already normalize audio (modern US produced media is often atrocious), modify gamma and brightness, and probably some other stuff. Oh and it's not as though I'm viewing on a color calibrated monitor in the first place.
The purists can purchase lossless physical copies. The rest of us would benefit from such improvements.
> Why? If you're spending a significant chunk of your bits just transmitting data that could be effectively recreated on the client for free, isn't that wasteful? Sure, maybe the grains wouldn't be at the exact same coordinates, but it's not like the director purposefully placed each grain in the first place.
I'm sorry your tech isn't good enough to recreate the original. That does not mean you get to change the original because your tech isn't up to the task. Update your task to better handle the original. That's like saying an image of the Starry Night doesn't retain the details, so we're going to smear the original to fit the tech better. No. Go fix the tech. And no, this is not fixing the tech. It is a band-aid to cover the flaws in the tech.
> I'm sorry your tech isn't good enough to recreate the original. That does not mean you get to change the original because your tech isn't up to the task.
The market has spoken and it says that people want to watch movies even when they don't have access to a 35mm projector or a projector than can handle digital cinema packages, so nobody is seeing the original outside a theater.
Many viewers are bandwidth limited, so there's tradeoffs ... if this film grain stuff improves available picture quality at a given bandwidth, that's a win. IMHO, Netflix blogs about codec things seem to focus on bandwidth reduction, so I'm never sure if users with ample bandwidth end up getting less quality or not; that's a valid question to ask.
Because the specks of grain aren't at the exact same coordinates? What differences are we talking about here exactly?
The differences are actual film grain vs some atrocious RGB noise artificially added by the streamer. How is that unclear? What else could we be talking about?
Right, the current implementation is bad.
In theory though, I don't see any reason why client-side grain that looks identical to the real thing shouldn't be achievable, with massive bandwidth savings in the process.
It won't be, like, pixel-for-pixel identical, but that was why I said no director is placing individual grain specks anyway.
If the original is an actual production shot on film, the film grain is naturally part of it. Removing it never looks good. If it is something shot on a digital camera and had grain added in post, then you can go back to before the grain was added and then do it client side without degradation. But you can never have identical when it originated on film. That's like saying you can take someone's freckles away and put them back in post just rearranged and call it the same.
> with massive bandwidth savings in the process
Let's be clear. The alternative isn't "higher bandwidth" it's "aggressive denoising during stream encode". If the studio is adding grain in post then describing that as a set of parameters will result in a higher quality experience for the vast majority of those viewing it in this day and age.
And yet here we are: DNR -> fancy grain -> DNR -> basic, approximated grain. Because noise doesn’t compress. And you get compression artifacts even in Blu-ray releases. What’s the point of applying fancy grain when what a lot viewers end up seeing is an ugly smudge?
Because it looks amazing in the editing studio. Just like the sound mix is incredible on the Atmos monitors in the sound mixing room, even though the home viewers have a soundbar at best and tiny stereo speakers in a flat panel typically. The dynamics and dialog channel will be fucked. But that’s user error.
Movies are best enjoyed in the theater.
This is exactly why theatrical releases are so important to movie producers, isn't it?
Theatrical release qualifies for certain awards and shiny statues. That's their concern. If a streaming platform wants to give them enough cash to beat out projected box office earnings, then they'll take it if they don't have any grandiose visions of golden statues.
I think at some point studios will give de-grained versions to Netflix directly.
The grain is there to hide the ugly smudge. that's the question they rather you didn't ask
Currently, in order to deal with noisy masters, Netflix has to either:
1. Denoise the master, then add AV1 FGS metadata to tell players how to reconstruct the noise in the master (which is what the blog post is about) to get back the original image the director saw and approved
2. Do nothing (which is what they were doing), and let some of the noise get blurred or erased by the quantization step of the encoding process, or worse, burn shittons of coding bits trying to describe the exact noise in the frame, which hurts visual quality of the things people actually look at
All of these imply changes to the image that the director decided on to get around the underlying fact that deliberately adding noise to an image is, from a signal processing perspective, really stupid. But if we are going to do it, we can at least ensure it happens as far down the chain as possible so that Netflix's encoding doesn't destroy the noise. That's the idea you responded to: have the production company deliver a master with FGS metadata instead of baked-in film grain.
I'll keep the film grain, I just want to be able to turn off laugh tracks.
To the comments hating on grain: everything naturally has some amount of noise or grain - even the best digital sensors. Heck, even your eyes do. It's useful beyond just aesthetics. It tends to increase perceived sharpness and hides flaws like color banding and compression artifacts.
That's not to say that all noise and grain is good. It can be unavoidable, due to inferior technology, or a result of poor creative choices. It can even be distracting. But the alternative where everything undergoes denoising (which many of our cameras do by default now) is much worse in my opinion. To my eyes, the smoothing that happens with denoising often looks unrealistic and far more distracting.
My issue is that grain is good based on the creative decisions of the creators of the content. It is not something that a group of nerds compressing 1s and 0s should be making
I agree. However, let's look at it practically. Let's assume someone is watching content streamed on a low bandwidth connection. As a content creator, what version of the compressed content would you rather your audience experience:
a) Compressed original with significant artifacts from the codec trying to represent original grain
b) A denoised version with fewer compression artifacts, but looks "smoothed" by the denoising
c) A denoised version with synthesized grain that looks almost as good as the original, though the grain doesn't exactly match
I personally think the FGS needs better grain simulation (to look more realistic), but even in its current state, I think I'd probably go with choice C. I'm all for showing the closest thing to the author's intent. We just need to remember that compression artifacts are not the author's intent.
In an ideal world where we can deliver full, uncompressed video to everyone, then obviously - don't mess with it at all!
For content that we're concerning ourselves with this level of detail, I'd prefer the old iTunes method of prefetching the file and not stream it. For typical YT content, streaming is fine. For typical sitcom or other content, streaming is fine. For something like a feature that I'm so concerned about the details of grain, I have no problem downloading to play a local version. No, not a torrent.
We live in a world of resource constraints. The nerds compressing 1s and 0s are very much concerned about creative intent. They noticed that compression wrecks that creative intent and wanted to better communicate it in a world with bandwidth limitations.
Case in point: the HBO into animation. It uses static from the old analog days. It looks like shit even at 4k because random noise is impossible to compress without the exact strategy outlined here (stripping it out and rendering it later).
The "at scale" part is the real story here. Film Grain Synthesis has been available in the usual AV1 encoders for a while, but required some amount of manual tweaking to avoid creating problems, meaning it was only used in production when you had a very limited catalog, or for particularly important titles. They do not provide a lot of details here about how they overcame those problems, but it is nice to see it being deployed more broadly.
> This grain, formed from tiny particles during the film’s development, is more than just a visual effect. It plays a key role in storytelling by enhancing the film’s depth and contributing to its realism.
I never understood the “grain = realism” thing. my real eyes don’t have grain. I do appreciate the role of grain as an artistic tool though, so this is still cool tech
The article points out the masking effect of grain, which hides the fake-looking compression artifacts, and also the familiarity/nostalgia aspect. But I will offer an additional explanation.
Look around you: nearly all surfaces have some kind of fine texture and are not visually uniform. When this is recorded as video, the fine texture is diminished due to things like camera optics, limited resolution, and compression smoothing. Film grain supplies some of the high frequency visual stimulus that was lost.
Our eyes and brains like that high frequency stimulation and aren't choosy about whether the exact noise pattern from the original scene is reproduced. That's why the x265 video encoder (which doesn't have grain synthesis since it produces H.265 video) has a psy-rd parameter that basically says, "try to keep the compressed video as 'energetic' as the original, even if the energy isn't in the exact same spot", and even a psy-rdoq parameter that says, "prefer higher 'energy' in general". These parameters can be adjusted to make a compressed video look better without needing to store more data.
> my real eyes don’t have grain.
They definitely do at night when it's dark out. There's a kind of "sparkling" or "static" that comes in faint light.
Fortunately, our eyes have way better sensitivity than cameras. But the "realism" just comes from how it was captured using the technology of the day. It's no different from phonograph hiss or the way a CRT signal blurs. The idea is to be "real" to the technology that the filmmaker used, and the way they knew their movie would be seen.
It's the same way Van Gogh's brush strokes were real to his paintings. You wouldn't want his oil paintings sanded down to become flat. It's the reality of the original medium. And so even when we have a digital print of the film, we want to retain as much of the reality of the original as we can.
Your Van Gogh analogy makes sense for old movies. It doesn't quite explain why we're still adding grain to new movies, except for those few which are purposefully evoking older movies.
Even modern cameras have grain. If you need to integrate your scene with motion graphics, background replacement, or vfx, you'll need to remove grain on part of the image, edit it, add the original grain back where possible and synthesize new grain elsewhere.
Often it can also make sense to modify the grain for aesthetics. Denoising usually produces a less detailed result, but what you can do is denoise only the color channels, not the brightness channel. Brightness noise looks normal to us, while color noise tends to look very artificial. But by keeping the brightness noise, you avoid losing detail to the denoiser.
We don't use obvious grain, usually. That's generally precisely to evoke something about the past -- for flashbacks, a period look, etc. A sense of grittiness like 70s movies, etc.
On the other hand, a small amount of constant grain or noise is intentionally often introduced because otherwise images feel too static and end up looking almost fake. Similarly, dithering is intentionally added to audio, like mastering CD's or tracks. It helps prevent artifacts in video and in audio.
People are always trying to rationalize and justify aesthetic preferences. The depth and nuance of your understanding of a thing will change how you perceive variations of that thing, whether it's guitar tonewoods, style of music, types of paint, flavor of beer, or the grain in film. If you know a lot about a subject, you can tell a lot about the history of a thing, and that's going to change how you feel about a thing.
A child watching a Buster Keaton skit and gasping and giggling and enjoying it is going to have a different subjective aesthetic experience of the media than a film critic who knows exactly what type of film and camera were used, and what the meaning of all the different abstractions imply about the scene, and the fabric of Keaton's costume, and so on, and so forth.
Subjective aesthetic preferences are in the realm of cognition - we need a formal theory of intelligence mapped to the human brain, and all of these subjective phenomena collapse into individualized data processing and initial conditions.
There's something about film grain contrasted against clean cel animation which might make it easier for people to suspend disbelief. They are conditioned to think that absence of grain is associated with unreal animation, particular types of media, and CGI. Home video and news and so forth had grain and low quality, so grain gets correlated with "real". In my view, there's nothing deeper than that - we're the product of our times. In 40 years, media will have changed, and it may be that film grain is associated with surrealism, or edited out completely, as it's fundamentally noise.
The way I see it is that grain makes the film look more detailed than it really is, it can also hide compression artefacts and blurriness.
I don't know the psychovisuals behind that. Maybe it adds some high frequencies that compression often washes out, or maybe acts like some kind of dithering.
As for your eyes, I am pretty sure that they have grain, that's how quantum physics work, you just don't perceive it because your brain filters it out. But again, I don't know how it interacts with film grain.
> As for your eyes, I am pretty sure that they have grain
And lots of it, actually. Just close your eyes or look at any non-textured surface. Tons of noise.
The decreasing signal-to-noise ratio is also highly noticeable when it gets darker.
Video signal without the noise or grain is annoying to watch as it makes everything in the ”out of focus” zone look smooth blurry. Your eyes want to focus yet it is an illusion of depth without an actual depth. Noise texture emphasizes that this is just a 2D plane after all so your eyes can rest and the viewer doesn’t feel like they need glasses. This is just my theory of it based on observation. No research behind it.
Mine do, at least when it's very dark
Yeah I've had visual snow, I think only when I'm tired, stood up too fast, dehydrated, etc. https://en.wikipedia.org/wiki/Visual_snow_syndrome
It's doubling back on itself. The film grain makes the footage look "cinematic" because it's how old movies looked.
This reminds me of modern windows having fake panes. They’re just strips that are applied to give the impressions that there are multiple smaller panes because people are used to that and it feels “correct”.
I have to imagine past glassmakers would have been absolutely enthralled by the ability we now have to make uniform, large sheets of glass, but here we are emulating the compromises they had to make because we are used to how it looks.
> They’re just strips that are applied to give the impressions that there are multiple smaller panes because people are used to that and it feels “correct”.
It is more than just 'feeling correct': windows and their various (sub-)elements that make them up (can) change the architectural proportions and how the building is perceived as a whole:
* https://www.youtube.com/watch?v=uAMyUoDz4Og
* https://www.youtube.com/watch?v=_c8Ahs9Tcnc&t=49
It is similar with columns: they're not just 'tall-and-narrow', but rather have certain proportions and shapes depending on the style and aesthetic/feeling one wishes to convey:
* https://en.wikipedia.org/wiki/Classical_order
And these proportions can even be 'fractal': the window panes related to windows as a whole, related to the building as a whole:
* https://www.youtube.com/watch?v=J-0XJpPnlrA&t=3m13s
* https://en.wikipedia.org/wiki/Golden_rectangle
* https://en.wikipedia.org/wiki/List_of_works_designed_with_th...
* https://www.nngroup.com/articles/golden-ratio-ui-design/
I strongly doubt that multiple smaller panes would have ever become a common style if we could have always made large glass panes. This is a perfect example of people becoming very used to a style forced by a technological limitation that is emulated even after the limitation doesn't exist.
> I strongly doubt that multiple smaller panes would have ever become a common style if we could have always made large glass panes.
Perhaps, but if you're going to have them anyways you might as well make a conscious choice as to how they add to the overall design of the structure.
It used to be a bigger deal (when digital cameras started being used) since people felt like digital video didn't look real/as good - movies shot on film were generally better looking (as crews were used shooting with it and digital video wasn't as sophisticated as today) and HAD grain.
It might be that there is a large part of the population that still has that association.
Cinephiles are also more likely to watch older (i.e. with grain) movies that ARE well shot and beautiful (which is why they are classics and watched by cinephiles) and not see bad film movies, only the cream of the crop, while being exposed to the whole gamut of quality when watching todays movies shot digitally. Would reinforce that grain = good while not being necessarily the case - and their opinion might be heard more than gen pop.
At any rate, it can be a neat tool to lower sharpness!
My vision is grainy because of visual snow. Which is why I turn off film grain in games, it stacks on my vision and makes a visual mess.
Grain = realism because real captured grain isn't total random noise. It's authentic noisy data. It's part of captured scene. It adds subtle tiny but real detail to the scene. Unless I am corrected here and that real grain is also total random noise.
If your eyes did have grain, then it would still be applied to watching an "ungrained" film, as you're still using the same eyes.
film grain adds realism in the same way that high frame rate films look wrong or vinyl sounds "warmer" or tube guitar amps sound "better" - It is what we are used to.
Film grain can create stochastic resonance with the underlying ground truth. In practice, this can improve the perceived image quality over having none.
Film grain and 24fps are both examples of people being far too attached to the technical limitations of film.
23.976 fps has been put on a pedestal as the "correct" look. Just look at the reaction to The Hobbit. However it does provide some objective advantages. 60 fps requires more lighting. Adding more lights means more electric setup and heat for actors in heavy makeup and costume. In post production that's more frames to edit.
> 60 fps requires more lighting.
Assuming the comparison uses the same image sensor for each.
23.976fps is only "correct" when telecining to a 59.94Hz format. 24fps round is the "correct" format in terms of actual filmmaking. I don't know of many films specifically shot at 23.976fps.
Isn't it strange how people accepted much bigger changes like sound and color but are now balking at higher frame rates?
grain and 24fps and widescreen trigger certain contextual emotions around the movie-watching experience. remove them and your brain contextualizes the video very differently.
this is likely the result of ~100 years of film-based filmmaking and projection. hell, we still call it filmmaking.
Yes, it is only the result of familiarity. We could gradually increase the frame rate of movies made in a year by 1 fps per year and then no one would even notice after 24 years every new movie would be 48fps.
When you talk on a cellphone the codec AMR-WB nominally captures 50 Hz - 7000 Hz. However that's only on the optional highest bitrate 23.85 Kbps. The most common bitrate 12.65 Kbps only goes up to 6400 Hz and synthesizes 6400 - 7000 Hz from lower frequencies and noise as it sounds better than not having the noise!
This sent me down a very interesting rabbit hole, thanks!
Film grain needs to die. Its time is past. Sepia photographs and running 16 FPS silent film at 24 FPS are already dead. Next, film grain.
Eastman Business Park in Rochester has been demolished.
Also, please stop putting dust and scratches on YouTube videos. Thank you.
Fake film grain maybe, but saying all film grain needs to die is like saying oil painting brush strokes need to die.
But why does filmgrain need to die?
> Picture this: you’re watching a classic film, and the subtle dance of film grain adds a layer of authenticity and nostalgia to every scene
It just adds visual noise that obscures details of the authentic scene, and nothing prevents nostalgia from being tied to many of the more prominent visual cues like old actors or your own old memories from when you watched it first...
> contributing to [film's] realism
But there is no grain in reality, so it does the opposite
Otherwise I'm glad AV1 marches along and instead of wasting bitrate encoding visual garbage has an algorithmic replacement mechanism- which also means you could turn it off easier.
Documentaries might care about accurately representing reality. For every other cinematic genre, "authenticity" is not an inherent goal. If film grain is part of the director's vision, then that's just as valid as a choice to have dramatic non-diagetic music playing in the background of a scene (which is highly inauthentic, but also highly effective at evoking emotion, which is the point of art).
I don't see how that is supposed to be relevant to the article's claim that grain adds authenticity and realism. Someone claiming that non-diegetic music adds to authenticity would also be weird.
I think what the author meant is that film grain is part of the authentic visual aesthetic of cinema produced using celluloid materials. A de-noised image is therefore inauthentic as that is contrary to the appearance of the original projected image or source/master elements.
> But there is no grain in reality, so it does the opposite
Well ackchually -- illumination is inherently random, so all time-bounded captures of a scene (including what your eyes do) are subject to shot noise: https://en.wikipedia.org/wiki/Shot_noise
You misackchuallied grain with any noise, for example, the bottom right square of the image at the wiki page is not grainy despite being technically shot-noisy
> But there is no grain in reality, so it does the opposite
The author did not do a particularly good job of selling the merits of film grain. I am not sure what he meant by that but I will say that it does increase the perceived sharpness of the image as well as perceived detail, even if it is just an illusion. The cinematographer Steve Yedlin describes it as giving the viewer’s eyes something to “latch” onto.
> It just adds visual noise that obscures details of the authentic scene
Does it add any more than modern video compression techniques? What constitutes noise in cinema, is somewhat subjective.
Which modern compression artifacts that are still visible at high bitrates do you have in mind that would similarly detail-obscuring?
I remember years ago when digital projection was just becoming the norm. I saw a movie (I don't remember which one) and during the opening scene I remember thinking "why are the credits jittering around? Oh. This is actually being projected on film!"
I'm in my early 50s so I remember film quite well. Just like vinyl or cassettes, I ain't going back and unless it is an artistic choice I don't want films to emulate what I consider to be an inferior technology.
It's a bit frustrating that the footage is first shot, then denoised in post, then renoised in post, then denoised in encoding and then renoised at decoding.
You worry too much, it's all fake. What you think is the "footage" will be many layers from various sources composited together.
Fake lights, fake shadows, fake sky, ...
Only if you are aware of it, which 99.9% of people consuming video content are not. It’s simply an unimportant implementation detail (from a viewer’s perspective who doesn’t really care about bitrate-as-cost).
Very interesting look into how AV1’s Film Grain Synthesis (FGS) is being scaled to production at Netflix. The part that stood out most to me was the smart decoupling of film grain from the core encoding process. Denoising before compression and then reintroducing grain during playback is such a clever inversion of the usual "preserve everything" strategy that often bloats bitrates.
What is the motivation for writing HN comments with AI?
Low quality AI comment
Ok, very cool. But I want Netflix to explain it related to Seinfeld, which at 10-12 feet looks fine, but up close looks insane. Blocky MJPEG + grain filter??
It's not like we're on Pentium II processors anymore -- I can filter just about anything with ShaderGlass [0] on a shitty computer (and some of the CRT shaders like crt-hyllian-curvature are brilliant, especially on old shows like NewsRadio that only exist on DVD) .. and I'm shocked that Netflix doesn't just have this built into their Apple TV app or whatever. I'm shocked PLEX doesn't have it! (that I know of)
I made a comment on a different post about imagining a world where local AI/LLM/whatever does some favorable processing for you, by you, on your device, of web content, to enhance your experience. I really believe media (streamers all the way down to open source devs) need to begin to incorporate whatever's out there that reduces friction and increases joy. It's all out there already! The heavy lifting has been done! Just make Family Matters look like how it looked when I was locking in on a Friday night for TGIF LOL
[0] https://github.com/mausimus/ShaderGlass
Yup, yet another example of the thing I'll never stop finding fascinating:
ANY noticeable percieved "flaw" in any creative media will eventually become an aesthetic choice.
I think there is a bit more to it. For example when developing a game when CRTs were popular they were using CRTs to view their game and making decisions based on what they saw on the CRT. If you display the same game with perfect square pixels it looks different. If the developers were viewing square pixels when developing the game they would make different decisions.
I don't think this conflicts with what I'm saying; I've seen what you talk about -- and yet in modern days people will emulate the "square pixel bad style" regardless.
Agree. Purely opining, but I assume that it's because of the emotional connection that artistic media has on people, despite the flaws.
People remember the emotions the artwork engendered, and thus the whole work is associated with the feelings, flaws and all. If the work is particularly widely known, the flaws can become a stand-in for the work itself.
I see this in video games - I'm fond of the NES-era "flaws" and limitations (palette limits, sprite limits, sound channel limits), but less connected to the Atari 2600 or SNES/PS1/NDS/etc flaws. Shovel Knight is charming; A Short Hike, while great, doesn't resonate on a style level.
Even if you ignore all of the emotional attachment aspects, I think you would expect this to happen simply because people who are good at their craft will learn how to work with the shortcomings of their tools. If your cameras unavoidably add grain to everything you shoot you'll find ways to make that look good, and some of the things you come up with might not work without grain.
For an example, watch Shogun. Director apparently thought that most of the screen being out of focus was a positive. Quite distracting.
I've noticed this in a few things the last few years. The top and bottom of the shot are out of focus and it is super distracting to me. Maybe it is meant to draw the eye to the middle of the frame.
Yup, notice that as well. A blurry vignette. Needle sharp in the center, but objects that are the same distance are blurry at the edges of the frame. At first I thought they use vintage lenses, but it's probably done in post.
I sometimes do something similar on photos, where I darken the edges a bit.
Or Snyder's terrible zombie movie Army of the Dead where he uses lenses with very shallow depth of field that makes almost everything look out of focus. It is very annoying.
I like this. Not really because I feel modern media should have added grain, but because for older media this is a method to get closer to the original but at much lower bitrates without excessive smoothing. What’s not to like?
Also, the author had me at God of Gamblers 2. So good. I will take him up on his recommendation to rewatch.
The original grain that is captured is actually a detail and not total random noise. I believe you can make up the vague sense of original scene if you could somehow extract that grain/noise alone.
It's like reducing an image to tiny dots with dithering (reminds of Atinkson dithering). Those grains are not a noise, they are a detail, actual data. That's why real grain looks good IMO.
This is just a fact of lossy compression: you want to throw away information that contributes less to the perception of the video so that you can describe it with fewer bits of information.
There are two possible advantages for this kind of grain synthesis. For Netflix, they could produce the same perceived quality at lower bitrates, which reduces costs per view and allows customers with marginally slow connections to get a higher quality version. For a consumer, the advantage would be getting more non-grain detail for a fixed bitrate.
You are right that if you subtract the dentists frame from the raw one, showing only the estimated noise, you would get some impression of the scene. I think there’s two reasons for this. Firstly, the places where the denoiser produced a blurry line that should be sharp may show up as faint lines. I don’t think this is ‘hidden information’ so much as it is information lost to lossy compression. In the same way, if you look at the difference between a raw image and one with compression, you may see some emphasized edges due to compression artefacts. Secondly, the less exposed regions of the film will have more noise so noisiness becomes a proxy for darkness, allowing some reproduction of the scene. I would expect this detail to be lost after adjusting for the piecewise linear function for grain intensity at different brightness levels.
Perhaps a third thing is the level of noise in the blacks and the ‘grain size’ or other statistical properties tell you about the kind of film being used, but I think those things are captured in the film grain simulation model.
Possibly there are some other artefacts like evidence of special effects, post processing, etc.
If you extract the grain (e.g. by subtracting a blurred version of the image), the result contains mostly noise, not meaningful scene information outside of some variation according to image brightness. Film grain is random, so the extracted "grain layer" doesn’t encode original image detail but film grain itself encodes relative lightness in its _density_.
Between 24fps and film grain people are way too attached to fundamentally inferior technology. With how strongly people resist frame rates faster than 24 I'm surprised people accepted color and sound, which were much bigger changes.
Everything is fake now. I want a technology which works with a raw film scans, not even compressing them to JPEG, which is a 1st step in loosing the details BTW. Motion detection, key frames, delta frames - fine. But with a lossless video. On a Blu Ray off course, i don't care much about streaming.
A 4K/24p film encoded with Apple ProRes 4444 XQ (not even ProRes RAW) is 716 GB per hour, so you would need to swap a total of 30 Blu-ray discs once every 4 minutes in order to watch a 2 hour movie.
I wonder how much you'd get with such a technology... truly uncompressed 4K video you're talking about something on the order of a few terabytes for a 90-minute movie, so way way bigger than the biggest 4K Blu-ray discs. Lossless compression would get you under that number, but far enough to matter?
Is film grain a nostalgia effect? If kids grew up without seeing film grain, how would they feel about it?
> Is film grain a nostalgia effect?
Film grain simulation can be an effect, but that's not the primary application being described here. As @ricardobeat succinctly put it elsewhere in this post, this process usually supports "a reproduction of the original grain pattern, compressed separately from the underlying image content. The result is closer to the original picture than the denoised/compressed version".
> If kids grew up without seeing film grain, how would they feel about it?
Film isn't quite dead yet, with ~7% of features in 2025 still being shot on film and maybe 1% more taking the digital→film→digital (a.k.a. "analog intermediate") route. Unless their media diet is all super-recent and low-quality, even today's kids will have most likely seen many movies shot on film with visible grain. Popular candidates would include the Harry Potter franchise, most episodes of the Star Wars franchise, early Marvel movies, many animated Disney and Studio Ghibli features, etc.
When I watch an 80 GB 4K movie from Blu-ray, the last thing I want is film grain, which makes it look like a VHS recording from the 90s.
Please also add more noise to the audio track.
happier and happier about leaving behind digital media to return to physical. to me this is literally slop. i want the uncompressed file stop selling me stepped on product
Your statement makes me think that one of the following is true:
1. You prefer Betamax or VHS to digital media (highly unlikely)
2. You own laserdiscs (limited to 480i)
3. You own 35mm prints of film.
Since all other formats film has been made available on are both digital media and compressed.
Uncompressed 4k video is ~5Gbps (3840 * 2160 * 3 * 24 * 8). A 2-hour movie clocks in at about 4.3TB. (3840 * 2160 * 3 * 24 * 60 * 60 * 2)
All that is 24fps.
That's without audio, which I assume you also want to be uncompressed.
TBF even as bloated as 48/24 FLAC is it's a rounding error at that point.
Physical media is what pioneered "stepping on" the product with things like Pan & Scan
agree
How much grain is there in IMAX films?
There's an influx of high-profile directors/films right now and in pipeline filmed for IMAX (F1: The Movie I think, Mission Impossible, etc) and Christopher Nolan's Odyssey coming next year shot entirely on IMAX film with newly developed smaller/quieter cameras made to accomplish it.
> How much grain is there in IMAX films?
I've read that a 15-perf 65mm IMAX negative shot with slower film stocks is "virtually grainless", even when viewed on a 70ft screen. Grain is apparently noticeable in IMAX films when large/fast stocks are used and pushed toward their limits, and (of course) when smaller-format film stocks have been blown up.
I love grain! 16mm vibes
"Your scientists were so preoccupied with whether or not they could, they didn't stop to think if they should."
When you ran out of things to innovate on.
They should use FPGA acceleration for this.
Fake grain really disgusts me.
This is not fake grain - it's a reproduction of the original grain pattern, compressed separately from the underlying image content. The result is closer to the original picture than the denoised / compressed version.
Depends on your definition of fake. I consider this to have crossed the line from lossy compression to a generative product.
[dead]
[flagged]
[flagged]
There's no need for low-quality comments cutting and pasting from your favourite LLM. You can literally just type it into your browser's search bar and learn what it is, or spend a minute reading the wiki page (https://en.wikipedia.org/wiki/AV1).