I investigated a gradient-descent-based concept adjacent to this back in like 2023 [0] inspired by the original 3DGS paper, I was pretty happy with the results I got with back-to-front alpha-blending and polar-coordinate gaussians for variable-curvature brush strokes. One of these days I should add another example gif to the repo besides just the Mona Lisa one that's currently there, but that's the only one I've got at the moment unfortunately.
As someone with an arts degree (not in painting though, but still) it feels really really weird to see how that animation builds up the image. It feels less random than the old "simulated annealing with vector shapes" techniques, ending up somewhere that feels both similar and extremely far removed from how a human would do it at the same.
It's like… the way a trained human would paint feels kind of like how MSB radix sort makes an array look more ordered, going from crude to refined. The way your algorithm builds up an i|age feels like I'm looking at a radix sort that alternates between a stable MSB and LSB radix sort, which is weird as heck. It's fascinating, really.
> Each layer is more translucent and specific than the one below it, augmenting the existing structure that's already been built up.
So interestingly, one distinguishing difference between oil paintings and acrylic paintings is that the former has more translucency, whereas the latter can be more easily layered on in a way that covers up the lower layers. This is what, according to one of my mentors in art school who was a painter, allows oil paintings to copy the feeling of real skin so well: it allows a painter who mastered the medium to create a similar subsurface scattering effect. That's also another reason why paintings in real life can look so different than images on a screen, aside from the much greater gamut than even the best color screens we have and the actual surface texture of the paint strokes. The downside is that any mistakes made on deeper layers will come through in higher layers. Especially if you paint using the techniques used by the old masters.
Anyway, this suggests your gaussian splats essentially are closer to "oil painting"-like. Makes me wonder if you could also encode this subsurface scattering and texture somehow, so that viewing the splats in 3D would recreate some of that missing effect of looking at a real painting!
Also, on the other end of the spectrum: do you think the algorithm could be modified to create SVG output? Then it might be useful as an artistic "upscaling" filter too.
(Also, speaking as someone with protanomaly, I'm guessing the links have some kind of turquise or light teal color to them, but it is effectively bright gray to me. And since they have almost the same brightness perception as the white text I genuinely cannot tell where the links are (except when the text calls them out). Since this is likely to affect anywhere between 5% to 10% of all men visiting your blog that might be worth changing.)
Creating a 3d scene and then using a light source could produce some interesting effects for sure. Since the layers are already in place, it's just a matter of setting up the scene.
And I haven't thought about making SVGs, but that's a really clever idea. I'm gonna add an issue to try this out.
Oh, and thanks for the heads up regarding readbility. I'll update the theme to account for that.
I am very much looking forward to see the results of trying it out, regardless of whether it works in practice! (negative results are also educational after all)
And thank you for looking into the accessibility issue regarding links!
Absolutely one of the most impressive paint "filters" I've seen.
How long is the frame time on the GPU implementation? Is there any hope for something like this to run at 60fps as a post process for a game? Can the data from a 3D scene be used to skip some of the initial analysis steps?
The results look way better than I was expecting. I don't know anything about Gaussians, but most of the "painting" effects I tried in image editors didn't produce satisfying results. This one looks great.
The effect looks decent in the foreground of some pictures, particularly the wolf and the cat. It really exaggerates the depth of field in the background, though, and those areas feel like someone ran a posterise after a blur, rather than brush strokes. Might just be a side effect of the author's predilection for bokeh, I guess. The Tokyo shot looks good throughout.
I think my main issue with it is that you don't end up with the impression of brush strokes on those areas - in a painting even big flat areas of colour tend to have clear evidence of brush strokes
That's because Gaussian splats are ellipses without any texture of their own (more or less), missing any detail that an actual brush stroke would have. Because the ellipses are so elongated in the finer details it feels like layered brush-strokes, but in the coarse background the "flatness" of the splats dominates. Compare to my stippling notebook[0], an even more simplified image filter (it's "just" tiled circles) with a similar "problem". Turn on the fake vignetting or tilt-shifting, and really turn it up for a similar effect.
If the goal is to create a paintlery illusion then there are undoubtedly ways to mitigate this lack of texture though.
Intriguing and nice writeup. This however tries to encode all the hidden painter's knowledge. Would it be easier to generate realistic images from paintings and use these pairs to finetune an image generation model?
You can already use genAI to do similar things. The whole point, however, was to see what I could do using traditional techniques. The other benefit is that it's something you can run locally without needing a lot of resources, and have full control over the shape of the output. You can also combine this with generative models, and have a model generate a picture, then have this repaint it. Ultimately the point was to have some fun and play around with combining these techniques.
> In the days of generative image models, it's still fun to see what can be achieved using traditional image transformation techniques.
> But the really fun part of the project was in combining a number of techniques, such as Perlin noise, wavelets, and edge detection, that I played around with previously in isolation. All these different tricks came together for this project, making it possible to build something greater than the sum of its parts. I find these are the most rewarding types of experiments where you can build on things you've previously learned and combine them in novel ways to make something new and unexpected. I hope you enjoyed the journey as much as I did working on the project.
I used my own photos for the input, I find images with the subject isolated to be more visually pleasing, hence most of my photos are in this style. But there wouldn't be any difference to the algorithm if the image was full of detail as you can see with my photo of Hong Kong at the end.
I investigated a gradient-descent-based concept adjacent to this back in like 2023 [0] inspired by the original 3DGS paper, I was pretty happy with the results I got with back-to-front alpha-blending and polar-coordinate gaussians for variable-curvature brush strokes. One of these days I should add another example gif to the repo besides just the Mona Lisa one that's currently there, but that's the only one I've got at the moment unfortunately.
[0] https://github.com/mkaic/painterbot
As someone with an arts degree (not in painting though, but still) it feels really really weird to see how that animation builds up the image. It feels less random than the old "simulated annealing with vector shapes" techniques, ending up somewhere that feels both similar and extremely far removed from how a human would do it at the same.
It's like… the way a trained human would paint feels kind of like how MSB radix sort makes an array look more ordered, going from crude to refined. The way your algorithm builds up an i|age feels like I'm looking at a radix sort that alternates between a stable MSB and LSB radix sort, which is weird as heck. It's fascinating, really.
Can you make the brush strokes look like brush strokes?
Neat, watching it paint as an animation is pretty fun.
> Each layer is more translucent and specific than the one below it, augmenting the existing structure that's already been built up.
So interestingly, one distinguishing difference between oil paintings and acrylic paintings is that the former has more translucency, whereas the latter can be more easily layered on in a way that covers up the lower layers. This is what, according to one of my mentors in art school who was a painter, allows oil paintings to copy the feeling of real skin so well: it allows a painter who mastered the medium to create a similar subsurface scattering effect. That's also another reason why paintings in real life can look so different than images on a screen, aside from the much greater gamut than even the best color screens we have and the actual surface texture of the paint strokes. The downside is that any mistakes made on deeper layers will come through in higher layers. Especially if you paint using the techniques used by the old masters.
Anyway, this suggests your gaussian splats essentially are closer to "oil painting"-like. Makes me wonder if you could also encode this subsurface scattering and texture somehow, so that viewing the splats in 3D would recreate some of that missing effect of looking at a real painting!
Also, on the other end of the spectrum: do you think the algorithm could be modified to create SVG output? Then it might be useful as an artistic "upscaling" filter too.
(Also, speaking as someone with protanomaly, I'm guessing the links have some kind of turquise or light teal color to them, but it is effectively bright gray to me. And since they have almost the same brightness perception as the white text I genuinely cannot tell where the links are (except when the text calls them out). Since this is likely to affect anywhere between 5% to 10% of all men visiting your blog that might be worth changing.)
Creating a 3d scene and then using a light source could produce some interesting effects for sure. Since the layers are already in place, it's just a matter of setting up the scene.
And I haven't thought about making SVGs, but that's a really clever idea. I'm gonna add an issue to try this out.
Oh, and thanks for the heads up regarding readbility. I'll update the theme to account for that.
I am very much looking forward to see the results of trying it out, regardless of whether it works in practice! (negative results are also educational after all)
And thank you for looking into the accessibility issue regarding links!
Absolutely one of the most impressive paint "filters" I've seen.
How long is the frame time on the GPU implementation? Is there any hope for something like this to run at 60fps as a post process for a game? Can the data from a 3D scene be used to skip some of the initial analysis steps?
The results look way better than I was expecting. I don't know anything about Gaussians, but most of the "painting" effects I tried in image editors didn't produce satisfying results. This one looks great.
The effect looks decent in the foreground of some pictures, particularly the wolf and the cat. It really exaggerates the depth of field in the background, though, and those areas feel like someone ran a posterise after a blur, rather than brush strokes. Might just be a side effect of the author's predilection for bokeh, I guess. The Tokyo shot looks good throughout.
It was more of a stylistic choice on my part. I could add more layers to flesh out background details. And it's Hong Kong actually. :)
I think my main issue with it is that you don't end up with the impression of brush strokes on those areas - in a painting even big flat areas of colour tend to have clear evidence of brush strokes
That's because Gaussian splats are ellipses without any texture of their own (more or less), missing any detail that an actual brush stroke would have. Because the ellipses are so elongated in the finer details it feels like layered brush-strokes, but in the coarse background the "flatness" of the splats dominates. Compare to my stippling notebook[0], an even more simplified image filter (it's "just" tiled circles) with a similar "problem". Turn on the fake vignetting or tilt-shifting, and really turn it up for a similar effect.
If the goal is to create a paintlery illusion then there are undoubtedly ways to mitigate this lack of texture though.
[0] https://observablehq.com/@jobleonard/a-fast-colored-stipple-...
Intriguing and nice writeup. This however tries to encode all the hidden painter's knowledge. Would it be easier to generate realistic images from paintings and use these pairs to finetune an image generation model?
You can already use genAI to do similar things. The whole point, however, was to see what I could do using traditional techniques. The other benefit is that it's something you can run locally without needing a lot of resources, and have full control over the shape of the output. You can also combine this with generative models, and have a model generate a picture, then have this repaint it. Ultimately the point was to have some fun and play around with combining these techniques.
From the article
> In the days of generative image models, it's still fun to see what can be achieved using traditional image transformation techniques.
> But the really fun part of the project was in combining a number of techniques, such as Perlin noise, wavelets, and edge detection, that I played around with previously in isolation. All these different tricks came together for this project, making it possible to build something greater than the sum of its parts. I find these are the most rewarding types of experiments where you can build on things you've previously learned and combine them in novel ways to make something new and unexpected. I hope you enjoyed the journey as much as I did working on the project.
I’d suggest for the examples to pick images that are not mostly bokeh.
I used my own photos for the input, I find images with the subject isolated to be more visually pleasing, hence most of my photos are in this style. But there wouldn't be any difference to the algorithm if the image was full of detail as you can see with my photo of Hong Kong at the end.
Super impressive results, and an awesome write up. Thanks :)
Glad you liked it!
Very cool
Thanks, it was a really fun thread to pull on. :)