I love seeing folks learn that container images are just fancy tar files and JSON and are therefore buildable by normal tools.
Along my own journey of demystification I made a few toy container image registries over the years that generated and served container images from nothing but the URL itself:
https://ko.kontain.me builds a go application on demand and serves it atop a minimal base image
https://apko.kontain.me builds a base image containing packages listed in the URL, again on demand.
The latest addition, https://git.kontain.me serves an image with the specified git repo already checked out in the image.
None of these should be used for anything serious but they were fun to make and play with. :)
random.kontain.me has been uncharacteristically useful in debugging image caching scenarios.
I sometimes wonder why there aren't more bespoke container tools (like yours). Would people be willing to pay for stuff like what you have built, if someone took the time to "productionize" it? Or is there no market?
Interesting... but also the "power" from container is precisely that the description itself, i.e. the Dockerfile, is sufficient. So I'm not sure what the main advantage of either providing the image itself via a registry or providing just the Dockerfile with the base image to another registry then adding steps add.
An image is immutable, many Dockerfiles are not idempotent. Running them multiple times at different times can lead to different results easily.
Building every time takes a lot of time. Both on download and execution time.
Also when building an image you may need to have access to resources that are unavailable where you actually want to run the image. So I see definitely reason for both to exist.
> we brought down image creation time to mere seconds, even for images that were multiple GiB in size
this sounds interesting; for e.g., was wondering the other day if we could build images without actually pulling base images.. everytime we compile, we copy artifact(s) onto a multi-hundred MB base image which definitely doesn't need to be pulled everytime.
You absolutely can just build things and put them on base images without building or pulling the base image at any point. This is a central feature of ko, a simple container build tool for most Go applications: https://ko.build
There are many ways to achieve this. If you are just copying artifacts, and not using RUN, then there is in principle no need to download the base image at all. Feel free to message me if you'd like to chat (email is in my profile).
One of the neat things about something like this is that you could, in theory, do OS builds (think stuff like bootable containers) where you can, deterministically, configure an operating system to boot into all within the confines of your web browser.
We already have image builders for stuff like Talos Linux and Incus OS. This is not out of the realm of possibility.
I'd say the limitations are relative to your use case: `docker build` works well enough in most cases, but sometimes it's not the right tool for the job (e.g., you cannot use `docker build` inside an AWS Lambda, you can get way better performance from specialized tools in some cases [1], etc)
Creating multi arch builds for starters. Maybe creating image from just composing layers instead of “running” a Dockerfile? The first I find cumbersome and is easily overcome by using buildx.
I haven’t tried lately, but I think running a build still requires running a docker engine. Buildx has builders that run as containers.
I joked once that the future was dockerizing every single react component and running them in a wasm port of k8s... I hope that premonition isn't coming true!
I can't imagine this happening in reality (i.e. people wanting to do this for normal production applications), but after reading your comment I must admit I am very tempted to try it for purely esoteric reasons.
The idea is that, in many cases, you can create a layer "by hand" without running actual Linux programs. Layers don't need to be pre-existing, the only requirement is that they can be built programmatically (inside the browser, in this case). The demo actually does that: it "manually" creates a layer from the user-specified entrypoint script, then creates an image from the pre-existing base image's layers and the new entrypoint layer.
In a more real scenario, you can e.g., turn pip wheels into layers without actually using docker's RUN command. All it takes is to massage the data from one archive format into another, programmatically. This unlocks lots of potential (e.g., it becomes embarrassingly parallel to build a container image comprised of pip wheels). Combine that with a good layer caching strategy and a registry that takes advantage of it, and you can have near-instant container builds for arbitrary sets of wheels.
I love seeing folks learn that container images are just fancy tar files and JSON and are therefore buildable by normal tools.
Along my own journey of demystification I made a few toy container image registries over the years that generated and served container images from nothing but the URL itself:
https://ko.kontain.me builds a go application on demand and serves it atop a minimal base image
https://apko.kontain.me builds a base image containing packages listed in the URL, again on demand.
The latest addition, https://git.kontain.me serves an image with the specified git repo already checked out in the image.
None of these should be used for anything serious but they were fun to make and play with. :)
random.kontain.me has been uncharacteristically useful in debugging image caching scenarios.
I sometimes wonder why there aren't more bespoke container tools (like yours). Would people be willing to pay for stuff like what you have built, if someone took the time to "productionize" it? Or is there no market?
AFAIK people only begrudgingly pay for Docker and it does way waaay more.
I have yet to receive any commercial interest in any of these tools :)
Interesting... but also the "power" from container is precisely that the description itself, i.e. the Dockerfile, is sufficient. So I'm not sure what the main advantage of either providing the image itself via a registry or providing just the Dockerfile with the base image to another registry then adding steps add.
An image is immutable, many Dockerfiles are not idempotent. Running them multiple times at different times can lead to different results easily.
Building every time takes a lot of time. Both on download and execution time.
Also when building an image you may need to have access to resources that are unavailable where you actually want to run the image. So I see definitely reason for both to exist.
Or am I misunderstanding your point?
we've built this with Coworker! just launched today. https://coworker.ai https://www.producthunt.com/products/coworker-ai
> we brought down image creation time to mere seconds, even for images that were multiple GiB in size
this sounds interesting; for e.g., was wondering the other day if we could build images without actually pulling base images.. everytime we compile, we copy artifact(s) onto a multi-hundred MB base image which definitely doesn't need to be pulled everytime.
You absolutely can just build things and put them on base images without building or pulling the base image at any point. This is a central feature of ko, a simple container build tool for most Go applications: https://ko.build
(I am a maintainer)
There are many ways to achieve this. If you are just copying artifacts, and not using RUN, then there is in principle no need to download the base image at all. Feel free to message me if you'd like to chat (email is in my profile).
You can, by using FUSE and lazy pulling files as they are opened. I'm working on doing this, myself.
One of the neat things about something like this is that you could, in theory, do OS builds (think stuff like bootable containers) where you can, deterministically, configure an operating system to boot into all within the confines of your web browser.
We already have image builders for stuff like Talos Linux and Incus OS. This is not out of the realm of possibility.
> and we sometimes resign ourselves to the limitations of docker build and friends
What are the limitations of `docker build`?
I'd say the limitations are relative to your use case: `docker build` works well enough in most cases, but sometimes it's not the right tool for the job (e.g., you cannot use `docker build` inside an AWS Lambda, you can get way better performance from specialized tools in some cases [1], etc)
[1]: See my sibling comment at https://news.ycombinator.com/item?id=48301041
Creating multi arch builds for starters. Maybe creating image from just composing layers instead of “running” a Dockerfile? The first I find cumbersome and is easily overcome by using buildx.
I haven’t tried lately, but I think running a build still requires running a docker engine. Buildx has builders that run as containers.
I joked once that the future was dockerizing every single react component and running them in a wasm port of k8s... I hope that premonition isn't coming true!
I can't imagine this happening in reality (i.e. people wanting to do this for normal production applications), but after reading your comment I must admit I am very tempted to try it for purely esoteric reasons.
And uses SOAP for identity and authorization.
I don't understand what this does. How can it "build" anything without a VM capable of running actual code?
Is "build" being used here in the sense of assembling pre-existing layers into an image? What would be the purpose of that?
The idea is that, in many cases, you can create a layer "by hand" without running actual Linux programs. Layers don't need to be pre-existing, the only requirement is that they can be built programmatically (inside the browser, in this case). The demo actually does that: it "manually" creates a layer from the user-specified entrypoint script, then creates an image from the pre-existing base image's layers and the new entrypoint layer.
In a more real scenario, you can e.g., turn pip wheels into layers without actually using docker's RUN command. All it takes is to massage the data from one archive format into another, programmatically. This unlocks lots of potential (e.g., it becomes embarrassingly parallel to build a container image comprised of pip wheels). Combine that with a good layer caching strategy and a registry that takes advantage of it, and you can have near-instant container builds for arbitrary sets of wheels.
GAME CHANGER