Would you consider `foo` a blue function and `bar` a red function? That doesn't seem particularly helpful to me.
The virality of async await is that once you mark a function async, then you can only call it from another async function, which forces you to mark more functions async, which in turn means that if you want to use blocking I/O APIs then you just can't because it's incompatible with your execution model because by daring to express asynchronicity of operations, you were forcefully opted into stackless coroutines.
That's what Zig solves, and that's what is real function coloring. People have written reimplementations of the same libraries multiple times because of it.
Just as an example. Note also how, coincidentally, this duplication of effort resulted in asyncio-redis being semi-abandoned and looking for maintainers. And you have to have both libraries because the asyncio one can't do blocking, and vice versa the other one can't do async.
Would you write two instances of essentially the same library just because one is missing an argument that gives it access to an `Io` interface? No, because you would just pass that extra argument around and nothing else would have to change.
> Would you consider `foo` a blue function and `bar` a red function? That doesn't seem particularly helpful to me.
In the sense of effect/capability typing, I think the answer is yes.
"Coloring" isn't magical, it's just a way to describe effects. Those effect can be described by keywords (`async` in JS, `throws` in Java, etc.) or special token parameters/types (what Zig does), but the consequences are the same: the effect propagates to the caller, and the caller becomes responsible for dealing with it.
Yeah, parameter passing can be seen as some kind of effect (like the Reader monad in Haskell). Passing parameters down the call stack is viral just like function coloring (if you need to do i/o and thus need to receive an io parameter, then your caller must receive an io parameter too, recursively; this is analogous to adding a keyword async in front of your function, and to its callers recursively). The solution is probably some sort of implicit parameter, again, like the Reader monad.
Note: async fn in Rust is also just a fn (in Rust's case, a fn that returns a future). It turns out that returning a future, or receiving as parameter a reference to the runtime, are equivalent in some sense
It's in a different league comparing to async-await abomination.
For one, you can store parameter on a struct thus working around the "virality" on a call site.
You have the following call stack (top to bottom): a -> b -> c -> d
Function "d" does IO, so a, b and c need to pass that IO down. It's the same as with async in terms of virality.
If I understood correctly, the main benefit here is that you don't have to dance around "can I call this function or is it going to block async runtime reactor?"
I would pick async/await style over this because (I think) extra function argument is higher visual load than "async fn".
If you want to print a string from your function, you must supply it to the function. The fact that your function now receives `str: []u8` doesn't make it str-colored.
On a side note: the entire "coloring" metaphor is extremely unfortunate and confusing IMO. It's somewhere at the top of my list (right beside the shroedinger's cat) of things which could've been explained better without dumbing them down.
That’s because str is not a meaningful effect, so we wouldn’t say that a function is “str-colored.”
Function coloring is indeed confusing, which is why I’ve been trying to ground this on effects instead. Effects are IMO easier to understand, and this is a straightforward example of effect typing.
I've been trying to beat this point in and failing. If a parameter type creates "colors", you can extrapolate that to an infinite set of colors in every single language and every single standard library, and the discussion on colors becomes meaningless.
Some people are so focused on categorical thinking that they are missing the forest for the trees.
The colors are a means of describing an observed outcome -- in Node's case, callback hell, in Rust's, 4 different standard libraries. Whatever it may be, the point is not that there are colors, it's the impact on there being colors.
> But there is a catch: with this new I/O approach it is impossible to write to a file without std.Io!
This sentence just makes me laugh, like it's some kind of "gotcha". It is the ENTIRE BASIS of the design!
> you can extrapolate that to an infinite set of colors in every single language and every single standard library, and the discussion on colors becomes meaningless.
It's more that discussion about most of them becomes meaningless, because they're trivial. We only care when it's hard to swap between "colours", so e.g. making it easy to call an Io function from a non-Io function "removes" the colouring problem.
> so e.g. making it easy to call an Io function from a non-Io function "removes" the colouring problem.
Exactly. In golang (which is also a cooperatively multithreaded runtime if I understand correctly), calling a function that needs IO does not infect the callers type signature.
Another poster up thread identified the exact problem: async/await contexts are not first-class values, they are second class citizens. If they were values then you could just stick the context in a struct/class and pass that around instead, and avoid having to refactor call chains every time something changes. It's their second class status that forces the "colouring" into the function signature itself at each point. This is also why ordinary first class values do not introduce colours, ie. you can hide new values/parameters inside other types that are already part of the function signature, thus halting the propagation/vitality of the change.
Of course, if these async contexts were first class citizens then you've basically just reinvented delimited continuations, and that introduces complications that compiler writers want to avoid, which is why async/await are second citizens.
Because async/await are not values, they are ways to run/structure your code. That's why they are so infectious. If you don't want the division the only solution is to make everything of one kind. Languages like C make everything sync/blocking, while languages like Go make everything async.
> the only solution is to make everything of one kind
That's not really the case. All you really need is a way to run async code from a sync function; "keep doing this async thing until it's done" is the primitive you need, and some languages/runtimes offer this.
Going by that reasoning then Rust solved the colored function problem too: when calling an `async` function from a sync one use `block_on`, while when calling a sync blocking function from an `async` one use `spawn_blocking`, but somehow people are not happy with this.
Yes, it's quite curious. Having used both block_on and spawn_blocking, and not being worried about what "colour" my function is, I am also quite confused about the fuss.
On a practical note, since Rust doesn't standardise on an async runtime, it would be more accurate to say tokio solved the coloured function problem, for whatever that means. Or any and everyone else that made it easy to call one coloured function from another.
They are computations that produce values. Computations can be reified as values. What do you think functions and threads are?
As I described above, "delimited continuations" are values that subsume async/await and many other kinds of effects. You can handle async/await like any other value if they were reified as delimited continuations, but this makes the compiler writer's life much more difficult.
Well said. Honestly I don't understand this "function coloring argument" at all in zig's context.
Function is the smallest unit of logic. And function parameter is the fundamental way to control it.so, in theory, __A function parameter is the smallest possible design choice you have to control async vs sync__
There is no way to reduce it further. As you need to encode this somehow, otherwise it will be implicit and very hard to debug.
functions can compose. Function parameters can be composed. Making this a solid design choice.
Zig has somehow achieved this feat of decomposing async/sync to a single parameter. This deserves an ovation
Other langs should take note.
> No, because you would just pass that extra argument around and nothing else would have to change.
> Semantically, passing std.Io to every function is no different from making every Node.js function async and returning a promise. Zig shifts function coloring from blocking/non-blocking choice to io/non-io.
> Complains are about how inconvenient it is to work with [blocking and non-blocking function handling] differences
"Extra argument" in this case is not much different from "extra async keyword", because you can only run I/O functions when having std.Io in scope.
They're similar, but if you're not writing Haskell it's super easy to make a variable always be in scope. You don't have to change all the intermediate functions to do it.
> The virality of async await is that once you mark a function async, then you can only call it from another async function
Rust calling async function in non-async function:
...
// Create the runtime
let rt = Runtime::new().unwrap();
// Get a handle from this runtime
let handle = rt.handle();
// Execute the future, blocking the current thread until completion
handle.block_on(async {
println!("hello");
});
Of course, spinning up a new runtime within the context of a boundary like that is probably wasteful (lots of new threads created if you’re not careful). But you could stash that runtime behind a OnceLock (you’d need to block_on the Handle I imagine rather than the Runtime directly, but doable).
And calling blocking from non-blocking:
let result = tokio::task::spawn_blocking(|| {
5
}).await;
This of course is basically essentially what Zig is doing, except instead of hidden global state it’s parameter passed. This is one area Zig does do better in - I wish Rust would default more to instance state instead of implicit global state.
I really don’t agree with the idea that this is functional colouring. Then we have to start talking about function colouring in a whole bunch of new contexts like with Zigs explicit passing of allocator. Or any other parameter that needs to be explicitly passed to use some kind of interface.
I think we should stick to talking about colouring when there is special calling conventions or syntax, which has the consequence of having to write separate libraries/modules for async code and non-async code.
That is the significant problem we have been seeing with many async implementation, and the one which Zig apparently fully solves.
> Then we have to start talking about function colouring in a whole bunch of new contexts like with Zigs explicit passing of allocator.
That's pretty much where we are though. If you have a function that isn't passed an allocator, and now it needs to call a function that does take an allocator, we're in the same place.
Rust's 'async' keyword changes the type of the return value, but you can just write the different return value yourself; it's 'coloured' purely by what it means to be returning a 'Future'.
No, it's entirely possible. And that's why we don't think about "has an allocator" as a colouring problem.
Likewise if JavaScript had an easy way to get a handle to its Runtime, and a function "block on promise" in its early days, we'd have never had all these "colouring" arguments.
If javascript worked differently we wouldn't have arguments about that significant feature of javascript.
How would you add "block on promise"? Javascript started off without the ability to block on anything, not even I/O because it didn't have I/O. Later when XMLHttpRequeest was added it was purely callback-based.
There are ways to make that API work but introducing the concept of blocking would be a big language change. It's not fixing a flaw, it's trading away some complications for a different pile of complications. There's still a conflict to resolve in language design when you want some functions that are synchronous and don't block and other functions that are asynchronous.
Edit: I guess javascript blocks on alert() but that doesn't seem like what you'd want "block on promise" to do...
This is effectively a special calling convention: Zig expects you to pass in a "token" object that communicates a kind of effect (I/O in this case). No token, no effect (modulo a soundness hole).
This is not a new pattern, and I think it's a pretty good one (and is arguably more ergonomic and general than syntax-level effects). But it's quintessential function coloring.
It does no such thing. you could pass a function a vtable and the vtable could have one implementation that calls an io stashed in the parent of the vtable, and a different vtable that doesnt and the function calling the vtable would be none the wiser. what is the color of the function that took the vtable?
this is not just academic; it would be for example the basis for mocked integration tests on a database or over the net api call.
That's a calling convention, with indirection. You need some kind of capability token for this kind of asynchronicity scheme; it doesn't matter how you get it, but it needs to be there.
To be clear, there's nothing wrong with this; it's just another way to encode capabilities/effects.
> what is the color of the function that took the vtable?
It also has the I/O effect. You don’t need to call or make use of an effect to be “tainted” by it; it just needs to be in the closure (or whatever scope is relevant in the language).
Intuitively: if you mark a function as async, it doesn’t stop being async “colored” just because you don’t actually perform any async operations in it. This is the same thing.
you might be talking about stackless coroutines, which are currently not part of zig, in which case, yes, the compiler might [0] have to instantiate different functions under the hood for the stackless-coro and the non-stackless-coro cases, with some mechanism to drop in an executor at the boundaries. until then its really hard to claim that the functions are colored.
[0] But even in that case there's not really coloring because if you provide a single io implementation there won't be different functions, even at the compiled level.
No, I really just mean from a PLT perspective. The sync/async implementation used under the hood doesn't matter; what matters is that an `Io` in the closure is a token type for communicating effects. No instantiated or enclosed token; no effects.
People seem to be really defensive about this, like it's a bad thing. It isn't! It's arguably a significantly cleaner way to handle what people confusingly call "coloring." But that doesn't make it not "coloring," because coloring is about effects and vitality, not about keywords and syntax.
1. the original author of the function coloring post wad not exactly a theorist so if you're applying some other idea of what coloring is, then you're muddying the waters.
2. here's what i have to say about your idea of what coloring is: That's all fine and good in theory, but in practice it makes no difference, at the user level, or at the compiler level.
I don’t know anything about the original author, but I do know what function coloring is. It’s a way to describe effects. An effect is a function color.
> That's all fine and good in theory, but in practice it makes no difference, at the user level, or at the compiler level
Every example given so far shows Io’s virality, so I don’t know how you can assert how it doesn’t make a difference. The entire point of the design appears (reasonably) to be to introduce a token object that conveys an effect, rather than requiring a runtime to intermediate that effect.
Again, none of this is bad. Effect typing is cool. But it is, by definition, a way to color a function.
(Maybe the confusion here stems from the fact that languages like Go appear to have it “both ways” without coloring. Go is able to do that because it has an intrusive runtime that intermediates asynchronous events. Zig can’t do the same thing without making the same compromises as Go vis a vis FFI performance and ABI compatibility.)
I think I almost agree with what I think you mean, but I'm not sure, hopefully you'll entertain a few questions?
Say you have two functions one that does some kind of IO, (say a pair of bidirectional read, write, from stdin and stdout) and another that returns a block of memory of a constant size. Would you say that one is colored, and the other isn't?
I'll also try to make an attempt at the idea I'm trying to figure out too. I genuinely can't predict which side you'll answer for, but I'm assuming you'll say that it's not colored, because while it does do IO, but only directly though the global file descriptors which have no baring on the calling conventions. Which means the decisions and impact about which color this IO function does ever apply to the callers and callees. This I think is the vitality you mean where the callers and callees are required to know [something] about the semantics? I probably agree you could call this coloring and defend it... It's interesting that this might be closer to the "accepted" definitions of what any language means when it says colored. But I don't think this understanding most people have about the concept of coloring. Most I see are arguing about the semantic effects, much more than the "accepted" definition. I have the distinct impression that when most say coloring, the most significant implication is when and where you resolve the specific color. Will it be sync or async code, when, where, and who gets to decide.
Or perhaps I'm wrong, and you would say that one function is colored, but the other isnt? Then I'd ask if you can tell it's colored, can you tell me which color?
In both cases I'd argue it's better to embrace the semantic widening of the concept of function coloring, because what I understand from how you describe it, I feel that saying this is coloring is strictly less enlightening about the reality of the code and system together, than calling this function alpha. With zig's new IO interface, I can't tell you what the RGB values are for the color, but I can confidently say, it's alpha channel is 0xff. You might not be able to tell me what color it is, but it without a doubt isn't a pure function, interacts with IO and does have a null byte in the alpha channel.
im no virologist but i don't think your definition of viral is reasonable. this is like saying "anything that wants to use a database needs a socket, so a socket is viral"
Look, either you move the program counter to a different place in memory (function call) or you push a task into an event loop. Even if you somehow elide all these differences, they're so different under the hood you'll always have to know in some circumstances. It's honestly wild we conflate them at all.
If we apply your logic, then every mainstream language has the coloring problem with not just two, but a multitude of colors.
// OMG we can't call this without passing the service
// This function is people-colored
public Person findPersonByName(PeopleService service, String name) {
// OMG we can't find without the service
service.find(name)
}
I think I know what you are talking about. In a typical three-tier enterprise application there's no clear separation of concerns between "services" and DAO/Repo...
And this is due to how most persistence libraries are designed: they eagerly execute side-effects, this breaks composability due to poor transaction control, so you end up leaking a lot of your business logic into your supposedly persistence layer.
Take a look at Scala's doobie[1]. Any doobie operation returns a `ConnectionIO`, which is only a description of an operation (free monad). With a proper doobie usage the DAO layer is an algebra of possible persistence-related operations, and the service layer implements business logic by combining primitive ConnectionIOs and interpreting them with full control of transaction boundaries.
The issues I've had with function colouring had to do with trying to compose code using (or expecting) blocking effects with those using async effects in NodeJS - if one library has a higher-order function that expects a non-async function and you have functionality which is provided to you as async, it can be very difficult to plumb them together! And if it's the other way around, it can be quite the performance killer (think how much faster better-sqlite3 is than alternatives). Zig's approach eliminates this problem, AFAICT.
If I had to choose between having to pass through an effect handler like `io` or write `async` everywere, the former seems like a better use of my time. It's explicit, but that can be good.
It also fits Zig well with the allocator. Code can expect an allocator or perhaps an allocator and `io`, or perhaps neither. It's analogous to Rust code that is core vs alloc/nostd vs std.
I am slightly amused that a "C-but-better" language is going to have an `io` passed through non-pure functions much like Haskell. It's that idea combined with Rust's pluggable async runtimes (and stackless concurrency) combined with Roc's "platforms" - but for systems programmers. Quite amazing.
Yes… like every other argument one function provides to another that it doesn’t pass directly through from one of its own arguments.
The function color analogy captures a pernicious viral dynamic with async where it allows only one-way composition. That just doesn’t exist with zig i/o.
Function colouring created a lot of angst when it first came about, particularly because of the difficulties of calling a function of one colour from another. Whether that was possible, what it actually meant, wasn't really well defined.
As other comments have said, there's nothing special about "colouring"; sync/async functions are a case where those above problems are tough, but simpler versions of the problem are everywhere and we don't freak out about them e.g. call a fallible function from an infallible function.
It really all turns on how easy it is to ultimately make the call to the other "function" colour. In Zig's case, if its easy to get an Io in a function that didn't take an Io, it's a non-issue. Likewise for the "fallible function call from infallible function": if it fails, do something that doesn't result in the infallible function failing (do something else? Terminate? Anything will do).
One way or another, I like how this is implemented. Explicitely passing dependencies like this, with a tight syntax, makes things easy to understand and write.
I haven't written any Zig but these demonstrations give me strong vibes of how I felt when I picked up Go more than 10y ago.
I believe Zig's new I/O is neither colored nor not colored, instead the design is orthogonal to that. You won't be able to make code magically async by just swapping the implementation of Io, instead you'll need some kind of runtime that allows functions to wait without blocking OS threads. Depending on how this is implemented (if it's possible at all) you'll have colored functions or not.
i never got warm with this async/await style of calling things. i much more prefer using threads/fibers/protothreads instead. Sure you have to be more expicit on creating and controlling these and also passing around data may get more complex, but at least they do not gloss over the underlying problems async and await tries to cover up
If you want to go down that route, any function that has, or doesn't have, any given resource is colored then.
Would you consider `foo` a blue function and `bar` a red function? That doesn't seem particularly helpful to me.The virality of async await is that once you mark a function async, then you can only call it from another async function, which forces you to mark more functions async, which in turn means that if you want to use blocking I/O APIs then you just can't because it's incompatible with your execution model because by daring to express asynchronicity of operations, you were forcefully opted into stackless coroutines.
That's what Zig solves, and that's what is real function coloring. People have written reimplementations of the same libraries multiple times because of it.
https://github.com/redis/redis-py https://github.com/jonathanslenders/asyncio-redis
Just as an example. Note also how, coincidentally, this duplication of effort resulted in asyncio-redis being semi-abandoned and looking for maintainers. And you have to have both libraries because the asyncio one can't do blocking, and vice versa the other one can't do async.
Would you write two instances of essentially the same library just because one is missing an argument that gives it access to an `Io` interface? No, because you would just pass that extra argument around and nothing else would have to change.
> Would you consider `foo` a blue function and `bar` a red function? That doesn't seem particularly helpful to me.
In the sense of effect/capability typing, I think the answer is yes.
"Coloring" isn't magical, it's just a way to describe effects. Those effect can be described by keywords (`async` in JS, `throws` in Java, etc.) or special token parameters/types (what Zig does), but the consequences are the same: the effect propagates to the caller, and the caller becomes responsible for dealing with it.
Yeah, parameter passing can be seen as some kind of effect (like the Reader monad in Haskell). Passing parameters down the call stack is viral just like function coloring (if you need to do i/o and thus need to receive an io parameter, then your caller must receive an io parameter too, recursively; this is analogous to adding a keyword async in front of your function, and to its callers recursively). The solution is probably some sort of implicit parameter, again, like the Reader monad.
Note: async fn in Rust is also just a fn (in Rust's case, a fn that returns a future). It turns out that returning a future, or receiving as parameter a reference to the runtime, are equivalent in some sense
> Passing parameters down the call stack is viral
It's in a different league comparing to async-await abomination. For one, you can store parameter on a struct thus working around the "virality" on a call site.
Huh?
You have the following call stack (top to bottom): a -> b -> c -> d
Function "d" does IO, so a, b and c need to pass that IO down. It's the same as with async in terms of virality.
If I understood correctly, the main benefit here is that you don't have to dance around "can I call this function or is it going to block async runtime reactor?"
I would pick async/await style over this because (I think) extra function argument is higher visual load than "async fn".
`greet` has Io within its closure, so I’m pretty sure this is consistent with the GP’s point.
If you want to print a string from your function, you must supply it to the function. The fact that your function now receives `str: []u8` doesn't make it str-colored.
On a side note: the entire "coloring" metaphor is extremely unfortunate and confusing IMO. It's somewhere at the top of my list (right beside the shroedinger's cat) of things which could've been explained better without dumbing them down.
That’s because str is not a meaningful effect, so we wouldn’t say that a function is “str-colored.”
Function coloring is indeed confusing, which is why I’ve been trying to ground this on effects instead. Effects are IMO easier to understand, and this is a straightforward example of effect typing.
I've been trying to beat this point in and failing. If a parameter type creates "colors", you can extrapolate that to an infinite set of colors in every single language and every single standard library, and the discussion on colors becomes meaningless.
Some people are so focused on categorical thinking that they are missing the forest for the trees.
The colors are a means of describing an observed outcome -- in Node's case, callback hell, in Rust's, 4 different standard libraries. Whatever it may be, the point is not that there are colors, it's the impact on there being colors.
> But there is a catch: with this new I/O approach it is impossible to write to a file without std.Io!
This sentence just makes me laugh, like it's some kind of "gotcha". It is the ENTIRE BASIS of the design!
> you can extrapolate that to an infinite set of colors in every single language and every single standard library, and the discussion on colors becomes meaningless.
It's more that discussion about most of them becomes meaningless, because they're trivial. We only care when it's hard to swap between "colours", so e.g. making it easy to call an Io function from a non-Io function "removes" the colouring problem.
> so e.g. making it easy to call an Io function from a non-Io function "removes" the colouring problem.
Exactly. In golang (which is also a cooperatively multithreaded runtime if I understand correctly), calling a function that needs IO does not infect the callers type signature.
In async rust, and in "async param" zig, it does.
Another poster up thread identified the exact problem: async/await contexts are not first-class values, they are second class citizens. If they were values then you could just stick the context in a struct/class and pass that around instead, and avoid having to refactor call chains every time something changes. It's their second class status that forces the "colouring" into the function signature itself at each point. This is also why ordinary first class values do not introduce colours, ie. you can hide new values/parameters inside other types that are already part of the function signature, thus halting the propagation/vitality of the change.
Of course, if these async contexts were first class citizens then you've basically just reinvented delimited continuations, and that introduces complications that compiler writers want to avoid, which is why async/await are second citizens.
> async/await contexts are not first-class values
Because async/await are not values, they are ways to run/structure your code. That's why they are so infectious. If you don't want the division the only solution is to make everything of one kind. Languages like C make everything sync/blocking, while languages like Go make everything async.
> the only solution is to make everything of one kind
That's not really the case. All you really need is a way to run async code from a sync function; "keep doing this async thing until it's done" is the primitive you need, and some languages/runtimes offer this.
Going by that reasoning then Rust solved the colored function problem too: when calling an `async` function from a sync one use `block_on`, while when calling a sync blocking function from an `async` one use `spawn_blocking`, but somehow people are not happy with this.
Yes, it's quite curious. Having used both block_on and spawn_blocking, and not being worried about what "colour" my function is, I am also quite confused about the fuss.
On a practical note, since Rust doesn't standardise on an async runtime, it would be more accurate to say tokio solved the coloured function problem, for whatever that means. Or any and everyone else that made it easy to call one coloured function from another.
> Because async/await are not values
They are computations that produce values. Computations can be reified as values. What do you think functions and threads are?
As I described above, "delimited continuations" are values that subsume async/await and many other kinds of effects. You can handle async/await like any other value if they were reified as delimited continuations, but this makes the compiler writer's life much more difficult.
Go is preemptive. Async/await is cooperative because you’re explicitly cooperating and yield.
> The virality of async await is that once you mark a function async, then you can only call it from another async function
That's not exactly true. Many languages return a curried version of it, which can be executed once you have a runtime.
Well said. Honestly I don't understand this "function coloring argument" at all in zig's context.
Function is the smallest unit of logic. And function parameter is the fundamental way to control it.so, in theory, __A function parameter is the smallest possible design choice you have to control async vs sync__
There is no way to reduce it further. As you need to encode this somehow, otherwise it will be implicit and very hard to debug.
functions can compose. Function parameters can be composed. Making this a solid design choice.
Zig has somehow achieved this feat of decomposing async/sync to a single parameter. This deserves an ovation Other langs should take note.
> in theory, __A function parameter is the smallest possible design choice you have to control async vs sync__
So we are just going to forget Go exists?
Disclaimer: I have not used golang, so i might be wrong.
as per my understanding, golang has it implicit. Like there is no way for a developer to tell if code will run under async context or not.
> No, because you would just pass that extra argument around and nothing else would have to change.
> Semantically, passing std.Io to every function is no different from making every Node.js function async and returning a promise. Zig shifts function coloring from blocking/non-blocking choice to io/non-io.
> Complains are about how inconvenient it is to work with [blocking and non-blocking function handling] differences
"Extra argument" in this case is not much different from "extra async keyword", because you can only run I/O functions when having std.Io in scope.
They're similar, but if you're not writing Haskell it's super easy to make a variable always be in scope. You don't have to change all the intermediate functions to do it.
> The virality of async await is that once you mark a function async, then you can only call it from another async function
Rust calling async function in non-async function:
https://docs.rs/tokio/latest/tokio/runtime/struct.Handle.htm...Of course, spinning up a new runtime within the context of a boundary like that is probably wasteful (lots of new threads created if you’re not careful). But you could stash that runtime behind a OnceLock (you’d need to block_on the Handle I imagine rather than the Runtime directly, but doable).
And calling blocking from non-blocking:
This of course is basically essentially what Zig is doing, except instead of hidden global state it’s parameter passed. This is one area Zig does do better in - I wish Rust would default more to instance state instead of implicit global state.So that's an escape in Rust and tokio.
But consider a language that doesn't have block_on, like javascript.
I really don’t agree with the idea that this is functional colouring. Then we have to start talking about function colouring in a whole bunch of new contexts like with Zigs explicit passing of allocator. Or any other parameter that needs to be explicitly passed to use some kind of interface.
I think we should stick to talking about colouring when there is special calling conventions or syntax, which has the consequence of having to write separate libraries/modules for async code and non-async code.
That is the significant problem we have been seeing with many async implementation, and the one which Zig apparently fully solves.
> Then we have to start talking about function colouring in a whole bunch of new contexts like with Zigs explicit passing of allocator.
That's pretty much where we are though. If you have a function that isn't passed an allocator, and now it needs to call a function that does take an allocator, we're in the same place.
Rust's 'async' keyword changes the type of the return value, but you can just write the different return value yourself; it's 'coloured' purely by what it means to be returning a 'Future'.
So its impossible to get an allocator through a different means from inside a function? It must be passed in?
No, it's entirely possible. And that's why we don't think about "has an allocator" as a colouring problem.
Likewise if JavaScript had an easy way to get a handle to its Runtime, and a function "block on promise" in its early days, we'd have never had all these "colouring" arguments.
If javascript worked differently we wouldn't have arguments about that significant feature of javascript.
How would you add "block on promise"? Javascript started off without the ability to block on anything, not even I/O because it didn't have I/O. Later when XMLHttpRequeest was added it was purely callback-based.
There are ways to make that API work but introducing the concept of blocking would be a big language change. It's not fixing a flaw, it's trading away some complications for a different pile of complications. There's still a conflict to resolve in language design when you want some functions that are synchronous and don't block and other functions that are asynchronous.
Edit: I guess javascript blocks on alert() but that doesn't seem like what you'd want "block on promise" to do...
This is effectively a special calling convention: Zig expects you to pass in a "token" object that communicates a kind of effect (I/O in this case). No token, no effect (modulo a soundness hole).
This is not a new pattern, and I think it's a pretty good one (and is arguably more ergonomic and general than syntax-level effects). But it's quintessential function coloring.
> Zig expects you to pass
It does no such thing. you could pass a function a vtable and the vtable could have one implementation that calls an io stashed in the parent of the vtable, and a different vtable that doesnt and the function calling the vtable would be none the wiser. what is the color of the function that took the vtable?
this is not just academic; it would be for example the basis for mocked integration tests on a database or over the net api call.
That's a calling convention, with indirection. You need some kind of capability token for this kind of asynchronicity scheme; it doesn't matter how you get it, but it needs to be there.
To be clear, there's nothing wrong with this; it's just another way to encode capabilities/effects.
> what is the color of the function that took the vtable?
It has the I/O effect.
> It has the I/O effect
it does not, because it can take a vtable that does not call i/o
concretely:
It also has the I/O effect. You don’t need to call or make use of an effect to be “tainted” by it; it just needs to be in the closure (or whatever scope is relevant in the language).
Intuitively: if you mark a function as async, it doesn’t stop being async “colored” just because you don’t actually perform any async operations in it. This is the same thing.
you might be talking about stackless coroutines, which are currently not part of zig, in which case, yes, the compiler might [0] have to instantiate different functions under the hood for the stackless-coro and the non-stackless-coro cases, with some mechanism to drop in an executor at the boundaries. until then its really hard to claim that the functions are colored.
[0] But even in that case there's not really coloring because if you provide a single io implementation there won't be different functions, even at the compiled level.
No, I really just mean from a PLT perspective. The sync/async implementation used under the hood doesn't matter; what matters is that an `Io` in the closure is a token type for communicating effects. No instantiated or enclosed token; no effects.
People seem to be really defensive about this, like it's a bad thing. It isn't! It's arguably a significantly cleaner way to handle what people confusingly call "coloring." But that doesn't make it not "coloring," because coloring is about effects and vitality, not about keywords and syntax.
1. the original author of the function coloring post wad not exactly a theorist so if you're applying some other idea of what coloring is, then you're muddying the waters.
2. here's what i have to say about your idea of what coloring is: That's all fine and good in theory, but in practice it makes no difference, at the user level, or at the compiler level.
I don’t know anything about the original author, but I do know what function coloring is. It’s a way to describe effects. An effect is a function color.
> That's all fine and good in theory, but in practice it makes no difference, at the user level, or at the compiler level
Every example given so far shows Io’s virality, so I don’t know how you can assert how it doesn’t make a difference. The entire point of the design appears (reasonably) to be to introduce a token object that conveys an effect, rather than requiring a runtime to intermediate that effect.
Again, none of this is bad. Effect typing is cool. But it is, by definition, a way to color a function.
(Maybe the confusion here stems from the fact that languages like Go appear to have it “both ways” without coloring. Go is able to do that because it has an intrusive runtime that intermediates asynchronous events. Zig can’t do the same thing without making the same compromises as Go vis a vis FFI performance and ABI compatibility.)
The vtable example is nonviral.
I don’t see how that can be the case, given that Io is in the closure. Anything that wants to do I/O needs that token; that’s what virality is.
I think I almost agree with what I think you mean, but I'm not sure, hopefully you'll entertain a few questions?
Say you have two functions one that does some kind of IO, (say a pair of bidirectional read, write, from stdin and stdout) and another that returns a block of memory of a constant size. Would you say that one is colored, and the other isn't?
I'll also try to make an attempt at the idea I'm trying to figure out too. I genuinely can't predict which side you'll answer for, but I'm assuming you'll say that it's not colored, because while it does do IO, but only directly though the global file descriptors which have no baring on the calling conventions. Which means the decisions and impact about which color this IO function does ever apply to the callers and callees. This I think is the vitality you mean where the callers and callees are required to know [something] about the semantics? I probably agree you could call this coloring and defend it... It's interesting that this might be closer to the "accepted" definitions of what any language means when it says colored. But I don't think this understanding most people have about the concept of coloring. Most I see are arguing about the semantic effects, much more than the "accepted" definition. I have the distinct impression that when most say coloring, the most significant implication is when and where you resolve the specific color. Will it be sync or async code, when, where, and who gets to decide.
Or perhaps I'm wrong, and you would say that one function is colored, but the other isnt? Then I'd ask if you can tell it's colored, can you tell me which color?
In both cases I'd argue it's better to embrace the semantic widening of the concept of function coloring, because what I understand from how you describe it, I feel that saying this is coloring is strictly less enlightening about the reality of the code and system together, than calling this function alpha. With zig's new IO interface, I can't tell you what the RGB values are for the color, but I can confidently say, it's alpha channel is 0xff. You might not be able to tell me what color it is, but it without a doubt isn't a pure function, interacts with IO and does have a null byte in the alpha channel.
im no virologist but i don't think your definition of viral is reasonable. this is like saying "anything that wants to use a database needs a socket, so a socket is viral"
> which has the consequence of having to write separate libraries/modules for async code and non-async code.
That is not a consequence of function coloring or syntax, it is a consequence of having multiple ways of performing IO.
Look, either you move the program counter to a different place in memory (function call) or you push a task into an event loop. Even if you somehow elide all these differences, they're so different under the hood you'll always have to know in some circumstances. It's honestly wild we conflate them at all.
I think this is a good kind of function coloring. It would avoid some scars I have from:
- seemingly harmless functions that unexpectedly end up writing four different files to disk.
- Packages that do I/O or start threads when you simply import them.
If we apply your logic, then every mainstream language has the coloring problem with not just two, but a multitude of colors.
EDIT: formattingThis is off topic. But I really hate “services”. They always are just some random collection of functions and aren’t a real object at all.
I think I know what you are talking about. In a typical three-tier enterprise application there's no clear separation of concerns between "services" and DAO/Repo... And this is due to how most persistence libraries are designed: they eagerly execute side-effects, this breaks composability due to poor transaction control, so you end up leaking a lot of your business logic into your supposedly persistence layer.
Take a look at Scala's doobie[1]. Any doobie operation returns a `ConnectionIO`, which is only a description of an operation (free monad). With a proper doobie usage the DAO layer is an algebra of possible persistence-related operations, and the service layer implements business logic by combining primitive ConnectionIOs and interpreting them with full control of transaction boundaries.
[1] https://typelevel.org/doobie/index.html
I’m sorry but I just don’t enjoy functional Scala.
Paraphrasing someone else: then keep enjoying the disfunctional one :-P
Haha you got a good chuckle out of me on that one.
I am actually really excited about this.
The issues I've had with function colouring had to do with trying to compose code using (or expecting) blocking effects with those using async effects in NodeJS - if one library has a higher-order function that expects a non-async function and you have functionality which is provided to you as async, it can be very difficult to plumb them together! And if it's the other way around, it can be quite the performance killer (think how much faster better-sqlite3 is than alternatives). Zig's approach eliminates this problem, AFAICT.
If I had to choose between having to pass through an effect handler like `io` or write `async` everywere, the former seems like a better use of my time. It's explicit, but that can be good.
It also fits Zig well with the allocator. Code can expect an allocator or perhaps an allocator and `io`, or perhaps neither. It's analogous to Rust code that is core vs alloc/nostd vs std.
I am slightly amused that a "C-but-better" language is going to have an `io` passed through non-pure functions much like Haskell. It's that idea combined with Rust's pluggable async runtimes (and stackless concurrency) combined with Roc's "platforms" - but for systems programmers. Quite amazing.
> much like Haskell
I have not mentioned this in the post, but it's surprising how monadic I/O operations are, including async/await approach
And it is certainly in FP land to pass I/O as parameter, because everything is a function
I don't care about the color of the function. What matters is if you'll have to write two versions of the function, and this seems to solve that.
With Zig I/O you can call i/o functions from non-i/o functions and non-i/o functions from i/o functions.
In the analogy of “What color is your function”, you can call blue functions from red functions and red functions from blue functions.
The pernicious viral nature of function coloring doesn’t apply.
> With Zig I/O you can call i/o functions from non-i/o functions and non-i/o functions from i/o functions
Yes, but you would either create a new std.Io just for that, or use a global/semi-global std.Io from somwhere else
Yes… like every other argument one function provides to another that it doesn’t pass directly through from one of its own arguments.
The function color analogy captures a pernicious viral dynamic with async where it allows only one-way composition. That just doesn’t exist with zig i/o.
Function colouring created a lot of angst when it first came about, particularly because of the difficulties of calling a function of one colour from another. Whether that was possible, what it actually meant, wasn't really well defined.
As other comments have said, there's nothing special about "colouring"; sync/async functions are a case where those above problems are tough, but simpler versions of the problem are everywhere and we don't freak out about them e.g. call a fallible function from an infallible function.
It really all turns on how easy it is to ultimately make the call to the other "function" colour. In Zig's case, if its easy to get an Io in a function that didn't take an Io, it's a non-issue. Likewise for the "fallible function call from infallible function": if it fails, do something that doesn't result in the infallible function failing (do something else? Terminate? Anything will do).
One way or another, I like how this is implemented. Explicitely passing dependencies like this, with a tight syntax, makes things easy to understand and write.
I haven't written any Zig but these demonstrations give me strong vibes of how I felt when I picked up Go more than 10y ago.
I mean, isn't it how it's always been, up until JS/python "single threaded" runtimes became what most people had ever encountered?
Single threaded is a bit old, it wasn't until recently that no one had multiple cores outside HPC.
I believe Zig's new I/O is neither colored nor not colored, instead the design is orthogonal to that. You won't be able to make code magically async by just swapping the implementation of Io, instead you'll need some kind of runtime that allows functions to wait without blocking OS threads. Depending on how this is implemented (if it's possible at all) you'll have colored functions or not.
Related discussion (Zig's New Async I/O): https://news.ycombinator.com/item?id=44545949
i never got warm with this async/await style of calling things. i much more prefer using threads/fibers/protothreads instead. Sure you have to be more expicit on creating and controlling these and also passing around data may get more complex, but at least they do not gloss over the underlying problems async and await tries to cover up