I prefer a slightly more general rule: Make Illegal States Unrepresentable. The "Parse, don't validate" rule is a special case of MISU.
What's the difference? MISU applies even when there's no parsing-like transformation happening. For example, if you have a variable that represents the current state of a network connection, and let's say it can be Disconnected, or Connected to some IP address (this is an oversimplification).
Then one way to do it would be
struct {
connected: bool,
peer_ip: int32
}
The trouble is that this allows us to represent an illegal/meaningless state: we're disconnected but there's still some junk old peer_ip hanging in there. Even worse, we might have written
Now we could have connected = true but peer_ip = None.
The solution is to use a sum type:
type connection =
Disconnected
| Connected of int32
(sorry for using made-up syntax; I hope it's clear to anyone familiar with Rust.)
"Make Illegal States Unrepresentable" applies throughout your program, at every interface between modules or functions in the program, including but not limited to parsing input.
But you can't hold both a `Connection<Connected>` and a `Connection<Disconnected>` in the same variable or pass them to the same function, which defeats the point.
It's the typestate pattern. The compiler enforces valid state transitions and so you can't just pass to the same function. It's useful to ensure that only some transitions are allowed. To be honest, the connection state presented by GP is a bad example. But you shouldn't say "it defeats the point".
A better example is an HTTP writer. One state is `Headers` with the methods `add_header()` and `body()` which returns the response in state `Body`. This enforces that one can't add headers after `body()` has been called. But you can finish the response in the `Body` state. If you do something not in this order, you get a compile error.
Personally I don’t like using nil to represent specific states in my domain. I try to only judiciously use it to represent the lack of knowledge about some state, like if I need to reach out to some external service I don’t control, that may or may not give me an answer for my query (and conversely, I don’t like putting “unknown” cases into my enums).
Yes, the type I wrote down is isomorphic to Option<int32>, so the compiler will ultimately generate the same machine code and do the same optimizations.
But defining a new type allows you to give the two cases more descriptive names. That saves you from having to remember and comment what the two cases represent. In a small but significant way, the code documents itself.
Also, doing it this way gives the Connection type its own name and identity separate from Option<int32>. So if you have a function that takes a Connection and some other Option<int32>, you can’t accidentally switch them: even though the two types are equivalent for optimizations and codegen, they’re NOT equivalent for the type checker.
It also makes function signatures more self-documenting: you can tell just from the type that the function takes or returns a Connection, and that helps in a small but real way to understand what it does.
I like programming like this a lot! So much so that the reason I personally like Rust a lot is simply that it’s a mainstream language with great tooling that has sum types and reasonable generics. It’s like if OCaml or Haskell had the community or tooling, or if Go or Python had the type features.
Yes, it'd probably be compiled into very machine code. However defining an enum for it puts the intention into the source code more explicitly, so that's considered the more idiomatic way. You'd probably have a `fn peer_id(&self) -> Option<Inet4Addr>` accessor that you'd use in the case where you'd want the Option.
You'd have to manually implement the traits to support the ergonomics of slices and iteration and costly reallocation if you need to pass ownership as a Vec:
I'd expect:
pub struct NonEmpty<T> {
v: Vec<T>,
}
The constructor would enforce the invariant and then you'd impl Deref and DerefMut for [T] to gain normal len/is_empty/indexing/iteration, passing as &[T] to other funcs and mutating values (which can't break the invariant).
To mutate length while preserving the invariant it's dealers choice e.g.
- add .into_vec() for unwrap/mutate/rewrap
- add invariant preserving mutators of your choice
Not sure switching between languages makes for a compelling argument:
"Look how easy it is to accidentally bypass the invariant of a rust newtype by transliterating the data shape into Haskell and deriving a new type". Uh, ok.
If comparing the "risk of accident" between a newtype wrapper whose only role is enforcing the the invariant versus manually reimplementing vector and iterator semantics to use a different layout... I'd say the newtype wins that.
It would be good advice to keep a newtype that enforces an invariant as a single purpose primitive type. A building block and not a place to add other features.
There might be times I'd prefer structural enforcement e.g. something serialisation related. Converting into a non-rust format is what they are doing in their "accident"!
DerefMut would allow you to call `clear()` on `v` violating the invariant
I'd be great is there were a way to shadow methods but even then guarantees would be poor since Vec might add a new method in the future which isn't covered by invariant checks
Deref/DerefMut enables implicit type coercion rather than exposing an interface. You can choose the target type and immutable/mutable but not parts of the target type.
You can use all the slice reference methods (that do not require ownership) with:
If you DerefMut to a Vec then you won't be able to preserve the invariant.
If you want control over methods to expose then you need wrapper methods for those you want. If you want to expose some of the traits the inner type implements then there are likely derive macros available e.g. with derive_more you could expose just indexing as:
This is great. Alexis King actually stated that, had she known how popular “Parse, Don’t Validate” had been, she would have written it in a language more widely used than Haskell.
With its crap type system Python might be even better, in a strange way.
Haskell's strong, static, non-reflective type system tends to make "parse, don't validate" produce code that also looks nicer. Which is great. So great that it steals a bit of the main message's valor.
In Python, though, it's really easy to just let your data be a dynamically typed list of dicts forever. So easy that parsing into something more strongly typed looks like a whole lot of extra effort. Upon looking at that sort of thing many a working Python programmer, myself included, hears the voice of GvR murmuring disparaging things about "academic" programmers down in the pit of their brain.
Which creates an opportunity to demonstrate all the ways the (arguably) more Pythonic way is actually a royal PITA when you try to make your code robust. Handling and reporting data validity errors gets scattered all over the code, which makes it annoying to maintain. Unit test suites get bloated because it's not obvious what inputs a function should be able to handle. Comments and docstrings to help keep track of this stuff begin to proliferate.
I write Python. While I read the original post and nodded along, nothing changed about how I worked until I started playing with Rust. Seeing a strongly typed language force the constraints and stop stupid errors was when the rationale finally clicked for me. Knowing that this function will never blow-up in an unexpected way changes how you think about problems.
You otherwise get accustomed to some wishy-washy blobs of data that get passed around and find it normal. Maybe you include some ad hoc guard rails here and there, which catches the egregious errors, but there is always some lingering uncertainty. Some string that should have been an int, missing key here, the object which never had the validation check, etc
It is like unit-testing - more-up front work, but I could never go back to a world where I did not get these automated assurances. Sadly, I am a grug-brain which could only feel the lesson from personal experience.
I’m a big fan of “Parse dont validate” and I feel like injecting that approach into python has so many benefits.
there’s all the code correctness stuff that we all love. But the biggest benefit is making it so much easier to maintain other parts of the stack.
Following a stack trace into somebody else’s functions and you see that the args are completely untyped or just a bunch of ‘dict[str, Any]’ is the worst. But if you see those inputs as more narrowly typed data classes it makes it so much easier to grok what the function is supposed to do
I had a very hard lesson in that once when I inherited a legacy Clojure application.
Previously I had been somewhat sympathetic to Rich Hickey’s impassioned monologues about making maps first-class and how everything gets easier when you let data just be itself. But when I had to learn my way around code that someone else had written according to that philosophy… hoo boy. Clojure quickly got demoted from being one of my favorite languages to merely being one of my favorite write-only languages.
I'm the author of typedload. For json-type of data that is loaded in dictionaries you can do A LOT of validation at runtime automatically if you define the types.
> [1] Other languages - like Go or Python - have a runtime check that raises some sort of exception or panic when lst[0] is accessed on an empty list or slice.
You are technically correct :) But that's not the point of the footnote, and you're right that I should update it. The point here is that Python and Go don't have the equivalent of "first", and have the programmer rely on the runtime error in lst[0]. Rust can affort to have a "first" method because of Option.
Thank you! I always found the quip "parse, don't validate" to be more confusing than helpful. Parsing doesn't remove the need for validation. All it does is ensures the type is right, but does nothing to validate the values for the domain.
Good article on Rust's strengths with types. If you like this, you may be curious how you might build your own parse capabilities. I like the Rust crates Winnow and Nom, and also the Rust traits From and Into.
Another take, from the primary example: This is what `unwrap()` is for. I understand that the author is looking at this from a correctness and safety(?) perspective. For practical purposes, I would unwrap here. If it's less trivial than the example, unwrap with a comment explaining why it's fine.
Another angle: Unfortunately, the `first()` method being fallible here is just an issue of using an imperfect method/datatype here. This is where the author gets in to a non-empty-vec custom type. Then you are balancing using a more correct type that takes custom wiring vs a std lib thing everyone understands and takes no setup. I would lean towards this setup if I were using this non_empty_vec.first() unwrap pattern a number of times in the code base; then the setup would be worth it, at least for my own code bases. If I were exposing this in a lib others would use, I would keep the standard Vec so as to be more transparent for others.
In both views: "This is what unwrap is for" does it for me in all cases I've encountered to date. Maybe for aerospace or safety critical systems, I would have a different take.
A third take: I notice this trend in the rust community. It's not my cup of tea. Keep things simple, easy to maintain, and don't let "correctness" get in the way. In this example, I don't think it gets in the way, but I have seen this mindset lead to it getting in the way, especially in embedded, where mapping the Owernership model to hardware ends up in messy patterns and surprising assertions about embedded-101 concepts like DMA being "unsolved", "no good way", "difficult" etc.
Rust provides tools to make sure specific logic is correct if it passes the compiler. People sometimes go overboard and assume you have to type-maxx your code, regardless of complexity added by doing so.
I was thinking about its implementation, too, and ended up deciding that it was probably a performance optimization.
`Vec<T>` stores all data on the heap, so getting anything out of it involves a pointer deref and possibly also an array bounds check. This `NonEmpty<T>` type keeps the first element of the list in a location that supports some low-level optimization that might make a significant difference in situations where accessing the first element is much more common than accessing subsequent elements.
For a untype language like js array, since it can be empty, you have to either always check the length, the item returned, or have a precondition to know the array is not empty.
All three of those cases is either code or context you’re holding in your head.
You absolutely do have to deal with this in your untyped language otherwise you're just letting edge cases and runtime errors happen. The syntax will be different, it will have to happen at runtime instead of compile time, the chosen wording and language to represent the concepts and the work may even vary, but you are not exempt.
Everyone has to deal with this. The only question is whether you deal with it in the type system, through lots of checks at runtime, or buggy behavior when the user encounters cases you didn't envision.
I lamented patterns of what I consider over-use of the type system in Rust (This applies to other languages as well). There is a balance: I prefer this over-use/strict use (e.g. implementing a custom non-empty Vec type) over not using typing at all, but a big margin!
I prefer a slightly more general rule: Make Illegal States Unrepresentable. The "Parse, don't validate" rule is a special case of MISU.
What's the difference? MISU applies even when there's no parsing-like transformation happening. For example, if you have a variable that represents the current state of a network connection, and let's say it can be Disconnected, or Connected to some IP address (this is an oversimplification).
Then one way to do it would be
The trouble is that this allows us to represent an illegal/meaningless state: we're disconnected but there's still some junk old peer_ip hanging in there. Even worse, we might have written Now we could have connected = true but peer_ip = None.The solution is to use a sum type:
(sorry for using made-up syntax; I hope it's clear to anyone familiar with Rust.)"Make Illegal States Unrepresentable" applies throughout your program, at every interface between modules or functions in the program, including but not limited to parsing input.
In rust this would simply be
or
But you can't hold both a `Connection<Connected>` and a `Connection<Disconnected>` in the same variable or pass them to the same function, which defeats the point.
It's the typestate pattern. The compiler enforces valid state transitions and so you can't just pass to the same function. It's useful to ensure that only some transitions are allowed. To be honest, the connection state presented by GP is a bad example. But you shouldn't say "it defeats the point".
A better example is an HTTP writer. One state is `Headers` with the methods `add_header()` and `body()` which returns the response in state `Body`. This enforces that one can't add headers after `body()` has been called. But you can finish the response in the `Body` state. If you do something not in this order, you get a compile error.
Thanks. This had me thinking to myself some. I found this https://cliffle.com/blog/rust-typestate/ too, which others may find useful..
Wouldn’t peer: Option<int32> be enough to represent this?
None = disconnected
Personally I don’t like using nil to represent specific states in my domain. I try to only judiciously use it to represent the lack of knowledge about some state, like if I need to reach out to some external service I don’t control, that may or may not give me an answer for my query (and conversely, I don’t like putting “unknown” cases into my enums).
Yes, the type I wrote down is isomorphic to Option<int32>, so the compiler will ultimately generate the same machine code and do the same optimizations.
But defining a new type allows you to give the two cases more descriptive names. That saves you from having to remember and comment what the two cases represent. In a small but significant way, the code documents itself.
Also, doing it this way gives the Connection type its own name and identity separate from Option<int32>. So if you have a function that takes a Connection and some other Option<int32>, you can’t accidentally switch them: even though the two types are equivalent for optimizations and codegen, they’re NOT equivalent for the type checker.
It also makes function signatures more self-documenting: you can tell just from the type that the function takes or returns a Connection, and that helps in a small but real way to understand what it does.
I like programming like this a lot! So much so that the reason I personally like Rust a lot is simply that it’s a mainstream language with great tooling that has sum types and reasonable generics. It’s like if OCaml or Haskell had the community or tooling, or if Go or Python had the type features.
Yes, it'd probably be compiled into very machine code. However defining an enum for it puts the intention into the source code more explicitly, so that's considered the more idiomatic way. You'd probably have a `fn peer_id(&self) -> Option<Inet4Addr>` accessor that you'd use in the case where you'd want the Option.
This opens you to the possible mistake of accidentally passing something of value None that then is mistakenly interpreted as a valid disconnect.
In some cases using None that way is totally fine, but it is a trade off and the cleaner way is to use a separate type
it's just a minimal example, in reality you would typically have various other data that exists exclusively in connected/unconnected states
Rawwrr, use dependent types and write your entire program in the signature!
Not sure that type is good advice:
You'd have to manually implement the traits to support the ergonomics of slices and iteration and costly reallocation if you need to pass ownership as a Vec:I'd expect:
The constructor would enforce the invariant and then you'd impl Deref and DerefMut for [T] to gain normal len/is_empty/indexing/iteration, passing as &[T] to other funcs and mutating values (which can't break the invariant).To mutate length while preserving the invariant it's dealers choice e.g.
- add .into_vec() for unwrap/mutate/rewrap
- add invariant preserving mutators of your choice
There's a whole follow up post about this: https://lexi-lambda.github.io/blog/2020/11/01/names-are-not-...
Not sure switching between languages makes for a compelling argument:
"Look how easy it is to accidentally bypass the invariant of a rust newtype by transliterating the data shape into Haskell and deriving a new type". Uh, ok.
If comparing the "risk of accident" between a newtype wrapper whose only role is enforcing the the invariant versus manually reimplementing vector and iterator semantics to use a different layout... I'd say the newtype wins that.
It would be good advice to keep a newtype that enforces an invariant as a single purpose primitive type. A building block and not a place to add other features.
There might be times I'd prefer structural enforcement e.g. something serialisation related. Converting into a non-rust format is what they are doing in their "accident"!
DerefMut would allow you to call `clear()` on `v` violating the invariant
I'd be great is there were a way to shadow methods but even then guarantees would be poor since Vec might add a new method in the future which isn't covered by invariant checks
clear() is a method of std::Vec not std::slice.
DerefMut to [T] not Vec<T>.
Which deref must I use to get most of the existing interface sans `retain()`?
Deref/DerefMut enables implicit type coercion rather than exposing an interface. You can choose the target type and immutable/mutable but not parts of the target type.
You can use all the slice reference methods (that do not require ownership) with:
https://doc.rust-lang.org/std/primitive.slice.htmlIf you DerefMut to a Vec then you won't be able to preserve the invariant.
If you want control over methods to expose then you need wrapper methods for those you want. If you want to expose some of the traits the inner type implements then there are likely derive macros available e.g. with derive_more you could expose just indexing as:
The reason I am asking is because I strongly believe to achieve the semantics of the type, just derefing to a Vec will not be sufficient.
This is great. Alexis King actually stated that, had she known how popular “Parse, Don’t Validate” had been, she would have written it in a language more widely used than Haskell.
With its rich type system, Haskell is the perfect language to demonstrate the dictum.
With its crap type system Python might be even better, in a strange way.
Haskell's strong, static, non-reflective type system tends to make "parse, don't validate" produce code that also looks nicer. Which is great. So great that it steals a bit of the main message's valor.
In Python, though, it's really easy to just let your data be a dynamically typed list of dicts forever. So easy that parsing into something more strongly typed looks like a whole lot of extra effort. Upon looking at that sort of thing many a working Python programmer, myself included, hears the voice of GvR murmuring disparaging things about "academic" programmers down in the pit of their brain.
Which creates an opportunity to demonstrate all the ways the (arguably) more Pythonic way is actually a royal PITA when you try to make your code robust. Handling and reporting data validity errors gets scattered all over the code, which makes it annoying to maintain. Unit test suites get bloated because it's not obvious what inputs a function should be able to handle. Comments and docstrings to help keep track of this stuff begin to proliferate.
I write Python. While I read the original post and nodded along, nothing changed about how I worked until I started playing with Rust. Seeing a strongly typed language force the constraints and stop stupid errors was when the rationale finally clicked for me. Knowing that this function will never blow-up in an unexpected way changes how you think about problems.
You otherwise get accustomed to some wishy-washy blobs of data that get passed around and find it normal. Maybe you include some ad hoc guard rails here and there, which catches the egregious errors, but there is always some lingering uncertainty. Some string that should have been an int, missing key here, the object which never had the validation check, etc
It is like unit-testing - more-up front work, but I could never go back to a world where I did not get these automated assurances. Sadly, I am a grug-brain which could only feel the lesson from personal experience.
I’m a big fan of “Parse dont validate” and I feel like injecting that approach into python has so many benefits.
there’s all the code correctness stuff that we all love. But the biggest benefit is making it so much easier to maintain other parts of the stack.
Following a stack trace into somebody else’s functions and you see that the args are completely untyped or just a bunch of ‘dict[str, Any]’ is the worst. But if you see those inputs as more narrowly typed data classes it makes it so much easier to grok what the function is supposed to do
I had a very hard lesson in that once when I inherited a legacy Clojure application.
Previously I had been somewhat sympathetic to Rich Hickey’s impassioned monologues about making maps first-class and how everything gets easier when you let data just be itself. But when I had to learn my way around code that someone else had written according to that philosophy… hoo boy. Clojure quickly got demoted from being one of my favorite languages to merely being one of my favorite write-only languages.
I'm the author of typedload. For json-type of data that is loaded in dictionaries you can do A LOT of validation at runtime automatically if you define the types.
On the footnote:
> [1] Other languages - like Go or Python - have a runtime check that raises some sort of exception or panic when lst[0] is accessed on an empty list or slice.
Rust has the same thing. Accessing a `Vec` by index goes via the index trait: https://doc.rust-lang.org/std/ops/trait.Index.html#tymethod....
Vec implements Index here: https://github.com/rust-lang/rust/blob/d080e7dff1b0fc5454154...
Vec's Index defers to slice's: https://github.com/rust-lang/rust/blob/d080e7dff1b0fc5454154...
Slice defers to... intrinsics: https://github.com/rust-lang/rust/blob/d080e7dff1b0fc5454154...
Which injects a bounds check: https://github.com/rust-lang/rust/blob/d080e7dff1b0fc5454154...
The bounds check: https://github.com/rust-lang/rust/blob/d080e7dff1b0fc5454154...
So I'd say the footnote is not correct.
You are technically correct :) But that's not the point of the footnote, and you're right that I should update it. The point here is that Python and Go don't have the equivalent of "first", and have the programmer rely on the runtime error in lst[0]. Rust can affort to have a "first" method because of Option.
"parse don't validate" can be rephrased: "expect the type validation from a parser"
Which is plainly moving the problem around, for types. The value validation is a much simpler problem, as a separate application-specific check.
Thank you! I always found the quip "parse, don't validate" to be more confusing than helpful. Parsing doesn't remove the need for validation. All it does is ensures the type is right, but does nothing to validate the values for the domain.
It is also about clear demarcation, of validated stuff vs not yet validated.
Crystal clear clarity is a nice thing to have. (As with everything, there are trade-offs)
Good article on Rust's strengths with types. If you like this, you may be curious how you might build your own parse capabilities. I like the Rust crates Winnow and Nom, and also the Rust traits From and Into.
Another take, from the primary example: This is what `unwrap()` is for. I understand that the author is looking at this from a correctness and safety(?) perspective. For practical purposes, I would unwrap here. If it's less trivial than the example, unwrap with a comment explaining why it's fine.
Another angle: Unfortunately, the `first()` method being fallible here is just an issue of using an imperfect method/datatype here. This is where the author gets in to a non-empty-vec custom type. Then you are balancing using a more correct type that takes custom wiring vs a std lib thing everyone understands and takes no setup. I would lean towards this setup if I were using this non_empty_vec.first() unwrap pattern a number of times in the code base; then the setup would be worth it, at least for my own code bases. If I were exposing this in a lib others would use, I would keep the standard Vec so as to be more transparent for others.
In both views: "This is what unwrap is for" does it for me in all cases I've encountered to date. Maybe for aerospace or safety critical systems, I would have a different take.
A third take: I notice this trend in the rust community. It's not my cup of tea. Keep things simple, easy to maintain, and don't let "correctness" get in the way. In this example, I don't think it gets in the way, but I have seen this mindset lead to it getting in the way, especially in embedded, where mapping the Owernership model to hardware ends up in messy patterns and surprising assertions about embedded-101 concepts like DMA being "unsolved", "no good way", "difficult" etc.
Rust provides tools to make sure specific logic is correct if it passes the compiler. People sometimes go overboard and assume you have to type-maxx your code, regardless of complexity added by doing so.
Disagree. Unwrap works, but it's on the other side of the signature. It would be nice if we could put it all in the callee.
Nonempty type wrappers are a stark reminder that we are missing out on refinement types.
Rust has a proposal for this, in the form of pattern types. It's on nightly, but quite far from being ready for prime time.
I was thinking about its implementation, too, and ended up deciding that it was probably a performance optimization.
`Vec<T>` stores all data on the heap, so getting anything out of it involves a pointer deref and possibly also an array bounds check. This `NonEmpty<T>` type keeps the first element of the list in a location that supports some low-level optimization that might make a significant difference in situations where accessing the first element is much more common than accessing subsequent elements.
...I'm so happy that I write in untyped languages so I don't have to deal with any of this.
But you kind of do.
For a untype language like js array, since it can be empty, you have to either always check the length, the item returned, or have a precondition to know the array is not empty.
All three of those cases is either code or context you’re holding in your head.
All that stuff is equivalent to a type system
You absolutely do have to deal with this in your untyped language otherwise you're just letting edge cases and runtime errors happen. The syntax will be different, it will have to happen at runtime instead of compile time, the chosen wording and language to represent the concepts and the work may even vary, but you are not exempt.
Congratulations on having to implement your checks at runtime, but stupid, repetitive, boring work is kind of why we invented compilers.
Everyone has to deal with this. The only question is whether you deal with it in the type system, through lots of checks at runtime, or buggy behavior when the user encounters cases you didn't envision.
Ignorance is bliss.
I lamented patterns of what I consider over-use of the type system in Rust (This applies to other languages as well). There is a balance: I prefer this over-use/strict use (e.g. implementing a custom non-empty Vec type) over not using typing at all, but a big margin!
Balance in all things?
Someone has to deal with it, either you or the (strongly typed) language.