I work in one of large tech companies, and I can attest that while the idea seems very neat in theory (esp if your schemas are typed), and even if you define an api for defining new building blocks, sooner or later people realize that they need to dynamically adjust parts of the pipeline, and they write components to dynamically set and resolve these, and then other components on top of these components, and then complements for composing components - and now you forced yourself into implementing a weird and hard to debug functional programming language in yaml which is not a place someone wants to find themselves in :'(
one lesson I learned from this: any bit of logic that defines a computation should prefer explicit imperative code (eg python) over configuration, because you are likely to eventually implement an imperative language in that configuration language anyway
> sooner or later people realize that they need to dynamically adjust parts of the pipeline
The customer is the hard part in all of this, but there is respite if you are patient and careful with the tech.
If you are in a situation where you need to go from one SQL database to another SQL database, the # of additional tools required should be zero. Using a merge statement & recursive CTEs per target table, you can transform any schema into any other. Most or all of the actual business logic can reside in the command text - how we filter & project data into the target system.
If we accept the SQL-to-SQL case has a good general solution, I would then ask if it is possible to refactor all problems such that they wind up with this shape in the middle. All of that nasty systems code could then be focused more on loading and extracting data into and out of this regime where it can be trivially sliced & diced. Once you have something in Postgres or SQL Server, you are at the top of the hill. Everything adapts to you at that point. Talking to another instance of yourself - or something that looks & talks like you - is trivial.
The other advantage with this path is that refactoring SQL scripts is something the customer (B2B) can directly manage in many situations. The entire pipeline can live in a single text file that you throw around an email chain. You don't have to teach them things like python, yaml or source control.
yeah, most projects when you spot a config file, its complexity will tend to scale with the increasing complexity of the domain you capture.
so either it's very small/mature and you don't have to worry too much, or in the active development case your config files are pretty much the instruction set of some kind of logical foggy vm... and eventually a whole environment of tools etc. will "compile down" to your config files and you get a pain knot to endlessly massage...
I worked on something very similar for inference on video streams. To avoid the limitations of the config files mentioned in a sibling comment, I added a tool to convert a config to plain Rust. Your primary focus has to be the quality of the Rust API, and the config files are syntactic sugar for the beginning or simpler projects.
I haven’t benchmarked this, but I have recently benchmarked Spark Streaming vs self-rolled Go vs Bento vs RisingWave (which is also in Rust) and RW matched/exceeded self-rolled, and absolutely demolished Bento and Spark. Not even in the same ballpark.
Highly recommend checking RisingWave out if you have real time streaming transformation use cases. It’s open source too.
The benchmark was some high throughput low latency JSON transformations.
Yes, they are similar. ArkFlow is mainly based on DataFusion. Bento actually comes from Benthos. Currently, the ArkFlow project is in the early stages and no performance comparison test has been conducted, but I believe that ArkFlow will outperform them in the long run.
What we found with RPCN (redpanda connect)/old benthos is that most systems are very slow and only cpu intensive things require manual CPU instruction optimizations like the snowflake connector we wrote (https://docs.redpanda.com/redpanda-connect/components/output...). The bulk of it is just about completeness. Go feels like the Perl of the 2020s. Cool little libs for just about everything.
Yes, Arroyo is entirely based on DataFusion, but ArkFlow is not exactly. In the future, ArkFlow will establish a plug-in ecosystem, allowing anyone to process data through plug-ins, not limited to DataFusion.
This isn't quite correct (I'm the creator of Arroyo). We use DataFusion to implement parts of our SQL support (in particular the planner and the expression interpreter) but we have our own dataflow and operators. By contrast Synnada[0] is directly built on DF.
A contrast between Arroyo and systems like Benthos and from what I can tell ArkFlow, is that Arroyo is a "stateful" stream processing engine, which means that we can support things like windows, aggregates, and joins, with exactly-once semantics and fault tolerance, at the cost of significant additional complexity[1].
No worries! We definitely rely heavily on DF (it’s an incredible project!). Part of what makes it so great is its modularity—it’s a toolkit for building sql systems, which is extremely cool.
High Performance: Built on Rust and Tokio async runtime, offering excellent performance and low latency
Multiple Data Sources: Support for Kafka, MQTT, HTTP, files, and other input/output sources
Powerful Processing Capabilities: Built-in SQL queries, JSON processing, Protobuf encoding/decoding, batch processing, and other processors
Extensible: Modular design, easy to extend with new input, output, and processor components
I think a stream processing engine written in rust will have better performance, lower latency, more stable services, lower memory footprint, and cost savings. At the same time, ArkFlow is based on DataFusion implementation, which will put ArkFlow on a strong open source community.
Rust is rather heavy on its copy/clone imposed semantics making it potentially less suitable for low-latency or large data volume processing workloads. Picking Rust for its performance potential only means that you're going to have a harder time beating other native performance-oriented stream processing engines written in either C or C++, if that is your goal of course.
This logic
> written in rust will have better performance, lower latency, ..., lower memory footprint
is flawed and is cargo-cult programming unless you say what are you objectively comparing it against and how you intend to achieve those goals. Picking the right™ language just for the sake of these goals won't get you too far.
> Rust is rather heavy on its copy/clone imposed semantics making it potentially less suitable for low-latency or large data volume processing workloads. Picking Rust for its performance potential only means that you're going to have a harder time beating other native performance-oriented stream processing engines written in either C or C++, if that is your goal of course.
There is absolutely nothing in Rust's semantics preventing you from writing high-performance data processing workloads in it, and in fact it's one of the best languages for that purpose. Beyond that, the usual barrier to entry for working on a product like this written in C++ is incredibly high in part because stability and safety are so critical for these products--which is one of the reasons that in practice they are often written in memory safe languages, where C++ is not even an option. Have you worked on any nontrivial Rust data processing product where "copy/clone imposed semantics" somehow prevented you from getting big performance wins? I'd be very curious to hear about this if so.
Stability and safety are the least of the concerns in data processing and database workloads. That's totally not the reason why we saw an increase of these systems during the 90s and early 00s written in Java or similar alternative languages. It was ease of use, low-entry bar into the ecosystem and generally developer pool accessibility. Otherwise, the cost is the main driver in infrastructure type of software and the reason why we see many of these rewritten exactly in C++. Rust is just another contender here, and it's usually because of the performance and a lot of hype recently, which is fair.
Why do you have to beat a native performance-oriented streaming engine written in C or C++?
Currently, most of the mainstream stream processing engines are written in Java. Sorry, I may not add qualifiers to make you misunderstandings.
Software does not have silver bullets, so does programming languages, and each has its own strengths. I also like to use go and Java to develop software.
So if you don't want to beat native engines in performance what is it that you're trying to solve but Java-based engines don't have? I think it's pretty important to set a vision upfront otherwise you're going to set yourself a trap for a quick failure.
I work in one of large tech companies, and I can attest that while the idea seems very neat in theory (esp if your schemas are typed), and even if you define an api for defining new building blocks, sooner or later people realize that they need to dynamically adjust parts of the pipeline, and they write components to dynamically set and resolve these, and then other components on top of these components, and then complements for composing components - and now you forced yourself into implementing a weird and hard to debug functional programming language in yaml which is not a place someone wants to find themselves in :'(
one lesson I learned from this: any bit of logic that defines a computation should prefer explicit imperative code (eg python) over configuration, because you are likely to eventually implement an imperative language in that configuration language anyway
> sooner or later people realize that they need to dynamically adjust parts of the pipeline
The customer is the hard part in all of this, but there is respite if you are patient and careful with the tech.
If you are in a situation where you need to go from one SQL database to another SQL database, the # of additional tools required should be zero. Using a merge statement & recursive CTEs per target table, you can transform any schema into any other. Most or all of the actual business logic can reside in the command text - how we filter & project data into the target system.
If we accept the SQL-to-SQL case has a good general solution, I would then ask if it is possible to refactor all problems such that they wind up with this shape in the middle. All of that nasty systems code could then be focused more on loading and extracting data into and out of this regime where it can be trivially sliced & diced. Once you have something in Postgres or SQL Server, you are at the top of the hill. Everything adapts to you at that point. Talking to another instance of yourself - or something that looks & talks like you - is trivial.
The other advantage with this path is that refactoring SQL scripts is something the customer (B2B) can directly manage in many situations. The entire pipeline can live in a single text file that you throw around an email chain. You don't have to teach them things like python, yaml or source control.
yeah, most projects when you spot a config file, its complexity will tend to scale with the increasing complexity of the domain you capture.
so either it's very small/mature and you don't have to worry too much, or in the active development case your config files are pretty much the instruction set of some kind of logical foggy vm... and eventually a whole environment of tools etc. will "compile down" to your config files and you get a pain knot to endlessly massage...
Thank you for your valuable experience, I will seriously think about what you said.
Pretty much my take any time I see all the convoluted Bicep and YAML we have since there's a bunch of conditional logic and more in our pipelines.
So far, this was exactly my experience as well. Well said.
I worked on something very similar for inference on video streams. To avoid the limitations of the config files mentioned in a sibling comment, I added a tool to convert a config to plain Rust. Your primary focus has to be the quality of the Rust API, and the config files are syntactic sugar for the beginning or simpler projects.
Very cool! Seems like a Rust version of something like Bento? [1] Have you done any benchmarking against similar stream processing tools?
[1] https://github.com/warpstreamlabs/bento
I haven’t benchmarked this, but I have recently benchmarked Spark Streaming vs self-rolled Go vs Bento vs RisingWave (which is also in Rust) and RW matched/exceeded self-rolled, and absolutely demolished Bento and Spark. Not even in the same ballpark.
Highly recommend checking RisingWave out if you have real time streaming transformation use cases. It’s open source too.
The benchmark was some high throughput low latency JSON transformations.
Thanks for your recommendation.
Yes, they are similar. ArkFlow is mainly based on DataFusion. Bento actually comes from Benthos. Currently, the ArkFlow project is in the early stages and no performance comparison test has been conducted, but I believe that ArkFlow will outperform them in the long run.
Benthos: https://github.com/redpanda-data/benthos
DataFusion: https://github.com/apache/datafusion
What we found with RPCN (redpanda connect)/old benthos is that most systems are very slow and only cpu intensive things require manual CPU instruction optimizations like the snowflake connector we wrote (https://docs.redpanda.com/redpanda-connect/components/output...). The bulk of it is just about completeness. Go feels like the Perl of the 2020s. Cool little libs for just about everything.
Yes, RPCN (redpanda connect)/old benthos is very cool and can solve most of the scenes. Let me tell you quietly that I am using it too.
Arroyo is another one based on DataFusion
Yes, Arroyo is entirely based on DataFusion, but ArkFlow is not exactly. In the future, ArkFlow will establish a plug-in ecosystem, allowing anyone to process data through plug-ins, not limited to DataFusion.
This isn't quite correct (I'm the creator of Arroyo). We use DataFusion to implement parts of our SQL support (in particular the planner and the expression interpreter) but we have our own dataflow and operators. By contrast Synnada[0] is directly built on DF.
A contrast between Arroyo and systems like Benthos and from what I can tell ArkFlow, is that Arroyo is a "stateful" stream processing engine, which means that we can support things like windows, aggregates, and joins, with exactly-once semantics and fault tolerance, at the cost of significant additional complexity[1].
[0] https://www.synnada.ai/ [1] https://www.arroyo.dev/blog/stateful-stream-processing
Arroyo has been designed with more comprehensive consideration.
Sorry, please forgive me for not knowing Arroyo completely.
No worries! We definitely rely heavily on DF (it’s an incredible project!). Part of what makes it so great is its modularity—it’s a toolkit for building sql systems, which is extremely cool.
Yes, whether it is DataFusion, Arroyo, or Bentos, these open source products have made me profit a lot.
High Performance: Built on Rust and Tokio async runtime, offering excellent performance and low latency Multiple Data Sources: Support for Kafka, MQTT, HTTP, files, and other input/output sources Powerful Processing Capabilities: Built-in SQL queries, JSON processing, Protobuf encoding/decoding, batch processing, and other processors Extensible: Modular design, easy to extend with new input, output, and processor components
Is there a product motivation; a deficiency you seek to rectify in existing solutions?
I think a stream processing engine written in rust will have better performance, lower latency, more stable services, lower memory footprint, and cost savings. At the same time, ArkFlow is based on DataFusion implementation, which will put ArkFlow on a strong open source community.
Are there benchmarks you can share? Not discounting Rust, just wondering if you're already seeing some obvious numbers.
Sorry, not yet, but this is the direction ArkFlow is working hard. Rust's own potential will also guide ArkFlow in this direction.
Rust is rather heavy on its copy/clone imposed semantics making it potentially less suitable for low-latency or large data volume processing workloads. Picking Rust for its performance potential only means that you're going to have a harder time beating other native performance-oriented stream processing engines written in either C or C++, if that is your goal of course.
This logic
> written in rust will have better performance, lower latency, ..., lower memory footprint
is flawed and is cargo-cult programming unless you say what are you objectively comparing it against and how you intend to achieve those goals. Picking the right™ language just for the sake of these goals won't get you too far.
> Rust is rather heavy on its copy/clone imposed semantics making it potentially less suitable for low-latency or large data volume processing workloads. Picking Rust for its performance potential only means that you're going to have a harder time beating other native performance-oriented stream processing engines written in either C or C++, if that is your goal of course.
There is absolutely nothing in Rust's semantics preventing you from writing high-performance data processing workloads in it, and in fact it's one of the best languages for that purpose. Beyond that, the usual barrier to entry for working on a product like this written in C++ is incredibly high in part because stability and safety are so critical for these products--which is one of the reasons that in practice they are often written in memory safe languages, where C++ is not even an option. Have you worked on any nontrivial Rust data processing product where "copy/clone imposed semantics" somehow prevented you from getting big performance wins? I'd be very curious to hear about this if so.
Stability and safety are the least of the concerns in data processing and database workloads. That's totally not the reason why we saw an increase of these systems during the 90s and early 00s written in Java or similar alternative languages. It was ease of use, low-entry bar into the ecosystem and generally developer pool accessibility. Otherwise, the cost is the main driver in infrastructure type of software and the reason why we see many of these rewritten exactly in C++. Rust is just another contender here, and it's usually because of the performance and a lot of hype recently, which is fair.
Why do you have to beat a native performance-oriented streaming engine written in C or C++?
Currently, most of the mainstream stream processing engines are written in Java. Sorry, I may not add qualifiers to make you misunderstandings.
Software does not have silver bullets, so does programming languages, and each has its own strengths. I also like to use go and Java to develop software.
So if you don't want to beat native engines in performance what is it that you're trying to solve but Java-based engines don't have? I think it's pretty important to set a vision upfront otherwise you're going to set yourself a trap for a quick failure.
Hi! Brother, I think I will seriously consider what you said and I am honored to communicate with you.
Welcome to follow the latest news from ArkFlow at any time and even participate.
Reminds me of Pathway https://pathway.com/
Good job, this is a rich reference.
Good job brother! What do you think you need to implement before it is production-ready?
Hi,brother! I'm still thinking, but it's certainly not now.
Rust is increasingly becoming the default language for building infrastructure.
I think stability, reliability and high performance are the foundation of infrastructure.