As someone who has made the mistake of using kafka in a non enterprise space - it really seems like the etcd problem where you need more time to run etcd than to run whatever service you're providing.
I previously helped clients setup and run Kafka clusters. Why they'd need Kafka was always our first question, never got a good answer from a single one of them. That's not to say that Kafka isn't useful, it is, in the right setting, but that settings is never "I need a queue". If you need a queue, great, go get RabbitMQ, ZMQ, Redis, SQS, named pipes, pretty anything but Kafka. It's not that Kafka can't do it, but you are making things harder than they needed to be.
Kafka isn’t a queue, it’s a distributed log. A partitioned topic can take very large volumes of message writes, persist them indefinitely, deliver them to any subscriber in-order and at-least-once (even for subscribers added after the message was published), and do all of that distributed and HA.
If you need all those things, there just are not a lot of options.
The way people choose to use feedback on HN never fails to suprise me - we've got a generally intelligent user base here, but the most common type of feedback voting isn't because something is wrong but rather a childish "I don't like it - I want to suppress this comment".
In this case it's something different - this was an honest question, and received two useful replies, so why downvote?! The mental model of people using Kafka is useful to know - in this case the published data being more log-like than stream-like since it's retained per a TTL policy, with each "subscriber" having their own controllable read index.
I'd started using it at v0.8 at a previous adtech company because my problem was "We generate terabytes of events a day we need to process and aggregate and bill on, how the hell do we move this data around reliably?"
The data team I'd inherited had started with NFS and shell scripts, before a brief detour into GlusterFS after NFS proved to be, well, NFS. GlusterFS was no better.
Using S3 was better, but we still hit data loss problems (on our end, not S3
's, to clear) which isn't great when you need to bill on some of that data.
Then I heard about Kafka, bought a copy of I <3 Logs, and decided that maybe Kafka was worth the complexity, and boom, it was. No more data loss, and a happier business management.
I was headhunted for my current company for my Kafka experience. First thing I realised when I looked at the product was - "Ah, we don't need Kafka for this."
But the VP responsible was insistent. So now I spend a lot of time doing education on how to use Kafka properly.
And the very first thing I start with is "Kafka is not a queue. It's a big dumb pipe that does very smart things to move data efficiently and with minimal risk of data loss - and the smartest thing it does, it choosing to be very dumb.
Want to synchronously know if your message was consumed? Kafka don't care. You need a queue."
Have you found a good Head for Kafka to easily query the Topics using a SQL like language? Especially something that can infer table schema from the Schema Registry.
I do not recommend Redis (janky implementation, subscribers drop randomly, java libraries are a crime against humanity) or RabbitMQ (memory issues), ZMQ is not a messaging queue, named pipes are not reliable and what the hell is SQS.
SQS is simple queueing service in AWS. It's ok, guarantees at least one time delivery, but I am not sure how useful it is for large volumes of messages (by this I just mean that we use it for low volume messages and I don't have experience when using larger volumes).
SQS is fantastic at exceptionally high total volumes of messages - you probably can't saturate it. But it's not great for streaming a list of ordered messages. SQS has a FIFO mode but performance will never be what you can get out of Kafka.
Also, SQS isn't pub/sub. Kafka and SQS really have very different use cases.
How have you had so many issues with Redis. We used it at a previous place it was basically bullet proof. That being said we didn’t use Java but Python. Idk.
lettuce - doesn't reconnect properly during redis restarts (1/10 chance)
jedis - subscriptions drop and stop receiving for no reason
however, my latest wrapper for jedis does seem to be holding up and haven't had too many issues, but I have a very robust checking for dropped connections.
IMO, recommending RabbitMQ depends on what language you are using and how well suited the available client libraries are to your use case.
I used RabbitMQ a few years back on a C++ project, and at the time (has anything changed?) the best supported C++ client library seemed to be AMQP-CPP which isn't multi-thread safe, therefore requiring an application interface layer to need to be written to address this in a performant way.
In our case we wanted to migrate a large legacy system from CORBA (point to point) to a more flexible bus-based architecture, so we also had to implement a CORBA-like RPC layer on top of Rabbit, including support for synchronous delivery failure detection, which required more infrastructure to be built on top of AMQP-CPP. In the end the migration was successful, but it felt like we were fighting AMQP-CPP a lot of the way.
I'm a bit hazy on the full details because it was a few years ago, but basically it gets more complicated because you subscribe by installing an async callback which needs to ack/nak messages, needing locking, and will be called from the context of the network event loop that also needs locking. If you do any real work in the message processing callback then you'll be blocking the event loop, so the callback has to defer processing by queuing C++ lambdas capturing the context, and running those in a thread pool.
> Why they'd need Kafka was always our first question, never got a good answer from a single one of them
"To follow the hype train, Bro" is often the real answer.
> If you need a queue, great, go get RabbitMQ, ZMQ, Redis, SQS, named pipes, pretty anything but Kafka.
Or just freaking MQTT.
MQTT has been battle-proven for 25 years, is simple and does perfectly the job if you do not ship GBs of blobs through your messaging system (which you should not do anyway).
It's resume-driven development. It honestly can make sense for both company and employee.
Companies get standard tech stacks people are happy to work with, because working with them gets people experience with tech stacks that are standard at many companies. It's a virtuous cycle.
And sure even if you need just a specific thing, it's often better to go slightly overkill for something that's got millions of stack overflow solutions for common issues figured out. Vs picking some niche thing that you are now 1 of like six total people in the entire world using in prod.
Obviously the dose makes the poison and don't use kafka for your small internal app thing and don't use k8s where docker will do, but also, probably use k8s if you need more than docker instead of using some weird other thing nobody will know about.
That's what happened where I worked. The people making the tech decisions were worried they weren't "keeping up" and committed us all-in on kafka. That decision cost the company millions.
Disclaimer: I'm a dev and I'm not very familiar with the actual maintenance of kafka clusters. But we run the aws managed service version (MSK), and it seems to just pretty much work.
We send terrabytes of data through kafka asynchronously, because of its HA properties and persistent log, allowing consumers to consume in their own time and put the data where it needs to be. So imagine, many apps across our entire stack have the same basic requirement, publish a lot of data which people want to analyse somewhere later. Kafka gives us a single mechanism to do that.
So now my question. I've never used MQTT before. What are the benefits of using MQTT in our setup vs using kafka?
I use MQTT daily. I'm not sure why the commenter suggested it; it is orthogonal to queueing or log streams.
MQTT is a publish/subscribe protocol for large scale distributed messaging, often used in small embedded devices or factories. It is made for efficient transfer of small, often byte sized payloads of IoT device data. It does not replace Kafka or RabbitMQ - messages should be read off of the MQTT broker as quickly as possible. ( I know this from experience - MQTT brokers get bogged down rapidly if there are too many messages "in flight")
A very common pattern is to use MQTT for communications, and then Kafka or RabbitMq for large scale queuing of those messages for downstream applications.
I presume one will want to use https://github.com/eclipse-mosquitto/mosquitto if going that route because I seem to recall the "mainstream" MQTT project doing a rugpull but since I'm not deeply in that community, I don't have substantiating links handy
MQTT and Kafka solve different problems. At my current company, we use both.
Kafka isn’t a queue. It’s overkill to use it as one.
Kafka is a great place to persist data for minutes, hours or days before it’s processed. It fully decouples producers and consumers. It’s also stupidly complex and very hard to operate reliably in an HA configuration.
MQTT is good for when data needs to leave or enter your cloud, but persistence is bolted on (at least it is in mosquitto), so a crash means lost data even though you got a PUBACK.
I'm really unsure where the drive for this is coming from tbh. (cough CFLT share price since IPO, big enterprise customers cough) If this was so desirable, everyone would've jumped ship to Pulsar already.
You're running a distributed system. They aren't simple.
Especially on AWS. AWS is really a double-bladed sword. Yeah, you'll get tutorials to set up whatever distributed system pretty quickly, but your nodes aren't nearly as reliable. Your networking isn't nearly as reliable. Your costs aren't nearly as reliable and administration. Headaches go up in the long run
This is what alot of people don't get. They think Kubernetes is a solution to everything. In reality all k8s is makes the mechanics a little easier.
But if you don't understand distributed systems, it almost makes it worse because its tempting to segment the system across dozens of microservices which all have to talk with each other and synchronize, and the whole thing becomes a buggy, slow clusterfuck.
Dozens of microservices, oh one should be so lucky. Try hundreds where the complexity of the whole system rises non-linerarly as a function of each microservice.
Looks interesting; does it take a different architectural approach than WarpStream did?
That’s not coded “you’re reinventing the wheel”; WarpStream had some significant drawbacks, so I’m truly curious about different approaches in the message-log-backed-by-blob-store space.
Architecturally, there are a lot of the same considerations. A key difference is that we offer streams as a cloud API, and it does not have WarpStream's BYOC split – where some stuff runs in your environment and metadata lives in their cloud – so we can offer lower latencies. We are also not trying to be Kafka API compatible, S2 has its own REST API.
The dimensions we focus on are number of streams (unlimited, so you can do granular streams like per user or session), internet accessibility (you can generate finely-scoped access tokens that can be safely used from clients like CLIs or browsers), and soon also massive read fanout for feed-like use cases.
LinkedIn recently announced that it transitioned from Kafka to Northguard.
Introducing Northguard and Xinfra: scalable log storage at LinkedIn [1] & LinkedIn: Stream Processing 4.16.25 [2]
Northguard doesn’t look like it’s been open sourced? I’d be curious to know how it compares to Apache Pulsar [0]. I feel like I see some similarities reading the LI blog post.
I haven't been following kafka for some years now, but i thought Linkedin were heavily invested in it. What happened? Also what happened to Confluent? Their team were ex-Linkedin members from what i remember?
NorthGuard looks like a clean sheet redesign of Kafka. The OSS Kafka community has taken a long time to implement things like KRaft, which addressed metadata scalability concerns by storing metadata in the brokers themselves (it used to be in a separate data store called ZooKeeper which was operationally complicated). NorthGuard also supports splitting and merging ranges of keys without repartitioning the entire existing dataset. The way records are assigned to partitions is a big problem in running Apache Kafka at scale because it requires predicting the key distribution and number of partitions ahead of time.
Confluent is 11 years old and IPOed several years ago. It was founded by 3 ex-LinkedIn people who originally designed Kafka. 2 of the 3 founders are still at the company. LinkedIn never used Confluent, Confluent was a company founded to sell an enterprise version of the open source project (and later a cloud version).
I worked on this while I was at LI and I think the major selling point back then was Replayability of messages but it was something similar to what you would get with Pub/Sub. We could also have multiple clients listening and processing same messages for their own purposes so you could use the same queue and have different clients process them as they wanted.
Its the ability to replay messages at later notice when needed.
At least this was the reason we decided to use Kafka instead of simple queues.
It was useful when we built new consumer types for the same data we already processed or we knew we gonna have later but cant build now due to prorities.
Kafka's ability to ingest the firehose and present it as a throttle-able consumable to many different applications is great. If you're thinking "just use a database", it's worth noting that SQL databases are _not well suited_ to drinking from a firehose of writes, and that distributed SQL in 2012 was not a thing. Kafka was one of the first systems that fully embraced the dropping of the C from CAP theorem, which was a big step forward for web applications at scale. If you bristle at that, know that using read-replicas of your postgres database present the same correctness problems.
These days though, unless I was at Fortune 100 scale, I'd absolutely turn to Redis Cluster Streams instead. So much simpler to manage and so much cheaper to run.
Also I like Kafka because I met two pretty Russian girls in San Francisco a decade back and the group we were in played a game where we described what the company we worked for did in the abstract, and then tried to guess the startup. They said "we write distributed streaming software", I guessed "confluent" immediately. At the time confluent was quite new and small. Fun night. Fun era.
For a long time I've wondered if we could just invent an extension for Postgres that allow physically ordered, append-only tables.
The main two things that makes Postgres less suitable for Kafka-type logs is that tables aren't very efficient for sequentially ordered data, and that deletion incurs bloat until vacuumed. You could solve both by providing a new table engine (table access method), although I'm not sure you can control heap storage placement to the degree desired for a physically ordered table. But you could also do a lot of tricks to make it delete faster (append only means no updates; just prune from the head without MVCC when provably safe against concurrent reads?) and make filtering faster.
Kafka is of course more than that, but I bet you can get quite far with this.
> turn to Redis Cluster Streams instead. So much simpler to manage and so much cheaper to run
I don't have any experience with Redis Cluster Streams. Could you please tell us how it is simpler to manage? IMO, installing and managing a Kafka cluster in a non Fortune 100 scale is simple enough: run 1 java command for zookeeper, run another java command for a broker (with recent version of Kafka, zookeeper is not needed anymore). The configuration files are not very simple but not very complicated either. When we have another machine, we can run another broker on it.
Redis Cluster Streams is cheaper to run because it's written in C, doesn't need a VN to run? Or because its messages are stored in RAM not SSD?
I haven’t used Kafka since the change to remove zookeeper, it’s likely they’re more or less on par now. Redis gets a win because most shops already have Redis, it’s already trusted and installed, just waiting for its first XADD.
> Kafka was one of the first systems that fully embraced the dropping of the C from CAP theorem, which was a big step forward for web applications at scale.
Could you expand on this - when does it drop C? Are you referring to cases where you write to Kafka without waiting for all replicas to acknowledge the write? (acks=1)
And why was it a big step - what other systems didn't embrace dropping the C?
Well at the time (and this is still largely true), people were very insistent on ACID compliance in databases. Obviously this made sense of many applications, but became a bottleneck at huge scale. Being able to be eventually consistent became a golden feature. It was worked around by using eg read replicas in production, as SQL replication breaks the Correctness in favor of the Availability. Kafka’s “acks=1” is part of the story yes, but simply writing events to be eventually processed also accomplishes “dropping correctness”.
Native support for dropping correctness in SQL is tricky, see Transaction Isolation Levels, but I mostly mean in overall web architecture, rather than specifically in one database or the other.
> that SQL databases are _not well suited_ to drinking from a firehose of writes
Now I’m wondering if we’re all overthinking this when we could just use rendezvous hashing and a bunch of database servers with a heap table called “eventlog” and be done with it…
NATS is very good. It's important to distinguish between core NATS and Jetstream, however.
Core NATS is an ephemeral message broker. Clients tell the server what subjects they want messages about, producers publish. NATS handles the routing. If nobody is listening, messages go nowhere. It's very nice for situations where lots of clients come and go. It's not reliable; it sheds messages when consumers get slow. No durability, so when a consumer disconnects, it will miss messages sent in its absence. But this means it's very lightweight. Subjects are just wildcard paths, so you can have billions of them, which means RPC is trivial: Send out a message and tell the receiver to post a reply to a randomly generated subject, then listen to that subject for the answer.
NATS organizes brokers into clusters, and clusters can form hub/spoke topologies where messages are routed between clusters by interest, so it's very scalable; if your cluster doesn't scale to the number of consumers, you can add another cluster that consumes the first cluster, and now you have two hubs/spokes. In short, NATS is a great "message router". You can build all sorts of semantics on top of it: RPC, cache invalidation channels, "actor" style processes, traditional pub/sub, logging, the sky is the limit.
Jetstream is a different technology that is built on NATS. With Jetstream, you can create streams, which are ordered sequences of messages. A stream is durable and can have settings like maximum retention by age and size. Streams are replicated, with each stream being a Raft group. Consumers follow from a position. In many ways it's like Kafka and Redpanda, but "on steroids", superficially similar but just a lot richer.
For example, Kafka is very strict about the topic being a sequence of messages that must be consumed exactly sequentially. If the client wants to subscribe to a subset of events, it must either filter client-side, or you have some intermediary that filters and writes to a topic that the consumer then consumes. With NATS, you can ask the server to filter.
Unlike Kafka, you can also nack messages; the server keeps track of what consumers have seen. Nacking means you lose ordering, as the nacked messages come back later. Jetstream also supports a Kafka-like strictly ordered mode. Unlike Kafka, clients can choose the routing behaviour, including worker style routing and deterministic partitioning.
Unlike Kafka's rigid networking model (consumers are assigned partitions and they consume the topic and that's it), as with NATS, you can set up complex topologies where streams get gatewayed and replicated. For example, you can streams in multiple regions, with replication, so that consumers only need to connect to the local region's hub.
While NATS/Jetstream has a lot of flexibility, I feel like they've compromised a bit on performance and scalability. Jetstream clusters don't scale to many servers (they recommend max 3, I think) and large numbers of consumers can make the server run really hot. I would also say that they made a mistake adopting nacking into the consuming model. The big simplification Kafka makes is that topics are strictly sequential, both for producing and consuming. This keeps the server simpler and forces the client to deal with unprocessable messages. Jetstream doesn't allow durable consumers to be strictly ordered; what the SDK calls an "ordered consumer" is just an ephemeral consumer. Furthermore, ephemeral consumers don't really exist. Every consumer will create server-side state. In our testing, we found that having more than a few thousand consumers is a really bad idea. (The newest SDK now offers a "direct fetch" API where you can consume a stream by position without registering a server-side consumer, but I've not yet tried it.)
Lastly, the mechanics of the server replication and connectivity is rather mysterious, and it's hard to understand when something goes wrong. And with all the different concepts — leaf nodes, leaf clusters, replicas, mirrors, clusters, gateways, accounts, domains, and so on — it's not easy to understand the best way to design a topology. The Kafka network model, by comparison, is very simple and straightforward, even if it's a lot less flexible. With Kafka, you can still build hub/spoke topologies yourself by reading from topics and writing to other topics, and while it's something you need to set up yourself, it's less magical, and easier to control and understand.
Where I work, we have used NATS extensively with great success. We also adopted Jetstream for some applications, but we've soured on it a bit, for the above reasons, and now use Redpanda (which is Kafka-compatible) instead. I still think JS is a great fit for certain types of apps, but I would definitely evaluate the requirements carefully first. Jetstream is different enough that it's definitely not just a "better Kafka".
We're particularly interested in NATS' feature of working with individual messages and have been bitten by Kafka's "either process the entire batch or put it back for later processing", which doesn't work for our needs.
Interested if Redpanda is doing better than either.
Redpanda is fantastic, but it has the exact same message semantics as Kafka. They don't even have their own client; you connect using the Kafka protocol. Very happy with it, but it does have the same "whole batch or nothing" approach.
NATS/Jetstream is amazing if it fits your use case and you don't need extreme scalability. As I said before, it offers a lot more flexibility. You can process a stream sequentially but also nack messages, so you get the best of both worlds. It has deduping (new messages for the same subject will mark older ones as deleted) and lots of other convenience goodies.
Thank you so much again. Yes, we are not Google scale, our main priority is durability and scalability but only up to a (I'd say fairly modest) point. I.e. be able to have one beefy NATS server do it all and only add a second one when things start getting bad. Even 3 servers we'd see as a strategic defeat + we have data but again, very far from Google scale.
We've looked at Redis streams but me and a few others are skeptical as Redis is not known for good durability practices (talking about the past; I've no idea if they pivoted well in the last years) and sadly none of us has any experience with MQTT -- though we heard tons of praise on that one.
But our story is: some tens of terabytes of data, no more than a few tens of millions of events / messages a day, aggressive folding of data in multiple relational DBs, and a very dynamic and DB-heavy UI (I will soon finish my Elixir<=>Rust SQLite3 wrapper so we're likely going to start sharding the DB-intensive customer data to separate SQLite3 databases and I'm looking forward to spearheading this effort; off-topic). For our needs NATS Jetstream sounds like the exactly perfect fit, though time will tell.
I still have the nagging feeling of missing out on still having not tried MQTT though...
At that scale, Jetstream should work well. In my experience, Jetstream's main performance weakness is the per-stream/consumer overhead: Too many and NATS ends up running too hot due to all the state updates and Raft traffic. (Each stream is a Raft group, but so is each consumer.)
If its tens of TB in a stream, then I've not personally stored that much data in a stream, but I don't see why it wouldn't handle it. Note that Jetstream has a maximum message size of 1MB (this is because Jetstream uses NATS for its client/server protocol, which has that limit), which was a real problem for one use case I had. Redpanda has essentially no upper limit.
Note that number of NATS servers isn't the same as the replication factor. You can have 3 servers and a replication factor of 2 if you want, which allows more flexibility. Both consumers and streams have their own replication factors.
The other option I have considered in the past is EMQX, which is a clustered MQTT system written in Erlang. It looks nice, but I've never used it in production, and it's one of those projects that nobody seems to be talking about, at least not in my part of the industry.
Well I work mainly with Elixir in the last 10-ish years (with a lot of Rust and some Golang here and there) so EMQX would likely be right up my alley.
Do you have any other recommendations? The time is right for us and I'll soon start evaluating. I only have NATS Jestream and MQTT on my radar so far.
Kafka I already used and rejected for the reasons above ("entire batch or nothing / later").
As for data, I meant tens of terabytes of traffic on busy days, sorry. Most of the time it's a few hundred gigs. (Our area is prone to spikes and the business hours matter a lot.) And again, that's total traffic. I don't think we'd have more than 10-30GB stored in our queue system, ever. Our background workers aggressively work through the backlog and chew data into manageable (and much smaller chunks) 24/7.
And as one of the seniors I am extremely vigilant of payload sizes. I had to settle on JSON for now but I push back, hard, on any and all extra data; anything and everything that can be loaded from DB or even caches is delegated as such with various IDs -- this also helps us with e.g. background jobs that are no longer relevant as certain entity's state moved too far forward due to user interaction and the enriching job no longer needs to run; when you have only references in your message payload, this enables and even forces the background job to load data exactly at the time of its run and not assume a potentially outdated state.
Anyhow, I got chatty. :)
Thank you. If you have other recommendations, I am willing to sacrifice a little weekend time to give them a cursory research. Again, utmost priority is 100% durability (as much as that is even possible of course) and mega ultra speed is not of the essence. We'll never have even 100 consumers per stream; I haven't ever seen more than 30 in our OTel tool dashboard.
EDIT: I should also say that our app does not have huge internal traffic; it's a lot (Python wrappers around AI / OCR / others is one group of examples) but not huge. As such, our priorities for a message queue are just "be super reliable and be able to handle an okay beating and never lose stuff" really. It's not like in e.g. finance where you might have dozens of Kafka clusters and workers that hand off data from one Kafka queue to another with a ton of processing in the meantime. We are very far from that.
Jetstream is written in Go and the Go SDK is very mature, and has all the support one needs to create streams and consumers; never used it from Elixir, though. EMQX's Go support looks less good (though since it's MQTT you can use any MQTT client).
Regarding data reliability, I've never lost production data with Jetstream. But I've had some odd behaviour locally where everything has just disappeared suddenly. I would be seriously anxious if I had TBs of stream data I couldn't afford to lose, and no way to regenerate it easily. It's possible to set up a consumer that backs up everything to (say) cloud storage, just in case. You can use Benthos to set up such a pipeline. I think I'd be less anxious with Kafka or Redpanda because of their reputation in being very solid.
Going back to the "whole batch or nothing", I do see this as a good thing myself. It means you are always processing in exact order. If you have to reject something, the "right" approach is an explicit dead-letter topic — you can still consume that one from the same consumer. But it makes the handling very explicit. With Jetstream, you do have an ordered stream, but the broker also tracks acid/nacks, which adds complexity. You get nacks even if you never do it manually; all messages have a configurable ack deadline, and if your consumer is too slow, the message will be automatically bounced. (The ack delay also means if a client crashes, the message will sit in the broker for up to the ack delay before it gets delivered to another consumer.)
But of course, this is super convenient, too. You can write simpler clients, and the complicated stuff is handled by the broker. But having written a lot of these pipelines, my philosophy these days is that — at least for "this must not be allowed to fail" processing, I prefer something that is explicit and simpler and less magical, even if it's a bit less convenient to write code for it. Just my 2 cents!
This is getting a bit long. Please do reach out (my email is in my profile) if you want to chat more!
I got really pissed off with their field CTO for essentially trying to pull the wool over my eyes regarding performance and reliability.
Essentially their base product (NATs) has a lot of performance but trades it off for reliability. So they add Jetstream to NATs to get reliability, but use the performance numbers of pure NATs.
I got burned by MongoDB for doing this to me, I won’t work with any technology that is marketed in such a disingenuous way again.
Yes, I meant Jetstream (I even typed it but second guessed myself, my mistake) I’m typing these when I get a moment as I’m at a wedding- so I apologise.
The issue in the docs was that there are no available Jetstream numbers, so I talked over a video call to the field CTO, who cited the base NATs numbers to me, and when I pressed him on if it was with Jetstream he said that it was without: so I asked for them with Jetstream enabled and he cited the same numbers back to me. Even when I pressed him again that “you just said those numbers are without Jetstream” he said that it was not an issue.
So, I got a bit miffed after the call ended, we spent about 45 minutes on the call and this was the main reason to have the call in the first place so I am a bit bent about it. Maybe its better now, this was a year ago.
This doesn’t really support your position as far as most readers are concerned - it sounds like a disconnect. If they didn’t do this in any ad copy or public docs it’s not really in Mongo territory.
I’m telling you why I am skeptical of any tech that intentionally obfuscates trade-offs, I’m not making a comparison on which of these is worse; and I don’t really care if people take my anecdote seriously either: because they should make their own conclusions.
However it might help people go in to a topic about performance and reliability from a more informed position.
I don't doubt your experience. But I think it might have been more just that guy, than NATS in general.
The other day i was listening to a podcast with their ceo from maybe 6 months ago, and he talked quite openly about how jetstream and consumers add considerable drag compared to normal pubsub. And, more generally, how users unexpectedly use and abuse nats, and how they've been able to improve things as a result.
The delivery guarantees section alone doesn’t make me trust it. You can do at least once or at most once with kafka. Exactly once is mostly a lie, it depends on the downstream system: unless going back to the same system, the best you can do is at least once with idempotancy
I wanted to write a comment on this topic, but after several tries this thread is where I ended up because it describes my sentiment as well.
The arguments in the article are very compelling. But as soon as you choose Kafka you realize the things you hate.
Many of the reasons are stupid things - like it uncovers otherwise unimportant bugs in your client code. Or that it just makes experimenting a hassle because it enforces poking around in lots of different places to do something. Or that writing and maintaining the compulsory integration test takes weeks of your time.
Sure - you can replay your data - but not until you have fixed all the issues for that special case in your receiving service.
I think maybe my main gripe (for us) was that it was a difficult to get an understanding what is actually inside your pipe. Much easier to have that in a solid state in s3?
At the end of they day you get annoyed because it slows you down. In particular when you are a small localized team.
Totally agree with this. I’ll add that replaying your data needs special tooling to 1) find the correct offsets on each topic, and 2) spin up whatever daemon will consume that data out-of-band from normal processing, and shut it down when completed.
I don’t remember where I read this, but someone made the observation that writing a stream processing system is about 3x harder than writing a batch system, exactly for all the reasons you mentioned. I’m looking at replacing some of our Kafka usage with a clickhouse table that’s ordered and partitioned by insertion time, because if I want to do stuff with that data stream, at least I can do a damn SQL query.
Yes I'll happily extend that to 10x more difficult.
At least compared to building a batched pipeline with SQL. I think you should really think hard whether you really need a streaming pipeline. And even if you find that you do, it may be worthwhile to make a batched pipeline as your first implementation.
I did exactly what you describe in my previous job. In the beginning with reluctance from our architects who wanted to keep banging the dead horse and did not understand the power of SQL "SQL is not real programming, engineers write java" (ok maybe I deserve a straw-man yellow card here, they don't deserve all of that). But I think they understood after a while.
With AWS Athena and Airflow. Good luck, consider me your distant moral support.
This might be hyperbolic, but I think Kafka (or at least the concept of event driven architecture for sharing data across many systems) is one of the most under-rated technologies. It's used at a lot of big corps but is never talked about.
Like modeling everything as graphs, it is often a trap. Just because a model is flexible enough to capture all of your use cases doesn't mean you should use it. In fact you should prefer less flexible more constrained models that are simpler.
Distributed ledgers like Bitcoin do store transitions as events, and that's because nodes need the transitions to valid the next state. So you might say that Bitcoin is a widely run piece of software, using an event driven architecture.
Not every system needs to have all of it's state transitions available for for efficient reading. And often times you can derive the state transition from the previous and next state if you really need them (Git does that). Even though Git can compute all of the state transitions for the system, it doesn't store events, it stores snapshots.
The biggest strength of Kafka in my opinion is consumer groups.I have been using it since 2016 in at least 3 projects and it never failed, not that big workloads though (~100 messages/sec max). However it is a bit difficult to monitor and manage using only the out of the box applications.
Startup founder here -- we tried it, and it feels bloated (Java!), bureaucratic and overcomplicated for what it is. Something like Redis queues or even ZMQ probably suffices for 90% of use cases. Maybe in hyper-scaled applications that need to be ultraperformant (e.g., realtime trading, massive streaming platforms) is where Kafka comes into play.
If you are using this sort of redis queue (https://redis.io/glossary/redis-queue/) with PUSH/POP vs fan-out you're working on a very different sort of problem than what Kafka is built for.
Like the article says, fan-out is a key design characteristic. There are "redis streams" now but they didn't exist back then. The durability story and cluster stories aren't as good either, I believe, so they can probably take you so far but won't be as generally suitable depending on where your system goes in the future. There are also things like RedPanda that speak Kafka w/o the Java.
However, if you CAN run on a single node w/o worrying about partitioning, you should do that as long as you can get away with it. Once you add multiple partitions ordering becomes hard to reason about and while there are things like message keys to address that, they have limitations and can lead to hotspotting and scaling bottlenecks.
But the push/pop based systems also aren't going to give you at-least-once guarantees (looks like Redis at least has a "pop+push" thing to move to a DIFFERENT list that a single consumer would manage but that seems like it gets hairy for scaling out even a little bit...).
I'm curious, what exactly feels bloated about Java? I don't feel like the Java language or runtime are particularly bloated, so I'm guessing you're referring to some practices/principles that you often see around Java software?
Whatever efficiency may hypothetically be possible with Java, you can in-fact spot a real world Java program in the wild by looking for the thing taking up 10x the memory it seems like it should need… when idle.
Yes yes I’m sure there are exceptions somewhere but I’ve been reading Java fans using benchmarks to try to convince me that I can’t tell which programs on my computer are Java just by looking for the weirdly slow ones, when I in fact very much could, for 25ish years.
Java programs have a feel and it’s “stuttery resource hog”. Whatever may be possible with the platform, that’s the real-world experience.
I held the same view as you when I was 22, more than 15 years ago.
With over 15 years of professional experience since then, my perspective has shifted: Java demonstrates its strength when stability, performance, and scalability are required (e.g. bloody enterprise)
A common misconception comes from superficial benchmarking. Many focus solely on memory consumption, which often provides a distorted picture of actual system efficiency.
I can point to EU-scale platforms that have reliably served over 100 million users for more than a decade without significant issues. The bottleneck is rarely the language itself, it is the depth of the team’s experience.
> Many focus solely on memory consumption, which often provides a distorted picture of actual system efficiency.
When other languages can do the same thing with an order of magnitude less RAM, any other efficencies in the system tend to be overshadowed by that and be the sticking point in peoples memories.
You may argue that holding on to this extra memory makes subsequent calls and reads quicker etc, but in my experience generally people are willing to sacrifice milliseconds to gain gigabytes of memory.
node is a notable exception. Compared to java node is a hellhole. the standard library is non-existent, most libraries are a buggy mess, the build system is horrible...in fact there is no reliable build system that solves all your typical problems in 1 app. The list goes on.
The JVM eats a chunk of memory in order to make its garbage collector more efficient. Think of it like Linux's page cache.
I haven't worked with too much Java, but I suspect that the distaste many have for it is due to its wide adoption by large organizations and the obfuscating "dressed up" tendency of the coding idioms used in large organizations.
The runtime isn't inherently slow, but maybe it's easier to write slow programs in Java.
Technically kind of true but at the same time Android apps are predominantly Java/Kotlin. It speaks more to Java just having a bad desktop story. But it’s also why Android devices need 2x the ram
While for typical backend situations, reference counting has a crazy high throughput overhead, doing atomic inc/decs left and right, that instantly trashes any kind of cache, and does it in the mutator thread that would do the actual work, for the negligible benefit of using less memory. Meanwhile a tracing GC can do (almost) all its work in another thread, not slowing down the actually important business task, and with generational GCs cleaning up is basically a no-op (of just saying that this region can now be reused).
It's a tradeoff as everything in IT.
Also, iPhone CPUs are always a generation ahead, than any android CPU, if not more. So it's not really Apples to oranges.
That would be a compelling counter if and only if languages like Java actually beat other languages in throughput. In practice that doesn’t seem to be the case and the reasons for that seem to be:
* languages like c++ and Rust simply don’t allocate as much as Java, instead using value types. Even C# is better here with value types being better integrated.
* languages like c++ and Rust do not force atomic reference counting. Rust even offers non atomic ref counting in the standard library. You also only need to atomic increment / decrement when ownership is being transferred to a thread - that isn’t quite as common depending on the structure of your code. Even swift doesn’t do too badly here because of the combination of compiler being able to prove the permission of eliding the need for reference counting altogether and offering escape hatches of data types that don’t need it.
* c++, Rust, and Swift can access lower level capabilities (eg SIMD and atomics) that let them get significantly higher throughput.
* Java’s memory model implies and requires the JVM to insert atomic accesses all over the place you wouldn’t expect (eg reading an integer field of a class is an atomic read and writing it is an atomic write). This is going to absolutely swamp any advantage of the GC. Additionally, a lot of Java code declares methods synchronized which requires taking a “global” lock on the object which is expensive and pessimistic for performance as compared with the fine-grained access other languages offer.
* there’s lots of research into ways of offering atomic reference counts more cheaply (called biased RC) which can safely avoid needing to do an atomic operation in places completely transparently and safely provided the conditions are met .
I’ve yet to see a Java program that actually gets higher throughput than Rust so the theoretical performance advantage you claim doesn’t appear to manifest in practice.
Of course with manual memory management you may be able to write more efficient programs, though it is not a given, and comes at the price of a more complicated and less flexible programming model. At least with Rust, it is actually memory safe, unlike c++.
- ref counting still has worse throughout than a tracing GC, even if it is single-threaded, and doesn't have to use atomic instructions. This may or may not matter, I'm not claiming it's worse, especially when used very rarely as is the case with typical c++/rust programs.
> You also only need to atomic increment / decrement when ownership is being transferred to a thread
Java can also do on-stack replacement.. sometimes.
- regarding lower level capabilities, java does have an experimental Vector API for simd. Atomics are readily available in the language.
- Java's memory model only requires 32-bit writes to be "atomic" (though in actuality the only requirement is to not tear - there is no happens before relation in the general case, and that's what is expensive), though in practice 64-bit is also atomic, both of which are free on modern hardware. Field acces is not different from what rust or c++ does, AFAIK in the general case. And `synchronized` is only used when needed - it's just syntactic convenience. This depends on the algorithm at hand, there is no difference between the same algorithm written in rust/c++ vs java from this perspective. If it's lockless, it will be lockless in Java as well. If it's not, than all of them will have to add a lock.
The point is not that manual memory can't be faster/more efficient. It's that it is not free, and comes at a non-trivial extra effort on developers side, which is not even a one-time thing, but applies for the lifetime of the program.
> ref counting still has worse throughout than a tracing GC, even if it is single-threaded, and doesn't have to use atomic instructions. This may or may not matter, I'm not claiming it's worse, especially when used very rarely as is the case with typical c++/rust programs.
That’s a bold claim to make that doesn’t seem to actually be true from my experience. Your 5ghz CPU can probably do ~20 billion non atomic reference adjustments whereas your GC system has to have atomics all over the place or it won’t work and atomics have parasitic performance on unrelated code due to bus locks and whatnot.
> Java can also do on-stack replacement.. sometimes
That’s not what this is. It’s called hybrid RC and it applies always provided you follow the rules.
> The point is not that manual memory can't be faster/more efficient. It's that it is not free, and comes at a non-trivial extra effort on developers side, which is not even a one-time thing, but applies for the lifetime of the program.
The argument here is not about developer productivity - the specific claim is that the Java GC lets you write higher throughput code than you would get with Rust or C++. That just isn’t true so you end up sacrificing throughput AND latency AND peak memory usage. You may not care and are fine with that tradeoff, but claiming you’re not making that tradeoff is not based on the facts.
> the specific claim is that the Java GC lets you write higher throughput code than you would get with Rust or C++
No, that has never been the specific claim - you can always write more efficient code with manual memory management, given enough time, effort and skill. I wasn't even the one who brought up c++ and rust. Like literally I write this twice in my comment.
What I'm talking about is reference counting as a GC technique vs tracing as a GC technique, all else being equal - it would be idiotic to compare these two if no other "variable" is fixed. (Oh and I didn't even mention the circular references problem, which means you basically have to add a tracing step either-way unless you restrict your language so that it can't express circular stuff).
As for the atomic part, sure, if all it would do is non-atomic increments then CPUs would be plenty happy. And you are right that depending on how the tracing GC is implemented, it will have a few atomic instructions. What you may miss is how often each run. On almost every access, vs every once in a while on a human timescale. Your OS scheduler will also occasionally trash the performance of your thread. But this is the actually apples to oranges comparison, and both techniques can do plenty of tweaks to hide certain tradeoffs, at the price of something else.
And I also mention that the above triad of time, skill and effort is not a given and is definitely not free.
In Rust there’s no forcing of any specific garbage collection mechanism. You’re free to do whatever and there’s many high performance crates to let you accomplish this. Even in Swift this is opt-in.
As for “skill” this is one thing that’s super hard to optimize for. All I can do is point to existence proofs that there’s no mainstream operating system, browser or other piece of high performance code written in Java and it’s all primarily C/C++ with some assembly with Rust starting to take over the C/C++ bits. And at the point where you’re relegating Java to being “business” logic, there’s plenty of languages that are better suited for that in terms of ergonomics.
Sure, but I think that people often fall into the trap of imagining a problem that nicely fits a RAII model, where each lifetime is statically knowable. This is either due to having a specific problem, or because we decided on a specific constraint.
Java is used in HFT (well, there are two types of "high frequency", one where general purpose CPUs are already too slow, where it obviously doesn't apply (neither do rust or c++)) - but sure, I wouldn't write a runtime or other piece of code in Java where absolute control over the hardware is required. But that's a small niche only. What about large distributed systems/algorithms? Why is Java over-represented in this niche (e.g. Kafka, Elasticsearch, etc)?
> And at the point where you’re relegating Java to being “business” logic, there’s plenty of languages that are better suited for that in terms of ergonomics.
> Java’s memory model implies and requires the JVM to insert atomic accesses all over the place you wouldn’t expect (eg reading an integer field of a class is an atomic read and writing it is an atomic write).
AFAIK that doesn’t really happen. They won’t insert atomic accesses anywhere on real hardware because the cpu is capable of doing that atomically anyway.
> Additionally, a lot of Java code declares methods synchronized which requires taking a “global” lock on the object which is expensive and pessimistic for performance as compared with the fine-grained access other languages offer.
What does this have to do with anything? Concurrency requires locks. Arc<T> is a global lock on references. “A lot” of Java objects don’t use synchronized. I’d even bet that 95-99% of them don’t.
> Concurrency requires locks. Arc<T> is a global lock on references
Concurrency does not require locks. There’s entire classes of lock free and wait free algorithms. Arc<T> is also not a lock - it uses atomics to manage the reference counts and no operation on an Arc needs to wait on a lock (it is a lock-free container).
> “A lot” of Java objects don’t use synchronized. I’d even bet that 95-99% of them don’t.
Almost all objects that are used in a concurrent context will likely feature synchronized, at least historically. That’s why Hashtable was split into HashMap (unsynchronized) and ConcurrentHashMap (no longer using synchronized). Thats why you have StringBuffer which was redone into StringBuilder.
Ok I mispoke on Arc because I was being hasty; but you're still being pedantic. Concurrency still requires locks. Wait/lock free algorithms can't cover the entirety of concurrency. Rust ships with plenty of locks in std::sync and to implement a ConcurrentHashMap in Rust you would still need to lock. In fact it doesn't even look like Rust supplies concurrent collections at all. So what are we even talking about here? This is still a far cry from "a lot of Java objects use global synchronized locks".
No, that’s an overly strong statement - concurrency doesn’t necessarily require locks even though they can be convenient to express it. You could have channels and queues to transfer data and ownership between threads. Not a lock in sight as queues and channels can be done lock free. The presence of locks in the Rust standard library says nothing other than it’s a very common concurrency tool, not that it’s absolutely required.
> and to implement a ConcurrentHashMap in Rust you would still need to lock
There’s many ways to implement concurrency safe hashmaps (if you explicitly needs such a data structure as the synchronization mechanism) without locks. Notably RCU is such a mechanism (really neat mechanism developed for the kernel although not super user friendly yet or common in userspace) and there are also generational garbage techniques available (kind of similar to tracing GC conceptually but implemented just for a single data structure). A common and popular crate in Rust for this is DashMap which doesn’t use locks and is a concurrency safe hashmap.
> A common and popular crate in Rust for this is DashMap which doesn’t use locks and is a concurrency safe hashmap.
Still not in the standard library. The only way in Rust is to use a global lock around map. Seems to be worse than the situation in Java. You could implement the same thing and use a third party library in Java too. So your original point of "everything uses a global lock" is "overly strong"
You’ve now degraded the conversation into a very very weird direction. You made a claim that concurrency required locks. It simply does not and I have an existence proof of Dashmap as a hashmap that doesn’t have any locks anywhere.
The strengths and weaknesses of the standard library aren’t relevant. But if we’re going there, the reason they’re not in the Rust standard library is likely in practice concurrent data structures are an anti pattern - putting a lock around a data structure doesn’t suddenly solve higher order race conditions which is something a lot of Java programmers seem to believe because the standard library encourages this kind of thinking.
As for “my comment” about “global lock” (your words not mine), it’s that the implicit lock that’s available on every object is a bad idea for highly concurrent code (not to mention the implicit overhead that implies for every part of the object graph regardless of it being needed anywhere). Don’t get me wrong - Java took a valiant effort to define a solid memory model for concurrency when the field was still young. Many of the ideas didn’t pan out and are antipatterns these days for high performing code. Of course none of that pertains to the original point of the conversation - tracing GCs have significantly more overhead in practice because they’re very difficult to be opt in, carry quite a penalty if not, Rc/Arc is much better as it’s possible to do opt-in when you need shared ownership (which isn’t always), and in practice loops don’t come up often enough to matter and when they do there’s still solutions. In other words tracing GCs drop huge amounts of performance on the floor and you can read all the comments to see how the claims are “it’s more efficient than Rc”, or “performance is free” or even “it doesn’t matter because the programmer is more efficient”. I’d buy the efficiency argument when the only alternative was C/C++ and came with serious memory safety baggage, but not any of the others and memory safety without sacrificing performance of C++ in my view is a solved problem with Rust.
It depends how you implement reference counting. In Rust the atomic inc-dec operations can be kept at a minimum (i.e. only for true changes in lifecycle ownership) because most accesses are validated at compile time by the borrow checker.
Is this an AI-generated answer? Most of these are not even true, although I still would prefer Go for micro-services. I'll address just a bunch and to be clear - I'm not even a big Java fan.
- Quarkus with GraalVM compiles your Java app to native code. There is no JIT or warm up, memory footprint is also low. By the way, the JVM Hotspot JIT can actually make your Java app faster than your Go or Rust app in many cases [citation needed] exactly due to the hot path optimizations it does.
- GC tuning - I don't even know who does this. Maybe Netflix or some trading shops? Almost no one does this nowadays and with the new JVM ZGC [0] coming up, nobody would need to.
> You can’t ship a minimal standalone binary without pulling in a JVM.
- You'd need JRE actually, e.g., 27 MB .MSI for Windows. That's probably the easiest thing to install today and if you do this via your package manager, you also get regular security fixes. Build tools like Gradle generate a fully ready-to-execute directory structure for your app. If you got the JRE on your system, it will run.
> Dependency management and classpath conflicts historically plagued Java
The keyword here is "historically". Please try Maven or Gradle today and enjoy the modern dependency management. It just works. I won't delve into Java 9 modules, but it's been ages since I last saw a class path issue.
> J2EE
Is someone still using this? It is super easy writing a web app with Java+Javalin for example. The Java library and frameworks ecosystem is super rich.
> “Write once, run anywhere” costs: The abstraction layers that make Java portable also add runtime weight and overhead.
Like I wrote above, the HotSpot JIT is actually doing the heavy lifting for your in real time. These claims are baseless without pointing to what "overhead" is meant in practice.
I believe Netflix has moved to ZGC with no tuning. Their default setup is to set the min/max heap to the same size, enable always pretouch, and to use transparent huge pages [0]. GC tuning is something of the past. Once automatic heap sizing for ZGC and G1 land you won’t even need to set the heap size [1][2]. They’ll still use more ram because the vm and jit, but the days of it holding on to ram when it doesn’t need it should be over.
> taking up 10x the memory it seems like it should need… when idle.
The JVM tends to hold onto memory in order to make things faster when it does wind up needing that memory for actual stuff. However, how much it holds on to, how the GC is setup, etc are all tunable parameters. Further, if it's holding onto memory that's not being used, these are prime candidates to be stored in virtual memory which is effectively free.
Note that the memory problem you mentioned is not really a problem in fact. It is just how managed memory works in Java. Just run .gc() and you'll see what I'm talking about. It reserves memory which you can see on the charts but it is not necessarily used memory.
Well, you might want to read up on how OSs handle memory under the hood, and that virtual memory != physical, and that task manager and stuff like that can't show the real memory usage.
Nonetheless, tracing GCs do have some memory overhead in exchange for better throughput. This is basically the same concept as using a buffer.
-----
And can you tell which of these websites use Java from "the feel"? AWS cloud infra, a significant chunk of Google, Apple's backends, Alibaba, Netflix?
I’ve never seen another basic tech used to develop other programs that’s so consistently obvious from its high resource use and slowness, aside from the modern web platform (Chrome, as you put it). It was even more obvious back when we had slower machines, of course, but Java still stands out. It may be able to calculate digits of Pi in a tight loop about as fast as C, but real programs are bloated and slow.
5% of "whose server-side programming language we know"
From the website.
And 76% of these websites is PHP, which seems to mean.. they can determine PHP more easily for a website (nonetheless, there are indeed a lot of WordPress sites, but not this amount).
Right, so Im assuming that as you are saying 'Half the web runs on java', maybe you know more about what websites are using in their backend? Care to share where you are getting this information from?
That doesn't match my experience in the last 15 years working for 3 companies (one was a big enterprise, one medium sized and one startup)
Maybe I have been lucky, or that the practice is more common in certain countries or eco systems? Java has been a very productive language for me, and the code has been far from the forced pattern usage that I have read horror stories about.
Have you gotten to use loom/virtual threads? I’ve heard pretty interesting stuff about em, but haven’t really spent the time to get into it yet. It’s pretty exciting and tbh gives me an easy elevator pitch to JVM world for people outside of it
If you have a use-case where you currently allocate ~1K threads mostly waiting on I/O switching to virtual threads is a one-liner ("Thread.ofVirtual()" instead of
"Thread.ofPlatform()"). No more golang envy for sure.
Depending on how much memory is used by the Thread stack (presumably 1M-512K by default, allegedly 128K with Alpine base images) that's your 1G-500M heap space usage improvement right off the bat.
The migration from JDK17 to JDK21 was uneventful in production. The only issue is limited monitoring as a thread dump will not show most virtual threads and the micrometer metrics will not even collect the total number of active virtual threads. It's supposed to work better in JDK24.
The Spring Framework directly supports virtual threads with "spring.threads.virtual.enabled=true" but I haven't tried it to comment.
But the perf is not reliable. If you want latency and throughput, idiomatic Rust will give you better properties. Interestingly even will Go for some reason has better latency guarantees I believe even though it’s GC is worse than Java.
There is not much point talking about throughput and latency in the abstract - they are very often opposing goals, you can make one better at the expense of the other.
Go's GC is tuned more for latency at the expense of throughput (not sure if it still applies, but Go was quite literally stopping the "business" mutator threads when utilisation got higher to be able to keep up with the load - Java's default GC is tuned for a more balanced approach, but it can deliver it at very high congestion rates as well. Plus it has a low-latency focused GC which has much better latency guarantees, and it trades off some throughput in a consistent manner, so you can choose what fits best). The reason it might sometimes be more efficient than Java is simply value types - it doesn't create as much garbage, so doesn't need as good a GC in certain settings.
Rust code can indeed be better at both metrics for a particular application, but it is not automatically true, e.g. if the requirements have funny lifetimes and you put a bunch of ARC's, then you might actually end up worse than a modern tracing GC could do. Also, future changes to the lifetimes may be more expensive (even though the compiler will guide you, you still have to make a lot of recursive changes all across the codebase, even if it might be a local change only in, say, Java), so for often changing requirements like most business software, it may not be the best choice (even though I absolutely love Rust).
The problem is that writing genuinely performant Java code requires that you drop most if not all of the niceties of writing Java. At that point, why write Java at all? Just find some other language that targets the JVM. But then you're already treading such DIY and frictionful waters that just adopting some other cross-platform language/runtime isn't the worst idea.
> The problem is that writing genuinely performant Java code requires that you drop most if not all of the niceties of writing Java
Such as? The only area where you have to "drop" features is high-frequency trading, where they often want to reach a steady-state for the trading interval with absolutely no allocations. But for HFT you would have to do serious tradeoffs for every language.
In my experience, vanilla java is more than fine for almost every application - you might just benchmark your code and maybe add some int arrays over an Integer list, but Java's GC is an absolute beast, you don't have to baby it at all.
>The problem is that writing genuinely performant Java code requires that you drop most if not all of the niceties of writing Java. At that point, why write Java at all?
The reason is quite well known. Supporting multiple languages is a cost. If you only have to support one language, everything is simpler and cheaper.
With Java, you can write elegant code these days, rely on ZGC, not really worry too much about GC and get excellent performance with quick development cycles for most of your use cases. Then with the same language and often in the same repo (monorepo is great) you can write smarter code for your hot path in a GC free manner and get phenomenal performance.
And you get that with only having one build system, one CI pipeline, one deployment system, some amazing profiling and monitoring tooling, a bunch of shared utility code that you don't have to duplicate, and a lot more benefits.
That's the reason to choose Java.
Of course, if you're truly into HFT space, then they'll be writing in C, C++ or on FPGAs.
Depends on the program (especially the framework used) and the GC being used. I can write a java program and set it up in a way that it runs faster than almost everything else. For example in a serverless architecture where you need fast startup and small programs you can choose __not__ to use a GC and run ephemeral Java scripts. It starts and finishes running faster than you can blink.
None of what you said are any of the reasons given that it WAS written in Java already [0] but rewrote it all in Go explicitly because of its performance, concurrency and single binary distribution characteristics.
Those were enough technical advantages to abandon any thought of a production-grade version of k8s in Java.
> the anti patterns weren’t enough we also observe how Kubernetes has over 20 main() functions in a monolithic “build” directory. We learn how Kubernetes successfully made vendoring even more challenging than it already was, and discuss the pitfalls with this design. We look at what it would take to begin undoing the spaghetti code that is the various Kubernetes binaries built from github.com/kubernetes/kubernetes
It seems to me that perhaps it wasn’t the languages fault but the authors.
If you have this stance to technology ("feels bloated", meme-level knowledge on Java), then I really hope that you are not responsible for technical decisions.
Publishing an event to Kafka puts it “out there” in a way that guarantees it won’t be lost and allows any number of interested consumers, including the data warehouse, to deal with it at their leisure (subject to retention period which is typically like 72h). For us, your Kafka topics and their schemas are as much a part of your API as your gRPC IDLs. Something like Redis or 0MQ feels more appropriate for internal coordination between instances of the same service, or at least a producer that has a specific consumer in mind.
It doesn't have to be 'hyper-scaled' to be needed, unless we have widely different definitions of hyper scale. Access logs from a few thousand servers with medium traffic will push you past any single instance service, and Kafka works great for that workload.
> Maybe in hyper-scaled applications that need to be ultraperformant (e.g., realtime trading, massive streaming platforms) is where Kafka comes into play.
Kafka is used because the Java folks don't want to learn something new due to job security, even though there are faster and compatible alternatives that exist today.
Rather use Redpanda, than continue to use Kafka and then complain about how resource intensive it is alongside zookeeper and all the circus the comes with it and make AWS smile as you're losing hundreds of thousands a month.
Very few people want to re-create their production AWS MSK cluster from scratch. And that's the only way currently. MSK can usually upgrade Kafka brokers with minor performance degradation but not for this particular change.
Couldn’t disagree more… if you go the ZMQ you are left alone handling many things you get in Kafka for free. If you have any sort of big data problems then good luck. You are going to reinvent the wheel.
It's neither. Only the thumbnail background is AI generated.
The look of the page -- that's Substack's default UI, you can't control it too much. The other images are created by me.
I'm simply curious what parts give that "cheap" look so I can improve. On Reddit I've had massive amounts of downvotes because they guess the content is AI, when in fact no AI is used in the creation process at all.
One guess I have is the bullet points + bolding combo. Most AIs use a ton of that, and rightly so, because it aids in readability.
Yeah, I think those things do add to the impression and you should get rid of them. I'd suggest gluing some of those shorter paragraphs into longer ones too.
Kafka was created at LinkedIn when it was already quite a large web platform, to solve the problem of distributing a unique newsfeed to a wide audience… so while they weren’t Google, they were still closer to Google than the targets of the “UNPHAT” expression. You can apply UNPHAT to most of the people using Kafka today, but it’s not fair to apply it to LinkedIn when they created it.
> In 2010, LinkedIn had 90 million members. Today, we serve over 1.2 billion members on LinkedIn. Unsurprisingly, this increase has created some challenges over the years, making it difficult to keep up with the rapid growth in the number, volume, and complexity of Kafka use cases. Supporting these use-cases meant running Kafka at a scale of over 32T records/day at 17 PB/day on 400K topics distributed across 10K+ machines within 150 clusters.
I believe these companies save literally every ounce of data they can find. Once you have the infra and teams for it, it seems easy to make a case for storing something.
Similarly, Uber has shared they push 89 GB/s through Kafka - 7.7 PB/s. People always ask me - what is a taxi/food-delivery app storing so much
17PB = about 567 of those drives... being totally filled... per day.
I was hoping somebody would come and say this is a simple spelling error or something.
The cost of the drives alone seems astronomical, let alone the logistics of the data center keeping up with storing that much data.
EDIT: I have just realised that they are probably only processing at this speed, rather than storing it, can anyone confirm if they store all the logs they process?
I would assume storage varies greatly. I know that LinkedIn quoted an average read fanout ratio of 5.5x in Kafka, meaning each byte was read 5.5x times. Assuming that is still true, we ought to divide by 6.5x to get to the daily write amount
That comes out to 87 disks a day. Assuming a 7 day retention period (this is on the high side), it’s not unthinkable to have a 600-1800 disk deployment (accounting for replication copies)
> That comes out to 87 disks a day. Assuming a 7 day retention period (this is on the high side), it’s not unthinkable to have a 600-1800 disk deployment (accounting for replication copies)
Yep. Whole week can be easily stored in 1-2 racks.
I read it and looked at the block diagrams. I still don't get it. You have "data integration problems". Many software share data. Use database. Problems solved.
My only complaint with this article is that it seems to be implying kafka that linkedIn's problem couldn't have been solved with a bunch of off-the-shelf tools.
So solve "ETLs into a data warehouse are hard to make low-latency and hard to manage in a large org" by... just hypothetical better "off the shelf tools". Or "don't want low latency because you're 'just' a recruiting tool, so who cares how quickly you can get insights into your business."
Go back to the article, it wasn't about event-sourcing or replacing a DB for application code.
> It’s just a social network for recruiters it’s not as complicated as they like to pretend.
Dismissing this as «just a social network» understates the real constraints: enormous scale, global privacy rules, graph queries, near-real-time feeds and abuse controls. Periodic DB queries can work at small scale, but at high volume they either arrive late or create bursts that starve the primary. Capturing changes once and pushing them through a distributed transaction log such as Kafka evens out load, improves data timeliness and lets multiple consumers process events safely and independently. It does add operational duties – schema contracts, idempotency and retention – yet those are well-understood trade-offs. The question is not push versus pull in the abstract, but which approach meets the timeliness, fan-out and reliability required.
> You don’t need push, it’s just a performance optimization that almost never justifies using a whole new tool.
It is not about drama but about fit for purpose at scale.
Pull can work well for modest workloads or narrow deltas, especially with DB features such as incremental materialised views or change tables. At large scale, periodic querying becomes costly and late: you either poll frequently and hammer the primary, or poll infrequently and accept stale data. Even with cursoring and jitter, polls create bursty load and poor tail latencies.
Push via change data capture into a distributed log such as Kafka addresses such pain points. The log decouples producers from consumers, smooths load, improves timeliness and lets multiple processors scale independently and replay for backfills. It also keeps the OLTP database focused on transactions rather than fan-out reads.
This is not free: push introduces operational work and design care – schema contracts, ordering being per-partition, duplicate delivery and idempotency, back-pressure and retention governance including data-protection deletes. The usual mitigations are the outbox pattern, idempotent consumers, DLQ's and documented data contracts. The data processing complexity now belongs in each consumer, not the data processing engine (e.g. a DB).
Compute–storage separation in modern databases raises single-cluster ceilings for storage and read scale, yet it does not solve single-writer limits or multi-region active-active writes. For heavy write fan-out and near-real-time propagation, a CDC-to-log pipeline remains the safer bet.
To sum it up, both pull and push are valid – engineering is all about each specific use case assessment and the trade-off analysis. For small or bounded scopes, a well-designed pull loop is simpler. As scale, fan-out and timeliness requirements grow, push delivers better timeliness, correctness and operability.
> LinkedIn used site activity data (e.g. someone liked this, someone posted this)1 for many things - tracking fraud/abuse, matching jobs to users, training ML models, basic features of the website (e.g who viewed your profile, the newsfeed), warehouse ingestion for offline analysis/reporting and etc.
Who controls the database? Is it the fraud/abuse team responsible for the migrations? Does the ML team tell the Newsfeed team to stop doing so many writes because it's slowing things down?
> According to LinkedIn's engineers, Kafka had become increasingly difficult to manage at LinkedIn's scale (32T records/day, 17 PB/day, 400K topics, 150 clusters).
Wtf is LinkedIn doing that they create 17 PB/day?????
In my experience, Apache Kafka must be understood not as an isolated messaging tool, but as a comprehensive data streaming platform. Its successful implementation demands a holistic approach that encompasses performance, governance, and lifecycle management. I have consistently found that simply adopting the technology without a robust supporting architecture is an ineffective practice that leads to operational challenges.
Based on my work managing large-scale Kafka environments across critical sectors, I have identified that their stability and efficiency are upheld by a set of essential practices and tools. These are the non-negotiable pillars for success:
Health Checks & Observability: Proactive cluster health monitoring and complete visibility into the data flow are paramount.
Failure Management: Implementing dedicated portals and processes for handling Dead-Letter Queues (DLQs) ensures that no critical information is lost during failures.
Automation & DevOps: I leverage Strimzi for Kubernetes-native cluster management, orchestrating it through ArgoCD and GitOps practices. This ensures consistent, secure, and repeatable deployments.
The correct application of these engineering principles allows for remarkable results. For instance, at a large fashion retail group, I successfully scaled an environment to handle a peak traffic of 480,000 TPS. This high-availability system is efficiently maintained by a lean operational team of just two junior-to-mid-level professionals.
From my perspective, success in adopting Kafka is determined by the business context and the maturity of the applied software engineering. The investment in a well-planned architecture and a robust support ecosystem has a clear return, paying for itself through a significant reduction in operational costs (OPEX) within an estimated two-year period.
Taming Kafka isn't about new, complex secrets. It's about applying the same robust software engineering and architecture fundamentals we've relied on for +50 years (Software Engineering). The platform is new (2011), the principles are not.
No, man, this wasn't done by an LLM. I actually do Kafka implementations and follow exactly the same script I described.
Perhaps the fact that I'm not a native English speaker may have caused this confusion. I just made sure my text was written correctly in English with a translator.
Non-native English speakers are 100% welcome on Hacker News, and the rest of us are amazed by how good your English usually is.
Automated comments are what people here feel strongly about rejecting. It isn't just the LLM-written ones that trigger such reactions, but also comments that have been filtered through translators/editors/checkers and so on. Readers are becoming hypersensitive to these, so it would be much better to just write in your own voice. The benefit of authenticity outweighs the cost of a few grammar or spelling errors.
As someone who has made the mistake of using kafka in a non enterprise space - it really seems like the etcd problem where you need more time to run etcd than to run whatever service you're providing.
I previously helped clients setup and run Kafka clusters. Why they'd need Kafka was always our first question, never got a good answer from a single one of them. That's not to say that Kafka isn't useful, it is, in the right setting, but that settings is never "I need a queue". If you need a queue, great, go get RabbitMQ, ZMQ, Redis, SQS, named pipes, pretty anything but Kafka. It's not that Kafka can't do it, but you are making things harder than they needed to be.
Kafka isn’t a queue, it’s a distributed log. A partitioned topic can take very large volumes of message writes, persist them indefinitely, deliver them to any subscriber in-order and at-least-once (even for subscribers added after the message was published), and do all of that distributed and HA.
If you need all those things, there just are not a lot of options.
Perhaps the best terse summary of Kafka I have come across yet.
Finally a valuable answer thank you
Why do you say log rather than just publish and subscribe?
Clients don’t have to subscribe to latest messages, but rather can request any available offset range.
the log stays on kafka for replay until your per log retention settings delete it
The way people choose to use feedback on HN never fails to suprise me - we've got a generally intelligent user base here, but the most common type of feedback voting isn't because something is wrong but rather a childish "I don't like it - I want to suppress this comment".
In this case it's something different - this was an honest question, and received two useful replies, so why downvote?! The mental model of people using Kafka is useful to know - in this case the published data being more log-like than stream-like since it's retained per a TTL policy, with each "subscriber" having their own controllable read index.
> Please don't comment about the voting on comments. It never does any good, and it makes boring reading.
https://news.ycombinator.com/newsguidelines.html#comments
Same goes for metacomments about voting.
What do you think about Temporal?
Okay for small numbers of high value jobs (e.g uber trips or food deliveries etc), prohibitively expensive for anything you need even a few k/sec of.
HERO
I'd started using it at v0.8 at a previous adtech company because my problem was "We generate terabytes of events a day we need to process and aggregate and bill on, how the hell do we move this data around reliably?"
The data team I'd inherited had started with NFS and shell scripts, before a brief detour into GlusterFS after NFS proved to be, well, NFS. GlusterFS was no better.
Using S3 was better, but we still hit data loss problems (on our end, not S3 's, to clear) which isn't great when you need to bill on some of that data.
Then I heard about Kafka, bought a copy of I <3 Logs, and decided that maybe Kafka was worth the complexity, and boom, it was. No more data loss, and a happier business management.
I was headhunted for my current company for my Kafka experience. First thing I realised when I looked at the product was - "Ah, we don't need Kafka for this."
But the VP responsible was insistent. So now I spend a lot of time doing education on how to use Kafka properly.
And the very first thing I start with is "Kafka is not a queue. It's a big dumb pipe that does very smart things to move data efficiently and with minimal risk of data loss - and the smartest thing it does, it choosing to be very dumb.
Want to synchronously know if your message was consumed? Kafka don't care. You need a queue."
Have you found a good Head for Kafka to easily query the Topics using a SQL like language? Especially something that can infer table schema from the Schema Registry.
Do you have a blog somewhere? Where do I learn how to use Kafka properly? I like the idea behind it, but its quirks...not so much.
ZMQ is not a managed queue. It’s networking library.
I do not recommend Redis (janky implementation, subscribers drop randomly, java libraries are a crime against humanity) or RabbitMQ (memory issues), ZMQ is not a messaging queue, named pipes are not reliable and what the hell is SQS.
SQS is simple queueing service in AWS. It's ok, guarantees at least one time delivery, but I am not sure how useful it is for large volumes of messages (by this I just mean that we use it for low volume messages and I don't have experience when using larger volumes).
SQS is fantastic at exceptionally high total volumes of messages - you probably can't saturate it. But it's not great for streaming a list of ordered messages. SQS has a FIFO mode but performance will never be what you can get out of Kafka.
Also, SQS isn't pub/sub. Kafka and SQS really have very different use cases.
Agreed, I was just trying to answer the parent's question as to what it is.
How have you had so many issues with Redis. We used it at a previous place it was basically bullet proof. That being said we didn’t use Java but Python. Idk.
lettuce - doesn't reconnect properly during redis restarts (1/10 chance) jedis - subscriptions drop and stop receiving for no reason
however, my latest wrapper for jedis does seem to be holding up and haven't had too many issues, but I have a very robust checking for dropped connections.
IMO, recommending RabbitMQ depends on what language you are using and how well suited the available client libraries are to your use case.
I used RabbitMQ a few years back on a C++ project, and at the time (has anything changed?) the best supported C++ client library seemed to be AMQP-CPP which isn't multi-thread safe, therefore requiring an application interface layer to need to be written to address this in a performant way.
In our case we wanted to migrate a large legacy system from CORBA (point to point) to a more flexible bus-based architecture, so we also had to implement a CORBA-like RPC layer on top of Rabbit, including support for synchronous delivery failure detection, which required more infrastructure to be built on top of AMQP-CPP. In the end the migration was successful, but it felt like we were fighting AMQP-CPP a lot of the way.
Out of curiousity what was the issue with just wrapping the AMQP-CPP pub/sub calls around a mutex?
I'm a bit hazy on the full details because it was a few years ago, but basically it gets more complicated because you subscribe by installing an async callback which needs to ack/nak messages, needing locking, and will be called from the context of the network event loop that also needs locking. If you do any real work in the message processing callback then you'll be blocking the event loop, so the callback has to defer processing by queuing C++ lambdas capturing the context, and running those in a thread pool.
> Why they'd need Kafka was always our first question, never got a good answer from a single one of them
"To follow the hype train, Bro" is often the real answer.
> If you need a queue, great, go get RabbitMQ, ZMQ, Redis, SQS, named pipes, pretty anything but Kafka.
Or just freaking MQTT.
MQTT has been battle-proven for 25 years, is simple and does perfectly the job if you do not ship GBs of blobs through your messaging system (which you should not do anyway).
It's resume-driven development. It honestly can make sense for both company and employee.
Companies get standard tech stacks people are happy to work with, because working with them gets people experience with tech stacks that are standard at many companies. It's a virtuous cycle.
And sure even if you need just a specific thing, it's often better to go slightly overkill for something that's got millions of stack overflow solutions for common issues figured out. Vs picking some niche thing that you are now 1 of like six total people in the entire world using in prod.
Obviously the dose makes the poison and don't use kafka for your small internal app thing and don't use k8s where docker will do, but also, probably use k8s if you need more than docker instead of using some weird other thing nobody will know about.
That's what happened where I worked. The people making the tech decisions were worried they weren't "keeping up" and committed us all-in on kafka. That decision cost the company millions.
> That decision cost the company millions.
And 5 years later the responsible of the decision left the company with a giant pile of mess behind him/her.
But let's see things positively: he can now add "Kafka at scale" on the CV.
That is exactly what happened.
> Or just freaking MQTT.
Disclaimer: I'm a dev and I'm not very familiar with the actual maintenance of kafka clusters. But we run the aws managed service version (MSK), and it seems to just pretty much work.
We send terrabytes of data through kafka asynchronously, because of its HA properties and persistent log, allowing consumers to consume in their own time and put the data where it needs to be. So imagine, many apps across our entire stack have the same basic requirement, publish a lot of data which people want to analyse somewhere later. Kafka gives us a single mechanism to do that.
So now my question. I've never used MQTT before. What are the benefits of using MQTT in our setup vs using kafka?
I use MQTT daily. I'm not sure why the commenter suggested it; it is orthogonal to queueing or log streams.
MQTT is a publish/subscribe protocol for large scale distributed messaging, often used in small embedded devices or factories. It is made for efficient transfer of small, often byte sized payloads of IoT device data. It does not replace Kafka or RabbitMQ - messages should be read off of the MQTT broker as quickly as possible. ( I know this from experience - MQTT brokers get bogged down rapidly if there are too many messages "in flight")
A very common pattern is to use MQTT for communications, and then Kafka or RabbitMq for large scale queuing of those messages for downstream applications.
> it is orthogonal to queueing or log streams.
That is currently the problem.
A lot of usage of Kafka I have seen in the wild are not for log stream or queing but deployed as a simple pub/sub messaging service because "why not".
Thank you.
I presume one will want to use https://github.com/eclipse-mosquitto/mosquitto if going that route because I seem to recall the "mainstream" MQTT project doing a rugpull but since I'm not deeply in that community, I don't have substantiating links handy
MQTT and Kafka solve different problems. At my current company, we use both.
Kafka isn’t a queue. It’s overkill to use it as one.
Kafka is a great place to persist data for minutes, hours or days before it’s processed. It fully decouples producers and consumers. It’s also stupidly complex and very hard to operate reliably in an HA configuration.
MQTT is good for when data needs to leave or enter your cloud, but persistence is bolted on (at least it is in mosquitto), so a crash means lost data even though you got a PUBACK.
KIP-932[0] adds queue semantics for Kafka. You may still want to use another queue though: as always, no one size fits all.
[0] https://cwiki.apache.org/confluence/display/KAFKA/KIP-932%3A...
I CAN drive a Ferrari to the grocery store.
Sure, it can do it. But it's not efficient or what it's good at.
I'm really unsure where the drive for this is coming from tbh. (cough CFLT share price since IPO, big enterprise customers cough) If this was so desirable, everyone would've jumped ship to Pulsar already.
I think our salesmen were happier to sell kafka, because it's enterprisey. Redis is better? Well, now we use kafka and redis.
If you're running a distributed system...
You're running a distributed system. They aren't simple.
Especially on AWS. AWS is really a double-bladed sword. Yeah, you'll get tutorials to set up whatever distributed system pretty quickly, but your nodes aren't nearly as reliable. Your networking isn't nearly as reliable. Your costs aren't nearly as reliable and administration. Headaches go up in the long run
This is what alot of people don't get. They think Kubernetes is a solution to everything. In reality all k8s is makes the mechanics a little easier.
But if you don't understand distributed systems, it almost makes it worse because its tempting to segment the system across dozens of microservices which all have to talk with each other and synchronize, and the whole thing becomes a buggy, slow clusterfuck.
Dozens of microservices, oh one should be so lucky. Try hundreds where the complexity of the whole system rises non-linerarly as a function of each microservice.
With Kafka, at least the naming is on point
You might like what we are building with https://s2.dev :)
Looks interesting; does it take a different architectural approach than WarpStream did?
That’s not coded “you’re reinventing the wheel”; WarpStream had some significant drawbacks, so I’m truly curious about different approaches in the message-log-backed-by-blob-store space.
Architecturally, there are a lot of the same considerations. A key difference is that we offer streams as a cloud API, and it does not have WarpStream's BYOC split – where some stuff runs in your environment and metadata lives in their cloud – so we can offer lower latencies. We are also not trying to be Kafka API compatible, S2 has its own REST API.
The dimensions we focus on are number of streams (unlimited, so you can do granular streams like per user or session), internet accessibility (you can generate finely-scoped access tokens that can be safely used from clients like CLIs or browsers), and soon also massive read fanout for feed-like use cases.
LinkedIn recently announced that it transitioned from Kafka to Northguard. Introducing Northguard and Xinfra: scalable log storage at LinkedIn [1] & LinkedIn: Stream Processing 4.16.25 [2]
[1]: https://www.linkedin.com/blog/engineering/infrastructure/int...
[2]: https://www.youtube.com/watch?v=RDV6-MUVEbQ
Northguard doesn’t look like it’s been open sourced? I’d be curious to know how it compares to Apache Pulsar [0]. I feel like I see some similarities reading the LI blog post.
0: https://pulsar.apache.org/
I haven't been following kafka for some years now, but i thought Linkedin were heavily invested in it. What happened? Also what happened to Confluent? Their team were ex-Linkedin members from what i remember?
NorthGuard looks like a clean sheet redesign of Kafka. The OSS Kafka community has taken a long time to implement things like KRaft, which addressed metadata scalability concerns by storing metadata in the brokers themselves (it used to be in a separate data store called ZooKeeper which was operationally complicated). NorthGuard also supports splitting and merging ranges of keys without repartitioning the entire existing dataset. The way records are assigned to partitions is a big problem in running Apache Kafka at scale because it requires predicting the key distribution and number of partitions ahead of time.
Confluent is 11 years old and IPOed several years ago. It was founded by 3 ex-LinkedIn people who originally designed Kafka. 2 of the 3 founders are still at the company. LinkedIn never used Confluent, Confluent was a company founded to sell an enterprise version of the open source project (and later a cloud version).
Confluent is expensive and I don’t believe LI used them; they did use OSS kafka. Im guessing that after being acquired by MS they explored other tech.
I worked on this while I was at LI and I think the major selling point back then was Replayability of messages but it was something similar to what you would get with Pub/Sub. We could also have multiple clients listening and processing same messages for their own purposes so you could use the same queue and have different clients process them as they wanted.
Its the ability to replay messages at later notice when needed.
At least this was the reason we decided to use Kafka instead of simple queues.
It was useful when we built new consumer types for the same data we already processed or we knew we gonna have later but cant build now due to prorities.
Kafka's ability to ingest the firehose and present it as a throttle-able consumable to many different applications is great. If you're thinking "just use a database", it's worth noting that SQL databases are _not well suited_ to drinking from a firehose of writes, and that distributed SQL in 2012 was not a thing. Kafka was one of the first systems that fully embraced the dropping of the C from CAP theorem, which was a big step forward for web applications at scale. If you bristle at that, know that using read-replicas of your postgres database present the same correctness problems.
These days though, unless I was at Fortune 100 scale, I'd absolutely turn to Redis Cluster Streams instead. So much simpler to manage and so much cheaper to run.
Also I like Kafka because I met two pretty Russian girls in San Francisco a decade back and the group we were in played a game where we described what the company we worked for did in the abstract, and then tried to guess the startup. They said "we write distributed streaming software", I guessed "confluent" immediately. At the time confluent was quite new and small. Fun night. Fun era.
For a long time I've wondered if we could just invent an extension for Postgres that allow physically ordered, append-only tables.
The main two things that makes Postgres less suitable for Kafka-type logs is that tables aren't very efficient for sequentially ordered data, and that deletion incurs bloat until vacuumed. You could solve both by providing a new table engine (table access method), although I'm not sure you can control heap storage placement to the degree desired for a physically ordered table. But you could also do a lot of tricks to make it delete faster (append only means no updates; just prune from the head without MVCC when provably safe against concurrent reads?) and make filtering faster.
Kafka is of course more than that, but I bet you can get quite far with this.
> turn to Redis Cluster Streams instead. So much simpler to manage and so much cheaper to run
I don't have any experience with Redis Cluster Streams. Could you please tell us how it is simpler to manage? IMO, installing and managing a Kafka cluster in a non Fortune 100 scale is simple enough: run 1 java command for zookeeper, run another java command for a broker (with recent version of Kafka, zookeeper is not needed anymore). The configuration files are not very simple but not very complicated either. When we have another machine, we can run another broker on it.
Redis Cluster Streams is cheaper to run because it's written in C, doesn't need a VN to run? Or because its messages are stored in RAM not SSD?
I haven’t used Kafka since the change to remove zookeeper, it’s likely they’re more or less on par now. Redis gets a win because most shops already have Redis, it’s already trusted and installed, just waiting for its first XADD.
Love the story!
> Kafka was one of the first systems that fully embraced the dropping of the C from CAP theorem, which was a big step forward for web applications at scale.
Could you expand on this - when does it drop C? Are you referring to cases where you write to Kafka without waiting for all replicas to acknowledge the write? (acks=1)
And why was it a big step - what other systems didn't embrace dropping the C?
Well at the time (and this is still largely true), people were very insistent on ACID compliance in databases. Obviously this made sense of many applications, but became a bottleneck at huge scale. Being able to be eventually consistent became a golden feature. It was worked around by using eg read replicas in production, as SQL replication breaks the Correctness in favor of the Availability. Kafka’s “acks=1” is part of the story yes, but simply writing events to be eventually processed also accomplishes “dropping correctness”.
Native support for dropping correctness in SQL is tricky, see Transaction Isolation Levels, but I mostly mean in overall web architecture, rather than specifically in one database or the other.
> that SQL databases are _not well suited_ to drinking from a firehose of writes
Now I’m wondering if we’re all overthinking this when we could just use rendezvous hashing and a bunch of database servers with a heap table called “eventlog” and be done with it…
> Kafka's ability to ingest the firehose and present it as a throttle-able consumable to many different applications is great.
I sue Kafka precisely for that. Redis Cluster Streams have AOF persistence logs as I see from the doc. How stable it is?
Does anyone use https://nats.io here? I have heard good things about it. I would love to hear about the comparisons between nats.io and kafka
NATS is very good. It's important to distinguish between core NATS and Jetstream, however.
Core NATS is an ephemeral message broker. Clients tell the server what subjects they want messages about, producers publish. NATS handles the routing. If nobody is listening, messages go nowhere. It's very nice for situations where lots of clients come and go. It's not reliable; it sheds messages when consumers get slow. No durability, so when a consumer disconnects, it will miss messages sent in its absence. But this means it's very lightweight. Subjects are just wildcard paths, so you can have billions of them, which means RPC is trivial: Send out a message and tell the receiver to post a reply to a randomly generated subject, then listen to that subject for the answer.
NATS organizes brokers into clusters, and clusters can form hub/spoke topologies where messages are routed between clusters by interest, so it's very scalable; if your cluster doesn't scale to the number of consumers, you can add another cluster that consumes the first cluster, and now you have two hubs/spokes. In short, NATS is a great "message router". You can build all sorts of semantics on top of it: RPC, cache invalidation channels, "actor" style processes, traditional pub/sub, logging, the sky is the limit.
Jetstream is a different technology that is built on NATS. With Jetstream, you can create streams, which are ordered sequences of messages. A stream is durable and can have settings like maximum retention by age and size. Streams are replicated, with each stream being a Raft group. Consumers follow from a position. In many ways it's like Kafka and Redpanda, but "on steroids", superficially similar but just a lot richer.
For example, Kafka is very strict about the topic being a sequence of messages that must be consumed exactly sequentially. If the client wants to subscribe to a subset of events, it must either filter client-side, or you have some intermediary that filters and writes to a topic that the consumer then consumes. With NATS, you can ask the server to filter.
Unlike Kafka, you can also nack messages; the server keeps track of what consumers have seen. Nacking means you lose ordering, as the nacked messages come back later. Jetstream also supports a Kafka-like strictly ordered mode. Unlike Kafka, clients can choose the routing behaviour, including worker style routing and deterministic partitioning.
Unlike Kafka's rigid networking model (consumers are assigned partitions and they consume the topic and that's it), as with NATS, you can set up complex topologies where streams get gatewayed and replicated. For example, you can streams in multiple regions, with replication, so that consumers only need to connect to the local region's hub.
While NATS/Jetstream has a lot of flexibility, I feel like they've compromised a bit on performance and scalability. Jetstream clusters don't scale to many servers (they recommend max 3, I think) and large numbers of consumers can make the server run really hot. I would also say that they made a mistake adopting nacking into the consuming model. The big simplification Kafka makes is that topics are strictly sequential, both for producing and consuming. This keeps the server simpler and forces the client to deal with unprocessable messages. Jetstream doesn't allow durable consumers to be strictly ordered; what the SDK calls an "ordered consumer" is just an ephemeral consumer. Furthermore, ephemeral consumers don't really exist. Every consumer will create server-side state. In our testing, we found that having more than a few thousand consumers is a really bad idea. (The newest SDK now offers a "direct fetch" API where you can consume a stream by position without registering a server-side consumer, but I've not yet tried it.)
Lastly, the mechanics of the server replication and connectivity is rather mysterious, and it's hard to understand when something goes wrong. And with all the different concepts — leaf nodes, leaf clusters, replicas, mirrors, clusters, gateways, accounts, domains, and so on — it's not easy to understand the best way to design a topology. The Kafka network model, by comparison, is very simple and straightforward, even if it's a lot less flexible. With Kafka, you can still build hub/spoke topologies yourself by reading from topics and writing to other topics, and while it's something you need to set up yourself, it's less magical, and easier to control and understand.
Where I work, we have used NATS extensively with great success. We also adopted Jetstream for some applications, but we've soured on it a bit, for the above reasons, and now use Redpanda (which is Kafka-compatible) instead. I still think JS is a great fit for certain types of apps, but I would definitely evaluate the requirements carefully first. Jetstream is different enough that it's definitely not just a "better Kafka".
> Jetstream clusters don't scale to many servers (they recommend max 3, I think)
Jetstream is even more limited than most Kafkas on number of streams https://github.com/nats-io/nats-server/discussions/5128#disc...
That's an amazing analysis, thank you so much!
What are your impressions of Red panda?
We're particularly interested in NATS' feature of working with individual messages and have been bitten by Kafka's "either process the entire batch or put it back for later processing", which doesn't work for our needs.
Interested if Redpanda is doing better than either.
Redpanda is fantastic, but it has the exact same message semantics as Kafka. They don't even have their own client; you connect using the Kafka protocol. Very happy with it, but it does have the same "whole batch or nothing" approach.
NATS/Jetstream is amazing if it fits your use case and you don't need extreme scalability. As I said before, it offers a lot more flexibility. You can process a stream sequentially but also nack messages, so you get the best of both worlds. It has deduping (new messages for the same subject will mark older ones as deleted) and lots of other convenience goodies.
Thank you so much again. Yes, we are not Google scale, our main priority is durability and scalability but only up to a (I'd say fairly modest) point. I.e. be able to have one beefy NATS server do it all and only add a second one when things start getting bad. Even 3 servers we'd see as a strategic defeat + we have data but again, very far from Google scale.
We've looked at Redis streams but me and a few others are skeptical as Redis is not known for good durability practices (talking about the past; I've no idea if they pivoted well in the last years) and sadly none of us has any experience with MQTT -- though we heard tons of praise on that one.
But our story is: some tens of terabytes of data, no more than a few tens of millions of events / messages a day, aggressive folding of data in multiple relational DBs, and a very dynamic and DB-heavy UI (I will soon finish my Elixir<=>Rust SQLite3 wrapper so we're likely going to start sharding the DB-intensive customer data to separate SQLite3 databases and I'm looking forward to spearheading this effort; off-topic). For our needs NATS Jetstream sounds like the exactly perfect fit, though time will tell.
I still have the nagging feeling of missing out on still having not tried MQTT though...
At that scale, Jetstream should work well. In my experience, Jetstream's main performance weakness is the per-stream/consumer overhead: Too many and NATS ends up running too hot due to all the state updates and Raft traffic. (Each stream is a Raft group, but so is each consumer.)
If its tens of TB in a stream, then I've not personally stored that much data in a stream, but I don't see why it wouldn't handle it. Note that Jetstream has a maximum message size of 1MB (this is because Jetstream uses NATS for its client/server protocol, which has that limit), which was a real problem for one use case I had. Redpanda has essentially no upper limit.
Note that number of NATS servers isn't the same as the replication factor. You can have 3 servers and a replication factor of 2 if you want, which allows more flexibility. Both consumers and streams have their own replication factors.
The other option I have considered in the past is EMQX, which is a clustered MQTT system written in Erlang. It looks nice, but I've never used it in production, and it's one of those projects that nobody seems to be talking about, at least not in my part of the industry.
Well I work mainly with Elixir in the last 10-ish years (with a lot of Rust and some Golang here and there) so EMQX would likely be right up my alley.
Do you have any other recommendations? The time is right for us and I'll soon start evaluating. I only have NATS Jestream and MQTT on my radar so far.
Kafka I already used and rejected for the reasons above ("entire batch or nothing / later").
As for data, I meant tens of terabytes of traffic on busy days, sorry. Most of the time it's a few hundred gigs. (Our area is prone to spikes and the business hours matter a lot.) And again, that's total traffic. I don't think we'd have more than 10-30GB stored in our queue system, ever. Our background workers aggressively work through the backlog and chew data into manageable (and much smaller chunks) 24/7.
And as one of the seniors I am extremely vigilant of payload sizes. I had to settle on JSON for now but I push back, hard, on any and all extra data; anything and everything that can be loaded from DB or even caches is delegated as such with various IDs -- this also helps us with e.g. background jobs that are no longer relevant as certain entity's state moved too far forward due to user interaction and the enriching job no longer needs to run; when you have only references in your message payload, this enables and even forces the background job to load data exactly at the time of its run and not assume a potentially outdated state.
Anyhow, I got chatty. :)
Thank you. If you have other recommendations, I am willing to sacrifice a little weekend time to give them a cursory research. Again, utmost priority is 100% durability (as much as that is even possible of course) and mega ultra speed is not of the essence. We'll never have even 100 consumers per stream; I haven't ever seen more than 30 in our OTel tool dashboard.
EDIT: I should also say that our app does not have huge internal traffic; it's a lot (Python wrappers around AI / OCR / others is one group of examples) but not huge. As such, our priorities for a message queue are just "be super reliable and be able to handle an okay beating and never lose stuff" really. It's not like in e.g. finance where you might have dozens of Kafka clusters and workers that hand off data from one Kafka queue to another with a ton of processing in the meantime. We are very far from that.
Those are the two I can think of.
Jetstream is written in Go and the Go SDK is very mature, and has all the support one needs to create streams and consumers; never used it from Elixir, though. EMQX's Go support looks less good (though since it's MQTT you can use any MQTT client).
Regarding data reliability, I've never lost production data with Jetstream. But I've had some odd behaviour locally where everything has just disappeared suddenly. I would be seriously anxious if I had TBs of stream data I couldn't afford to lose, and no way to regenerate it easily. It's possible to set up a consumer that backs up everything to (say) cloud storage, just in case. You can use Benthos to set up such a pipeline. I think I'd be less anxious with Kafka or Redpanda because of their reputation in being very solid.
Going back to the "whole batch or nothing", I do see this as a good thing myself. It means you are always processing in exact order. If you have to reject something, the "right" approach is an explicit dead-letter topic — you can still consume that one from the same consumer. But it makes the handling very explicit. With Jetstream, you do have an ordered stream, but the broker also tracks acid/nacks, which adds complexity. You get nacks even if you never do it manually; all messages have a configurable ack deadline, and if your consumer is too slow, the message will be automatically bounced. (The ack delay also means if a client crashes, the message will sit in the broker for up to the ack delay before it gets delivered to another consumer.)
But of course, this is super convenient, too. You can write simpler clients, and the complicated stuff is handled by the broker. But having written a lot of these pipelines, my philosophy these days is that — at least for "this must not be allowed to fail" processing, I prefer something that is explicit and simpler and less magical, even if it's a bit less convenient to write code for it. Just my 2 cents!
This is getting a bit long. Please do reach out (my email is in my profile) if you want to chat more!
I dont have kafka experience, but nats is absolutely amazing. Just a complete pleasure to use, in every way.
https://www.synadia.com/blog/nats-and-kafka-compared
I got really pissed off with their field CTO for essentially trying to pull the wool over my eyes regarding performance and reliability.
Essentially their base product (NATs) has a lot of performance but trades it off for reliability. So they add Jetstream to NATs to get reliability, but use the performance numbers of pure NATs.
I got burned by MongoDB for doing this to me, I won’t work with any technology that is marketed in such a disingenuous way again.
Don't implement any distributive technology until aphyr has put it through the paces, and even then... Pilot
https://aphyr.com/about
"Unavailable Due to the UK Online Safety Act"
:(
You mean Jetstream?
Can you point to where they are using core NATS numbers to describe Jetstream?
Yes, I meant Jetstream (I even typed it but second guessed myself, my mistake) I’m typing these when I get a moment as I’m at a wedding- so I apologise.
The issue in the docs was that there are no available Jetstream numbers, so I talked over a video call to the field CTO, who cited the base NATs numbers to me, and when I pressed him on if it was with Jetstream he said that it was without: so I asked for them with Jetstream enabled and he cited the same numbers back to me. Even when I pressed him again that “you just said those numbers are without Jetstream” he said that it was not an issue.
So, I got a bit miffed after the call ended, we spent about 45 minutes on the call and this was the main reason to have the call in the first place so I am a bit bent about it. Maybe its better now, this was a year ago.
This doesn’t really support your position as far as most readers are concerned - it sounds like a disconnect. If they didn’t do this in any ad copy or public docs it’s not really in Mongo territory.
I don’t really care.
I’m telling you why I am skeptical of any tech that intentionally obfuscates trade-offs, I’m not making a comparison on which of these is worse; and I don’t really care if people take my anecdote seriously either: because they should make their own conclusions.
However it might help people go in to a topic about performance and reliability from a more informed position.
I don't doubt your experience. But I think it might have been more just that guy, than NATS in general.
The other day i was listening to a podcast with their ceo from maybe 6 months ago, and he talked quite openly about how jetstream and consumers add considerable drag compared to normal pubsub. And, more generally, how users unexpectedly use and abuse nats, and how they've been able to improve things as a result.
It’s deceptive if true, why are you trying to spin it as it’s ok cause the deception were not published
Its not ok if there was deception, but it sounds just as likely its a communication disconnect in their call. We only have one side of it.
There is a good comparison between NATS, Kakfa, and others here: https://docs.nats.io/nats-concepts/overview/compare-nats
Maybe needs a neutral party comparison :)
The delivery guarantees section alone doesn’t make me trust it. You can do at least once or at most once with kafka. Exactly once is mostly a lie, it depends on the downstream system: unless going back to the same system, the best you can do is at least once with idempotancy
It’s on the NATS website and “nats” appears in the URL three times, so maybe this isn’t the most objective source.
It was created to teach me the concept of love-hate relationships
I wanted to write a comment on this topic, but after several tries this thread is where I ended up because it describes my sentiment as well.
The arguments in the article are very compelling. But as soon as you choose Kafka you realize the things you hate.
Many of the reasons are stupid things - like it uncovers otherwise unimportant bugs in your client code. Or that it just makes experimenting a hassle because it enforces poking around in lots of different places to do something. Or that writing and maintaining the compulsory integration test takes weeks of your time.
Sure - you can replay your data - but not until you have fixed all the issues for that special case in your receiving service.
I think maybe my main gripe (for us) was that it was a difficult to get an understanding what is actually inside your pipe. Much easier to have that in a solid state in s3?
At the end of they day you get annoyed because it slows you down. In particular when you are a small localized team.
Totally agree with this. I’ll add that replaying your data needs special tooling to 1) find the correct offsets on each topic, and 2) spin up whatever daemon will consume that data out-of-band from normal processing, and shut it down when completed.
I don’t remember where I read this, but someone made the observation that writing a stream processing system is about 3x harder than writing a batch system, exactly for all the reasons you mentioned. I’m looking at replacing some of our Kafka usage with a clickhouse table that’s ordered and partitioned by insertion time, because if I want to do stuff with that data stream, at least I can do a damn SQL query.
Yes I'll happily extend that to 10x more difficult.
At least compared to building a batched pipeline with SQL. I think you should really think hard whether you really need a streaming pipeline. And even if you find that you do, it may be worthwhile to make a batched pipeline as your first implementation.
I did exactly what you describe in my previous job. In the beginning with reluctance from our architects who wanted to keep banging the dead horse and did not understand the power of SQL "SQL is not real programming, engineers write java" (ok maybe I deserve a straw-man yellow card here, they don't deserve all of that). But I think they understood after a while.
With AWS Athena and Airflow. Good luck, consider me your distant moral support.
This might be hyperbolic, but I think Kafka (or at least the concept of event driven architecture for sharing data across many systems) is one of the most under-rated technologies. It's used at a lot of big corps but is never talked about.
Like modeling everything as graphs, it is often a trap. Just because a model is flexible enough to capture all of your use cases doesn't mean you should use it. In fact you should prefer less flexible more constrained models that are simpler.
Distributed ledgers like Bitcoin do store transitions as events, and that's because nodes need the transitions to valid the next state. So you might say that Bitcoin is a widely run piece of software, using an event driven architecture.
Not every system needs to have all of it's state transitions available for for efficient reading. And often times you can derive the state transition from the previous and next state if you really need them (Git does that). Even though Git can compute all of the state transitions for the system, it doesn't store events, it stores snapshots.
Apache Kafka was originally developed by LinkedIn engineers, primarily Jay Kreps, Neha Narkhede, and Jun Rao, around 2010.
It was later open-sourced in 2011 and became a top-level project under the Apache Software Foundation in 2012.
The creators went on to co-found Confluent, a company that provides commercial support and enterprise features around Kafka.
The biggest strength of Kafka in my opinion is consumer groups.I have been using it since 2016 in at least 3 projects and it never failed, not that big workloads though (~100 messages/sec max). However it is a bit difficult to monitor and manage using only the out of the box applications.
I would take a hard look at Kinesis/Pubsub for these data volumes; should cost in the tens of dollars monthly.
Startup founder here -- we tried it, and it feels bloated (Java!), bureaucratic and overcomplicated for what it is. Something like Redis queues or even ZMQ probably suffices for 90% of use cases. Maybe in hyper-scaled applications that need to be ultraperformant (e.g., realtime trading, massive streaming platforms) is where Kafka comes into play.
If you are using this sort of redis queue (https://redis.io/glossary/redis-queue/) with PUSH/POP vs fan-out you're working on a very different sort of problem than what Kafka is built for.
Like the article says, fan-out is a key design characteristic. There are "redis streams" now but they didn't exist back then. The durability story and cluster stories aren't as good either, I believe, so they can probably take you so far but won't be as generally suitable depending on where your system goes in the future. There are also things like RedPanda that speak Kafka w/o the Java.
However, if you CAN run on a single node w/o worrying about partitioning, you should do that as long as you can get away with it. Once you add multiple partitions ordering becomes hard to reason about and while there are things like message keys to address that, they have limitations and can lead to hotspotting and scaling bottlenecks.
But the push/pop based systems also aren't going to give you at-least-once guarantees (looks like Redis at least has a "pop+push" thing to move to a DIFFERENT list that a single consumer would manage but that seems like it gets hairy for scaling out even a little bit...).
> and it feels bloated (Java!)
I'm curious, what exactly feels bloated about Java? I don't feel like the Java language or runtime are particularly bloated, so I'm guessing you're referring to some practices/principles that you often see around Java software?
Whatever efficiency may hypothetically be possible with Java, you can in-fact spot a real world Java program in the wild by looking for the thing taking up 10x the memory it seems like it should need… when idle.
Yes yes I’m sure there are exceptions somewhere but I’ve been reading Java fans using benchmarks to try to convince me that I can’t tell which programs on my computer are Java just by looking for the weirdly slow ones, when I in fact very much could, for 25ish years.
Java programs have a feel and it’s “stuttery resource hog”. Whatever may be possible with the platform, that’s the real-world experience.
I held the same view as you when I was 22, more than 15 years ago.
With over 15 years of professional experience since then, my perspective has shifted: Java demonstrates its strength when stability, performance, and scalability are required (e.g. bloody enterprise)
A common misconception comes from superficial benchmarking. Many focus solely on memory consumption, which often provides a distorted picture of actual system efficiency.
I can point to EU-scale platforms that have reliably served over 100 million users for more than a decade without significant issues. The bottleneck is rarely the language itself, it is the depth of the team’s experience.
> Many focus solely on memory consumption, which often provides a distorted picture of actual system efficiency.
When other languages can do the same thing with an order of magnitude less RAM, any other efficencies in the system tend to be overshadowed by that and be the sticking point in peoples memories.
You may argue that holding on to this extra memory makes subsequent calls and reads quicker etc, but in my experience generally people are willing to sacrifice milliseconds to gain gigabytes of memory.
node is a notable exception. Compared to java node is a hellhole. the standard library is non-existent, most libraries are a buggy mess, the build system is horrible...in fact there is no reliable build system that solves all your typical problems in 1 app. The list goes on.
The JVM eats a chunk of memory in order to make its garbage collector more efficient. Think of it like Linux's page cache.
I haven't worked with too much Java, but I suspect that the distaste many have for it is due to its wide adoption by large organizations and the obfuscating "dressed up" tendency of the coding idioms used in large organizations.
The runtime isn't inherently slow, but maybe it's easier to write slow programs in Java.
you know why you don’t see many non-Java programs on your computer taking up 10x memory? because no one uses them to write anything :)
jokes aside, we got a shift in the industry where many java programs were replaced by electron-like programs which now take 20x memory
Technically kind of true but at the same time Android apps are predominantly Java/Kotlin. It speaks more to Java just having a bad desktop story. But it’s also why Android devices need 2x the ram
That has nothing to do with Java. The Android runtime is NOT Java/OpenJdk.
No but it does speak to the memory overhead of tracing GC vs ref counting as garbage collection strategies.
Which is very important in... embedded settings.
While for typical backend situations, reference counting has a crazy high throughput overhead, doing atomic inc/decs left and right, that instantly trashes any kind of cache, and does it in the mutator thread that would do the actual work, for the negligible benefit of using less memory. Meanwhile a tracing GC can do (almost) all its work in another thread, not slowing down the actually important business task, and with generational GCs cleaning up is basically a no-op (of just saying that this region can now be reused).
It's a tradeoff as everything in IT.
Also, iPhone CPUs are always a generation ahead, than any android CPU, if not more. So it's not really Apples to oranges.
That would be a compelling counter if and only if languages like Java actually beat other languages in throughput. In practice that doesn’t seem to be the case and the reasons for that seem to be:
* languages like c++ and Rust simply don’t allocate as much as Java, instead using value types. Even C# is better here with value types being better integrated.
* languages like c++ and Rust do not force atomic reference counting. Rust even offers non atomic ref counting in the standard library. You also only need to atomic increment / decrement when ownership is being transferred to a thread - that isn’t quite as common depending on the structure of your code. Even swift doesn’t do too badly here because of the combination of compiler being able to prove the permission of eliding the need for reference counting altogether and offering escape hatches of data types that don’t need it.
* c++, Rust, and Swift can access lower level capabilities (eg SIMD and atomics) that let them get significantly higher throughput.
* Java’s memory model implies and requires the JVM to insert atomic accesses all over the place you wouldn’t expect (eg reading an integer field of a class is an atomic read and writing it is an atomic write). This is going to absolutely swamp any advantage of the GC. Additionally, a lot of Java code declares methods synchronized which requires taking a “global” lock on the object which is expensive and pessimistic for performance as compared with the fine-grained access other languages offer.
* there’s lots of research into ways of offering atomic reference counts more cheaply (called biased RC) which can safely avoid needing to do an atomic operation in places completely transparently and safely provided the conditions are met .
I’ve yet to see a Java program that actually gets higher throughput than Rust so the theoretical performance advantage you claim doesn’t appear to manifest in practice.
The main topic here was Swift vs Android's Java.
Of course with manual memory management you may be able to write more efficient programs, though it is not a given, and comes at the price of a more complicated and less flexible programming model. At least with Rust, it is actually memory safe, unlike c++.
- ref counting still has worse throughout than a tracing GC, even if it is single-threaded, and doesn't have to use atomic instructions. This may or may not matter, I'm not claiming it's worse, especially when used very rarely as is the case with typical c++/rust programs.
> You also only need to atomic increment / decrement when ownership is being transferred to a thread
Java can also do on-stack replacement.. sometimes.
- regarding lower level capabilities, java does have an experimental Vector API for simd. Atomics are readily available in the language.
- Java's memory model only requires 32-bit writes to be "atomic" (though in actuality the only requirement is to not tear - there is no happens before relation in the general case, and that's what is expensive), though in practice 64-bit is also atomic, both of which are free on modern hardware. Field acces is not different from what rust or c++ does, AFAIK in the general case. And `synchronized` is only used when needed - it's just syntactic convenience. This depends on the algorithm at hand, there is no difference between the same algorithm written in rust/c++ vs java from this perspective. If it's lockless, it will be lockless in Java as well. If it's not, than all of them will have to add a lock.
The point is not that manual memory can't be faster/more efficient. It's that it is not free, and comes at a non-trivial extra effort on developers side, which is not even a one-time thing, but applies for the lifetime of the program.
> ref counting still has worse throughout than a tracing GC, even if it is single-threaded, and doesn't have to use atomic instructions. This may or may not matter, I'm not claiming it's worse, especially when used very rarely as is the case with typical c++/rust programs.
That’s a bold claim to make that doesn’t seem to actually be true from my experience. Your 5ghz CPU can probably do ~20 billion non atomic reference adjustments whereas your GC system has to have atomics all over the place or it won’t work and atomics have parasitic performance on unrelated code due to bus locks and whatnot.
> Java can also do on-stack replacement.. sometimes
That’s not what this is. It’s called hybrid RC and it applies always provided you follow the rules.
> The point is not that manual memory can't be faster/more efficient. It's that it is not free, and comes at a non-trivial extra effort on developers side, which is not even a one-time thing, but applies for the lifetime of the program.
The argument here is not about developer productivity - the specific claim is that the Java GC lets you write higher throughput code than you would get with Rust or C++. That just isn’t true so you end up sacrificing throughput AND latency AND peak memory usage. You may not care and are fine with that tradeoff, but claiming you’re not making that tradeoff is not based on the facts.
> the specific claim is that the Java GC lets you write higher throughput code than you would get with Rust or C++
No, that has never been the specific claim - you can always write more efficient code with manual memory management, given enough time, effort and skill. I wasn't even the one who brought up c++ and rust. Like literally I write this twice in my comment.
What I'm talking about is reference counting as a GC technique vs tracing as a GC technique, all else being equal - it would be idiotic to compare these two if no other "variable" is fixed. (Oh and I didn't even mention the circular references problem, which means you basically have to add a tracing step either-way unless you restrict your language so that it can't express circular stuff).
As for the atomic part, sure, if all it would do is non-atomic increments then CPUs would be plenty happy. And you are right that depending on how the tracing GC is implemented, it will have a few atomic instructions. What you may miss is how often each run. On almost every access, vs every once in a while on a human timescale. Your OS scheduler will also occasionally trash the performance of your thread. But this is the actually apples to oranges comparison, and both techniques can do plenty of tweaks to hide certain tradeoffs, at the price of something else.
And I also mention that the above triad of time, skill and effort is not a given and is definitely not free.
In Rust there’s no forcing of any specific garbage collection mechanism. You’re free to do whatever and there’s many high performance crates to let you accomplish this. Even in Swift this is opt-in.
As for “skill” this is one thing that’s super hard to optimize for. All I can do is point to existence proofs that there’s no mainstream operating system, browser or other piece of high performance code written in Java and it’s all primarily C/C++ with some assembly with Rust starting to take over the C/C++ bits. And at the point where you’re relegating Java to being “business” logic, there’s plenty of languages that are better suited for that in terms of ergonomics.
Sure, but I think that people often fall into the trap of imagining a problem that nicely fits a RAII model, where each lifetime is statically knowable. This is either due to having a specific problem, or because we decided on a specific constraint.
Java is used in HFT (well, there are two types of "high frequency", one where general purpose CPUs are already too slow, where it obviously doesn't apply (neither do rust or c++)) - but sure, I wouldn't write a runtime or other piece of code in Java where absolute control over the hardware is required. But that's a small niche only. What about large distributed systems/algorithms? Why is Java over-represented in this niche (e.g. Kafka, Elasticsearch, etc)?
> And at the point where you’re relegating Java to being “business” logic, there’s plenty of languages that are better suited for that in terms of ergonomics.
That's subjective.
> Java’s memory model implies and requires the JVM to insert atomic accesses all over the place you wouldn’t expect (eg reading an integer field of a class is an atomic read and writing it is an atomic write).
AFAIK that doesn’t really happen. They won’t insert atomic accesses anywhere on real hardware because the cpu is capable of doing that atomically anyway.
> Additionally, a lot of Java code declares methods synchronized which requires taking a “global” lock on the object which is expensive and pessimistic for performance as compared with the fine-grained access other languages offer.
What does this have to do with anything? Concurrency requires locks. Arc<T> is a global lock on references. “A lot” of Java objects don’t use synchronized. I’d even bet that 95-99% of them don’t.
> Concurrency requires locks. Arc<T> is a global lock on references
Concurrency does not require locks. There’s entire classes of lock free and wait free algorithms. Arc<T> is also not a lock - it uses atomics to manage the reference counts and no operation on an Arc needs to wait on a lock (it is a lock-free container).
> “A lot” of Java objects don’t use synchronized. I’d even bet that 95-99% of them don’t.
Almost all objects that are used in a concurrent context will likely feature synchronized, at least historically. That’s why Hashtable was split into HashMap (unsynchronized) and ConcurrentHashMap (no longer using synchronized). Thats why you have StringBuffer which was redone into StringBuilder.
Ok I mispoke on Arc because I was being hasty; but you're still being pedantic. Concurrency still requires locks. Wait/lock free algorithms can't cover the entirety of concurrency. Rust ships with plenty of locks in std::sync and to implement a ConcurrentHashMap in Rust you would still need to lock. In fact it doesn't even look like Rust supplies concurrent collections at all. So what are we even talking about here? This is still a far cry from "a lot of Java objects use global synchronized locks".
No, that’s an overly strong statement - concurrency doesn’t necessarily require locks even though they can be convenient to express it. You could have channels and queues to transfer data and ownership between threads. Not a lock in sight as queues and channels can be done lock free. The presence of locks in the Rust standard library says nothing other than it’s a very common concurrency tool, not that it’s absolutely required.
> and to implement a ConcurrentHashMap in Rust you would still need to lock
There’s many ways to implement concurrency safe hashmaps (if you explicitly needs such a data structure as the synchronization mechanism) without locks. Notably RCU is such a mechanism (really neat mechanism developed for the kernel although not super user friendly yet or common in userspace) and there are also generational garbage techniques available (kind of similar to tracing GC conceptually but implemented just for a single data structure). A common and popular crate in Rust for this is DashMap which doesn’t use locks and is a concurrency safe hashmap.
> A common and popular crate in Rust for this is DashMap which doesn’t use locks and is a concurrency safe hashmap.
Still not in the standard library. The only way in Rust is to use a global lock around map. Seems to be worse than the situation in Java. You could implement the same thing and use a third party library in Java too. So your original point of "everything uses a global lock" is "overly strong"
You’ve now degraded the conversation into a very very weird direction. You made a claim that concurrency required locks. It simply does not and I have an existence proof of Dashmap as a hashmap that doesn’t have any locks anywhere.
The strengths and weaknesses of the standard library aren’t relevant. But if we’re going there, the reason they’re not in the Rust standard library is likely in practice concurrent data structures are an anti pattern - putting a lock around a data structure doesn’t suddenly solve higher order race conditions which is something a lot of Java programmers seem to believe because the standard library encourages this kind of thinking.
As for “my comment” about “global lock” (your words not mine), it’s that the implicit lock that’s available on every object is a bad idea for highly concurrent code (not to mention the implicit overhead that implies for every part of the object graph regardless of it being needed anywhere). Don’t get me wrong - Java took a valiant effort to define a solid memory model for concurrency when the field was still young. Many of the ideas didn’t pan out and are antipatterns these days for high performing code. Of course none of that pertains to the original point of the conversation - tracing GCs have significantly more overhead in practice because they’re very difficult to be opt in, carry quite a penalty if not, Rc/Arc is much better as it’s possible to do opt-in when you need shared ownership (which isn’t always), and in practice loops don’t come up often enough to matter and when they do there’s still solutions. In other words tracing GCs drop huge amounts of performance on the floor and you can read all the comments to see how the claims are “it’s more efficient than Rc”, or “performance is free” or even “it doesn’t matter because the programmer is more efficient”. I’d buy the efficiency argument when the only alternative was C/C++ and came with serious memory safety baggage, but not any of the others and memory safety without sacrificing performance of C++ in my view is a solved problem with Rust.
It depends how you implement reference counting. In Rust the atomic inc-dec operations can be kept at a minimum (i.e. only for true changes in lifecycle ownership) because most accesses are validated at compile time by the borrow checker.
[flagged]
Is this an AI-generated answer? Most of these are not even true, although I still would prefer Go for micro-services. I'll address just a bunch and to be clear - I'm not even a big Java fan.
- Quarkus with GraalVM compiles your Java app to native code. There is no JIT or warm up, memory footprint is also low. By the way, the JVM Hotspot JIT can actually make your Java app faster than your Go or Rust app in many cases [citation needed] exactly due to the hot path optimizations it does.
- GC tuning - I don't even know who does this. Maybe Netflix or some trading shops? Almost no one does this nowadays and with the new JVM ZGC [0] coming up, nobody would need to.
> You can’t ship a minimal standalone binary without pulling in a JVM.
- You'd need JRE actually, e.g., 27 MB .MSI for Windows. That's probably the easiest thing to install today and if you do this via your package manager, you also get regular security fixes. Build tools like Gradle generate a fully ready-to-execute directory structure for your app. If you got the JRE on your system, it will run.
> Dependency management and classpath conflicts historically plagued Java
The keyword here is "historically". Please try Maven or Gradle today and enjoy the modern dependency management. It just works. I won't delve into Java 9 modules, but it's been ages since I last saw a class path issue.
> J2EE
Is someone still using this? It is super easy writing a web app with Java+Javalin for example. The Java library and frameworks ecosystem is super rich.
> “Write once, run anywhere” costs: The abstraction layers that make Java portable also add runtime weight and overhead.
Like I wrote above, the HotSpot JIT is actually doing the heavy lifting for your in real time. These claims are baseless without pointing to what "overhead" is meant in practice.
---
0 - https://inside.java/2023/11/28/gen-zgc-explainer/ or https://www.youtube.com/watch?v=dSLe6G3_JmE
> GC tuning, Netflix
I believe Netflix has moved to ZGC with no tuning. Their default setup is to set the min/max heap to the same size, enable always pretouch, and to use transparent huge pages [0]. GC tuning is something of the past. Once automatic heap sizing for ZGC and G1 land you won’t even need to set the heap size [1][2]. They’ll still use more ram because the vm and jit, but the days of it holding on to ram when it doesn’t need it should be over.
[0] https://netflixtechblog.com/bending-pause-times-to-your-will...
[1] https://openjdk.org/jeps/8329758
[2] https://openjdk.org/jeps/8359211
conflicts are a necessary evil with a massive dependency ecosystem
> taking up 10x the memory it seems like it should need… when idle.
The JVM tends to hold onto memory in order to make things faster when it does wind up needing that memory for actual stuff. However, how much it holds on to, how the GC is setup, etc are all tunable parameters. Further, if it's holding onto memory that's not being used, these are prime candidates to be stored in virtual memory which is effectively free.
Note that the memory problem you mentioned is not really a problem in fact. It is just how managed memory works in Java. Just run .gc() and you'll see what I'm talking about. It reserves memory which you can see on the charts but it is not necessarily used memory.
Well, you might want to read up on how OSs handle memory under the hood, and that virtual memory != physical, and that task manager and stuff like that can't show the real memory usage.
Nonetheless, tracing GCs do have some memory overhead in exchange for better throughput. This is basically the same concept as using a buffer.
-----
And can you tell which of these websites use Java from "the feel"? AWS cloud infra, a significant chunk of Google, Apple's backends, Alibaba, Netflix?
"Java is bloated because I only look at the bloated examples."
Is C++ bloated because of the memory Chrome uses?
When all your examples in actual use are bloated…
I’ve never seen another basic tech used to develop other programs that’s so consistently obvious from its high resource use and slowness, aside from the modern web platform (Chrome, as you put it). It was even more obvious back when we had slower machines, of course, but Java still stands out. It may be able to calculate digits of Pi in a tight loop about as fast as C, but real programs are bloated and slow.
Sounds like a classic case of confirmation bias.
Especially that like half of the web runs on Java, you just have absolutely no idea when it silently does its job perfectly.
> Especially that like half of the web runs on Java
Source?
W3 seems to think its more like ~5%
https://w3techs.com/technologies/comparison/pl-java
5% of "whose server-side programming language we know"
From the website.
And 76% of these websites is PHP, which seems to mean.. they can determine PHP more easily for a website (nonetheless, there are indeed a lot of WordPress sites, but not this amount).
Right, so Im assuming that as you are saying 'Half the web runs on java', maybe you know more about what websites are using in their backend? Care to share where you are getting this information from?
K.
Have you ever run on-prem Atlassian products or any enterprise JVM apps?
They hog RAM, are slow, and are a bitch to configure.
Java the language and Java the runtime are fine.
The way most Java code is written is terrible Enterprise factory factory factory.
That doesn't match my experience in the last 15 years working for 3 companies (one was a big enterprise, one medium sized and one startup)
Maybe I have been lucky, or that the practice is more common in certain countries or eco systems? Java has been a very productive language for me, and the code has been far from the forced pattern usage that I have read horror stories about.
Have you gotten to use loom/virtual threads? I’ve heard pretty interesting stuff about em, but haven’t really spent the time to get into it yet. It’s pretty exciting and tbh gives me an easy elevator pitch to JVM world for people outside of it
If you have a use-case where you currently allocate ~1K threads mostly waiting on I/O switching to virtual threads is a one-liner ("Thread.ofVirtual()" instead of "Thread.ofPlatform()"). No more golang envy for sure.
Depending on how much memory is used by the Thread stack (presumably 1M-512K by default, allegedly 128K with Alpine base images) that's your 1G-500M heap space usage improvement right off the bat.
The migration from JDK17 to JDK21 was uneventful in production. The only issue is limited monitoring as a thread dump will not show most virtual threads and the micrometer metrics will not even collect the total number of active virtual threads. It's supposed to work better in JDK24.
The Spring Framework directly supports virtual threads with "spring.threads.virtual.enabled=true" but I haven't tried it to comment.
But the perf is not reliable. If you want latency and throughput, idiomatic Rust will give you better properties. Interestingly even will Go for some reason has better latency guarantees I believe even though it’s GC is worse than Java.
There is not much point talking about throughput and latency in the abstract - they are very often opposing goals, you can make one better at the expense of the other.
Go's GC is tuned more for latency at the expense of throughput (not sure if it still applies, but Go was quite literally stopping the "business" mutator threads when utilisation got higher to be able to keep up with the load - Java's default GC is tuned for a more balanced approach, but it can deliver it at very high congestion rates as well. Plus it has a low-latency focused GC which has much better latency guarantees, and it trades off some throughput in a consistent manner, so you can choose what fits best). The reason it might sometimes be more efficient than Java is simply value types - it doesn't create as much garbage, so doesn't need as good a GC in certain settings.
Rust code can indeed be better at both metrics for a particular application, but it is not automatically true, e.g. if the requirements have funny lifetimes and you put a bunch of ARC's, then you might actually end up worse than a modern tracing GC could do. Also, future changes to the lifetimes may be more expensive (even though the compiler will guide you, you still have to make a lot of recursive changes all across the codebase, even if it might be a local change only in, say, Java), so for often changing requirements like most business software, it may not be the best choice (even though I absolutely love Rust).
This presupposes the use case is such that this even matters. Obviously that is the case sometimes, but in the vast majority of cases it is not.
For many applications deferred garbage collection is acceptable.
Worse latency every ten minutes tends to be fine.
This is more of a meme, than reality. There are architecture astronauts for every platform and shitty code knows no bounds, regardless of language.
The problem is that writing genuinely performant Java code requires that you drop most if not all of the niceties of writing Java. At that point, why write Java at all? Just find some other language that targets the JVM. But then you're already treading such DIY and frictionful waters that just adopting some other cross-platform language/runtime isn't the worst idea.
> The problem is that writing genuinely performant Java code requires that you drop most if not all of the niceties of writing Java
Such as? The only area where you have to "drop" features is high-frequency trading, where they often want to reach a steady-state for the trading interval with absolutely no allocations. But for HFT you would have to do serious tradeoffs for every language.
In my experience, vanilla java is more than fine for almost every application - you might just benchmark your code and maybe add some int arrays over an Integer list, but Java's GC is an absolute beast, you don't have to baby it at all.
>The problem is that writing genuinely performant Java code requires that you drop most if not all of the niceties of writing Java. At that point, why write Java at all?
The reason is quite well known. Supporting multiple languages is a cost. If you only have to support one language, everything is simpler and cheaper.
With Java, you can write elegant code these days, rely on ZGC, not really worry too much about GC and get excellent performance with quick development cycles for most of your use cases. Then with the same language and often in the same repo (monorepo is great) you can write smarter code for your hot path in a GC free manner and get phenomenal performance.
And you get that with only having one build system, one CI pipeline, one deployment system, some amazing profiling and monitoring tooling, a bunch of shared utility code that you don't have to duplicate, and a lot more benefits.
That's the reason to choose Java.
Of course, if you're truly into HFT space, then they'll be writing in C, C++ or on FPGAs.
> what exactly feels bloated about Java?
https://docs.spring.io/spring-framework/docs/2.5.x/javadoc-a...
Kafka does not use Spring.
Starting up a Java program takes much longer than it should and that affects perception.
Depends on the program (especially the framework used) and the GC being used. I can write a java program and set it up in a way that it runs faster than almost everything else. For example in a serverless architecture where you need fast startup and small programs you can choose __not__ to use a GC and run ephemeral Java scripts. It starts and finishes running faster than you can blink.
It may affect developer perception but I'm pretty sure my users don't notice and don't care.
With AOT tho that should be somewhat moot.
I am just explaining why it has that reputation.
> I'm curious, what exactly feels bloated about Java?
Everything.
Why do you think Kubernetes is NOT written in Java?
... Because it came from Google?
Golang has little to distinguish itself technically. It has a more modern std lib (for now) and isn't Oracle.
Which aren't trivial, but they aren't Trump cards.
> ... Because it came from Google?
Nope.
None of what you said are any of the reasons given that it WAS written in Java already [0] but rewrote it all in Go explicitly because of its performance, concurrency and single binary distribution characteristics.
Those were enough technical advantages to abandon any thought of a production-grade version of k8s in Java.
[0] https://archive.fosdem.org/2019/schedule/event/kubernetesclu...
> rewrote it all in Go explicitly because of its performance
Because someone wanted a new, shiny toy.
> the anti patterns weren’t enough we also observe how Kubernetes has over 20 main() functions in a monolithic “build” directory. We learn how Kubernetes successfully made vendoring even more challenging than it already was, and discuss the pitfalls with this design. We look at what it would take to begin undoing the spaghetti code that is the various Kubernetes binaries built from github.com/kubernetes/kubernetes
It seems to me that perhaps it wasn’t the languages fault but the authors.
If you have this stance to technology ("feels bloated", meme-level knowledge on Java), then I really hope that you are not responsible for technical decisions.
Publishing an event to Kafka puts it “out there” in a way that guarantees it won’t be lost and allows any number of interested consumers, including the data warehouse, to deal with it at their leisure (subject to retention period which is typically like 72h). For us, your Kafka topics and their schemas are as much a part of your API as your gRPC IDLs. Something like Redis or 0MQ feels more appropriate for internal coordination between instances of the same service, or at least a producer that has a specific consumer in mind.
Have you tried Redpanda?
It doesn't have to be 'hyper-scaled' to be needed, unless we have widely different definitions of hyper scale. Access logs from a few thousand servers with medium traffic will push you past any single instance service, and Kafka works great for that workload.
> Maybe in hyper-scaled applications that need to be ultraperformant (e.g., realtime trading, massive streaming platforms) is where Kafka comes into play.
Kafka is used because the Java folks don't want to learn something new due to job security, even though there are faster and compatible alternatives that exist today.
Rather use Redpanda, than continue to use Kafka and then complain about how resource intensive it is alongside zookeeper and all the circus the comes with it and make AWS smile as you're losing hundreds of thousands a month.
I thought Kafka ditched the zookeeper
I wish; KIP-500[1] was the "let's use native Raft" implementation but I have never once in my life seen anyone using Kafka in KIP-500 mode
1: https://cwiki.apache.org/confluence/display/kafka/kip-500:+r...
Confluent has shared they've migrated thousands of Kafka clusters (their whole cloud fleet) to KIP-500
https://www.confluent.io/blog/zookeeper-to-kraft-with-conflu...
Very few people want to re-create their production AWS MSK cluster from scratch. And that's the only way currently. MSK can usually upgrade Kafka brokers with minor performance degradation but not for this particular change.
Couldn’t disagree more… if you go the ZMQ you are left alone handling many things you get in Kafka for free. If you have any sort of big data problems then good luck. You are going to reinvent the wheel.
(I was wondering if this was some sort of generated ripoff but the author worked on Kafka for 6 years: https://x.com/BdKozlovski.)
What do you mean by "generated ripoff"? Are you saying it read like AI?
I mean that it looked like either AI generated or blogspam or both. Not the writing though, more the look of the page. Happy to be wrong.
It's neither. Only the thumbnail background is AI generated.
The look of the page -- that's Substack's default UI, you can't control it too much. The other images are created by me.
I'm simply curious what parts give that "cheap" look so I can improve. On Reddit I've had massive amounts of downvotes because they guess the content is AI, when in fact no AI is used in the creation process at all.
One guess I have is the bullet points + bolding combo. Most AIs use a ton of that, and rightly so, because it aids in readability.
Yeah, I think those things do add to the impression and you should get rid of them. I'd suggest gluing some of those shorter paragraphs into longer ones too.
lots of people not considering UNPHAT - https://gist.github.com/rponte/67c78e6b3ee349a6e14cb8fb155b7... / https://stevenschwenke.de/pragmaticSoftwareEngineeringUNPHAT - some solutions didn't exist in the past. you evaluate your landscape and your problem, and try to find a good fit.
Kafka was created at LinkedIn when it was already quite a large web platform, to solve the problem of distributing a unique newsfeed to a wide audience… so while they weren’t Google, they were still closer to Google than the targets of the “UNPHAT” expression. You can apply UNPHAT to most of the people using Kafka today, but it’s not fair to apply it to LinkedIn when they created it.
> 17 PB/day
Are they really generating data at this scale?
I cant even imagine a system which creates and stores this much data this fast.
Where does this 17 PB/day number come from? I didn't quote any numbers directly.
Looking at the 2012 paper, it implies a 1.35TB per day (they store 9.5TB across all topics at 7d retention)
> In 2010, LinkedIn had 90 million members. Today, we serve over 1.2 billion members on LinkedIn. Unsurprisingly, this increase has created some challenges over the years, making it difficult to keep up with the rapid growth in the number, volume, and complexity of Kafka use cases. Supporting these use-cases meant running Kafka at a scale of over 32T records/day at 17 PB/day on 400K topics distributed across 10K+ machines within 150 clusters.
https://www.linkedin.com/blog/engineering/infrastructure/int...
~197 GB/s ... nice.
I believe these companies save literally every ounce of data they can find. Once you have the infra and teams for it, it seems easy to make a case for storing something.
Similarly, Uber has shared they push 89 GB/s through Kafka - 7.7 PB/s. People always ask me - what is a taxi/food-delivery app storing so much
The biggest capacity HD available today is 30TB.
17PB = about 567 of those drives... being totally filled... per day.
I was hoping somebody would come and say this is a simple spelling error or something.
The cost of the drives alone seems astronomical, let alone the logistics of the data center keeping up with storing that much data.
EDIT: I have just realised that they are probably only processing at this speed, rather than storing it, can anyone confirm if they store all the logs they process?
I would assume storage varies greatly. I know that LinkedIn quoted an average read fanout ratio of 5.5x in Kafka, meaning each byte was read 5.5x times. Assuming that is still true, we ought to divide by 6.5x to get to the daily write amount
That comes out to 87 disks a day. Assuming a 7 day retention period (this is on the high side), it’s not unthinkable to have a 600-1800 disk deployment (accounting for replication copies)
> That comes out to 87 disks a day. Assuming a 7 day retention period (this is on the high side), it’s not unthinkable to have a 600-1800 disk deployment (accounting for replication copies)
Yep. Whole week can be easily stored in 1-2 racks.
Sorry I meant to reply to this post above, who linked the blog post which mentions the 17PB/S
Post: https://news.ycombinator.com/item?id=45002089
Why was it named that is also a question.
> Jay Kreps chose to name the software after the author Franz Kafka because it is "a system optimized for writing", and he liked Kafka's work.
From Wikipedia.
Kafkaesque to configure and get running for simple tasks.
A system optimized for writing could also describe the machine in Kafka's "In the Penal Colony".
That’s funny, I assumed it was called Kafka because the act of processing items in a queue over and over could be described as kafkaesque.
It is called Kafka because it can write.
It's a bit like calling a dictation software "Hitler" because he also liked to dictate.
Thats a brilliant idea, although if I ever create a dictation software I am going to call it 'Mussolini'.
> Thats a brilliant idea
I know, Claude was also enthusiastic about it.
I read it and looked at the block diagrams. I still don't get it. You have "data integration problems". Many software share data. Use database. Problems solved.
My only complaint with this article is that it seems to be implying kafka that linkedIn's problem couldn't have been solved with a bunch of off-the-shelf tools.
What off the shelf tools in 2012 would you propose, exactly?
Sounds like MQTT?
MQTT wouldn't give you the persistence or the decoupling of fast and slow consumers.
Make it less event-orchestrated and use a db. It’s just a social network for recruiters it’s not as complicated as they like to pretend.
You don’t need push, it’s just a performance optimization that almost never justifies using a whole new tool.
So solve "ETLs into a data warehouse are hard to make low-latency and hard to manage in a large org" by... just hypothetical better "off the shelf tools". Or "don't want low latency because you're 'just' a recruiting tool, so who cares how quickly you can get insights into your business."
Go back to the article, it wasn't about event-sourcing or replacing a DB for application code.
> It’s just a social network for recruiters it’s not as complicated as they like to pretend.
Dismissing this as «just a social network» understates the real constraints: enormous scale, global privacy rules, graph queries, near-real-time feeds and abuse controls. Periodic DB queries can work at small scale, but at high volume they either arrive late or create bursts that starve the primary. Capturing changes once and pushing them through a distributed transaction log such as Kafka evens out load, improves data timeliness and lets multiple consumers process events safely and independently. It does add operational duties – schema contracts, idempotency and retention – yet those are well-understood trade-offs. The question is not push versus pull in the abstract, but which approach meets the timeliness, fan-out and reliability required.
> You don’t need push, it’s just a performance optimization that almost never justifies using a whole new tool.
It is not about drama but about fit for purpose at scale.
Pull can work well for modest workloads or narrow deltas, especially with DB features such as incremental materialised views or change tables. At large scale, periodic querying becomes costly and late: you either poll frequently and hammer the primary, or poll infrequently and accept stale data. Even with cursoring and jitter, polls create bursty load and poor tail latencies.
Push via change data capture into a distributed log such as Kafka addresses such pain points. The log decouples producers from consumers, smooths load, improves timeliness and lets multiple processors scale independently and replay for backfills. It also keeps the OLTP database focused on transactions rather than fan-out reads.
This is not free: push introduces operational work and design care – schema contracts, ordering being per-partition, duplicate delivery and idempotency, back-pressure and retention governance including data-protection deletes. The usual mitigations are the outbox pattern, idempotent consumers, DLQ's and documented data contracts. The data processing complexity now belongs in each consumer, not the data processing engine (e.g. a DB).
Compute–storage separation in modern databases raises single-cluster ceilings for storage and read scale, yet it does not solve single-writer limits or multi-region active-active writes. For heavy write fan-out and near-real-time propagation, a CDC-to-log pipeline remains the safer bet.
To sum it up, both pull and push are valid – engineering is all about each specific use case assessment and the trade-off analysis. For small or bounded scopes, a well-designed pull loop is simpler. As scale, fan-out and timeliness requirements grow, push delivers better timeliness, correctness and operability.
The only correct answer to the question asked is "I don't know the context, I need more information". Anything else is being a bad engineer.
Your solution to a queue and publish subscribe problem is to use a database?
Adding onto this.
> LinkedIn used site activity data (e.g. someone liked this, someone posted this)1 for many things - tracking fraud/abuse, matching jobs to users, training ML models, basic features of the website (e.g who viewed your profile, the newsfeed), warehouse ingestion for offline analysis/reporting and etc.
Who controls the database? Is it the fraud/abuse team responsible for the migrations? Does the ML team tell the Newsfeed team to stop doing so many writes because it's slowing things down?
And what would that off-the-shelf software have been?
So now LinkedIn has dropped Kafka and wrote their own called Northguard. More info here:
https://www.infoq.com/news/2025/06/linkedin-northguard-xinfr...
> According to LinkedIn's engineers, Kafka had become increasingly difficult to manage at LinkedIn's scale (32T records/day, 17 PB/day, 400K topics, 150 clusters).
Wtf is LinkedIn doing that they create 17 PB/day?????
[flagged]
In my experience, Apache Kafka must be understood not as an isolated messaging tool, but as a comprehensive data streaming platform. Its successful implementation demands a holistic approach that encompasses performance, governance, and lifecycle management. I have consistently found that simply adopting the technology without a robust supporting architecture is an ineffective practice that leads to operational challenges.
Based on my work managing large-scale Kafka environments across critical sectors, I have identified that their stability and efficiency are upheld by a set of essential practices and tools. These are the non-negotiable pillars for success:
Health Checks & Observability: Proactive cluster health monitoring and complete visibility into the data flow are paramount.
Failure Management: Implementing dedicated portals and processes for handling Dead-Letter Queues (DLQs) ensures that no critical information is lost during failures.
Automation & DevOps: I leverage Strimzi for Kubernetes-native cluster management, orchestrating it through ArgoCD and GitOps practices. This ensures consistent, secure, and repeatable deployments.
The correct application of these engineering principles allows for remarkable results. For instance, at a large fashion retail group, I successfully scaled an environment to handle a peak traffic of 480,000 TPS. This high-availability system is efficiently maintained by a lean operational team of just two junior-to-mid-level professionals.
From my perspective, success in adopting Kafka is determined by the business context and the maturity of the applied software engineering. The investment in a well-planned architecture and a robust support ecosystem has a clear return, paying for itself through a significant reduction in operational costs (OPEX) within an estimated two-year period.
Taming Kafka isn't about new, complex secrets. It's about applying the same robust software engineering and architecture fundamentals we've relied on for +50 years (Software Engineering). The platform is new (2011), the principles are not.
This appears to be a LLM generated comment; if my assumption is correct - please do not do this here. Thank you.
No, man, this wasn't done by an LLM. I actually do Kafka implementations and follow exactly the same script I described.
Perhaps the fact that I'm not a native English speaker may have caused this confusion. I just made sure my text was written correctly in English with a translator.
Non-native English speakers are 100% welcome on Hacker News, and the rest of us are amazed by how good your English usually is.
Automated comments are what people here feel strongly about rejecting. It isn't just the LLM-written ones that trigger such reactions, but also comments that have been filtered through translators/editors/checkers and so on. Readers are becoming hypersensitive to these, so it would be much better to just write in your own voice. The benefit of authenticity outweighs the cost of a few grammar or spelling errors.
[dead]