S3 is up there as one of my favorite tech products ever. Over the years I've used it for all sorts of things but most recently I've been using it to roll my own DB backup system.
One of the things that shocks me about the system is the level of object durability. A few years ago I was taking an AWS certification course and learned that their durability number means that one can expect to loose data about once every 10,000 years. Since then anytime I talk about S3's durability I bring up that example and it always seems to convey the point for a layperson.
And it's "simplicity" is truly elegant. When I first started using S3 I thought of it as a dumb storage location but as I learned more I realized that it had some wild features that they all managed to hide properly so when you start it looks like a simple system but you can gradually get deeper and deeper until your S3 bucket is doing some pretty sophisticated stuff.
Last thing I'll say is, you know your API is good when "S3 compatable API" is a selling point of your competitors.
> Last thing I'll say is, you know your API is good when "S3 compatable API" is a selling point of your competitors.
Counter-point: You know that you're the dominant player. See: .psd, .pdf, .xslx. Not particularly good file types, yet widely supported by competitor products.
Photoshop, PDF and Excel are all products that were truly much better than their competitors at the time of their introduction.
Every file format accumulates cruft over thirty years, especially when you have hundreds of millions of users and you have to expand the product for use cases the original developers never imagined. But that doesn’t mean the success wasn’t justified.
It’s designed for that level of durability, but it’s only as good as a single change or correlated set of hardware failures that can quickly change the theoretical durability model. Or even corrupting data is possible too.
You're totally correct, but these products also need to be specifically designed against these failure cases (i.e. it's more than just MTTR + MTTF == durability). You (of course) can't just run deployments without validating that the durability property is satisfied throughout the change.
Yep! There’s a lot of checksum verification, carefully orchestrated deployments, hardware diversity, erasure code selection, the list goes on and on. I help run a multi-exabyte storage system - I’ve seen a few things.
I've used it for server backups too, just a simple webserver. Built a script that takes the webserver files, config files and makes a database dump, packages it all into a .tar.gz file on monday mornings, and uploads it to S3 using a "write only into this bucket" access key. In S3 I had it set up so it sends me an email whenever a new file was added, and that anything older than 3 weeks is put into cold storage.
Of course, I lost that script when the server crashed, the one thing I didn't back up properly.
What it means is in any given year, you have a 1 in 10,000 chance that a data loss event occurs. It doesn’t stack like that.
If you had light bulbs that lasted 1,000 hrs on average, and you had 10k light bulbs, and turned them all on at once, then they would all last 1,000 hours on average. Some would die earlier and some later, but the top line number does not tell you anything about the distribution, only the average (mean). That’s what MTTF is; the mean time for a given part to where it has a greater likelihood to have failed by then vs not. It doesn’t tell you if the distribution of light bulbs burning out is 10 hrs or 500 hrs wide. it’s the latter, you’ll start seeing bulbs out within 750 hrs, but if the former it’d be 995 hrs before anything burned out.
I generally see object storage systems advertise 11 9s of availability. You would usually see a commercial distributed file system (obviously stuff like Ceph and Lustre will depend on your specific configuration) advertise less (to trade off performance for durability).
it's really not that impressive, but you have to use erasure coding (chop the data D in X parts, use these to generate Y extra pieces, and store all X+Y of them) iso replication (store D n times)
> but as I learned more I realized that it had some wild features that they all managed to hide properly so when you start it looks like a simple system but you can gradually get deeper and deeper until your S3 bucket is doing some pretty sophisticated stuff.
Over the years working on countless projects I’ve come to realize that the more “easy” something looks to an end user, the more work it took to make it that way. It takes a lot of work to create and polish something to the state where you’d call it graceful, elegant and beautiful.
There are exceptions for sure, but often times hidden under every delightful interface is an iceberg of complexity. When something “just works” you know there was a hell of a lot of effort that went into making it so.
It's funny—S3 started as a "simple" storage service, and now it's handling entire table abstractions. Reminds me how SQL was declared dead every few years, yet here we are, building ever more complex data solutions on top of supposedly simple foundations.
I instinctively distrust any software or protocol that implies it is "simple" in its name: SNMP, SMTP, TFTP, SQS, etc. They're usually the cause of an equal or more amount of headaches than alternatives.
Maybe such solutions are a reaction to previous more "complex" solutions, and they do indeed start simple, but inevitably get swallowed by the complexity monster with age.
TFTP is probably the exception to that rule. All the other protocols started out easily enough and added more and more cruft. TFTP stayed the way it's always been - minimalist, terrifyingly awful at most things, handy for a few corner cases. If you know when to use it and when to use something like SCP, you're golden.
If TFTP had gone the way of SNMP, we'd have 'tftp <src> <dest> --proto tcp --tls --retries 8 --log-type json' or some horrendous mess like that.
> I’ve seen other examples where customers guess at new APIs they hope that S3 will launch, and have scripts that run in the background probing them for years! When we launch new features that introduce new REST verbs, we typically have a dashboard to report the call frequency of requests to it, and it’s often the case that the team is surprised that the dashboard starts posting traffic as soon as it’s up, even before the feature launches, and they discover that it’s exactly these customer probes, guessing at a new feature.
This surprises me; has anyone done something similar and benefitted from it? It's the sort of thing where I feel like you'd maybe get a result 1% of the time if that, and then only years later when everyone has moved on from the problem they were facing at the time...
Anyone can create a CRUD API. It takes a _lot_ of work to make a CRUD API that scales with high availability and a reasonable consistency model. The vast majority of engineers would take months or years to write demo.
If you don't believe me, you might want to reconsider how skilled the average developer _really_ is.
That’s when you really know you hid all the complexity well. When people call your globally replicated data store with granular permissions, sophisticated data retention policies, versioning, and manage to have, what, seven (ten?) nines or something, “simple”.
No problem. I’m sure ChatGPT could cook up a replacement in a weekend. Like Dropbox it’s just rsync with some scripts that glue it together. How hard could it possibly be?
I mean people serve entire websites right out of s3 buckets. Using it as a crude CDN of sorts.
little history: When we were getting ready to do an API at DigitalOcean I got asked "uhm... how should it feel?" I thought about that for about 11 seconds and said "if all our APIs feel as good as S3, it should be fine" - it's a good API.
It's only sad that the SDKs are often on the heavy side, I remember that the npm package used to be multiple megabytes as it bundled large parts of the core AWS SDK. Nowadays, I believe it's still around half a megabyte.
I have a feeling that economies of scale have a point of diminishing returns. At what point does it become more costly and complicated to store your data on S3 versus just maintaining a server with RAID disks somewhere?
S3 is an engineering marvel, but it's an insanely complicated backend architecture just to store some files.
That's going to depend a lot on what your needs are, particularly in terms of redundancy and durability. S3 takes care of a lot of that for you.
One server with a RAID array can survive, usually, 1 or maybe 2 drive failures. The remaining drives in the array will have to do more work when a failed drive is replaced and data is copied to the new array member. This sometimes leads to additional failures before replacement completes, because all the drives in the array are probably all the same model bought at the same time and thus have similar manufacturing quality and materials. This is part of why it's generally said that RAID != backup.
You can make a local backup to something like another server with its own storage, external drives, or tape storage. Capacity, recovery time, and cost varies a lot across the available options here. Now you're protected against the original server failing, but you're not protected against location-based impacts - power/network outages, weather damage/flooding, fire, etc.
You can make a remote backup. That can be in a location you own / control, or you can pay someone else to use their storage.
Each layer of redundancy adds cost and complexity.
AWS says they can guarantee 99.999999999% durability and 99.99% availability. You can absolutely design your own system that meets those thresholds, but that is far beyond what one server with a RAID array can do.
How many businesses or applications really need 99.999999999% durability and 99.99% availability?
Is your whole stack organized to deliver the forementioned durability and availability?
There are a lot of companies who their livelihood depends on their proprietary data, and loss of that data would be a company-ending-event. I'm not sure how the calculus works out exactly, but having additional backups and types of backups to reduce risk is probably one of the smaller business expenses one can pick up. Sending a couple TB of data to three+ cloud providers on top of your physical backups is in the tens of dollars per month.
I think that this is, to Andy's point, basically about simplicity. It's not that your business necessarily needs 11 9s of durability for continuity purposes, but it sure is nice that you never have to think about the durability of the storage layer (vs. even something like EBS where 5 9s of durability isn't quite enough to go from "improbable" to "impossible").
Different people and organizations will have different needs, as indicated in the first sentence of my post. For some use cases one server is totally fine, but it's good to think through your use cases and understand how loss of availability or loss of data would impact you, and how much you're willing to pay to avoid that.
I'll note that data durability is a bit of a different concern than service availability. A service being down for some amount of time sucks, but it'll probably come back up at some point and life moves on. If data is lost completely, it's just gone. It's going to have to be re-created from other sources, generated fresh, or accepted as irreplaceable and lost forever.
Some use cases can tolerate losing some or all of the data. Many can't, so data durability tends to be a concern for non-trivial use cases.
There is a diminishing return of what percentage you save, sure. But amazon will always be at that edge. They already have amortized the equipment, labour, administration, electricity, storage, cooling, etc.
They also already have support for storage tiering, replication, encryption, ACLs, integration with other services (from web access to sending notifications of storage events to lambda, sqs, etc). Uou get all of this whether you're saving 1 eight bit file or trillions of gigabyte sized ones.
There are reasons why you may need to roll your own storage setup (regulatory, geographic, some other unique reason), but you'll never be more economical than S3, especially if the storage is mostly sitting idle.
Probably never. The complexity is borne by Amazon. Even before any of the development begins if you want a RAID setup with some sort of decent availability you've already multiplied your server costs by the number of replicas you'd need. It's a Sisyphean task that also has little value for most people.
Much like twitter it's conceptually simple but it's a hard problem to solve at any scale beyond a toy.
Lots of comments here talking about how great S3 is.
Anyone willing to give a cliff notes about what's good about it?
I've been running various sites and apps for a decade, but have never touched S3 because the bandwidth costs are 1, sometimes 2 orders of magnitude more expensive than other static hosting solutions.
There are a few things about S3 that I find extremely powerful.
The biggest is, if I need to store some data. I know what the data is (so I don't need to worry about needing to in a moment notice traverse a file structure for example, I know my filenames), I can store that data, I don't need to figure out how much space I need ahead of time and it is there when I need it. Maybe it automatically moves to another storage tier to save me some money but I can reliably assume it will be there when I need it. Just that simplicity alone is worth a lot, I never need to think later that I need to expand some space, possibly introducing downtime depending on the setup, maybe dealing with partitions, etc.
Related to that is static hosting. I have ran a CDN and other static content out of S3 with cloud front in front of it. The storage cost was almost non existent due to how little actual data we were talking about and only paid for cloudfront costs when there were requests. If nothing was being used it was almost "free". Even when being used it was very cheap for my use cases.
Creating daily inventory reports in S3 is awesome.
But the thing that really is almost "magic" once you understand its quirks. Athena (and quick sight built on top of that and similar tools). The ability to store data in S3 like inventory reports that I already mentioned, access logs, cloud watch logs, or any structured data that you may not need to query often enough to warrant a full long running database. It may cost you a few dollars to run your Athena query and it is not going to be super quick, but if you know what you're looking for it is amazing.
S3 has often fallen into a "catch all" solution for me whenever I need to store data large enough that I don't want to keep it in a database (RDBMS or Redis).
Need to save a file somewhere? Dump it in S3. It's generally affordable (obviously dependent on scale and use), fast, easy, and super configurable.
Being able to expose something to the outside, or with a presigned URL is a huge advantage as well.
Off the top of my head, I think of application storage generally in this tier ordering (just off the top of my head based on the past few years of software development, no real deep thought here):
1. General application data that needs to be read, written, and related - RDBMS
2. Application data that needs to be read and written fast, no relations - Redis
3. Application data that is mostly stored and read - S3
Replace any of those with an equivalent storage layer.
One underappreciated feature of S3 - that allowed it to excel in workloads like the Tables feature described in the article - is that it's able to function as the world's highest throughout network filesystem. And you don't have to do anything to configure it (as the article points out). By storing data on S3, you get to access the full cross-sectional bandwidth of EC2, which is colossal. For effectively all workloads, you will max out your network connection before S3's. This enables workloads that can't scale anywhere else. Things like data pipelines generating unplanned hundred-terabit-per-second traffic spikes with hotspots that would crash any filesystem cluster I've ever seen. And you don't have to pay a lot for it- once you're done using the bandwidth, you can archive the data elsewhere or delete it.
You've totally hit the nail on the head. This is the real moat of S3, the fact that they have so much front-end throughput available from the gigantic buildout that folks can take advantage of without any capacity pre-planning.
S3 is great for being able to stick files somewhere and not have to think about any of the surrounding infrastructure on an ongoing basis [1]. You don't have to worry about keeping a RAID server, swapping out disks when one fails, etc.
For static hosting, it's fine, but as you say, it's not necessarily the cheapest, though you can bring the cost down by sticking a CDN (Cloudflare/CloudFront) in front of it. There are other use cases where it really shines though.
[1]: I say ongoing basis because you will need to figure out your security controls, etc. at the beginning so it's not totally no-thought.
Since everything you need to run "a full SPA" is to serve some static files over an internet connection I'm not sure how that tells you anything interesting about the platform. It's basically the simplest thing a web server can do.
We've used Netlify at previous projects, we used it because it was easy. No AWS accounts or knowledge needed, just push to master, let the CI build (it was a Gatsby site) and it was live.
I think Netlify is great but to me it's overkill if you just have a static site.
I understand that Netlify is much simpler to get started with and setting up an AWS account is somewhat more complex. If you have several sites, it's worth spending the time to learn.
I really enjoy using S3 to serve arbitrary blobs. It perfectly solves the problem space for my use cases.
I avoid getting tangled in authentication mess by simply naming my files using type4 GUIDs and dumping them in public buckets. The file name is effectively the authentication token and expiration policies are used to deal with the edges.
This has been useful for problems like emailing customers gigantic reports and transferring build artifacts between systems. Having a stable URL that "just works" everywhere easily pays for the S3 bill in terms of time & frustration saved.
My favorite use case for S3 API-compatible solutions: I often run into systems that generate lots of arbitrary data that only have temporary importance. A common example might be intermediate build artifacts, or testing ephemera (browser screenshots, etc). Things that are needed for X number of months and then just need to disappear.
Yeah, we can dump those to a filesystem. But then we have to ask which filesystem? What should the directory layout be? If there are millions or billions of objects, walking the whole tree gets expensive. Do we write a script to clean everything up? Run it via cron or some other job runner?
With S3, you just write your artifact to S3 with a TTL and it gets deleted automagically when it should. No cron jobs, no walking the whole tree. And you can set up other lifecycle options if you need it moved to other (cheaper) storage later on, backups, versioning, and whatever else.
For on-prem, you have Minio, Garage, or SeaweedFS. These are pretty nice to deploy the servers however you need for the level of reliability/durability you require.
"I think one thing that we’ve learned from the work on Tables is that it’s these _properties of storage_ that really define S3 much more than the object API itself."
Between the above philosophy, S3 Tables, and Express One Zone (SSD perf), it makes me really curious about what other storage modalities S3 moves towards supporting going forward.
S3 was one of the first offerings coming out of AWS right? It’s pretty legendary and a great concept to begin with. You can tell by how much sense it makes and then trying to wrap your ahead around the web dev world pre-S3.
? The web dev world pre-S3 was pretty much the same, but you stored your files on a regular server (and set up your own redundancy and backup strategy). Not that much different to be honest from an end user's point of view.
If you read the article, they say that exactly. Its by Dr Werner Vogels, they know exactly what goes into S3, since they are a principal engineer on the project.
if only metadata could be queried without processing a csv output file first, imagine storing thumbnails in there even! copied objects had actual events, not something you have to dig cloudtrail for, you could get last update time from a bucket to make caching easier
whoops. I was wrong! you can store base64 encoded metadata in the object then run a HEAD request to get it, but its limited to 2kb.
also, you ~can~ query the metadata, but its latency is more suited to batch processing and not lambdas
Table stakes is a poker term meaning the absolute minimum amount you are allowed to bet. So the title translates to "In S3, simplicity is the bare minimum" or "In S3, simplicity is so important that if we didn't have it, we might as well not even have S3."
It's a risky idiom in general because it's often used to prevent debate. "Every existing product has feature X, so feature X is table stakes." "Why are we testing whether we really need this feature, it's table stakes!" My observation has been that "table stakes" features are often the best ones to reject. (Not so in the case of this title, though)
S3 is up there as one of my favorite tech products ever. Over the years I've used it for all sorts of things but most recently I've been using it to roll my own DB backup system.
One of the things that shocks me about the system is the level of object durability. A few years ago I was taking an AWS certification course and learned that their durability number means that one can expect to loose data about once every 10,000 years. Since then anytime I talk about S3's durability I bring up that example and it always seems to convey the point for a layperson.
And it's "simplicity" is truly elegant. When I first started using S3 I thought of it as a dumb storage location but as I learned more I realized that it had some wild features that they all managed to hide properly so when you start it looks like a simple system but you can gradually get deeper and deeper until your S3 bucket is doing some pretty sophisticated stuff.
Last thing I'll say is, you know your API is good when "S3 compatable API" is a selling point of your competitors.
> Last thing I'll say is, you know your API is good when "S3 compatable API" is a selling point of your competitors.
Counter-point: You know that you're the dominant player. See: .psd, .pdf, .xslx. Not particularly good file types, yet widely supported by competitor products.
Photoshop, PDF and Excel are all products that were truly much better than their competitors at the time of their introduction.
Every file format accumulates cruft over thirty years, especially when you have hundreds of millions of users and you have to expand the product for use cases the original developers never imagined. But that doesn’t mean the success wasn’t justified.
PDF is not a product. I get what you are day but I can’t say that I’ve ever liked Adobe Acrobat
PDF is a product, just like PostScript was.
It’s designed for that level of durability, but it’s only as good as a single change or correlated set of hardware failures that can quickly change the theoretical durability model. Or even corrupting data is possible too.
You're totally correct, but these products also need to be specifically designed against these failure cases (i.e. it's more than just MTTR + MTTF == durability). You (of course) can't just run deployments without validating that the durability property is satisfied throughout the change.
Yep! There’s a lot of checksum verification, carefully orchestrated deployments, hardware diversity, erasure code selection, the list goes on and on. I help run a multi-exabyte storage system - I’ve seen a few things.
I've used it for server backups too, just a simple webserver. Built a script that takes the webserver files, config files and makes a database dump, packages it all into a .tar.gz file on monday mornings, and uploads it to S3 using a "write only into this bucket" access key. In S3 I had it set up so it sends me an email whenever a new file was added, and that anything older than 3 weeks is put into cold storage.
Of course, I lost that script when the server crashed, the one thing I didn't back up properly.
> their durability number means that one can expect to loose data about once every 10,000 years
What does that mean? If I have 1 million objects, I loose 100 per year?
What it means is in any given year, you have a 1 in 10,000 chance that a data loss event occurs. It doesn’t stack like that.
If you had light bulbs that lasted 1,000 hrs on average, and you had 10k light bulbs, and turned them all on at once, then they would all last 1,000 hours on average. Some would die earlier and some later, but the top line number does not tell you anything about the distribution, only the average (mean). That’s what MTTF is; the mean time for a given part to where it has a greater likelihood to have failed by then vs not. It doesn’t tell you if the distribution of light bulbs burning out is 10 hrs or 500 hrs wide. it’s the latter, you’ll start seeing bulbs out within 750 hrs, but if the former it’d be 995 hrs before anything burned out.
Amazon claims 99.999999999% durability.
If you have ten million objects, you should lose one every 10k years or so.
How does that compare to competitors and things like distributed file systems?
I generally see object storage systems advertise 11 9s of availability. You would usually see a commercial distributed file system (obviously stuff like Ceph and Lustre will depend on your specific configuration) advertise less (to trade off performance for durability).
it's really not that impressive, but you have to use erasure coding (chop the data D in X parts, use these to generate Y extra pieces, and store all X+Y of them) iso replication (store D n times)
I’ve never worked with AWS, but have a certification from GCP and currently use Azure.
What do you see as special for S3? Isn’t it just another bucket?
> but as I learned more I realized that it had some wild features that they all managed to hide properly so when you start it looks like a simple system but you can gradually get deeper and deeper until your S3 bucket is doing some pretty sophisticated stuff.
Over the years working on countless projects I’ve come to realize that the more “easy” something looks to an end user, the more work it took to make it that way. It takes a lot of work to create and polish something to the state where you’d call it graceful, elegant and beautiful.
There are exceptions for sure, but often times hidden under every delightful interface is an iceberg of complexity. When something “just works” you know there was a hell of a lot of effort that went into making it so.
The durability is not so good when you have a lot of objects
It's funny—S3 started as a "simple" storage service, and now it's handling entire table abstractions. Reminds me how SQL was declared dead every few years, yet here we are, building ever more complex data solutions on top of supposedly simple foundations.
I instinctively distrust any software or protocol that implies it is "simple" in its name: SNMP, SMTP, TFTP, SQS, etc. They're usually the cause of an equal or more amount of headaches than alternatives.
Maybe such solutions are a reaction to previous more "complex" solutions, and they do indeed start simple, but inevitably get swallowed by the complexity monster with age.
TFTP is probably the exception to that rule. All the other protocols started out easily enough and added more and more cruft. TFTP stayed the way it's always been - minimalist, terrifyingly awful at most things, handy for a few corner cases. If you know when to use it and when to use something like SCP, you're golden.
If TFTP had gone the way of SNMP, we'd have 'tftp <src> <dest> --proto tcp --tls --retries 8 --log-type json' or some horrendous mess like that.
> I’ve seen other examples where customers guess at new APIs they hope that S3 will launch, and have scripts that run in the background probing them for years! When we launch new features that introduce new REST verbs, we typically have a dashboard to report the call frequency of requests to it, and it’s often the case that the team is surprised that the dashboard starts posting traffic as soon as it’s up, even before the feature launches, and they discover that it’s exactly these customer probes, guessing at a new feature.
This surprises me; has anyone done something similar and benefitted from it? It's the sort of thing where I feel like you'd maybe get a result 1% of the time if that, and then only years later when everyone has moved on from the problem they were facing at the time...
Maybe this is a faster way of getting AWS feature requests heard.
I'm going to write a script that keeps trying to call ecs:MakeFargateCacheImages.
It could also be hackers, as when a new service launches is exactly when it will be most buggy. And the contents of S3 are a big payoff.
S3 is the simplest CRUD app you could create.
It's essentially just the 4 functions of C.R.U.D done to a file.
Most problems in tech are not that simple.
Note: not knocking the service. just pointing out not all things are so inherently basic (and valuable at the same time).
Anyone can create a CRUD API. It takes a _lot_ of work to make a CRUD API that scales with high availability and a reasonable consistency model. The vast majority of engineers would take months or years to write demo.
If you don't believe me, you might want to reconsider how skilled the average developer _really_ is.
That's the public interface. The underlying architecture is where the power is.
Isn't that the most impressive part? That the abstraction makes it seem so simple
A file system is simple. Open, read, close. Most tech problems are not that simple. How hard could a filesystem be?
Locking, checks after unclean shutdown, sparse files, high performance, reliabilty.... are all things that make filesystems harder.
That’s when you really know you hid all the complexity well. When people call your globally replicated data store with granular permissions, sophisticated data retention policies, versioning, and manage to have, what, seven (ten?) nines or something, “simple”.
No problem. I’m sure ChatGPT could cook up a replacement in a weekend. Like Dropbox it’s just rsync with some scripts that glue it together. How hard could it possibly be?
I mean people serve entire websites right out of s3 buckets. Using it as a crude CDN of sorts.
It’s a modern marvel.
little history: When we were getting ready to do an API at DigitalOcean I got asked "uhm... how should it feel?" I thought about that for about 11 seconds and said "if all our APIs feel as good as S3, it should be fine" - it's a good API.
The API, absolutely.
It's only sad that the SDKs are often on the heavy side, I remember that the npm package used to be multiple megabytes as it bundled large parts of the core AWS SDK. Nowadays, I believe it's still around half a megabyte.
I don't know if the npm is the same way, but the java sdk now has options by service. So you can include just s3 instead of all of aws
I have a feeling that economies of scale have a point of diminishing returns. At what point does it become more costly and complicated to store your data on S3 versus just maintaining a server with RAID disks somewhere?
S3 is an engineering marvel, but it's an insanely complicated backend architecture just to store some files.
That's going to depend a lot on what your needs are, particularly in terms of redundancy and durability. S3 takes care of a lot of that for you.
One server with a RAID array can survive, usually, 1 or maybe 2 drive failures. The remaining drives in the array will have to do more work when a failed drive is replaced and data is copied to the new array member. This sometimes leads to additional failures before replacement completes, because all the drives in the array are probably all the same model bought at the same time and thus have similar manufacturing quality and materials. This is part of why it's generally said that RAID != backup.
You can make a local backup to something like another server with its own storage, external drives, or tape storage. Capacity, recovery time, and cost varies a lot across the available options here. Now you're protected against the original server failing, but you're not protected against location-based impacts - power/network outages, weather damage/flooding, fire, etc.
You can make a remote backup. That can be in a location you own / control, or you can pay someone else to use their storage.
Each layer of redundancy adds cost and complexity.
AWS says they can guarantee 99.999999999% durability and 99.99% availability. You can absolutely design your own system that meets those thresholds, but that is far beyond what one server with a RAID array can do.
How many businesses or applications really need 99.999999999% durability and 99.99% availability? Is your whole stack organized to deliver the forementioned durability and availability?
There are a lot of companies who their livelihood depends on their proprietary data, and loss of that data would be a company-ending-event. I'm not sure how the calculus works out exactly, but having additional backups and types of backups to reduce risk is probably one of the smaller business expenses one can pick up. Sending a couple TB of data to three+ cloud providers on top of your physical backups is in the tens of dollars per month.
I think that this is, to Andy's point, basically about simplicity. It's not that your business necessarily needs 11 9s of durability for continuity purposes, but it sure is nice that you never have to think about the durability of the storage layer (vs. even something like EBS where 5 9s of durability isn't quite enough to go from "improbable" to "impossible").
Different people and organizations will have different needs, as indicated in the first sentence of my post. For some use cases one server is totally fine, but it's good to think through your use cases and understand how loss of availability or loss of data would impact you, and how much you're willing to pay to avoid that.
I'll note that data durability is a bit of a different concern than service availability. A service being down for some amount of time sucks, but it'll probably come back up at some point and life moves on. If data is lost completely, it's just gone. It's going to have to be re-created from other sources, generated fresh, or accepted as irreplaceable and lost forever.
Some use cases can tolerate losing some or all of the data. Many can't, so data durability tends to be a concern for non-trivial use cases.
> One server with a RAID array can survive, usually, 1 or maybe 2 drive failures.
Standard RAID configurations can only handle 2 failures, but there are libraries and filesystems allowing arbitrarily high redundancy.
As long as it's all in one server, there's still a lot of situations that can immediately cut through all that redundancy.
As long as it's all in one physical location, there's still fire and weather as ways to immediately cut through all that redundancy.
There is a diminishing return of what percentage you save, sure. But amazon will always be at that edge. They already have amortized the equipment, labour, administration, electricity, storage, cooling, etc.
They also already have support for storage tiering, replication, encryption, ACLs, integration with other services (from web access to sending notifications of storage events to lambda, sqs, etc). Uou get all of this whether you're saving 1 eight bit file or trillions of gigabyte sized ones.
There are reasons why you may need to roll your own storage setup (regulatory, geographic, some other unique reason), but you'll never be more economical than S3, especially if the storage is mostly sitting idle.
There are a few stories from companies that moved away from S3, like Dropbox, and who shared their investments and expenses.
The long and short of it is that getting anywhere near the redundancy, reliability, performance etc of S3, you're spending A Lot of money.
Probably never. The complexity is borne by Amazon. Even before any of the development begins if you want a RAID setup with some sort of decent availability you've already multiplied your server costs by the number of replicas you'd need. It's a Sisyphean task that also has little value for most people.
Much like twitter it's conceptually simple but it's a hard problem to solve at any scale beyond a toy.
Lots of comments here talking about how great S3 is.
Anyone willing to give a cliff notes about what's good about it?
I've been running various sites and apps for a decade, but have never touched S3 because the bandwidth costs are 1, sometimes 2 orders of magnitude more expensive than other static hosting solutions.
There are a few things about S3 that I find extremely powerful.
The biggest is, if I need to store some data. I know what the data is (so I don't need to worry about needing to in a moment notice traverse a file structure for example, I know my filenames), I can store that data, I don't need to figure out how much space I need ahead of time and it is there when I need it. Maybe it automatically moves to another storage tier to save me some money but I can reliably assume it will be there when I need it. Just that simplicity alone is worth a lot, I never need to think later that I need to expand some space, possibly introducing downtime depending on the setup, maybe dealing with partitions, etc.
Related to that is static hosting. I have ran a CDN and other static content out of S3 with cloud front in front of it. The storage cost was almost non existent due to how little actual data we were talking about and only paid for cloudfront costs when there were requests. If nothing was being used it was almost "free". Even when being used it was very cheap for my use cases.
Creating daily inventory reports in S3 is awesome.
But the thing that really is almost "magic" once you understand its quirks. Athena (and quick sight built on top of that and similar tools). The ability to store data in S3 like inventory reports that I already mentioned, access logs, cloud watch logs, or any structured data that you may not need to query often enough to warrant a full long running database. It may cost you a few dollars to run your Athena query and it is not going to be super quick, but if you know what you're looking for it is amazing.
S3 has often fallen into a "catch all" solution for me whenever I need to store data large enough that I don't want to keep it in a database (RDBMS or Redis).
Need to save a file somewhere? Dump it in S3. It's generally affordable (obviously dependent on scale and use), fast, easy, and super configurable.
Being able to expose something to the outside, or with a presigned URL is a huge advantage as well.
Off the top of my head, I think of application storage generally in this tier ordering (just off the top of my head based on the past few years of software development, no real deep thought here):
1. General application data that needs to be read, written, and related - RDBMS
2. Application data that needs to be read and written fast, no relations - Redis
3. Application data that is mostly stored and read - S3
Replace any of those with an equivalent storage layer.
One underappreciated feature of S3 - that allowed it to excel in workloads like the Tables feature described in the article - is that it's able to function as the world's highest throughout network filesystem. And you don't have to do anything to configure it (as the article points out). By storing data on S3, you get to access the full cross-sectional bandwidth of EC2, which is colossal. For effectively all workloads, you will max out your network connection before S3's. This enables workloads that can't scale anywhere else. Things like data pipelines generating unplanned hundred-terabit-per-second traffic spikes with hotspots that would crash any filesystem cluster I've ever seen. And you don't have to pay a lot for it- once you're done using the bandwidth, you can archive the data elsewhere or delete it.
You've totally hit the nail on the head. This is the real moat of S3, the fact that they have so much front-end throughput available from the gigantic buildout that folks can take advantage of without any capacity pre-planning.
S3 is great for being able to stick files somewhere and not have to think about any of the surrounding infrastructure on an ongoing basis [1]. You don't have to worry about keeping a RAID server, swapping out disks when one fails, etc.
For static hosting, it's fine, but as you say, it's not necessarily the cheapest, though you can bring the cost down by sticking a CDN (Cloudflare/CloudFront) in front of it. There are other use cases where it really shines though.
[1]: I say ongoing basis because you will need to figure out your security controls, etc. at the beginning so it's not totally no-thought.
I found out last year that you can actually run a full SPA using S3 and a CDN. It’s kind of a nuts platform
Since everything you need to run "a full SPA" is to serve some static files over an internet connection I'm not sure how that tells you anything interesting about the platform. It's basically the simplest thing a web server can do.
I use S3+Cloudfront for static sites and Cloudflare workers if it needed.
It's always crazy to me that people will run a could be static site on Netlify/Vercel/etc.
We've used Netlify at previous projects, we used it because it was easy. No AWS accounts or knowledge needed, just push to master, let the CI build (it was a Gatsby site) and it was live.
I think Netlify is great but to me it's overkill if you just have a static site.
I understand that Netlify is much simpler to get started with and setting up an AWS account is somewhat more complex. If you have several sites, it's worth spending the time to learn.
I really enjoy using S3 to serve arbitrary blobs. It perfectly solves the problem space for my use cases.
I avoid getting tangled in authentication mess by simply naming my files using type4 GUIDs and dumping them in public buckets. The file name is effectively the authentication token and expiration policies are used to deal with the edges.
This has been useful for problems like emailing customers gigantic reports and transferring build artifacts between systems. Having a stable URL that "just works" everywhere easily pays for the S3 bill in terms of time & frustration saved.
My favorite use case for S3 API-compatible solutions: I often run into systems that generate lots of arbitrary data that only have temporary importance. A common example might be intermediate build artifacts, or testing ephemera (browser screenshots, etc). Things that are needed for X number of months and then just need to disappear.
Yeah, we can dump those to a filesystem. But then we have to ask which filesystem? What should the directory layout be? If there are millions or billions of objects, walking the whole tree gets expensive. Do we write a script to clean everything up? Run it via cron or some other job runner?
With S3, you just write your artifact to S3 with a TTL and it gets deleted automagically when it should. No cron jobs, no walking the whole tree. And you can set up other lifecycle options if you need it moved to other (cheaper) storage later on, backups, versioning, and whatever else.
For on-prem, you have Minio, Garage, or SeaweedFS. These are pretty nice to deploy the servers however you need for the level of reliability/durability you require.
"I think one thing that we’ve learned from the work on Tables is that it’s these _properties of storage_ that really define S3 much more than the object API itself."
Between the above philosophy, S3 Tables, and Express One Zone (SSD perf), it makes me really curious about what other storage modalities S3 moves towards supporting going forward.
> In S3 simplicity is table stakes
The S3 API is not "simple."
Authentication being a good part of that.
S3 was one of the first offerings coming out of AWS right? It’s pretty legendary and a great concept to begin with. You can tell by how much sense it makes and then trying to wrap your ahead around the web dev world pre-S3.
? The web dev world pre-S3 was pretty much the same, but you stored your files on a regular server (and set up your own redundancy and backup strategy). Not that much different to be honest from an end user's point of view.
S3 is the closest thing to magic I have seen. It deserves a Whitman, but Gemini will have to do:
A lattice woven, intricate and deep, Where countless objects safely lie asleep.
To lean upon its strength, its constant hum, A silent guardian, 'til kingdom come.
Versioning whispers tales of changes past, While lifecycle rules decree what shadows last.
The endless hunger of a world online, S3's silent symphony, a work divine.
Consider then, this edifice of thought, A concept forged, meticulously wrought.
More than a bucket, more than simple store, A foundational layer, forevermore.
Would have been good if they mentioned they meant Amazon S3. It took me a while to figure out what this was about.
Initially I thought this was about S3 standby mode.
I can guarantee you, nothing is simple about S3. I bet engineers spend months on a single configuration change to the underlying subsystems.
If you read the article, they say that exactly. Its by Dr Werner Vogels, they know exactly what goes into S3, since they are a principal engineer on the project.
It’s Vogels’ blog but this is a guest post by Andy Warfield.
My mistake, I stand corrected.
That’s what makes it a good abstraction.
if only metadata could be queried without processing a csv output file first, imagine storing thumbnails in there even! copied objects had actual events, not something you have to dig cloudtrail for, you could get last update time from a bucket to make caching easier
When S3 Tables launched they made the Metadata available using this technology. So you can query it like an Apache Iceberg table.
https://aws.amazon.com/blogs/aws/introducing-queryable-objec...
Are you talking about getting metadata from many objects in the bucket simultaneously? You might be interested in S3 Metadata https://docs.aws.amazon.com/AmazonS3/latest/userguide/UsingM....
whoops. I was wrong! you can store base64 encoded metadata in the object then run a HEAD request to get it, but its limited to 2kb. also, you ~can~ query the metadata, but its latency is more suited to batch processing and not lambdas
tf is that title supposed to mean?
Table stakes is a poker term meaning the absolute minimum amount you are allowed to bet. So the title translates to "In S3, simplicity is the bare minimum" or "In S3, simplicity is so important that if we didn't have it, we might as well not even have S3."
It's a risky idiom in general because it's often used to prevent debate. "Every existing product has feature X, so feature X is table stakes." "Why are we testing whether we really need this feature, it's table stakes!" My observation has been that "table stakes" features are often the best ones to reject. (Not so in the case of this title, though)
Somehow gambling is directly correlated to using S3? That’s the best I got.