Not a data lake expert here, but I think there's a parallel with telemetry/logging for video games. The author talks about mass "ingestion without purpose," which is what I've sometimes flubbed in the past—during playtesting, we'd collect piles of data about where the players won/lost/moved/whatever, and there was so darned much of it that we'd just toss our hands up and not do anything with it.
The article suggests (sensibly!) that the right thing to do is to actually go in and design what's being logged with purpose (i.e., everything should answer a known question). But I don't always know what I'm looking for when I'm testing a thing. At the risk of using LLMs as a hammer that hammers everything, I wonder if they'd help with that problem?
I've pointed Codex at Posthog logs, for instance, and it's been able to say, "yeah, well, look at this pile of residential proxies hitting your site." I wonder if it'd similarly be able to say, "hey, I notice a pattern where everyone who picks the Cleric class stops playing after level 3"?
It used to be normal for video game playtesting departments to simply film the player playing the game. Things that are important are usually on screen if your game doesn't suck. It may be tempting to record everything that happens in the game world, but if you can't work it out from what's on screen, neither can your prospective players.
I'm building a product and exploring these questions in that domain. I'm thinking about it a little differently. I have a known set of hardware sensors that I'm collecting all the data from. Per sensor the data has a strong schema with a timestamp, what is interesting is the insights that can be gleaned across sensors. So what sensors do you have that you want to record, have a strong schema against that sensor, now have something you can use to tie across sensors, now you can use LLMs to query the data and draw interesting insights
Yeah, my impression is that even the best LLMs are pretty bad at analysing big data directly.
It's not even the hallucinations that are the biggest problem, it's more that they're so bad at managing their context windows that they end up ignoring huge chunks of the data without realising that they're doing it. You end up with a result that looks plausible but is often extremely misleading.
A better approach seems to be getting the LLM to write code to use more traditional analysis techniques (eg. iterative k-means, or whatever). That way you can at least be confident that you've looked at all the data rather than just a small slice of it.
The paradigm shift from ETL to ELT is what went wrong with Data Lakes. An enormous amount of business logic and effort was historically accumulated in the Extract-Transform stage (usually in COBOL or SQL), and then Loaded into a Data Mart for end-users.
When ELT became the corporate "Strategy", Extract-Load was technically easy and had a lot of Big Data evangelism and exuberance behind it. The money flowed easily at the beginning, but budgets aren't unlimited and corporate knowledge isn't easy to retain. So Data Lakes became Swamps of just Extract-Load, and business users were told to transform the data themselves.
As someone who lived through it for 12 years at an Insurance company, I like to think of it as ETL becoming Extract-Load-GoFuckYourself.
> The paradigm shift from ETL to ELT is what went wrong with Data Lakes.
It wasn't exactly the shift from ETL to ELT, it was the stated goal of putting "T" on read with no intermediate schema.
In my experience, it's always (since way before data lakes) been a multi-step process and I don't know how it's even possible to do the data lake strategy without re-creating a ton of transforms over and over:
1-Extract
2-Possibly do some transforms early due to specific operational/technical reasons
3-Load - even before "ELT" became a term, load without transform was happening due to constraints, like the transforms requiring data from separate operational systems to even be possible
4-Transform 1 - Foundational stuff like converting unique operational schemas (due to technical or functional reasons) into more natural schemas, easier processing in later steps
5-Transform 2 - Unifying core data that spans multiple operational systems (e.g. order fulfillment txn appears naturally in different systems but for different/overlapping business reasons and with some differences in identifiers and attributes)
6-Transform 3 - First pass at foundational target state
ive lived etl and elt eras. i would never, ever go back to etl. having your business logic baked into ssis or some god forsaken piece of software was truly awful. being stuck in the rigidity of the integration or package that transformed the data in flight until some poor underpaid engineer is whipped to go make a change to it and it breaks the whole package... having to migrate legacy stuff and pick through an ssis pack age only to find no one remembers why logic is the way that it is makes me wanna find a ledge. etl was a terrible paradigm. maybe people see it through rose tinted glasses because "usable data" was what landed. servicing and scaling over a thousand etl pipelines is the worst ball torture ever, trust me. for the most part i dont think anyone needs to care much about the how/why with data lakes, a competent data strategy involves taking landed elt data and transforming it into usable reporting/integration data that becomes self service. at my last org the whole damn data lake concept freed one small subset of IT that focused on data needs and transformed the organization: all business units could finally hire their own sql analysts because data was consolidated in a warehouse and was being synchronized from many systems and applications and the all those dreams of self-serve reporting came alive. it was a monumental organizational step that changed the face of the org forever.
not much of a dbt nerd myself but sql is at least a language analysts understand, so if they go peep dynamic tables / materialized views on whats transforming the data they can get a wrangle on it. this requires no expertise in file movement and threading and surrounding whatever, the business nerds can stay business nerding and influence and at some orgs have the latitude to control the materialization logic without needing to be an engineer. it's a much better fit since analysts tend to be closer to business needs.
i dont mind blasting data in blob storage. theres a couple use cases where the "lake" thing works out, extending a postgres db to read the same data store via pglake (or duckdb + ducklake) is actually pretty slick. i build some apps that also have report pages that do some pretty heavy analytics. powerbi is effectively dead to me now.
where i think it all went wrong is literally just nomenclature. such stupid namings in the space, self importance galore. datalake is already dumb, theres ten trillion more terribly named things in the "data engineering" space, and this was an era that was largely swamped by linkedin/medium/enshittification (we got snowflake astroturfing their own subreddit trying to get people to pay for their dumb cortex ai) so data engineering is really hard to take seriously sometimes. not to mention the existence of what i consider blip in history saas services like fivetran (still not sure how these guys are still in business). it was a career step towards software for me, but looking back it is a pretty funny space shrouded in too many services and licensing. one finds it hard not to champion for postgres, duckdb, etc.
with that said, data lakes have been ridiculously effective at my last org. moving engineering full time into real time replication lords and having a whole warehouse full of everything you can possibly need enabled even greater heights for us. integrations got to just delta system vs system and true them up on 1-5 minute schedules with literally just a job that runs a select from some view which does the delta compare. doesnt sound like much but when you work at one of those dumb orgs that pays for like 75 different platforms, suddenly it started only taking us days to onboard and wire in a whole platform into the fold at the org and build out middlewares, it's actually hilarious. all of that is possible because of data lakes for the most part, though we didn't always do the "lake" thing and just materialized stuff into tables. being able to make live reporting from just about every system and bring data together acrossed many systems actually insulates us from hemmorhaging way too much money on stupid shit like salesforce and a new erp. you get to buy best in class platforms for whatever function and then your erp is just.. a damn multipage app. all that was possible with our datalake constantly having data dumped into it and going thru the infamous and inevitable end game pipeline: the road to reporting
Whenever I encounter a "data lake" it's just a bunch of random CSV and JSON files on S3, and a bunch of half baked Python scripts to query it. And usually somebody tells me it's "big data", and that's why it is has to be that way. So I check the size and it would easily fit on a laptop from 15 years ago.
The first time (and all others) I heard "data lake" my brain instantly translated it to "somone elses problem". As in the very name implies a carelessness to just dump whatever and somone else will take care of it.
The people who told us to push all of our data to the data lake are the ones who are now telling us to feed all of our knowledge to AI. In both cases, the dream was to replace us with computation, either ML or LLM's. We survived the first one relatively unscathed, don't know about the second one yet, at least for my own job.
It’s hard to take an article seriously that uses a snowflake ceo maligning Hadoop as evidence of something. (Note, I am already a believer that data lake efforts usually ended badly.)
Regardless of whether their criticism is factual, it’s absurd to use a direct competitors assessment of the competing tech as a credible source!
If you do that in the opening paragraphs of your paper, it’s wrecked all credibility for everything that follows.
Not a data lake expert here, but I think there's a parallel with telemetry/logging for video games. The author talks about mass "ingestion without purpose," which is what I've sometimes flubbed in the past—during playtesting, we'd collect piles of data about where the players won/lost/moved/whatever, and there was so darned much of it that we'd just toss our hands up and not do anything with it.
The article suggests (sensibly!) that the right thing to do is to actually go in and design what's being logged with purpose (i.e., everything should answer a known question). But I don't always know what I'm looking for when I'm testing a thing. At the risk of using LLMs as a hammer that hammers everything, I wonder if they'd help with that problem?
I've pointed Codex at Posthog logs, for instance, and it's been able to say, "yeah, well, look at this pile of residential proxies hitting your site." I wonder if it'd similarly be able to say, "hey, I notice a pattern where everyone who picks the Cleric class stops playing after level 3"?
It used to be normal for video game playtesting departments to simply film the player playing the game. Things that are important are usually on screen if your game doesn't suck. It may be tempting to record everything that happens in the game world, but if you can't work it out from what's on screen, neither can your prospective players.
I'm building a product and exploring these questions in that domain. I'm thinking about it a little differently. I have a known set of hardware sensors that I'm collecting all the data from. Per sensor the data has a strong schema with a timestamp, what is interesting is the insights that can be gleaned across sensors. So what sensors do you have that you want to record, have a strong schema against that sensor, now have something you can use to tie across sensors, now you can use LLMs to query the data and draw interesting insights
You could even ask the LLM what other sensors fit here and how would it expand my knowledge in collaboration with the other sensors and data feeds.
> look at this pile of residential proxies hitting your site
How does it know that?
Yeah, my impression is that even the best LLMs are pretty bad at analysing big data directly.
It's not even the hallucinations that are the biggest problem, it's more that they're so bad at managing their context windows that they end up ignoring huge chunks of the data without realising that they're doing it. You end up with a result that looks plausible but is often extremely misleading.
A better approach seems to be getting the LLM to write code to use more traditional analysis techniques (eg. iterative k-means, or whatever). That way you can at least be confident that you've looked at all the data rather than just a small slice of it.
The paradigm shift from ETL to ELT is what went wrong with Data Lakes. An enormous amount of business logic and effort was historically accumulated in the Extract-Transform stage (usually in COBOL or SQL), and then Loaded into a Data Mart for end-users.
When ELT became the corporate "Strategy", Extract-Load was technically easy and had a lot of Big Data evangelism and exuberance behind it. The money flowed easily at the beginning, but budgets aren't unlimited and corporate knowledge isn't easy to retain. So Data Lakes became Swamps of just Extract-Load, and business users were told to transform the data themselves.
As someone who lived through it for 12 years at an Insurance company, I like to think of it as ETL becoming Extract-Load-GoFuckYourself.
> The paradigm shift from ETL to ELT is what went wrong with Data Lakes.
It wasn't exactly the shift from ETL to ELT, it was the stated goal of putting "T" on read with no intermediate schema.
In my experience, it's always (since way before data lakes) been a multi-step process and I don't know how it's even possible to do the data lake strategy without re-creating a ton of transforms over and over:
1-Extract
2-Possibly do some transforms early due to specific operational/technical reasons
3-Load - even before "ELT" became a term, load without transform was happening due to constraints, like the transforms requiring data from separate operational systems to even be possible
4-Transform 1 - Foundational stuff like converting unique operational schemas (due to technical or functional reasons) into more natural schemas, easier processing in later steps
5-Transform 2 - Unifying core data that spans multiple operational systems (e.g. order fulfillment txn appears naturally in different systems but for different/overlapping business reasons and with some differences in identifiers and attributes)
6-Transform 3 - First pass at foundational target state
7-Transform 4 - Possibly more, or possibly not
8-Consume valuable data
ive lived etl and elt eras. i would never, ever go back to etl. having your business logic baked into ssis or some god forsaken piece of software was truly awful. being stuck in the rigidity of the integration or package that transformed the data in flight until some poor underpaid engineer is whipped to go make a change to it and it breaks the whole package... having to migrate legacy stuff and pick through an ssis pack age only to find no one remembers why logic is the way that it is makes me wanna find a ledge. etl was a terrible paradigm. maybe people see it through rose tinted glasses because "usable data" was what landed. servicing and scaling over a thousand etl pipelines is the worst ball torture ever, trust me. for the most part i dont think anyone needs to care much about the how/why with data lakes, a competent data strategy involves taking landed elt data and transforming it into usable reporting/integration data that becomes self service. at my last org the whole damn data lake concept freed one small subset of IT that focused on data needs and transformed the organization: all business units could finally hire their own sql analysts because data was consolidated in a warehouse and was being synchronized from many systems and applications and the all those dreams of self-serve reporting came alive. it was a monumental organizational step that changed the face of the org forever.
not much of a dbt nerd myself but sql is at least a language analysts understand, so if they go peep dynamic tables / materialized views on whats transforming the data they can get a wrangle on it. this requires no expertise in file movement and threading and surrounding whatever, the business nerds can stay business nerding and influence and at some orgs have the latitude to control the materialization logic without needing to be an engineer. it's a much better fit since analysts tend to be closer to business needs.
i dont mind blasting data in blob storage. theres a couple use cases where the "lake" thing works out, extending a postgres db to read the same data store via pglake (or duckdb + ducklake) is actually pretty slick. i build some apps that also have report pages that do some pretty heavy analytics. powerbi is effectively dead to me now.
where i think it all went wrong is literally just nomenclature. such stupid namings in the space, self importance galore. datalake is already dumb, theres ten trillion more terribly named things in the "data engineering" space, and this was an era that was largely swamped by linkedin/medium/enshittification (we got snowflake astroturfing their own subreddit trying to get people to pay for their dumb cortex ai) so data engineering is really hard to take seriously sometimes. not to mention the existence of what i consider blip in history saas services like fivetran (still not sure how these guys are still in business). it was a career step towards software for me, but looking back it is a pretty funny space shrouded in too many services and licensing. one finds it hard not to champion for postgres, duckdb, etc.
with that said, data lakes have been ridiculously effective at my last org. moving engineering full time into real time replication lords and having a whole warehouse full of everything you can possibly need enabled even greater heights for us. integrations got to just delta system vs system and true them up on 1-5 minute schedules with literally just a job that runs a select from some view which does the delta compare. doesnt sound like much but when you work at one of those dumb orgs that pays for like 75 different platforms, suddenly it started only taking us days to onboard and wire in a whole platform into the fold at the org and build out middlewares, it's actually hilarious. all of that is possible because of data lakes for the most part, though we didn't always do the "lake" thing and just materialized stuff into tables. being able to make live reporting from just about every system and bring data together acrossed many systems actually insulates us from hemmorhaging way too much money on stupid shit like salesforce and a new erp. you get to buy best in class platforms for whatever function and then your erp is just.. a damn multipage app. all that was possible with our datalake constantly having data dumped into it and going thru the infamous and inevitable end game pipeline: the road to reporting
Whenever I encounter a "data lake" it's just a bunch of random CSV and JSON files on S3, and a bunch of half baked Python scripts to query it. And usually somebody tells me it's "big data", and that's why it is has to be that way. So I check the size and it would easily fit on a laptop from 15 years ago.
The first time (and all others) I heard "data lake" my brain instantly translated it to "somone elses problem". As in the very name implies a carelessness to just dump whatever and somone else will take care of it.
Bingo. Double points if the infra surrounding this costs more than a maxed out MBP a month in spend
The people who told us to push all of our data to the data lake are the ones who are now telling us to feed all of our knowledge to AI. In both cases, the dream was to replace us with computation, either ML or LLM's. We survived the first one relatively unscathed, don't know about the second one yet, at least for my own job.
It’s hard to take an article seriously that uses a snowflake ceo maligning Hadoop as evidence of something. (Note, I am already a believer that data lake efforts usually ended badly.)
Regardless of whether their criticism is factual, it’s absurd to use a direct competitors assessment of the competing tech as a credible source!
If you do that in the opening paragraphs of your paper, it’s wrecked all credibility for everything that follows.
I have the same feeling about quoting Gartner studies, tbh.
The paper is actually poor in data and mostly requotes other studies.
I'd say Bob Muglia is pretty credible