I think D4M can be the real and universal magic box alternative mentioned in the article where "magic box knows how to do only one thing: it reads from Kafka, builds Iceberg tables, and keeps them compacted" [1]. D4M can be even better than the proposed TableFlow solution because as it stands now TableFlow cannot fully perform all the listed 12 points that automates all of the annoying parts about generating and maintaining Iceberg tables. Above all D4M is founded by sound mathematics similar to SQL [2].
[1] D4M: Dynamic Distributed Dimensional Data Model:
I think D4M can be the real and universal magic box alternative mentioned in the article where "magic box knows how to do only one thing: it reads from Kafka, builds Iceberg tables, and keeps them compacted" [1]. D4M can be even better than the proposed TableFlow solution because as it stands now TableFlow cannot fully perform all the listed 12 points that automates all of the annoying parts about generating and maintaining Iceberg tables. Above all D4M is founded by sound mathematics similar to SQL [2].
[1] D4M: Dynamic Distributed Dimensional Data Model:
https://d4m.mit.edu/
[2] Mathematics of Big Data: Spreadsheets, Databases, Matrices, and Graphs:
https://mitpress.mit.edu/9780262038393/mathematics-of-big-da...