The takeaways at the very bottom of the page are valuable:
> Overall, having spent a significant amount of time building this project, scaling it up to the size it’s at now, as well as analysing the data, the main conclusion is that it is not worth building your own solution, and investing this much time. When I first started building this project 3 years ago, I expected to learn way more surprising and interesting facts. There were some, and it’s super interesting to look through those graphs, however retrospectively, it did not justify the hundreds of hours I invested in this project.
The whole "quantified self" movement might be more about OCD and perfectionism than anything else.
I've been wearing an Apple Watch for close to 10 years. I've tracked my weight as well along those years but nothing crazy like OP. The Apple watch tracked plenty.
I had some strange symptoms and two doctors insisted I had a weak heart and potential heart failure. This was shocking! Turns out I do have a really "weak" rhythm, but heart failure is when your heart is progressively getting worse in it's pumping. I don't even remember which metric he looked at in my Apple health - but basically my heart has always been this way. A doctor looking at a single data point might think I have abnormally low blood pressure/heart rate, but if I've had this for 10 years with no change, the medical assessment is very different - it means nothing. Sometimes boring data is exactly what you need. For this reason, I will probably always wear an Apple watch (or equivalent) moving forward.
Data can feel useless for 10 years until one day it becomes critical. The benefit is spiky and uneven.
But you didn’t spend hundreds of hours on it, so when it did happen to be useful it seemed like an outsized benefit.
I would wager that for most people, most data about themselves will be useless and not worth collecting.
Of course you can’t know what data will be useless or not, so unless the cost of collecting it is minimal or nil (wearing a smart watch, writing down your weight each day/week), it’s probably not worth it.
Spending hundreds of hours to build a solution to capture all data about yourself to find interesting patterns has a huge assumption baked into it: that there are interesting patterns to find.
About 9 years ago I had a run-in with stomach cancer. After a few months of chemo and a 7 hour operation I was eventually declared cancer free and have been ever since, but still have to live with the consequences of the treatment and be vigilant for any signs of it returning.
I still suffer intermittent stomach aches, especially in the early hours of the morning, and had a terrible time trying to decide if they were getting better or worse over time.
Our narrative voice is awful at detecting long term trends and tends to overcompensate for particularly good or bad patches so it was impossible for me to judge and I started keeping records of how bad the aches were each day.
Long story short, the average severity was mostly decreasing over time and the average time between bad aches was slowly increasing but it would have been impossible to tell if this was happening without keeping detailed records because it wasn't consistent - some months were much worse than others and completely skewed my perception of long term trends.
While most people hopefully won't ever need to do something like this, it did make me realise just how bad we are at picking up on long term trends so I can definitely see keeping daily records of, for instance, average daily happiness being eye-opening.
Sounds like you had a very good reason to be tracking your stomachaches. But almost everyone else aside from you And people In similar situations would be wasting their time doing that, I would imagine.
Yes, when you have an obvious reason to track some data, do it: I do not think anyone questions that.
Proactively capturing and tracking everything you can to prep for any future is too much work that would really steal your time from, you know, actually living a life.
Like anything else, I think it comes down to having a good use case.
I've gotten deep into weightlifting/bodybuilding over the past couple of years, and that's the kind of hobby where micro-optimizations and data tracking can have a pretty big impact on results (and sort of necessary, you can't fly blind with things like diet, especially)
E.g. I track and weigh everything I eat, take body measuraments on a weekly basis, Dexa scans every few months, etc - for me it's worth it because I know what I want to do with the data. If I didn't have a goal, all that tracking would clearly be overkill.
How long have you been tracking? Can you share an insight you've had from your data?
I've been weight lighting for ten years and initially tried to track things (down to how many reps I did of which exercise, with how much weight) and quickly came to the conclusion that is want worth it for me.
I initially came to the same conclusion. Though I lifted in accord with decent training principles regarding reps and sets, I didn't track for years. As I entered middle age, I started keeping a training log (just one big org file in emacs), mostly out of curiousity. As I entered my 50s, I experienced what Haruki Murakami references in "What I Talk About When I Talk About Running" --- Fat is easy to gain and hard to lose. Muscle is hard to gain and easy to lose.
Now I track a couple of critical metrics and it's working great. I weigh first thing every day, track all kcals (even if I overeat), plan and track workouts. I write my own plans pulled from principles in these books (don't work for the company, just a satisfied customer) https://muscleandstrengthpyramids.com/
I don't use the vast majority of the info in those books as I'm just a hobbyist who wants to be healthy and strong.
The biggest shift came from learning I was doing waaay too much training volume at the gym while trying to lose fat too quickly; a fine recipe for injury. Now, when I'm in a fat loss phase, I try to lose it as slowly as possible while still making progress. Strength training and fat loss is a very long very slow marathon, not a sprint.
Perhaps paradoxically, the awareness that's come from tracking has helped me relax. No need to major in the minors; pretty good is pretty good.
The tools I use are a scale, loseit, and org-mode.
The goal in bodybuilding during a gaining phase is to be in a very slight calories surplus (200-300 calories above maintenance, at most) to maximize the amount of time you're building muscle before you need to cut again (bring calories back to a deficit to shed body fat).
Tracking scale weight is difficult because shifts in water weight and hydration can swing the scale 5+ pounds in either direction without any change in body fat. So I pair scale weight with a 7-point skin caliper measurements taken on a weekly basis, along with waist circumference, in order to infer whether body fat is trending up or down. And also take weekly progress photos of 6 angles/poses with consistent lighting, which I share with a coach.
And then you pair that with weighing and logging everything you eat, and you can make small adjustments to your meal plan on a monthly basis to try to stay in that 200-300 calorie per day surplus for as long as possible. (Although most bodybuilding coaches adjust diet based purely on how your physique is changing in weekly check-in photos without the need for measurements, but I like extra data)
> down to how many reps I did of which exercise, with how much weight)
I also do this. Track every exercise, every weight, number of reps. It's necessary for knowing whether you're progressively overloading over long periods of time. Progressive overload becomes harder to measure once you're past newbie gains because you can't increase weight every week, so some weeks the goal is just to squeeze out an extra couple of reps. Which adds up over time
This is obviously excessive for 99% of people. But I enjoy doing it as a hobby. I would absolutely not recommend this level of tracking for health reasons (not necessary) - I find enjoyment in the process.
I track the reps weights of every exercise (in my own app). But the historical values are only useful up to last couple of weeks just to now if the general trends go up and what is stalled. Unless your goals are the numbers themselves and not health, I don’t think there is a reason to track everything. But it is fun.
Right, but you don't necessarily need to spend hundreds of hours to capture most of this; the data is already out there. If there were a tool that could collect it all in one place and give you insights with minimal effort that would be pretty neat.
agree. I would never do what OP did. But I also won't throw out my smart watch (Context: other people in the thread said they stopped using them because the data was useless)
The tricky part is the maintenance drag and subtle risk that you'll start making decisions to optimize your life by proxy metrics just because you have them.
> Data can feel useless for 10 years until one day it becomes critical. The benefit is spiky and uneven.
Not sure if in your case the data was critical, since the doctor likely would have just had you wear a monitor for a while after to come to the same conclusion.
My first car had a broken speedometer. It was a "hand" style like a clock. Instead of moving to the speed, it would spin 365 degrees. The faster the car went, the faster it would spin. Turns out, I acclimated to it and generally knew what speed i was going (relatively speaking).
The lesson, I think, is everything is relative. Even a dashboard with flawed data that is "consistent" can highlight anomalies. And often, that's all you really need out of them. (Or the lack of anomaly)
This is the obvious "benefit" of hindsight. Yes, you accidentally had access to data that provided historical patterns you exactly could use.
But, for anyone who does, there is another 1000 who do not when something hits them: many illnesses develop gradually, and all of our tests (thousands of blood tests, scans and imaging tech...) would benefit from having historical data when we were "ok".
Similarly, you probably did not have more data than what Apple provided to help narrow the problem you still had, right?
And if everyone was put under so many tests, we'd actually be "solving" a bunch of non-issues for people over-reacting to small deviations from "normal" range.
Apple watch helps you with a few parameters — not to be discounted — but I don't really see it as a counter.
But I see people start min-maxing these numbers as a replacement for big picture health goals.
From the outside, I see someone spending a lot of time focusing on numbers while they are actually regularly stressed, who doesn’t get good sleep, and has somewhat bare minimum exercise.
Collecting data is great but don’t sink so much effort into it until you have a problem.
I had a similar epiphany a few years back when I started wearing a step-tracker/sleep monitor.
It was kinda interesting to see how many times I woke up, or track hours, but to be honest I realised after a few months that when my tracker said "You had good sleep", or "You had bad sleep" I was already aware - I woke up smiling, or grumpy depending on how I'd done.
I didn't ever look at the data and think "I want to go to bed now to catch up on the four hours I missed yesterday". I continued to have mostly consistent hours, but if I was doing something interesting I'd stay awake, and if I was tired I'd go to bed earlier naturally. The graphs and data wasn't providing anything of value, or encouraging me to change my behaviour in any significant way.
I've found one very interesting thing from these trackers - namely how even small amount of alcohol destroy sleep quality metrics. One beer is enough for my sleep scores to drop by like 20-30% and it's consistent and reproducible for me every time. Whether it actually matters - I don't know, but it made me drink much less (from maybe once a week to maybe once every two months) which is good outcome I guess.
Eh, I found several interesting things from various tracking tools. Take a nap? Sleep is destroyed this night. Exercise in the evening? Same. Not something I’d pay attention to without noticing the chart afterwards.
There’s also the motivation factor. I’m not sure of the total %, but I certainly did some exercising just to fill the daily goal. Nothing life-changing, but for the price of a cheapo apple watch se once every 5 years or so, more than worth it.
It’s not unlike simplistic time tracking on my iphone. I spent a lot of time on bullshit websites. Obviously I knew it was happening, but the sheer magnitude was surprising. It’s akin to acute pain letting you know there’s a health problem vs something brewing in the background that you are vaguely aware of, but have no motivation to truly care about - one is far more noticeable than the other
Being aware, and being aware that you are aware are very similar things that are subtly different.
I was aware that alcohol affects your next day, even a little. That's because people always say that alcohol is bad for you (surprise surprise). I heard this, so you could say that I was aware. I generally thought about this as "a hangover is bad for you." and was somewhat dismissive of the "even a single drink has a bad effect" mantra.
I did some experimenting, and slowly realized that even a single drink can indeed have an impact on the next day. It's not a hangover, but an impact that I could feel nonetheless. I needed to do some light stats and a lot more journaling to build this awareness. I am now aware that I am aware.
Same. I had a Garmin for about 6 months and I eventually just stopped wearing it and sold it. Knowing how many steps I took today, checking it several times a day to see if I was meeting my goal, knowing how many vertical feet I skied.. none of this data ended up meaning anything to me.
The best use I got from my Apple Watch was to use the companion app of my gym routine tracker (to track current loads and personal best) and play music so I didn't have to bring the phone to the gym.
That was it, I got extremely annoyed by notifications so over time just disabled them. Also for some reason the heart rate monitor glitched a couple times, got alerts about my BPM at 180+ while I was sitting on the couch.
Eventually I just stopped using it and now sits in some drawer.
> it did not justify the hundreds of hours I invested in this project.
I agree with this but minimizing the cost changes the ROI.
Personally, I've discovered useful insights tracking various life metrics. But I also found quickly diminishing returns after a few weeks or months -- if an association isn't obvious within that timeframe it's either too much effort to isolate or too slow or small to matter.
At various points I've tracked calories, macronutrients, weight, allergens, supplements, sleep, exercise volume, exercise timing, nighttime screen use, spending/budget, air quality, and mood. Now I know what kind of cooking wrecks the air quality in my house, what foods I don't digest well, what various protein/carb/fat ratios look like on my plate, how much effort it takes to improve fitness, that exercise in the morning or early afternoon improves my sleep while exercise in the evening harms it, and that any alcohol or caffeine wreck my sleep while screens at night have no measurable effect. But once I understand the associations I can alter my behavior and move on.
> The whole "quantified self" movement might be more about OCD and perfectionism than anything else.
I would agree that continuing to track metrics every day long after they've stopped yielding new insights is often compulsive behavior. But I think that's an argument for time-boxing experiments, not necessarily avoiding them altogether.
I did something similar - I put 18 years of comments on reddit, HN, slashdot, and 3 years of LLM chats in the system. I ended with a similar conclusion - it was less useful than I expected. My intent was to do RAG over my corpus, have a LLM get direct access to what I commented over the years, but unfortunately this much information has a negative effect on LLM creativity. Its responses started to fall in line too much with my ideas and it lost its spark. In the end my conclusion was that all that data was facing towards the past while I desire LLMs to improve in the other temporal direction.
I did the same but with GPT embeddings. My primary problem was different though. I wanted to find when I talked about a related subject somewhere. Search works really well.
I have built a much more limited version of this that combines my journals, sketches, photos, geolocation etc. I found it useful for noticing patterns in my behaviour and in my irrationality. Journaling helped the most, because recording my feelings means acknowledging them and reviewing them.
Above all, it's just interesting. I enjoy reading about the day-by-day progression of a crush or my brutally honest feelings about a trip that produced stunning pictures. It weaves nuance into my history.
For me it's about managing ADHD-like (never diagnosed, and I don't care to assume) patterns, coupled with self-accountability. I have a self-improving dashboard (it kicks off Claude to propose additions based my positive/negative feedback on past attempts, and then builds them) that feels quite helpful. E.g. one of the things it added was fitbit integration that shoves my step count and resting heart rate in my face every day, and it's helped me drive my step count back up towards where I want it to be.
I do think it's not worth spending a whole lot of time on, though - hence why the first thing I did was add that mechanism to have Claude build it for me, with me mostly glancing at a plan and saying yes/no. It's the perfect thing to vibe-code - if it breaks, I revert a commit and it doesn't matter because nobody depends on it but me.
It's important to tell the readers how long you've been doing this - especially to those that also manage ADHD or ADHD-like symptoms.
Why? Because those individuals tend to spin something up, tell everyone about it (online, and offline) and then stop doing it few days later.
The result then ends up being a false signal for others in the same boat. People who read it, feel a spark of recognition ("someone like me actually figured this out"), and then invest real time, energy, maybe money, into replicating something the author themselves quietly abandoned two weeks later.
Just a small heads up from someone who used to get burned in the past :)
> Why? Because those individuals tend to spin something up, tell everyone about it (online, and offline) and then stop doing it few days later.
That's definitely me (most recent ones: using engineering notebook techniques but for my own life, and WOOP method), but I recognize that feeling like I've found THE solution when I'm only a few days into it, so I tend to wait and see, or if I tell someone I say "...but ask me again in a week or a month if I'm still doing it." (At least with the engineering notebook, I can still go back and use it to remember what steps and settings I used in KiCad or use WOOP on a new goal at any time. So it's not a total loss.)
I will say one thing that I have stuck with and is pretty useful is a morning checking and an evening checklist. I'm currently using a paper version with the days of March in the columns and the checklists in the rows, and X them off as I go. A slash for the one I'm doing now/next and X when it's done. Leave it blank (or write N) if I choose to skip it. As a back-up, when I can't get around to make a paper version (I'm planning to type in the steps in a spreadsheet so I can just revise and print it each month) I keep the lists in two Google Keep checklists. Those are great because you can reset the checklist each day for reuse, and you can drag to reorder it as you edit it, and you can indent one level to organize it a bit. The disadvantage is I might get distracted by notifications and stuff on my phone.
I have data going back many years, but this recent effort is a few months old at this point. It's however notably an iteration that has reduced the amount of time I spend on collating and reviewing data, by automating away most of my previous manual effort, including most of the coding, and so I do suspect I'll stick with this for a very long time. A significant part of the prompt to the Claude part of it is to focus a substantial portion of plans on how to automate little things that costs me time, and it's doing a decent job at that.
I've absolutely not figured it out, but I now have an agent throwing stuff at the wall (with guidance from read access to e.g. my journal and a few other data sources) to figure it out for me, and it's gotten steadily better.
The difference is that I know what my "symptoms" are, but I don't know whether the combination of them are sufficient to justify an ADHD diagnosis. I don't like to assume because there's a tendency for a lot people in tech to self-diagnose and in the process sometimes diminish the difficultly others have with them.
It's pure armchair psychology, but this type of project always makes me think about anxiety. Who really needs this level of self observation and control? At the same time, I really enjoy reading about it and I find the window into somebody else's world intriguing.
> Who really needs this level of self observation and control
I liked doing similar things in the past. There's no anxiety in the equation, just pure curiosity. How many times have I done a thing a month/year? I was always curious about stuff like this, much like the OP. There's also the hacker spirit in play - designing the apps for tracking stuff.
You're equating two different things here: "building a custom web application" with "quantified self". These are two entirely different things.
Felix's statement isn't a condemnation of quantified self. I think that sometimes, when you're applying algorithms that aren't well-studied, you get pointless or bogus data. I feel much this way about sleep tracking algorithms from the likes of Apple and Whoop. Vitals, too, although it seems remarkably good at detecting when I'm about to get sick.
As a person in the older demographic of HN, having an Apple Watch quietly collect useful data over years and store it in a database has been immensely useful in helping resolve medical issues.
I used to really be into QS, and if I'm training for a marathon, I'm still studying graphs and drawing conclusions. I'm still into QS, but now I just silently log data to Apple Health and use it for medical histories or to identify certain trends (vo2max, cardio fitness, blood pressure, etc) a couple times a year.
I never spent as much time as OP on this but I did collect a lot of data during peak quantified self. I liked having automatic data collection and being able to see trends, for example. I stopped because of a few issues, not related to time:
* Hardware companies went out of business, stopped supporting devices, etc. It became obvious that there was no long term commitment to make good quality hardware that lasted a long time.
* Many devices and/or data collection was consolidated big, data hungry companies Google and Apple. Competitors have similar anti-consumer uses of data. I don't want any of these companies to have my data.
* Related to the last one, limited to no offline or local only data collection.
It is very hard to gather most of this data with off the shelf hardware and keep your data private.
This provides some insight into startup ideas around privacy, local first, offline only self data collection. I agree this kind of personal life data is something you don't want to contribute unknowingly to big data. I could see wanting to share it particularly around a health problem where only massive compute has a chance at providing answers.
> however retrospectively, it did not justify the hundreds of hours I invested in this project.
Trying to extrapolate this conclusion to the entire "quantified self" movement is not correct. The issue is the time cost, not the act itself. If a trusted company came along (as if...) that sucked up this much data to allow you to answer these questions with minimal effort, I'm sure this would be a different story.
Anything at the fringes of tech with no tried and true solution requires hundreds of hours of effort. The author's conclusions are also personal, there are other styles of living and conclusions to be drawn that change the calculus on whether to do it or not.
I ended up doing something similar to this project a year or so ago: for nearly an entire year, I tracked every single thing I ate or drank.
Who knows how many hours I spent scanning nutritional facts on the backs of boxes, estimating amounts of liquids, and even tracking sips of water. And weighing myself! Thank goodness I used a "smart" scale at least, and I didn't have to worry about carefully inputting my weight to an app each time.
But the whole project was an exercise in perfectionism. "I have to remember this sandwich and log it the next time I'm alone" made me anxious, but once I logged it, I felt a sense of completion. The database and my personal history are now at nirvana. Everything is complete.
All for me to learn things every human alive knows today: eat more food and you'll gain weight; eat less food and you'll lose weight. Yes, I can now tell you the exact average difference in calories I'll eat, statistically speaking, on a day that I have adderall in the morning vs a day that I don't. Yes, there's a similar (but much smaller) difference in average calories per day if I have caffeine in the morning as well. And I can tell you that I generally eat an additional 200-400 calories per day on a Fri-Sun than on a Mon-Thu. Wow, groundbreaking.
I've always had a lot of water, but matching foodanddrink.csv to my HealthKit data showed that I have more water on days that I walk more steps. Mildly interesting to see it written out for me? I guess. But was it worth cataloguing every cup of water? Absolutely not.
Was any of [gestures broadly at me pulling out my phone and cataloguing each item I ate] necessary to learn that? Of course not. It gave me a chance to look back at the database and say "Wowee! I did that! Every day for a year, wow, I'm so cool!" and not much else.
extensive tracking of self-related metrics to improve ones health is the equivalent to reading tons of self-help books to improve ones life/social skills/...
We already mostly know what makes people happy/healthy: personal connections, physical activity, healthy diet and some sort of purpose/goal in life that goes beyond day-to-day activities.
The problem is that these things generally require (hard) work and can be unpleasant sometimes, so humans do what humans do and spend unreasonable amounts of time doing the more pleasant things such as reading and gathering info rather than applying these and what they already know.
(That's not to say that a project like this can't be fun or lead to insights, especially across longer time spans, but i feel like all of the questions in the first paragraph have fairly obvious answers if you know yourself at all, that don't require extensive tracking of stats to get)
Agreed, I have tendency to do this to “fix” something in my life, after doing it for 2 weeks I already know, just the consciously pay attention to something even without recording make it very obvious what’s going on.
I did something similar to pull data from my Garmin watch. This meant writing all manner of code to pull data out of FIT files (interesting and often infuriating self-describing file format), coming up with schemas to hold that data to make it queryable, adding visualisations, performing analysis, pattern matching, etc.
The end result is nothing really useful, I had a bunch of scripts that semi-automated some jobs that would have taken 1 minute to do manually and only ever needed to be done a max of five times a day, but I learned a load of things along the way. Often these were useful lessons that can be applied to many other things when developing software.
In a similar vein I've gone to lots of trouble to build a cooling system for my homelab rack (ESP32 to control PWM fans, Dallas 1-wire for reading temp/humidity, exposing measurements as metrics for scraping/observability, designing things to deal with the different voltages involved, etc). I could have just gone and bought an off-the-shelf solution from AC Infinity and installed it in minutes but where would the fun in that be.
yep, I do a simple version of this in Google Sheets. Very useful to be able to "Ctrl-F" your life, especially when combined with Google Maps location history.
> The whole "quantified self" movement might be more about OCD and perfectionism than anything else.
As someone who has dealt with OCD and perfectionism, I think that could be the case for a lot of people. And the urge to obsessively track everything can be debilitating.
I too learned this the hard way. I still haven't figured out why this is the case - like, is the inherent incidental irreducible complexity of human life too high, so it dominates majority of our actions?
> The whole "quantified self" movement might be more about OCD and perfectionism than anything else.
Quantified Self, at least in the intended form, is focused on testing specific theories, not on collecting large amounts of data and trying to find something interesting in them.
If you are not sick, tracking your life like this is, like he said, is useless. Enjoy your life!
On the other hand, if you are sick like me, charting your long term heath data from doctors visits and photographing skin issues can lead to great discoveries. I have been diagnosed with Erythrocytosis and a susceptibility to mycobacteria infection which caused pulmonary nodules and skin lesions. Only after showing my data collection to my doctor. Since I have mental illness they constantly over looked my physical issues so I needed hard data to convince them of my ideas.
For those curious, I have an minor IL12B deficiency and a partial immune deficiency leading to mildly elevated levels of DexoyATP which is partially corrected with zinc supplements.
This is disappointing.... Last year i noticed large chunks of my life were being monitored via many spreadsheets and i endeavoured to bring them all together in to one Oracle DB. My plan was to eventually put some ui and graphing on top very similar to OP but seeing this is making me think twice.
I kinda disagree. Doing something about open questions which are bothering you can be valuable in itself, even if the result of the process is useless in the end. And usually you are also learning some valuable things along the way, or just push your life into a necessary direction. That isn't related to OCD or perfectionism, it's just gaining more control and insight about some aspects of your life.
A simple back of the envelope calculation shows that Felix causes between 70 and 110 tonnes of CO2 emissions per year just from flying.
Paris accord says 1.5t per person per year, from all activities, Felix's flying alonre is ~10-15x current European yearly per person emissions and ~50-75x those compatible with +1.5C.
If you want to reduce air travel for environmental reasons, then tax it more.
Shaming individuals doesn't seem to be productive or helpful.
Air travel works for people if the benefits outweigh the costs. The only thing that changes behavior is to change the costs.
And even if costs were 10x there are still plenty of people who will fly tons, because it would still be economically productive. There are always going to be people who fly 10x more than others, because certain jobs and roles simply require it.
> If you want to reduce air travel for environmental reasons, then tax it more.
> Shaming individuals doesn't seem to be productive or helpful.
First, none of us have any power to "tax it more" so this is a dead end of discussion. Second, people have agency and we can hold them accountable socially for negative actions even if they are abiding by the current laws (or tax regime). This happens all the time, because laws don't fully align with morality in a culture. Suggesting that we should leave such things to the sole discretion of the economy and taxes describes a strange unhuman-like society that we don't live in.
> Suggesting that we should leave such things to the sole discretion of the economy and taxes describes a strange unhuman-like society that we don't live in.
Well, the choice is: either we leave it to economy and taxes which work, or we'll rely on shaming which doesn't work. To put it another way - it depends if you want to actually fix the problem or just want to have a moral high ground and don't really care about solving the problem.
> First, none of us have any power to "tax it more" so this is a dead end of discussion.
You probably have more power over that than changing a whole population's behavior.
I'd wager you probably have more power over that than changing just one person's behavior (Felix).
> Second, people have agency and we can hold them accountable socially for negative actions even if they are abiding by the current laws (or tax regime).
And people are holding you accountable for the shaming. This leads primarily to polarization, not change. Polarized folks are more likely to increase the negative behavior.
Do you think that comparing someone's CO2 emissions with the average and pointing out that it is much higher is value-free, just a totally neutral observation for no reason? That the commenter is fine with it? Or even that it's a good thing?
But I'll also respond to your questions: my purpose is to show that your claim that the original comment was "free from judgement" is wrong. I'm not neutral, I'm attempting to show that your claim is obviously false, that it's not plausible at all. Of course I'm trying to judge a comment that seems wrong.
So now that I've replied honestly to your questions, will you reply honestly to mine? Repeating:
> So you think the commenter was neutral? No judgment? Again, what was the purpose of the comment then?
Because if the purpose wasn't to shame the person for their carbon footprint, I can't imagine what else it possibly could have been.
But you see: not one comment here is neutral. It would be silly to expect a comment to be neutral, such a comment wouldn't be written in the first place. I think the original comment expressed the point while staying as neutral as possible.
> So you think the commenter was neutral?
Yes, it stated some facts.
> No judgment?
Yes, it contained no explicit value judgement. Any value judgement we bring into it is our own.
> Again, what was the purpose of the comment then?
How would I know the purpose of someone else's comments? I don't even really know what my purpose is debating here with you. I certainly don't see myself persuading you of anything :)
What you write doesn't make any sense. You say it's "silly to expect a comment to be neutral" but the comment is "as neutral as possible" and then answer if the commenter was neutral with "yes". Those aren't consistent.
I don't know what definitions of neutral or value judgment you're using, but I hope you can use this as a learning opportunity. The original comment has the obvious implicit judgment that a greater CO2 footprint is a bad thing. This is shared context. It is so obvious it doesn't need to be explicitly stated, any more than "murdering people is bad". The purpose of the comment is clearly to shame the person for having such a high carbon footprint, otherwise there's no purpose in bringing it up. I don't know what your purpose was in trying to deny that. But if you genuinely didn't understand before, I hope now you do, and that this has been helpful in improving your reading comprehension or understanding of shared/implicit context.
> The original comment has the obvious implicit judgment that a greater CO2 footprint is a bad thing.
No it did not contain any judgement. That is your reading and not the comment.
> This is shared context.
The world is a large place and different people share different context.
> The purpose of the comment is clearly to shame the person for having such a high carbon footprint, otherwise there's no purpose in bringing it up.
Informing perhaps?
I wonder what your gripe is: Do you think it's inappropriate to mention the CO2 at all? Or do you think that the poster should have said it in a different way? In what way?
I think a broad tax would just make it more difficult for middle and lower class to fly. Tax the business/first class and frequent flier, but don't push people who can already barely afford to fly out.
> I think a broad tax would just make it more difficult for middle and lower class to fly. Tax the business/first class and frequent flier, but don't push people who can already barely afford to fly out.
No, it's the other way around. If flying is bad then poor people absolutely have to be priced out of it. I know that it sounds absolutely awful, but unfortunately taxing just business and first class won't do much for reducing emissions if flying will be affordable for and used by billions of people.
I appreciate this concern, but "poor people exist" is not a valid reason to continue destroying the environment, imho. It's not just people that have to bear the repercussions.
That said there are probably some work-arounds, tax free twice a year, tax rebate or some-such.
You wrote one of the solutions as if it conflicts with the other one.
Let's raise the tax on an activity according to its negative side effects, while pointing out individuals that do a lot of it and dont take personal responsibility.
If you run a company or companies on two coasts and have a wife and family on another continent (say she has her own career and can't move), then what exactly are you supposed to do?
I don't know this guy's personal life, but the people I know who fly tons fit into this profile. E.g. the wife can't move because she's a tenured professor at her university, and he's got to be at both offices regularly. He's best qualified to run the company/companies, and he's not going to get divorced to reduce his CO2 emissions.
What exactly is the solution you propose? What personal responsibility do you expect them to take? You think he should get divorced? Only see his wife and kids four times a year? Have his company/companies suffer because he can't be there in person? Quit his jobs?
And let's be clear, there are lots of jobs that require tons of air travel. If you're a highly specialized repair technician for certain equipment, all you do is constantly fly around the world fixing equipment wherever it is. If you're a CEO of a multinational company, you're constantly flying around to different offices. Are you looking for "personal responsibility" here too? How?
I can't tell if you're serious or if you really think someone who has a family on another continent and is running two companies on opposite coasts is some kind of victim of their circumstances and needs a special keyboard warrior to comment on HN in support of them, lest they face the consequences of a little more tax that they'll never miss or some social shaming.
I'm sorry, I don't want HN to to be the place where we get into a fight over the mildest inconvenience for people who are already living extravagant lifestyles.
What I'm opposed to is some hand-wavy demand to "take personal responsibility" without suggesting exactly what they're supposed to do and whether any tradeoffs involved are reasonable.
And please don't call people names. You can write comments here without calling other people "keyboard warriors". Nor is it helpful to try to shut down some viewpoint by claiming that somebody doesn't need any extra support.
And I think most people would consider not seeing their family more than e.g. four times a year more than just the "mildest inconvenience".
> Shaming individuals doesn't seem to be productive or helpful.
I don't see how much support from history for that viewpoint. Some examples of positive societal change driven in part by shaming individuals: drink-driving, civil rights, sexual harassment, automobile safety, the slave trade, McCarthyism.
All those cases also have huge penalties or effective costs associated with them. Is there an accurate "shame first, then penalties came later" stand point?
Automobile safety in my life has only changed after fines.
Sexual harrassment still happens and doesn't seem to be helped by shaming someone as much as firing them. Though we often don't have the guts or legal backing to publically shame someone.
Most people don't spend a lot of time wistfully considering their CO2 usage, myself included. The religious zeal by which people feel the need to tell me about the ever-warming planet is honestly more off-putting than most actual religions.
Of the very few "f*cks" I can give in my life, I prefer to spend mine as I choose rather than being scolded for not giving mine to the pressing issues that others deem important.
The thing is that everyone alive today and in the future is footing the bill for that indifference. It's nice that you don't care, but it's not something I'd brag about.
Is the OP flying a private jet or something? Unless he is, it's a useless metric. The people flying private are responsible for a 1000x a regular persons emissions. It's offensive to suggest regular salaried people are supposed to be "doing something" in this CO2 effort.
It's true that a single private jet is causing tons of CO2 emissions. But in the end, all consumers control the market.
You can jump one link further in the chain to regions with much lower emissions. Somalia/Congo emit over 200x less CO2 per capita in comparison to the US. Do you think that's fair for them if the "regular salaried people" don't care?
If responsibility always gets dismissed by pointing to someone emitting more, nothing changes.
I too don't want to be a martyr hermit who don't travel and consume just to cur my CO2 emissions in half, while someone else generates 10x of that living a full life. That's just dumb, and I consider this whole movement rich people telling poor people that they should be considerate and sacrifice themselves.
If you truly care about the planet, don't have children.
It's an "all hands on deck" situation, yet everyone seems to think that someone else has to do something. You don't need to be a martyr, just to make an effort.
At the very least don't brag about not giving a crap.
> If you truly care about the planet, don't have children.
That's a fallacy; people care about the planet precisely because of children. I don't care about the planet for its own sake; I care because of the humans who inhabit it and their future lives.
Also, humanity spent 100,000 years without flying around the globe, and I doubt they were all living hermit martyr lives.
I don't feel it's properly engaging in good faith to say that I don't care. I don't specifically care, in the "feeling shame" sense that the GP had mentioned. As I pointed out, we all have a limited number of things we can realistically care about.
The fact that I happen to care about other things more than this specific flavor of global catastrophe is morally OK.
The way I see it, "from each according to their ability" is the right approach here. If you can afford 30% less flying, red meat etc, then it's your duty to do it. You don't need to make your cat go vegan; just... do what you can, even if imperfectly.
Shame is not necessary, but callous indifference is not acceptable. There is a middle ground where you treat it like a habit to improve, like a step count.
And this is it. This is why we are where we are today. That it is seen as taking a religious zeal to realize how flying very frequently is disastrous for the climate. That's our bar and what we have to work with. Yes, we are properly fucked.
I would totally do this without any shame if I had the need/desire. CO2 isn't going to be solved by well intentioned individuals making absolutely no impact. It will be a generic solution that solves it for everyone, or it won't be solved at all.
I'm also not going to take shorter showers when people are farming in a desert and shipping the crops to China.
You might think this makes me a terrible person. That's probably good. Because it will help people understand what we're up against and what needs to happen to actually solve the problem.
I refuse to worsen my life for an absolutely minuscule abstract benefit to someone, eventually, maybe. I am going to continue eating meat and flying when I want to, and I don't feel the least bit bad about it. It's unfortunate that climate change is happening, but my personal actions are not meaningfully contributing. My choices do not have cascading effects on others; whatever I do, the climate will change in the same manner as it otherwise would have. In that light, I am not going to reduce my QOL just to feel self righteous for no actual benefit to anyone.
You may wonder how this is consistent with my propensity to recycle and follow traffic laws and not do crimes and other socially beneficial minor things to which a comparison could be made to CO2 output: because those all have a greater benefit:cost ratio than flying and eating meat, by far. I am including the benefits of living in a high-trust society in that analysis.
I work as an airline pilot, and I just recently calculated mine to be 65,658 tonnes in my 10-year career so far (not quite so bad when calculated per seat-mile, but still eye opening!).
https://jameshard.ing/pilot/#statistics
There is a section comparing flight emissions to US citizen average total emissions. This might make him feel good, but only 30% of an average American's emissions come from transportation and just a sliver from flying, so it's very likely his total emissions are much higher.
I believe you misunderstand what "average" is: Americans (or people in general), on average, do not fly much.
Some — eg. those who've moved for work — probably fly 10-15x the average, and 10-15x of people fly 1/100th of average. Flying cost and availability correlate with average needs, but that presupposes that some people have higher needs altogether.
Exactly, I stopped reading when I saw the flying stats. There are people who still haven't clocked where our climate is headed.
I get that you may have to see family abroad or maybe indulge for a holiday, but this is "I'm using an airplane to commute" kind of level.
And here I am trying to book my train tickets to go to London instead of flying even though it costs three times as much just to avoid a few kg of CO2 (among other things), it's making me angry.
Depends where you take the train to London, it is a much nicer experience anyhow than going to airports and people should consider that as well (ignoring climate stuff)!
On the price, the very annoying thing is that fuel for planes is not taxed! Changing this would require quite some effort (falls under some specific laws, that are old and nobody wants to touch, etc.) but I think everybody should just ask "honest tax on fuels!" as this will make less people say (or thin) "but climate change is a hoax". Planes are just unfair competition to other transport due to taxes!
I agree re: fuel taxes, but it’s a complete nonstarter: passengers would be voting against their wallets, and airlines would lobby against it since it’s a vote against their business model.
An alternate approach that would be seen as consumer and business-friendly would be subsidizing companies with a certain level of fuel efficiency per passenger mile, targeted above current levels.
Eh, other people throw litter on the floor and rob elderly folks in their homes. Those people hardly ever get caught, but neither you nor I are are going to start copying their actions.
Isn't this a drop in the ocean? Why would any 'normal' person forgo flying? How much CO2 emissions have 'world leaders' produced going to summits, or Taylor Swift and her fans flaying to concerts or war flights?
Couple of things: 1) NO ONE is suggesting any one forego flying altogether, or skipping their once-a-year overseas vacation or periodic family visit. 2) THIS level of flying is not normal and is exactly the kind of harmful behavior people have in mind when they complain about frequent flyers. 3) Whinging about summits and Taylor Swift is just a bad faith red herring argument. Obviously less flying is better, no matter by who. To the extent it's related to the topic at all, it bolsters the case for less air travel.
I don't know why someone going on a vacation would have moral high ground over someone that HAS TO travel for his work. If you are scientist you absolutely have to fly a lot to visit lots of conferences, disseminate your work, provide lectures etc.
Everything any one person does is a 'drop in the ocean'. Thankfully, we organize and do things collectively very well - it's in our fundamental nature going back to non-human ancestors, and there is a long, rich history of how much we accomplish. Alsmost nothing that has ever been accomplished has been done without a lot of people doing it together.
The general concern around Taylor Swift's emissions has always struck me as shortsighted. Her Eras tour is estimated to have generated around $5bn in economic uplift in the US, at an estimated 10,000 tonnes CO2e for her personal travel. Even if the total footprint is higher, that is thousands of times lower than the emissions intensity of an industry like fast fashion. From an environmental point of view, attending a Taylor Swift show is a much less carbon-intensive way to spend your money than ordering from Temu.
In my experience, tracking objective things like "nutrition" and "sleep hours" is immensely useful to reflect on what went wrong, and tracking subjective things like "mood" or "stress" is useless given hedonic adaptation or heavy swings that make problems obvious, and not need tracking.
What's key is be able to visualize metrics easily on the data and frictionless data entry, I've got a decent setup with iPhone Action + Obsidian + QuickAdd scripts on Obsidian Sync (mobile + laptop). for visualization I use Obsidian Bases and Obsidian notes that run Dataview code blocks and Chart.js, couldn't be happier.
I could track things that are not interesting to reflect on like vitamin D supplementation for accountability but I've never bothered, especially if it's taken ~daily.
The language you use to describe this is fun, I make an app for self tracking called Reflect and would love your opinion of it, even if it doesnt suit your needs exactly.
Strongly agree with this. I’ve been using Apple’s “mood” log for about two years now, and it is extremely helpful for me to have a concrete view of the history of my general affect.
“This entire month I’ve been feeling good, I want to pinpoint why,” or “it’s clear since stressor X entered my life, my affect is lower; how can I resolve this?”
These long term trends are harder for me to track without data. It might be easy for others, but not me!
As someone with Schizoaffective Disorder Bipolar Type, if you are not diagnosed with a mood disorder, tracking "swings in you mood" when you have no clinical disorder seems like a disorder of its own.
I have had people tell me they were "manic". Then I showed them videos I took when I was manic and they see what I mean when I tell them they are not manic.
We have come to a place where we do not want even normal fluctuation in mood, and that is a illness of its own, but it is a cultural illness.
Maybe for some it's a lot more extreme than for others, but even if it's not so dramatic as to be categorized as a mental illness wouldn't you want to know if, say, there were a direct correlation between whether you went for your morning run and your mood later in the day?
Is this something that needs to be tracked to bring into your awareness? We have a memory storage device sitting on top of our spine. When I drink I feel drunk. Easy. If the change is noticeable you will notice it and remember it.
I am just trying to save you time and escape the cycle of "optimizations" which is where all this data logging leads.
Yes? As I wrote, the mere act of writing down your feelings forces you to acknowledge them and see patterns in them. Sometimes while writing something down I realise "wait, I've been through this before" or "every time this person is around, I feel this way". It helped me be more self-aware, for my own good.
It turns out that our memory storage device uses a very lossy form of compression. Memories get simplified and distorted over time. Heck, I can't even remember when something started hurting, so how should I notice a year-long pattern of thinking around a certain topic?
I get that everyone wants to be cynical about this, but you really can't deny that both the visualization and sheer scale of data is impressive. The way the "my life in weeks" is done is also very cool, I'll be stealing that for myself.
Bravo. This is my dream and also my nightmare. I was super into the quantified self movement a while back, before Apple Watch, writing Withings Scale API wrappers in Ruby and Fitbit days.
Every time I try to seriously track metrics of my life, the excitement of the insight gets worn away by the friction of recording and managing. I expect LLMs can help reduce the cost of this by an order of magnitude but then, as you mention, the question is, what do you do / change / learn because of the data?
I recently started tracking nutrition macros with an iOS app MacroFactor which I really like. This is the first time taking my weight doesn't feel like a IDK SHRUG moment and I can actually map my food intake to my weight.
Finances is probably the other highly actionable data source that is such high friction to manage (downloading CSVs, OFXs, monthly...) that it has always been a false start for me. I finally wrote a service to talk to Plaid directly and I successfully used it to categorize my business expenses at tax time. I finally have programmatic access to my bank account data!
You conclusion is definitely a cautionary take:
> the main conclusion is that it is not worth building your own solution, and investing this much time.
But, perhaps a subset of that data you find useful.
For the few times that I have tracked myself doing various activities, I have found that I didn't even need to do anything with the data, I was actively changing habits whilst I knew I was being monitored. For example, I wore a GCM for a few weeks and I found it difficult to live my "normal" day to day lifestyle. I wanted to do the things I felt were "right" and "healthy". Whether or not that is a good thing or not is a different topic, but just my 2c.
I ended up building something similar but way smaller in scope -- just a cron job that pulls Apple Health exports, bank CSV dumps, and git commit stats into a single SQLite db. The queries that actually proved useful were embarrassingly simple, stuff like correlating sleep hours with commit frequency or spotting that my spending spikes every time I start a new side project. The real 80/20 of personal analytics is just getting your data into one place where you can join across sources. Everything beyond that is diminishing returns.
Honestly the hardest part was normalizing the date and amount formats across banks. Each one exports slightly differently. I wrote a small Python script per bank that maps their CSV columns to a common schema (date, description, amount, category) and then bulk-inserts into SQLite. Once you have the schema figured out it takes maybe 20 minutes per bank. The manual step is downloading the CSV each month, but most banks let you set a date range so it is just a few clicks.
The value rarely shows up where you expected it to.
I kept a rough log of my sleep and mood for about a year with no specific goal. Mostly forgot about it. Then I had a weirdly bad few months and went back to look — turns out there was a pretty clear pattern I would've never noticed in the moment.
Maybe the framing of "was it worth it" is the wrong question. It's less like an investment with a return and more like keeping receipts. Useless 99% of the time, then suddenly you really need one.
As someone who dislikes being tracked, I find it disconcerting to have this level of data stored in 3rd party services, but joining up multiple services to give one coherent picture is cool. How cool is questionable, trying to correlate health data with location data is going to give a strange picture, and I question how relating health data to weather is useful. (Do I have lower blood pressure on rainy days?)
Forgive me for I am being sceptical. The might be some insight here I have not considered, but I'd feel a lot more comfortable if it was all self-hosted / self-collected data.
The appeal of a single queryable database for personal data makes a lot of sense - the hard part is always the ingestion and normalization, not the storage.
One pattern I have seen work well for the business version of this: a "company intelligence" database where everything known about a prospect company gets accumulated in one place over time. Homepage content, job postings, news mentions, funding history, tech stack signals, all deduplicated and queryable.
The challenge on the B2B side is the same as personal data: the data comes in from 8 different sources in 8 different formats, often with conflicts (two sources disagree on headcount, three sources have different founding dates). Your approach of controlling the schema from the start rather than trying to normalize later is the right call. Schema drift is what kills most long-term data projects.
What storage engine are you using? And how do you handle temporal data - do you snapshot state over time or just keep the latest version of each entity?
This reminds me of someone I knew slightly back in the late 70s: Robert Shields, who kept the world's longest diary [0]. He was eccentric but very nice...but he wrote 33 million words, filling 90+ boxes, of every detail of his life for decades, only quitting when a stroke rendered him unable to type about ten years before his death. I doubt he tried to reach any conclusions based on his journaling, though.
This was far more interesting than I first thought it would be when clicking the link. In particular, the place/time and life events and such being presented this way told a story and was fun.
> Probably obvious for many, but I didn't realize ACs don't transport any air into the room, but just moves it around
I had the same epiphany as you days after acquiring a CO2 monitor. Most people notice poor indoor air quality from proxies such as humidity and temperature. AC (without ventilation) eliminates these and tricks our senses very effectively, giving us cool and fresh feeling indoor spaces full of CO2 and devoid of oxygen.
Capturing data was the main bottleneck I ran into years ago when I tried something similar with a little MS Access database. I spent lots of time making it pretty and defining fields and data types, but actually typing stuff in? I lost interest quickly.
It's possible the friction could be reduced here by having some kind of Generative AI try to help capture data, but then you'd have to verify that it was being done correctly... honestly, I think it's simply not practical for most people to do this.
Real life is messy. How much time you want to spend on recordkeeping to make it seem less messy or make you feel like you have more control is up to you. But sometimes it's better to embrace the mess and let go of control, in my opinion. Chances are, no one's going to care about whatever you do here in 100 years. YMMV as always.
I can see the value of doing this but I think that this is a case of the 80-20 rule. Just the low effort 80, which could be keeping a journal where you reflect on how you felt a given day, why that might be, and what steps to take to avoid it from happening (or keep it happening) should get you pretty far.
Most of the things that really hinder my productiveness and happiness are the same things that everyone tells you about; sleep, diet, sunlight, not being stuck indoors all day, socializing. And most of these can be improved by making changes to my habits.
Other than that, is that I think that it's important to get into the habit of self reflection. Having a feedback loop on my own life has helped me find out what works and what doesn't. It's too easy to just go through the days slogging through and not making any changes because you don't even realize what's going on.
An interesting experiment, I think I'm too uncomfortable leaking data I don't yet know why someone would curate to me free of charge until I knew.
If there was a FOSS suite like Home Assistant that would do a few of those things I might try it out, especially the weather (I would add air quality) correlation to mood and other subjective states.
The hardest part of this kind of personal data system is retrieval not storage. At some point you have more data than fits in a prompt, so you need to decide what's relevant per query. Did you build any ranking or filtering logic, or do you query specific tables directly?
Flying stats dashboards always amuse me. I get it, for the non-pilot it's kind of like a status thing, "Well I traveled X times in 2025!". As a pilot though, I have gobs of stats I could put up there but from flying for 15 years I realize there's not really anything meaningful in there other than "Gee whiz, I flew a little more/less than last year." I know some other professional pilots do track some of their stats a little closer as they try to optimize for hours flown:hours paid but I've never cared to hyper optimize my schedule in that way.
Tangent, I've always had notes and have this longing to store things but at the same time I like "moving on" too. For ex. I used to have windows phones and saw I still have an accessible One Note from 10 years ago... tempted to read through it but also I've changed as a person.
Another tangent, recently bought a paper shredder, started shredding through boxes of mail I have kept since they have personal info like cc statements but on the same idea of moving on/reducing stuff I'm hoarding whether it's data or physical.
Waiting for a neckband/choker wearable that can take transcription from my vocal cord nerve, so I can operate a chatbot silently. Yes, scientists have made impulse-to-speech lab equipment.
Tried the manual tracking a few times and it does not tend to stick. Passive capture is the approach that ever worked for me. Having different apps and devices with permissions to track you and good APIs that enable integrating the data seems to be the way to go.
Did you think of building some proactive AI tools to make use of all this centralized data?
This is fascinating. One question I always have with large personal datasets like this: at what point did you start getting genuinely surprising insights versus confirming things you already suspected?
380k datapoints sounds incredible but I imagine the real challenge is turning that into decisions that actually change behavior
Related to this, I highly recommend anyone to install github.com/ActivityWatch/activitywatch, it's an amazing tool to keep track of your computer use completely locally. I think there are lots of possibilities with data analysis/AI aimed to improved one self's life
Wow, this sounds amazing! I am a statistics and chart lover, I love tracking various types of data, but this? I can't even imagine how much time it must have taken to input all the data. Huge respect to you! Keep going.
"GitHub Open Source Contributions" aside, everything else falls into the consumption/intake/internal categories; thin data on production/output/external ones.
I went through a similar process and came to the same conclusion, although with a slightly different twist. The project became the point and I almost stopped noticing the data towards the end.
Interesting approach.
One thing I noticed when people build systems like this is that the real challenge becomes decision-making rather than storage, deciding what actually deserves to become structured data.
Curious how you decide what goes into the database vs what stays unstructured.
I had an idea similar to this where you could add information about yourself and answer daily questions and get paid by companies who access this data. This could be an ethical way to share information benefitting all parties.
You and Stephen Wolfram. Ironically when I ask Mathematica "what is the mass of Stephen Wolfram" it cannot tell me. I'm honestly not trying to pick on him but I wanted to calculate his average density. What a rip off.
This might sound harsh but for someone who is keen on investing time to track so many things, you should invest some time in learning how to make better visualizations. A few tweaks here and there would really improve what you have.
> Back in 2019, I started collecting all kinds of metrics about my life. Every single day for the last 3 years I tracked over 100 different data types - ranging from fitness & nutrition to social life, computer usage and weather.
I know this is the type of person i would not like.
You'll get crap here for it, but I agree. This is someone that has so little to worry about that he has to look for things. Then you look at his travel habits and jobs and you realize he's just one more rich bored guy showing off as best as possible.
The takeaways at the very bottom of the page are valuable:
> Overall, having spent a significant amount of time building this project, scaling it up to the size it’s at now, as well as analysing the data, the main conclusion is that it is not worth building your own solution, and investing this much time. When I first started building this project 3 years ago, I expected to learn way more surprising and interesting facts. There were some, and it’s super interesting to look through those graphs, however retrospectively, it did not justify the hundreds of hours I invested in this project.
The whole "quantified self" movement might be more about OCD and perfectionism than anything else.
/edit: quantified, not qualified
A counter example:
I've been wearing an Apple Watch for close to 10 years. I've tracked my weight as well along those years but nothing crazy like OP. The Apple watch tracked plenty.
I had some strange symptoms and two doctors insisted I had a weak heart and potential heart failure. This was shocking! Turns out I do have a really "weak" rhythm, but heart failure is when your heart is progressively getting worse in it's pumping. I don't even remember which metric he looked at in my Apple health - but basically my heart has always been this way. A doctor looking at a single data point might think I have abnormally low blood pressure/heart rate, but if I've had this for 10 years with no change, the medical assessment is very different - it means nothing. Sometimes boring data is exactly what you need. For this reason, I will probably always wear an Apple watch (or equivalent) moving forward.
Data can feel useless for 10 years until one day it becomes critical. The benefit is spiky and uneven.
But you didn’t spend hundreds of hours on it, so when it did happen to be useful it seemed like an outsized benefit.
I would wager that for most people, most data about themselves will be useless and not worth collecting.
Of course you can’t know what data will be useless or not, so unless the cost of collecting it is minimal or nil (wearing a smart watch, writing down your weight each day/week), it’s probably not worth it.
Spending hundreds of hours to build a solution to capture all data about yourself to find interesting patterns has a huge assumption baked into it: that there are interesting patterns to find.
About 9 years ago I had a run-in with stomach cancer. After a few months of chemo and a 7 hour operation I was eventually declared cancer free and have been ever since, but still have to live with the consequences of the treatment and be vigilant for any signs of it returning.
I still suffer intermittent stomach aches, especially in the early hours of the morning, and had a terrible time trying to decide if they were getting better or worse over time.
Our narrative voice is awful at detecting long term trends and tends to overcompensate for particularly good or bad patches so it was impossible for me to judge and I started keeping records of how bad the aches were each day.
Long story short, the average severity was mostly decreasing over time and the average time between bad aches was slowly increasing but it would have been impossible to tell if this was happening without keeping detailed records because it wasn't consistent - some months were much worse than others and completely skewed my perception of long term trends.
While most people hopefully won't ever need to do something like this, it did make me realise just how bad we are at picking up on long term trends so I can definitely see keeping daily records of, for instance, average daily happiness being eye-opening.
Sounds like you had a very good reason to be tracking your stomachaches. But almost everyone else aside from you And people In similar situations would be wasting their time doing that, I would imagine.
Yes, when you have an obvious reason to track some data, do it: I do not think anyone questions that.
Proactively capturing and tracking everything you can to prep for any future is too much work that would really steal your time from, you know, actually living a life.
Like anything else, I think it comes down to having a good use case.
I've gotten deep into weightlifting/bodybuilding over the past couple of years, and that's the kind of hobby where micro-optimizations and data tracking can have a pretty big impact on results (and sort of necessary, you can't fly blind with things like diet, especially)
E.g. I track and weigh everything I eat, take body measuraments on a weekly basis, Dexa scans every few months, etc - for me it's worth it because I know what I want to do with the data. If I didn't have a goal, all that tracking would clearly be overkill.
How long have you been tracking? Can you share an insight you've had from your data?
I've been weight lighting for ten years and initially tried to track things (down to how many reps I did of which exercise, with how much weight) and quickly came to the conclusion that is want worth it for me.
I initially came to the same conclusion. Though I lifted in accord with decent training principles regarding reps and sets, I didn't track for years. As I entered middle age, I started keeping a training log (just one big org file in emacs), mostly out of curiousity. As I entered my 50s, I experienced what Haruki Murakami references in "What I Talk About When I Talk About Running" --- Fat is easy to gain and hard to lose. Muscle is hard to gain and easy to lose. Now I track a couple of critical metrics and it's working great. I weigh first thing every day, track all kcals (even if I overeat), plan and track workouts. I write my own plans pulled from principles in these books (don't work for the company, just a satisfied customer) https://muscleandstrengthpyramids.com/ I don't use the vast majority of the info in those books as I'm just a hobbyist who wants to be healthy and strong. The biggest shift came from learning I was doing waaay too much training volume at the gym while trying to lose fat too quickly; a fine recipe for injury. Now, when I'm in a fat loss phase, I try to lose it as slowly as possible while still making progress. Strength training and fat loss is a very long very slow marathon, not a sprint. Perhaps paradoxically, the awareness that's come from tracking has helped me relax. No need to major in the minors; pretty good is pretty good. The tools I use are a scale, loseit, and org-mode.
The goal in bodybuilding during a gaining phase is to be in a very slight calories surplus (200-300 calories above maintenance, at most) to maximize the amount of time you're building muscle before you need to cut again (bring calories back to a deficit to shed body fat).
Tracking scale weight is difficult because shifts in water weight and hydration can swing the scale 5+ pounds in either direction without any change in body fat. So I pair scale weight with a 7-point skin caliper measurements taken on a weekly basis, along with waist circumference, in order to infer whether body fat is trending up or down. And also take weekly progress photos of 6 angles/poses with consistent lighting, which I share with a coach.
And then you pair that with weighing and logging everything you eat, and you can make small adjustments to your meal plan on a monthly basis to try to stay in that 200-300 calorie per day surplus for as long as possible. (Although most bodybuilding coaches adjust diet based purely on how your physique is changing in weekly check-in photos without the need for measurements, but I like extra data)
> down to how many reps I did of which exercise, with how much weight)
I also do this. Track every exercise, every weight, number of reps. It's necessary for knowing whether you're progressively overloading over long periods of time. Progressive overload becomes harder to measure once you're past newbie gains because you can't increase weight every week, so some weeks the goal is just to squeeze out an extra couple of reps. Which adds up over time
This is obviously excessive for 99% of people. But I enjoy doing it as a hobby. I would absolutely not recommend this level of tracking for health reasons (not necessary) - I find enjoyment in the process.
I track the reps weights of every exercise (in my own app). But the historical values are only useful up to last couple of weeks just to now if the general trends go up and what is stalled. Unless your goals are the numbers themselves and not health, I don’t think there is a reason to track everything. But it is fun.
True, if you have a current and real need for the data, then it makes sense to collect it. But that’s an entirely different scenario.
Right, but you don't necessarily need to spend hundreds of hours to capture most of this; the data is already out there. If there were a tool that could collect it all in one place and give you insights with minimal effort that would be pretty neat.
agree. I would never do what OP did. But I also won't throw out my smart watch (Context: other people in the thread said they stopped using them because the data was useless)
The tricky part is the maintenance drag and subtle risk that you'll start making decisions to optimize your life by proxy metrics just because you have them.
> Data can feel useless for 10 years until one day it becomes critical. The benefit is spiky and uneven.
Not sure if in your case the data was critical, since the doctor likely would have just had you wear a monitor for a while after to come to the same conclusion.
My first car had a broken speedometer. It was a "hand" style like a clock. Instead of moving to the speed, it would spin 365 degrees. The faster the car went, the faster it would spin. Turns out, I acclimated to it and generally knew what speed i was going (relatively speaking).
The lesson, I think, is everything is relative. Even a dashboard with flawed data that is "consistent" can highlight anomalies. And often, that's all you really need out of them. (Or the lack of anomaly)
This is the obvious "benefit" of hindsight. Yes, you accidentally had access to data that provided historical patterns you exactly could use.
But, for anyone who does, there is another 1000 who do not when something hits them: many illnesses develop gradually, and all of our tests (thousands of blood tests, scans and imaging tech...) would benefit from having historical data when we were "ok".
Similarly, you probably did not have more data than what Apple provided to help narrow the problem you still had, right?
And if everyone was put under so many tests, we'd actually be "solving" a bunch of non-issues for people over-reacting to small deviations from "normal" range.
Apple watch helps you with a few parameters — not to be discounted — but I don't really see it as a counter.
Collecting data is great.
But I see people start min-maxing these numbers as a replacement for big picture health goals.
From the outside, I see someone spending a lot of time focusing on numbers while they are actually regularly stressed, who doesn’t get good sleep, and has somewhat bare minimum exercise.
Collecting data is great but don’t sink so much effort into it until you have a problem.
This was great because you didn't put much effort into it. Whereas the conclusion was about "it is not worth building your own solution".
So what happened with your symptoms?
I had a similar epiphany a few years back when I started wearing a step-tracker/sleep monitor.
It was kinda interesting to see how many times I woke up, or track hours, but to be honest I realised after a few months that when my tracker said "You had good sleep", or "You had bad sleep" I was already aware - I woke up smiling, or grumpy depending on how I'd done.
I didn't ever look at the data and think "I want to go to bed now to catch up on the four hours I missed yesterday". I continued to have mostly consistent hours, but if I was doing something interesting I'd stay awake, and if I was tired I'd go to bed earlier naturally. The graphs and data wasn't providing anything of value, or encouraging me to change my behaviour in any significant way.
I've found one very interesting thing from these trackers - namely how even small amount of alcohol destroy sleep quality metrics. One beer is enough for my sleep scores to drop by like 20-30% and it's consistent and reproducible for me every time. Whether it actually matters - I don't know, but it made me drink much less (from maybe once a week to maybe once every two months) which is good outcome I guess.
Eh, I found several interesting things from various tracking tools. Take a nap? Sleep is destroyed this night. Exercise in the evening? Same. Not something I’d pay attention to without noticing the chart afterwards.
There’s also the motivation factor. I’m not sure of the total %, but I certainly did some exercising just to fill the daily goal. Nothing life-changing, but for the price of a cheapo apple watch se once every 5 years or so, more than worth it.
It’s not unlike simplistic time tracking on my iphone. I spent a lot of time on bullshit websites. Obviously I knew it was happening, but the sheer magnitude was surprising. It’s akin to acute pain letting you know there’s a health problem vs something brewing in the background that you are vaguely aware of, but have no motivation to truly care about - one is far more noticeable than the other
Being aware, and being aware that you are aware are very similar things that are subtly different.
I was aware that alcohol affects your next day, even a little. That's because people always say that alcohol is bad for you (surprise surprise). I heard this, so you could say that I was aware. I generally thought about this as "a hangover is bad for you." and was somewhat dismissive of the "even a single drink has a bad effect" mantra.
I did some experimenting, and slowly realized that even a single drink can indeed have an impact on the next day. It's not a hangover, but an impact that I could feel nonetheless. I needed to do some light stats and a lot more journaling to build this awareness. I am now aware that I am aware.
> an impact that I could feel nonetheless. I needed to do some light stats and a lot more journaling to build this awareness.
If you could feel it why beed the stats?
I don't think I was clear.
There is a difference between knowing something, and believing it to be true.
I know that sometimes I feel good when I wake up. I know people say that drinking makes you feel not good when you wake up, sometimes.
It takes a bit of observation and some statistical sampling to connect those two together. Now I know it, and also believe it to be true.
Perhaps "aware" vs "aware that I am aware" was a bad way of phrasing it.
It's like any good cryptographic challenge - easier to validate the answer than to come up with it from scratch
Same. I had a Garmin for about 6 months and I eventually just stopped wearing it and sold it. Knowing how many steps I took today, checking it several times a day to see if I was meeting my goal, knowing how many vertical feet I skied.. none of this data ended up meaning anything to me.
The best use I got from my Apple Watch was to use the companion app of my gym routine tracker (to track current loads and personal best) and play music so I didn't have to bring the phone to the gym.
That was it, I got extremely annoyed by notifications so over time just disabled them. Also for some reason the heart rate monitor glitched a couple times, got alerts about my BPM at 180+ while I was sitting on the couch.
Eventually I just stopped using it and now sits in some drawer.
> it did not justify the hundreds of hours I invested in this project.
I agree with this but minimizing the cost changes the ROI.
Personally, I've discovered useful insights tracking various life metrics. But I also found quickly diminishing returns after a few weeks or months -- if an association isn't obvious within that timeframe it's either too much effort to isolate or too slow or small to matter.
At various points I've tracked calories, macronutrients, weight, allergens, supplements, sleep, exercise volume, exercise timing, nighttime screen use, spending/budget, air quality, and mood. Now I know what kind of cooking wrecks the air quality in my house, what foods I don't digest well, what various protein/carb/fat ratios look like on my plate, how much effort it takes to improve fitness, that exercise in the morning or early afternoon improves my sleep while exercise in the evening harms it, and that any alcohol or caffeine wreck my sleep while screens at night have no measurable effect. But once I understand the associations I can alter my behavior and move on.
> The whole "quantified self" movement might be more about OCD and perfectionism than anything else.
I would agree that continuing to track metrics every day long after they've stopped yielding new insights is often compulsive behavior. But I think that's an argument for time-boxing experiments, not necessarily avoiding them altogether.
I did something similar - I put 18 years of comments on reddit, HN, slashdot, and 3 years of LLM chats in the system. I ended with a similar conclusion - it was less useful than I expected. My intent was to do RAG over my corpus, have a LLM get direct access to what I commented over the years, but unfortunately this much information has a negative effect on LLM creativity. Its responses started to fall in line too much with my ideas and it lost its spark. In the end my conclusion was that all that data was facing towards the past while I desire LLMs to improve in the other temporal direction.
I did the same but with GPT embeddings. My primary problem was different though. I wanted to find when I talked about a related subject somewhere. Search works really well.
I have built a much more limited version of this that combines my journals, sketches, photos, geolocation etc. I found it useful for noticing patterns in my behaviour and in my irrationality. Journaling helped the most, because recording my feelings means acknowledging them and reviewing them.
Above all, it's just interesting. I enjoy reading about the day-by-day progression of a crush or my brutally honest feelings about a trip that produced stunning pictures. It weaves nuance into my history.
Yours is qualitatively different because you’re not quantifying your life.
A good thing.
For me it's about managing ADHD-like (never diagnosed, and I don't care to assume) patterns, coupled with self-accountability. I have a self-improving dashboard (it kicks off Claude to propose additions based my positive/negative feedback on past attempts, and then builds them) that feels quite helpful. E.g. one of the things it added was fitbit integration that shoves my step count and resting heart rate in my face every day, and it's helped me drive my step count back up towards where I want it to be.
I do think it's not worth spending a whole lot of time on, though - hence why the first thing I did was add that mechanism to have Claude build it for me, with me mostly glancing at a plan and saying yes/no. It's the perfect thing to vibe-code - if it breaks, I revert a commit and it doesn't matter because nobody depends on it but me.
It's important to tell the readers how long you've been doing this - especially to those that also manage ADHD or ADHD-like symptoms.
Why? Because those individuals tend to spin something up, tell everyone about it (online, and offline) and then stop doing it few days later.
The result then ends up being a false signal for others in the same boat. People who read it, feel a spark of recognition ("someone like me actually figured this out"), and then invest real time, energy, maybe money, into replicating something the author themselves quietly abandoned two weeks later.
Just a small heads up from someone who used to get burned in the past :)
> Why? Because those individuals tend to spin something up, tell everyone about it (online, and offline) and then stop doing it few days later.
That's definitely me (most recent ones: using engineering notebook techniques but for my own life, and WOOP method), but I recognize that feeling like I've found THE solution when I'm only a few days into it, so I tend to wait and see, or if I tell someone I say "...but ask me again in a week or a month if I'm still doing it." (At least with the engineering notebook, I can still go back and use it to remember what steps and settings I used in KiCad or use WOOP on a new goal at any time. So it's not a total loss.)
I will say one thing that I have stuck with and is pretty useful is a morning checking and an evening checklist. I'm currently using a paper version with the days of March in the columns and the checklists in the rows, and X them off as I go. A slash for the one I'm doing now/next and X when it's done. Leave it blank (or write N) if I choose to skip it. As a back-up, when I can't get around to make a paper version (I'm planning to type in the steps in a spreadsheet so I can just revise and print it each month) I keep the lists in two Google Keep checklists. Those are great because you can reset the checklist each day for reuse, and you can drag to reorder it as you edit it, and you can indent one level to organize it a bit. The disadvantage is I might get distracted by notifications and stuff on my phone.
I have data going back many years, but this recent effort is a few months old at this point. It's however notably an iteration that has reduced the amount of time I spend on collating and reviewing data, by automating away most of my previous manual effort, including most of the coding, and so I do suspect I'll stick with this for a very long time. A significant part of the prompt to the Claude part of it is to focus a substantial portion of plans on how to automate little things that costs me time, and it's doing a decent job at that.
I've absolutely not figured it out, but I now have an agent throwing stuff at the wall (with guidance from read access to e.g. my journal and a few other data sources) to figure it out for me, and it's gotten steadily better.
> managing ADHD-like (never diagnosed, and I don't care to assume)
What's the difference here between assuming and saying 'ADHD-like' instead?
The difference is that I know what my "symptoms" are, but I don't know whether the combination of them are sufficient to justify an ADHD diagnosis. I don't like to assume because there's a tendency for a lot people in tech to self-diagnose and in the process sometimes diminish the difficultly others have with them.
It's pure armchair psychology, but this type of project always makes me think about anxiety. Who really needs this level of self observation and control? At the same time, I really enjoy reading about it and I find the window into somebody else's world intriguing.
> Who really needs this level of self observation and control
I liked doing similar things in the past. There's no anxiety in the equation, just pure curiosity. How many times have I done a thing a month/year? I was always curious about stuff like this, much like the OP. There's also the hacker spirit in play - designing the apps for tracking stuff.
You're equating two different things here: "building a custom web application" with "quantified self". These are two entirely different things.
Felix's statement isn't a condemnation of quantified self. I think that sometimes, when you're applying algorithms that aren't well-studied, you get pointless or bogus data. I feel much this way about sleep tracking algorithms from the likes of Apple and Whoop. Vitals, too, although it seems remarkably good at detecting when I'm about to get sick.
As a person in the older demographic of HN, having an Apple Watch quietly collect useful data over years and store it in a database has been immensely useful in helping resolve medical issues.
I used to really be into QS, and if I'm training for a marathon, I'm still studying graphs and drawing conclusions. I'm still into QS, but now I just silently log data to Apple Health and use it for medical histories or to identify certain trends (vo2max, cardio fitness, blood pressure, etc) a couple times a year.
I never spent as much time as OP on this but I did collect a lot of data during peak quantified self. I liked having automatic data collection and being able to see trends, for example. I stopped because of a few issues, not related to time:
* Hardware companies went out of business, stopped supporting devices, etc. It became obvious that there was no long term commitment to make good quality hardware that lasted a long time.
* Many devices and/or data collection was consolidated big, data hungry companies Google and Apple. Competitors have similar anti-consumer uses of data. I don't want any of these companies to have my data.
* Related to the last one, limited to no offline or local only data collection.
It is very hard to gather most of this data with off the shelf hardware and keep your data private.
This provides some insight into startup ideas around privacy, local first, offline only self data collection. I agree this kind of personal life data is something you don't want to contribute unknowingly to big data. I could see wanting to share it particularly around a health problem where only massive compute has a chance at providing answers.
> however retrospectively, it did not justify the hundreds of hours I invested in this project.
Trying to extrapolate this conclusion to the entire "quantified self" movement is not correct. The issue is the time cost, not the act itself. If a trusted company came along (as if...) that sucked up this much data to allow you to answer these questions with minimal effort, I'm sure this would be a different story.
Anything at the fringes of tech with no tried and true solution requires hundreds of hours of effort. The author's conclusions are also personal, there are other styles of living and conclusions to be drawn that change the calculus on whether to do it or not.
I ended up doing something similar to this project a year or so ago: for nearly an entire year, I tracked every single thing I ate or drank.
Who knows how many hours I spent scanning nutritional facts on the backs of boxes, estimating amounts of liquids, and even tracking sips of water. And weighing myself! Thank goodness I used a "smart" scale at least, and I didn't have to worry about carefully inputting my weight to an app each time.
But the whole project was an exercise in perfectionism. "I have to remember this sandwich and log it the next time I'm alone" made me anxious, but once I logged it, I felt a sense of completion. The database and my personal history are now at nirvana. Everything is complete.
All for me to learn things every human alive knows today: eat more food and you'll gain weight; eat less food and you'll lose weight. Yes, I can now tell you the exact average difference in calories I'll eat, statistically speaking, on a day that I have adderall in the morning vs a day that I don't. Yes, there's a similar (but much smaller) difference in average calories per day if I have caffeine in the morning as well. And I can tell you that I generally eat an additional 200-400 calories per day on a Fri-Sun than on a Mon-Thu. Wow, groundbreaking.
I've always had a lot of water, but matching foodanddrink.csv to my HealthKit data showed that I have more water on days that I walk more steps. Mildly interesting to see it written out for me? I guess. But was it worth cataloguing every cup of water? Absolutely not.
Was any of [gestures broadly at me pulling out my phone and cataloguing each item I ate] necessary to learn that? Of course not. It gave me a chance to look back at the database and say "Wowee! I did that! Every day for a year, wow, I'm so cool!" and not much else.
extensive tracking of self-related metrics to improve ones health is the equivalent to reading tons of self-help books to improve ones life/social skills/...
We already mostly know what makes people happy/healthy: personal connections, physical activity, healthy diet and some sort of purpose/goal in life that goes beyond day-to-day activities. The problem is that these things generally require (hard) work and can be unpleasant sometimes, so humans do what humans do and spend unreasonable amounts of time doing the more pleasant things such as reading and gathering info rather than applying these and what they already know. (That's not to say that a project like this can't be fun or lead to insights, especially across longer time spans, but i feel like all of the questions in the first paragraph have fairly obvious answers if you know yourself at all, that don't require extensive tracking of stats to get)
This whole site is a rich persons humblebrag. That renders all of it moot as far as value.
Agreed, I have tendency to do this to “fix” something in my life, after doing it for 2 weeks I already know, just the consciously pay attention to something even without recording make it very obvious what’s going on.
Sometimes it's the journey not the destination.
I did something similar to pull data from my Garmin watch. This meant writing all manner of code to pull data out of FIT files (interesting and often infuriating self-describing file format), coming up with schemas to hold that data to make it queryable, adding visualisations, performing analysis, pattern matching, etc.
The end result is nothing really useful, I had a bunch of scripts that semi-automated some jobs that would have taken 1 minute to do manually and only ever needed to be done a max of five times a day, but I learned a load of things along the way. Often these were useful lessons that can be applied to many other things when developing software.
In a similar vein I've gone to lots of trouble to build a cooling system for my homelab rack (ESP32 to control PWM fans, Dallas 1-wire for reading temp/humidity, exposing measurements as metrics for scraping/observability, designing things to deal with the different voltages involved, etc). I could have just gone and bought an off-the-shelf solution from AC Infinity and installed it in minutes but where would the fun in that be.
I wouldnt have minded if I kept a simple daily journal with a photo a day. If I had that now with LLMs I could ask it "what year did xyz happen".
yep, I do a simple version of this in Google Sheets. Very useful to be able to "Ctrl-F" your life, especially when combined with Google Maps location history.
Having set up Analytics on all my websites, I noticed absolutely zero change in sales volume therefore Analytics is a waste of time.
> The whole "quantified self" movement might be more about OCD and perfectionism than anything else.
As someone who has dealt with OCD and perfectionism, I think that could be the case for a lot of people. And the urge to obsessively track everything can be debilitating.
One interesting actionable takeaway fact is that if you do such a project spending so much time on it, your life outside of it likely sucks.
I too learned this the hard way. I still haven't figured out why this is the case - like, is the inherent incidental irreducible complexity of human life too high, so it dominates majority of our actions?
> The whole "quantified self" movement might be more about OCD and perfectionism than anything else.
Quantified Self, at least in the intended form, is focused on testing specific theories, not on collecting large amounts of data and trying to find something interesting in them.
See, for example:
https://gwern.net/zeo/zeo#what-qs-is-not-just-data-gathering
Yep. Same goes for elaborate note-taking and productivity systems. The simpler, the less bookkeeping and cognitive overhead, the better.
*quantified self
If you are not sick, tracking your life like this is, like he said, is useless. Enjoy your life!
On the other hand, if you are sick like me, charting your long term heath data from doctors visits and photographing skin issues can lead to great discoveries. I have been diagnosed with Erythrocytosis and a susceptibility to mycobacteria infection which caused pulmonary nodules and skin lesions. Only after showing my data collection to my doctor. Since I have mental illness they constantly over looked my physical issues so I needed hard data to convince them of my ideas.
For those curious, I have an minor IL12B deficiency and a partial immune deficiency leading to mildly elevated levels of DexoyATP which is partially corrected with zinc supplements.
Cynical take: I didn't need to read to the bottom to know it was useless to track the correlation between drinking alcohol and dancing
This is disappointing.... Last year i noticed large chunks of my life were being monitored via many spreadsheets and i endeavoured to bring them all together in to one Oracle DB. My plan was to eventually put some ui and graphing on top very similar to OP but seeing this is making me think twice.
The whole "this is a movement" thing might be more about mental health issues than anything else.
I kinda disagree. Doing something about open questions which are bothering you can be valuable in itself, even if the result of the process is useless in the end. And usually you are also learning some valuable things along the way, or just push your life into a necessary direction. That isn't related to OCD or perfectionism, it's just gaining more control and insight about some aspects of your life.
> it's just gaining more control
Yes it is! And if you control everything, you won‘t make mistakes.
A simple back of the envelope calculation shows that Felix causes between 70 and 110 tonnes of CO2 emissions per year just from flying.
Paris accord says 1.5t per person per year, from all activities, Felix's flying alonre is ~10-15x current European yearly per person emissions and ~50-75x those compatible with +1.5C.
If you want to reduce air travel for environmental reasons, then tax it more.
Shaming individuals doesn't seem to be productive or helpful.
Air travel works for people if the benefits outweigh the costs. The only thing that changes behavior is to change the costs.
And even if costs were 10x there are still plenty of people who will fly tons, because it would still be economically productive. There are always going to be people who fly 10x more than others, because certain jobs and roles simply require it.
> If you want to reduce air travel for environmental reasons, then tax it more.
> Shaming individuals doesn't seem to be productive or helpful.
First, none of us have any power to "tax it more" so this is a dead end of discussion. Second, people have agency and we can hold them accountable socially for negative actions even if they are abiding by the current laws (or tax regime). This happens all the time, because laws don't fully align with morality in a culture. Suggesting that we should leave such things to the sole discretion of the economy and taxes describes a strange unhuman-like society that we don't live in.
> Suggesting that we should leave such things to the sole discretion of the economy and taxes describes a strange unhuman-like society that we don't live in.
Well, the choice is: either we leave it to economy and taxes which work, or we'll rely on shaming which doesn't work. To put it another way - it depends if you want to actually fix the problem or just want to have a moral high ground and don't really care about solving the problem.
> First, none of us have any power to "tax it more" so this is a dead end of discussion.
You probably have more power over that than changing a whole population's behavior.
I'd wager you probably have more power over that than changing just one person's behavior (Felix).
> Second, people have agency and we can hold them accountable socially for negative actions even if they are abiding by the current laws (or tax regime).
And people are holding you accountable for the shaming. This leads primarily to polarization, not change. Polarized folks are more likely to increase the negative behavior.
> Shaming individuals doesn't seem to be productive or helpful.
I don't see any shaming. It was all matter of fact, free from judgement.
What do you think was the purpose of the comment?
Do you think that comparing someone's CO2 emissions with the average and pointing out that it is much higher is value-free, just a totally neutral observation for no reason? That the commenter is fine with it? Or even that it's a good thing?
No, it does not mean the commenter is fine with it. Regardless of their feelings, the commenter had the good sense not to include any shaming.
So you think the commenter was neutral? No judgment? Again, what was the purpose of the comment then?
What is the purpose of your comment? Are you neutral? No judgement?
I asked you first.
But I'll also respond to your questions: my purpose is to show that your claim that the original comment was "free from judgement" is wrong. I'm not neutral, I'm attempting to show that your claim is obviously false, that it's not plausible at all. Of course I'm trying to judge a comment that seems wrong.
So now that I've replied honestly to your questions, will you reply honestly to mine? Repeating:
> So you think the commenter was neutral? No judgment? Again, what was the purpose of the comment then?
Because if the purpose wasn't to shame the person for their carbon footprint, I can't imagine what else it possibly could have been.
Ok, fair.
But you see: not one comment here is neutral. It would be silly to expect a comment to be neutral, such a comment wouldn't be written in the first place. I think the original comment expressed the point while staying as neutral as possible.
> So you think the commenter was neutral?
Yes, it stated some facts.
> No judgment?
Yes, it contained no explicit value judgement. Any value judgement we bring into it is our own.
> Again, what was the purpose of the comment then?
How would I know the purpose of someone else's comments? I don't even really know what my purpose is debating here with you. I certainly don't see myself persuading you of anything :)
What you write doesn't make any sense. You say it's "silly to expect a comment to be neutral" but the comment is "as neutral as possible" and then answer if the commenter was neutral with "yes". Those aren't consistent.
I don't know what definitions of neutral or value judgment you're using, but I hope you can use this as a learning opportunity. The original comment has the obvious implicit judgment that a greater CO2 footprint is a bad thing. This is shared context. It is so obvious it doesn't need to be explicitly stated, any more than "murdering people is bad". The purpose of the comment is clearly to shame the person for having such a high carbon footprint, otherwise there's no purpose in bringing it up. I don't know what your purpose was in trying to deny that. But if you genuinely didn't understand before, I hope now you do, and that this has been helpful in improving your reading comprehension or understanding of shared/implicit context.
> The original comment has the obvious implicit judgment that a greater CO2 footprint is a bad thing.
No it did not contain any judgement. That is your reading and not the comment.
> This is shared context.
The world is a large place and different people share different context.
> The purpose of the comment is clearly to shame the person for having such a high carbon footprint, otherwise there's no purpose in bringing it up.
Informing perhaps?
I wonder what your gripe is: Do you think it's inappropriate to mention the CO2 at all? Or do you think that the poster should have said it in a different way? In what way?
I think a broad tax would just make it more difficult for middle and lower class to fly. Tax the business/first class and frequent flier, but don't push people who can already barely afford to fly out.
> I think a broad tax would just make it more difficult for middle and lower class to fly. Tax the business/first class and frequent flier, but don't push people who can already barely afford to fly out.
No, it's the other way around. If flying is bad then poor people absolutely have to be priced out of it. I know that it sounds absolutely awful, but unfortunately taxing just business and first class won't do much for reducing emissions if flying will be affordable for and used by billions of people.
I appreciate this concern, but "poor people exist" is not a valid reason to continue destroying the environment, imho. It's not just people that have to bear the repercussions.
That said there are probably some work-arounds, tax free twice a year, tax rebate or some-such.
You wrote one of the solutions as if it conflicts with the other one.
Let's raise the tax on an activity according to its negative side effects, while pointing out individuals that do a lot of it and dont take personal responsibility.
If you run a company or companies on two coasts and have a wife and family on another continent (say she has her own career and can't move), then what exactly are you supposed to do?
I don't know this guy's personal life, but the people I know who fly tons fit into this profile. E.g. the wife can't move because she's a tenured professor at her university, and he's got to be at both offices regularly. He's best qualified to run the company/companies, and he's not going to get divorced to reduce his CO2 emissions.
What exactly is the solution you propose? What personal responsibility do you expect them to take? You think he should get divorced? Only see his wife and kids four times a year? Have his company/companies suffer because he can't be there in person? Quit his jobs?
And let's be clear, there are lots of jobs that require tons of air travel. If you're a highly specialized repair technician for certain equipment, all you do is constantly fly around the world fixing equipment wherever it is. If you're a CEO of a multinational company, you're constantly flying around to different offices. Are you looking for "personal responsibility" here too? How?
I can't tell if you're serious or if you really think someone who has a family on another continent and is running two companies on opposite coasts is some kind of victim of their circumstances and needs a special keyboard warrior to comment on HN in support of them, lest they face the consequences of a little more tax that they'll never miss or some social shaming.
I'm sorry, I don't want HN to to be the place where we get into a fight over the mildest inconvenience for people who are already living extravagant lifestyles.
I think you've misunderstood me.
I suggested raising taxes in the first place.
What I'm opposed to is some hand-wavy demand to "take personal responsibility" without suggesting exactly what they're supposed to do and whether any tradeoffs involved are reasonable.
And please don't call people names. You can write comments here without calling other people "keyboard warriors". Nor is it helpful to try to shut down some viewpoint by claiming that somebody doesn't need any extra support.
And I think most people would consider not seeing their family more than e.g. four times a year more than just the "mildest inconvenience".
> I think you've misunderstood me.
Knowing this completely changes the tone, I’d missed that you were that same commenter too.
> Shaming individuals doesn't seem to be productive or helpful.
I don't see how much support from history for that viewpoint. Some examples of positive societal change driven in part by shaming individuals: drink-driving, civil rights, sexual harassment, automobile safety, the slave trade, McCarthyism.
All those cases also have huge penalties or effective costs associated with them. Is there an accurate "shame first, then penalties came later" stand point?
Automobile safety in my life has only changed after fines. Sexual harrassment still happens and doesn't seem to be helped by shaming someone as much as firing them. Though we often don't have the guts or legal backing to publically shame someone.
> drink-driving, civil rights, sexual harassment, automobile safety, the slave trade, McCarthyism.
This hasn’t been a good few years for your examples.
That was my main takeaway as well. How do you do this without any shame? My man is emitting an amount of CO2 on par with an small African country
Most people don't spend a lot of time wistfully considering their CO2 usage, myself included. The religious zeal by which people feel the need to tell me about the ever-warming planet is honestly more off-putting than most actual religions.
Of the very few "f*cks" I can give in my life, I prefer to spend mine as I choose rather than being scolded for not giving mine to the pressing issues that others deem important.
The thing is that everyone alive today and in the future is footing the bill for that indifference. It's nice that you don't care, but it's not something I'd brag about.
Is the OP flying a private jet or something? Unless he is, it's a useless metric. The people flying private are responsible for a 1000x a regular persons emissions. It's offensive to suggest regular salaried people are supposed to be "doing something" in this CO2 effort.
It's true that a single private jet is causing tons of CO2 emissions. But in the end, all consumers control the market. You can jump one link further in the chain to regions with much lower emissions. Somalia/Congo emit over 200x less CO2 per capita in comparison to the US. Do you think that's fair for them if the "regular salaried people" don't care? If responsibility always gets dismissed by pointing to someone emitting more, nothing changes.
I too don't want to be a martyr hermit who don't travel and consume just to cur my CO2 emissions in half, while someone else generates 10x of that living a full life. That's just dumb, and I consider this whole movement rich people telling poor people that they should be considerate and sacrifice themselves.
If you truly care about the planet, don't have children.
It's an "all hands on deck" situation, yet everyone seems to think that someone else has to do something. You don't need to be a martyr, just to make an effort.
At the very least don't brag about not giving a crap.
I don't give a crap, in the most boastful way possible. Sorry if my zealotry isn't your zealotry.
Thank you for destroying the Earth!
Thank God I'm a single human and don't have nearly that much influence over the Earth.
"No individual rain drop ever considers itself responsible for the flood."
> If you truly care about the planet, don't have children.
That's a fallacy; people care about the planet precisely because of children. I don't care about the planet for its own sake; I care because of the humans who inhabit it and their future lives.
Also, humanity spent 100,000 years without flying around the globe, and I doubt they were all living hermit martyr lives.
I don't feel it's properly engaging in good faith to say that I don't care. I don't specifically care, in the "feeling shame" sense that the GP had mentioned. As I pointed out, we all have a limited number of things we can realistically care about.
The fact that I happen to care about other things more than this specific flavor of global catastrophe is morally OK.
The way I see it, "from each according to their ability" is the right approach here. If you can afford 30% less flying, red meat etc, then it's your duty to do it. You don't need to make your cat go vegan; just... do what you can, even if imperfectly.
Shame is not necessary, but callous indifference is not acceptable. There is a middle ground where you treat it like a habit to improve, like a step count.
And this is it. This is why we are where we are today. That it is seen as taking a religious zeal to realize how flying very frequently is disastrous for the climate. That's our bar and what we have to work with. Yes, we are properly fucked.
I would totally do this without any shame if I had the need/desire. CO2 isn't going to be solved by well intentioned individuals making absolutely no impact. It will be a generic solution that solves it for everyone, or it won't be solved at all.
I'm also not going to take shorter showers when people are farming in a desert and shipping the crops to China.
You might think this makes me a terrible person. That's probably good. Because it will help people understand what we're up against and what needs to happen to actually solve the problem.
"Take less flights" isn't the solution.
I refuse to worsen my life for an absolutely minuscule abstract benefit to someone, eventually, maybe. I am going to continue eating meat and flying when I want to, and I don't feel the least bit bad about it. It's unfortunate that climate change is happening, but my personal actions are not meaningfully contributing. My choices do not have cascading effects on others; whatever I do, the climate will change in the same manner as it otherwise would have. In that light, I am not going to reduce my QOL just to feel self righteous for no actual benefit to anyone.
You may wonder how this is consistent with my propensity to recycle and follow traffic laws and not do crimes and other socially beneficial minor things to which a comparison could be made to CO2 output: because those all have a greater benefit:cost ratio than flying and eating meat, by far. I am including the benefits of living in a high-trust society in that analysis.
"A society grows great when old men plant trees whose shade they know they shall never sit in."
Instead we're razing the forests and bragging about our indifference.
I work as an airline pilot, and I just recently calculated mine to be 65,658 tonnes in my 10-year career so far (not quite so bad when calculated per seat-mile, but still eye opening!). https://jameshard.ing/pilot/#statistics
There is a section comparing flight emissions to US citizen average total emissions. This might make him feel good, but only 30% of an average American's emissions come from transportation and just a sliver from flying, so it's very likely his total emissions are much higher.
I believe you misunderstand what "average" is: Americans (or people in general), on average, do not fly much.
Some — eg. those who've moved for work — probably fly 10-15x the average, and 10-15x of people fly 1/100th of average. Flying cost and availability correlate with average needs, but that presupposes that some people have higher needs altogether.
Exactly, I stopped reading when I saw the flying stats. There are people who still haven't clocked where our climate is headed.
I get that you may have to see family abroad or maybe indulge for a holiday, but this is "I'm using an airplane to commute" kind of level.
And here I am trying to book my train tickets to go to London instead of flying even though it costs three times as much just to avoid a few kg of CO2 (among other things), it's making me angry.
Depends where you take the train to London, it is a much nicer experience anyhow than going to airports and people should consider that as well (ignoring climate stuff)!
On the price, the very annoying thing is that fuel for planes is not taxed! Changing this would require quite some effort (falls under some specific laws, that are old and nobody wants to touch, etc.) but I think everybody should just ask "honest tax on fuels!" as this will make less people say (or thin) "but climate change is a hoax". Planes are just unfair competition to other transport due to taxes!
I agree re: fuel taxes, but it’s a complete nonstarter: passengers would be voting against their wallets, and airlines would lobby against it since it’s a vote against their business model.
Still worth trying, IMO.
An alternate approach that would be seen as consumer and business-friendly would be subsidizing companies with a certain level of fuel efficiency per passenger mile, targeted above current levels.
Surprise surprise. Felix is as rich and out of touch as most rich and out of touch people.
> And here I am ...
Reminds me of the soggy straw memes floating around now. I've been having those why bother? thoughts as well.
Eh, other people throw litter on the floor and rob elderly folks in their homes. Those people hardly ever get caught, but neither you nor I are are going to start copying their actions.
I wasn't thinking about assaulting the elderly, but flying more often to see family and friends.
On the scale of unacceptability, you're firmly in failing to wipe down the hand basin after using it territory. You get a pass.
I stopped reading at "San Francisco was always scary to walk"...
Isn't this a drop in the ocean? Why would any 'normal' person forgo flying? How much CO2 emissions have 'world leaders' produced going to summits, or Taylor Swift and her fans flaying to concerts or war flights?
Couple of things: 1) NO ONE is suggesting any one forego flying altogether, or skipping their once-a-year overseas vacation or periodic family visit. 2) THIS level of flying is not normal and is exactly the kind of harmful behavior people have in mind when they complain about frequent flyers. 3) Whinging about summits and Taylor Swift is just a bad faith red herring argument. Obviously less flying is better, no matter by who. To the extent it's related to the topic at all, it bolsters the case for less air travel.
28 flights per year.
That's 14 round trip flights. A little over once a month. Not at all surprising if once a month. Or divorced and kid is in another state.
If we're generous and assume each trip is actually 2 flights (i.e. connecting flights), then it's 7 per year.
Yeah, more than typical. But not insane.
> Not at all surprising if once a month.
Typo: Should have been "Not at all surprising if for work".
I don't know why someone going on a vacation would have moral high ground over someone that HAS TO travel for his work. If you are scientist you absolutely have to fly a lot to visit lots of conferences, disseminate your work, provide lectures etc.
Why do anything for the greater good at all then? (Also there's a big gap between "forgo flying" and "fly every 2 weeks for 7 years")
Everything any one person does is a 'drop in the ocean'. Thankfully, we organize and do things collectively very well - it's in our fundamental nature going back to non-human ancestors, and there is a long, rich history of how much we accomplish. Alsmost nothing that has ever been accomplished has been done without a lot of people doing it together.
The general concern around Taylor Swift's emissions has always struck me as shortsighted. Her Eras tour is estimated to have generated around $5bn in economic uplift in the US, at an estimated 10,000 tonnes CO2e for her personal travel. Even if the total footprint is higher, that is thousands of times lower than the emissions intensity of an industry like fast fashion. From an environmental point of view, attending a Taylor Swift show is a much less carbon-intensive way to spend your money than ordering from Temu.
People just want a scapegoat that allows them to say they have no responsibility
yeah America treats flying as like taking a bus, so I am not at all surprised
> yeah America treats flying as like taking a bus, so I am not at all surprised
What does this have to do with Felix?
Exactly, you'd never see any airlines get popular in Europe on the back of fares costing less than a bus ticket
Unrelated link: https://xcancel.com/Ryanair/status/776292730179682304
In my experience, tracking objective things like "nutrition" and "sleep hours" is immensely useful to reflect on what went wrong, and tracking subjective things like "mood" or "stress" is useless given hedonic adaptation or heavy swings that make problems obvious, and not need tracking.
What's key is be able to visualize metrics easily on the data and frictionless data entry, I've got a decent setup with iPhone Action + Obsidian + QuickAdd scripts on Obsidian Sync (mobile + laptop). for visualization I use Obsidian Bases and Obsidian notes that run Dataview code blocks and Chart.js, couldn't be happier.
I could track things that are not interesting to reflect on like vitamin D supplementation for accountability but I've never bothered, especially if it's taken ~daily.
The language you use to describe this is fun, I make an app for self tracking called Reflect and would love your opinion of it, even if it doesnt suit your needs exactly.
https://apps.apple.com/us/app/reflect-track-anything/id64638...
I think it's good to track mood swings, because it makes you notice them. After a while it makes you call out your own BS.
Strongly agree with this. I’ve been using Apple’s “mood” log for about two years now, and it is extremely helpful for me to have a concrete view of the history of my general affect.
“This entire month I’ve been feeling good, I want to pinpoint why,” or “it’s clear since stressor X entered my life, my affect is lower; how can I resolve this?”
These long term trends are harder for me to track without data. It might be easy for others, but not me!
As someone with Schizoaffective Disorder Bipolar Type, if you are not diagnosed with a mood disorder, tracking "swings in you mood" when you have no clinical disorder seems like a disorder of its own.
I have had people tell me they were "manic". Then I showed them videos I took when I was manic and they see what I mean when I tell them they are not manic.
We have come to a place where we do not want even normal fluctuation in mood, and that is a illness of its own, but it is a cultural illness.
Maybe for some it's a lot more extreme than for others, but even if it's not so dramatic as to be categorized as a mental illness wouldn't you want to know if, say, there were a direct correlation between whether you went for your morning run and your mood later in the day?
Is this something that needs to be tracked to bring into your awareness? We have a memory storage device sitting on top of our spine. When I drink I feel drunk. Easy. If the change is noticeable you will notice it and remember it.
I am just trying to save you time and escape the cycle of "optimizations" which is where all this data logging leads.
Yes? As I wrote, the mere act of writing down your feelings forces you to acknowledge them and see patterns in them. Sometimes while writing something down I realise "wait, I've been through this before" or "every time this person is around, I feel this way". It helped me be more self-aware, for my own good.
It turns out that our memory storage device uses a very lossy form of compression. Memories get simplified and distorted over time. Heck, I can't even remember when something started hurting, so how should I notice a year-long pattern of thinking around a certain topic?
The message here was "journaling is a useful form of introspection".
> Please respond to the strongest plausible interpretation of what someone says, not a weaker one that's easier to criticize. Assume good faith.
I get that everyone wants to be cynical about this, but you really can't deny that both the visualization and sheer scale of data is impressive. The way the "my life in weeks" is done is also very cool, I'll be stealing that for myself.
Bravo. This is my dream and also my nightmare. I was super into the quantified self movement a while back, before Apple Watch, writing Withings Scale API wrappers in Ruby and Fitbit days.
Every time I try to seriously track metrics of my life, the excitement of the insight gets worn away by the friction of recording and managing. I expect LLMs can help reduce the cost of this by an order of magnitude but then, as you mention, the question is, what do you do / change / learn because of the data?
I recently started tracking nutrition macros with an iOS app MacroFactor which I really like. This is the first time taking my weight doesn't feel like a IDK SHRUG moment and I can actually map my food intake to my weight.
Finances is probably the other highly actionable data source that is such high friction to manage (downloading CSVs, OFXs, monthly...) that it has always been a false start for me. I finally wrote a service to talk to Plaid directly and I successfully used it to categorize my business expenses at tax time. I finally have programmatic access to my bank account data!
You conclusion is definitely a cautionary take: > the main conclusion is that it is not worth building your own solution, and investing this much time.
But, perhaps a subset of that data you find useful.
For the few times that I have tracked myself doing various activities, I have found that I didn't even need to do anything with the data, I was actively changing habits whilst I knew I was being monitored. For example, I wore a GCM for a few weeks and I found it difficult to live my "normal" day to day lifestyle. I wanted to do the things I felt were "right" and "healthy". Whether or not that is a good thing or not is a different topic, but just my 2c.
Totally makes sense. Eventually being confronted with the measurement is its own motivator. Some sort of “future self” awareness or something.
Why don’t you just query Palantir DB by your human ID? It shows your entire life data and much more.
That made me laugh, but it was a sad laugh, because it is true.
can't afford a subscription
I ended up building something similar but way smaller in scope -- just a cron job that pulls Apple Health exports, bank CSV dumps, and git commit stats into a single SQLite db. The queries that actually proved useful were embarrassingly simple, stuff like correlating sleep hours with commit frequency or spotting that my spending spikes every time I start a new side project. The real 80/20 of personal analytics is just getting your data into one place where you can join across sources. Everything beyond that is diminishing returns.
How hard was setting up the bank CSV dumps ?
Honestly the hardest part was normalizing the date and amount formats across banks. Each one exports slightly differently. I wrote a small Python script per bank that maps their CSV columns to a common schema (date, description, amount, category) and then bulk-inserts into SQLite. Once you have the schema figured out it takes maybe 20 minutes per bank. The manual step is downloading the CSV each month, but most banks let you set a date range so it is just a few clicks.
The value rarely shows up where you expected it to.
I kept a rough log of my sleep and mood for about a year with no specific goal. Mostly forgot about it. Then I had a weirdly bad few months and went back to look — turns out there was a pretty clear pattern I would've never noticed in the moment.
Maybe the framing of "was it worth it" is the wrong question. It's less like an investment with a return and more like keeping receipts. Useless 99% of the time, then suddenly you really need one.
As someone who dislikes being tracked, I find it disconcerting to have this level of data stored in 3rd party services, but joining up multiple services to give one coherent picture is cool. How cool is questionable, trying to correlate health data with location data is going to give a strange picture, and I question how relating health data to weather is useful. (Do I have lower blood pressure on rainy days?)
Forgive me for I am being sceptical. The might be some insight here I have not considered, but I'd feel a lot more comfortable if it was all self-hosted / self-collected data.
The appeal of a single queryable database for personal data makes a lot of sense - the hard part is always the ingestion and normalization, not the storage.
One pattern I have seen work well for the business version of this: a "company intelligence" database where everything known about a prospect company gets accumulated in one place over time. Homepage content, job postings, news mentions, funding history, tech stack signals, all deduplicated and queryable.
The challenge on the B2B side is the same as personal data: the data comes in from 8 different sources in 8 different formats, often with conflicts (two sources disagree on headcount, three sources have different founding dates). Your approach of controlling the schema from the start rather than trying to normalize later is the right call. Schema drift is what kills most long-term data projects.
What storage engine are you using? And how do you handle temporal data - do you snapshot state over time or just keep the latest version of each entity?
This reminds me of someone I knew slightly back in the late 70s: Robert Shields, who kept the world's longest diary [0]. He was eccentric but very nice...but he wrote 33 million words, filling 90+ boxes, of every detail of his life for decades, only quitting when a stroke rendered him unable to type about ten years before his death. I doubt he tried to reach any conclusions based on his journaling, though.
[0] https://en.wikipedia.org/wiki/Robert_Shields_%28diarist%29
This was far more interesting than I first thought it would be when clicking the link. In particular, the place/time and life events and such being presented this way told a story and was fun.
> Probably obvious for many, but I didn't realize ACs don't transport any air into the room, but just moves it around
I had the same epiphany as you days after acquiring a CO2 monitor. Most people notice poor indoor air quality from proxies such as humidity and temperature. AC (without ventilation) eliminates these and tricks our senses very effectively, giving us cool and fresh feeling indoor spaces full of CO2 and devoid of oxygen.
This looks like it requires a heavy amount of discipline to track everything consistently over time. How do you build that into your daily routine?
Capturing data was the main bottleneck I ran into years ago when I tried something similar with a little MS Access database. I spent lots of time making it pretty and defining fields and data types, but actually typing stuff in? I lost interest quickly.
It's possible the friction could be reduced here by having some kind of Generative AI try to help capture data, but then you'd have to verify that it was being done correctly... honestly, I think it's simply not practical for most people to do this.
Real life is messy. How much time you want to spend on recordkeeping to make it seem less messy or make you feel like you have more control is up to you. But sometimes it's better to embrace the mess and let go of control, in my opinion. Chances are, no one's going to care about whatever you do here in 100 years. YMMV as always.
I can see the value of doing this but I think that this is a case of the 80-20 rule. Just the low effort 80, which could be keeping a journal where you reflect on how you felt a given day, why that might be, and what steps to take to avoid it from happening (or keep it happening) should get you pretty far.
Most of the things that really hinder my productiveness and happiness are the same things that everyone tells you about; sleep, diet, sunlight, not being stuck indoors all day, socializing. And most of these can be improved by making changes to my habits.
Other than that, is that I think that it's important to get into the habit of self reflection. Having a feedback loop on my own life has helped me find out what works and what doesn't. It's too easy to just go through the days slogging through and not making any changes because you don't even realize what's going on.
An interesting experiment, I think I'm too uncomfortable leaking data I don't yet know why someone would curate to me free of charge until I knew. If there was a FOSS suite like Home Assistant that would do a few of those things I might try it out, especially the weather (I would add air quality) correlation to mood and other subjective states.
The hardest part of this kind of personal data system is retrieval not storage. At some point you have more data than fits in a prompt, so you need to decide what's relevant per query. Did you build any ranking or filtering logic, or do you query specific tables directly?
> 2025 Update: I’ve stopped collecting data and working on this project, but I will keep this page alive.
I wonder why? Also, perhaps worthwhile putting this at the very top of the page, rather than the very bottom.
Flying stats dashboards always amuse me. I get it, for the non-pilot it's kind of like a status thing, "Well I traveled X times in 2025!". As a pilot though, I have gobs of stats I could put up there but from flying for 15 years I realize there's not really anything meaningful in there other than "Gee whiz, I flew a little more/less than last year." I know some other professional pilots do track some of their stats a little closer as they try to optimize for hours flown:hours paid but I've never cared to hyper optimize my schedule in that way.
Tangent, I've always had notes and have this longing to store things but at the same time I like "moving on" too. For ex. I used to have windows phones and saw I still have an accessible One Note from 10 years ago... tempted to read through it but also I've changed as a person.
Another tangent, recently bought a paper shredder, started shredding through boxes of mail I have kept since they have personal info like cc statements but on the same idea of moving on/reducing stuff I'm hoarding whether it's data or physical.
I thought you created a database from scratch, got me excited! (I’m a db guy)
He did!
The database is the data, collected into a coherent structure.
The database management system is (optionally) software. :)
I wonder how much time you spend daily on tracking things / data entry
Just check one of the pie charts and it'll tell you!
Waiting for a neckband/choker wearable that can take transcription from my vocal cord nerve, so I can operate a chatbot silently. Yes, scientists have made impulse-to-speech lab equipment.
Tried the manual tracking a few times and it does not tend to stick. Passive capture is the approach that ever worked for me. Having different apps and devices with permissions to track you and good APIs that enable integrating the data seems to be the way to go.
Did you think of building some proactive AI tools to make use of all this centralized data?
I store my travel in YAML files in git.
https://edwardbetts.com/agenda/trip/past https://edwardbetts.com/agenda/trip/stats
This is fascinating. One question I always have with large personal datasets like this: at what point did you start getting genuinely surprising insights versus confirming things you already suspected?
380k datapoints sounds incredible but I imagine the real challenge is turning that into decisions that actually change behavior
"Unsurprisingly, I'm exposed to less solar energy in Winter"
Never heard it phrased like this, and wondering if this phrasing explains the rest of the project as well :)
Related to this, I highly recommend anyone to install github.com/ActivityWatch/activitywatch, it's an amazing tool to keep track of your computer use completely locally. I think there are lots of possibilities with data analysis/AI aimed to improved one self's life
A shame the Android app hasn't been maintained in a while
I strongly recommend reading "The Right to Oblivion" for a different perspective. Great book too.
Wow, this sounds amazing! I am a statistics and chart lover, I love tracking various types of data, but this? I can't even imagine how much time it must have taken to input all the data. Huge respect to you! Keep going.
"GitHub Open Source Contributions" aside, everything else falls into the consumption/intake/internal categories; thin data on production/output/external ones.
> I walk more than twice as much when I'm in New York, compared to any other city
This is why I moved to Tokyo. Even if I want to avoid exercising I still take many steps
And here I am, feeling slightly ashamed at taking 1 or 2 flights a year. These "commute" statistics are staggering.
Taking “Know thyself” to a whole new level. I’d love to have these stats on me, if it could be done by inference, rather than conscious effort.
The step count in NYC stands out like One World Trade Center compared to the rest of the building when looking at the skyline ;-)
That last takeaway "it's not worth it" hits hard.
I went through a similar process and came to the same conclusion, although with a slightly different twist. The project became the point and I almost stopped noticing the data towards the end.
If anyone is interested in doing this sort of thing themselves, I make an iPhone app called Reflect meant for this exact purpose
https://apps.apple.com/us/app/reflect-track-anything/id64638...
I'd like to make the same but with Owntracks instead of Swarm and ActivityWatch instead of RescueTime.
Mmm... yes. It definitely looks like he's in the market for a new mattress.
It's all nice and all, but I'm just sitting here thinking "How can one afford all this flying around"?
Spoiler alert. You're reading a rich guys humblebrag site.
Forget that, I couldn't even fly that much if I was paid to
Interesting approach. One thing I noticed when people build systems like this is that the real challenge becomes decision-making rather than storage, deciding what actually deserves to become structured data. Curious how you decide what goes into the database vs what stays unstructured.
How do you sync the steps data from iOS to your own server automatically?
I had an idea similar to this where you could add information about yourself and answer daily questions and get paid by companies who access this data. This could be an ethical way to share information benefitting all parties.
New Zealand almost made the world map.
You and Stephen Wolfram. Ironically when I ask Mathematica "what is the mass of Stephen Wolfram" it cannot tell me. I'm honestly not trying to pick on him but I wanted to calculate his average density. What a rip off.
No need to spy on him, he publishes everything.
Bro queried his entire database, added a LIMIT 10 and it only produced 2 rows. Just a little humor in good faith.
This might be true if you said it about most people but clearly not him.
He's rich. You don't owe him good faith.
Understood - good faith rescinded, bad faith extended.
I don't know if you're joking, but this is exactly how it should work.
Hope they made backups :)
Dude. Backup that database.
Stop bragging, mine fits into a csv file.
There are exercises and supplements you can take that improve things slightly
You mean removing sense of humour? No thanks. But I can see they work.
To be fair, CSV files can be huge nowadays.
Don’t ask how I know…
So it didnt end up working too well seeing that the latest data is from 4 years ago.
This might sound harsh but for someone who is keen on investing time to track so many things, you should invest some time in learning how to make better visualizations. A few tweaks here and there would really improve what you have.
> Back in 2019, I started collecting all kinds of metrics about my life. Every single day for the last 3 years I tracked over 100 different data types - ranging from fitness & nutrition to social life, computer usage and weather.
I know this is the type of person i would not like.
You'll get crap here for it, but I agree. This is someone that has so little to worry about that he has to look for things. Then you look at his travel habits and jobs and you realize he's just one more rich bored guy showing off as best as possible.
Yeah, we've all put our whole lives into a single database. It's called the United States Government.