Why are they reporting correlations as percentages? That's incredibly misleading. If one wants a more intuitive measure of concordance than the correlation, one could use the coefficient of determination, which is the correlation squared.
This number tells you the percentage of the variation that is accounted for by the tracker. For a deeper introduction to correlations, see e.g. [1].
Let's rewrite the summary with the correct figures (and reduce the ridiculous number of sigfigs):
> The analysis reveals that commonly worn fitness trackers are the most accurate for measuring heart rate, providing a strong accuracy of 58 %. However, they are only moderately accurate for tracking step count and energy expenditure, 47 % and 32 %, respectively.
The highest level of accuracy is provided by the Apple Watch, which exhibits 74 % accuracy for heart rate and 50 % for energy expenditure. Garmin is the most accurate for tracking step count, as it offers 68 % accuracy.
On top of this inaccuracy in actual measurements, many fitness tracker brands invent their own metrics.
"How did you sleep?" I asked a friend who stayed over, - "I don't know" they replied as they raise their smartwatch to their face. "Pretty well, apparently! I got a sleep score of 84."
I've also heard of an acquaintance that they cancelled plans because their (Garmin™) Body Battery was too low.
Instinctively I flinch whenever people place such importance in what is essentially an arbitrary techno-horoscope. But I also find it hard to make an argument against it as long as it inspires healthy behaviour.
> When it comes to energy expenditure, Apple is the only fitness tracker providing a strong accuracy of 71.02%. Jawbone, Fitbit, and Polar yield only moderate accuracy, ranging between 50.23% and 65.57%.
I don't buy it, no brand can be accurate on energy expenditure because it depends on a lot more things than what the watch can monitor.
Even in lab they have to put big mask on people while exercising to get something somewhat accurate.
On watch, calorie counter is a function of time and heart rate. Yet most watch brand can't even get heart rate right beside apple and Huawei.
Tried all the brand, went back to using a 10€ Casio and put a H10 monitor when doing sport.
There is a YouTube channel named "The Quantified Scientist" which offers technical and more in-depth reviews of varied fitness trackers: https://www.youtube.com/@TheQuantifiedScientist. I would very much recommend it if you intend to buy a wearable and care about accuracy
I use an Apple Watch, and while I don’t necessarily care about its accuracy in terms of steps walked or other specific details, I do value the insights it gives me into my daily activity. Even if it’s not entirely accurate, I get a reasonable estimation of how much I’ve moved, stood, and worked out. My watch motives me to me more active.
"The evaluations presented are based on general observations and should not be interpreted as definitive assessments of specific brands or their features."
The most recent paper used for this meta-study is still pretty old, and this technology changes. The study tells you almost nothing that could inform a buying decision regarding today's Apple Watch vs. Garmin Epix Pro (as examples). The sensors on the market today didn't exist and were not evaluated in any of the papers.
67% is good enough for me. I am not using it as a medical device, and I am not an elite athlete. I just enjoy doing exercise, and every now and again I like to review the stats from my watch.
It's like the fat % on my bathroom scale. I know it's not perfect (in fact body % measuring is incredibly hard with massive error margins even in the most expensive/intrusive tests) but I'm not looking for exacts, I'm looking for trends and I've seen them where I expected to. So yes, the % may be bogus but I can use all that data to see trends which is all that matters (to me).
>The analysis reveals that commonly worn fitness trackers are the most accurate for measuring heart rate, providing a strong accuracy of 76.35%. However, they are only moderately accurate for tracking step count and energy expenditure, 68.75% and 56.63%, respectively.
>The highest level of accuracy is provided by the Apple Watch, which exhibits 86.31% accuracy for heart rate and 71.02% for energy expenditure. Garmin is the most accurate for tracking step count, as it offers 82.58% accuracy.
Not surprised, it’s a shame the folks who obsess over the numbers from these devices will be the ones least likely to understand what this actually means though.
I've had an iPhone for many years and just recently discovered the Health app.
My phone has a lot of my "steps", an estimate of my calories, data about my walking stability, and tons of other stuff. You don't even need a watch to get +/- 65% accuracy on anything except heart rate.
I remember one time when I got my daily step counter alert when I was eating chips on a sofa. I chuckled.
For step counting they really seem to only track certain hand motions, I though they would somehow notice how feet hits the ground and vibrations travel through the body to the wrist.
That's better than all SEO tools and probably most analytics imo (only measured SEO tools myself) ... the value in tracking is mainly in spotting trends.
Do they list the models of the brands they are talking about anywhere? For example, they mention Garmin but there is a huge variety of Garmin models with varying accuracy. I'm surprised I can't find it anywhere. They mention gathering data ("The majority of commercially available devices measure these basic metrics, while other metrics typically vary between different brands and models.") on different models but then group them together under a single brand? Why? Frankly, this analysis seems a bit half-assed.
I find fitness trackers/smartwatches expensive, useless, and frankly, creepy.
Why do I need to pay $400 for an Apple Watch that sometimes doesn't have enough battery to make it through the day, and aggressively insists on tracking every movement of mine on the grounds of 'fitness'?
Seriously, that 10 hour stand goal and 'streaks' and what other achievements are just to make sure this GPS and heart rate measuring device is on your wrist every day of every week.
It's utterly incapable of measuring anything fitness related, perhaps with the exception of some cardio stuff, but it's useless for any kind of weight training.
It's baffling to me that Apple somehow got the population at large to pay for its overpriced ankle monitor and got them to upgrade to the newest model every year.
Why are they reporting correlations as percentages? That's incredibly misleading. If one wants a more intuitive measure of concordance than the correlation, one could use the coefficient of determination, which is the correlation squared.
This number tells you the percentage of the variation that is accounted for by the tracker. For a deeper introduction to correlations, see e.g. [1].
Let's rewrite the summary with the correct figures (and reduce the ridiculous number of sigfigs):
> The analysis reveals that commonly worn fitness trackers are the most accurate for measuring heart rate, providing a strong accuracy of 58 %. However, they are only moderately accurate for tracking step count and energy expenditure, 47 % and 32 %, respectively.
The highest level of accuracy is provided by the Apple Watch, which exhibits 74 % accuracy for heart rate and 50 % for energy expenditure. Garmin is the most accurate for tracking step count, as it offers 68 % accuracy.
----
[1]: https://entropicthoughts.com/the-surprising-richness-of-corr...
On top of this inaccuracy in actual measurements, many fitness tracker brands invent their own metrics.
"How did you sleep?" I asked a friend who stayed over, - "I don't know" they replied as they raise their smartwatch to their face. "Pretty well, apparently! I got a sleep score of 84."
I've also heard of an acquaintance that they cancelled plans because their (Garmin™) Body Battery was too low.
Instinctively I flinch whenever people place such importance in what is essentially an arbitrary techno-horoscope. But I also find it hard to make an argument against it as long as it inspires healthy behaviour.
> When it comes to energy expenditure, Apple is the only fitness tracker providing a strong accuracy of 71.02%. Jawbone, Fitbit, and Polar yield only moderate accuracy, ranging between 50.23% and 65.57%.
I don't buy it, no brand can be accurate on energy expenditure because it depends on a lot more things than what the watch can monitor.
Even in lab they have to put big mask on people while exercising to get something somewhat accurate.
On watch, calorie counter is a function of time and heart rate. Yet most watch brand can't even get heart rate right beside apple and Huawei.
Tried all the brand, went back to using a 10€ Casio and put a H10 monitor when doing sport.
There is a YouTube channel named "The Quantified Scientist" which offers technical and more in-depth reviews of varied fitness trackers: https://www.youtube.com/@TheQuantifiedScientist. I would very much recommend it if you intend to buy a wearable and care about accuracy
As long as any error is consistent, I don't really mind. Mostly I use it as a way to track progress over time anyway.
I use an Apple Watch, and while I don’t necessarily care about its accuracy in terms of steps walked or other specific details, I do value the insights it gives me into my daily activity. Even if it’s not entirely accurate, I get a reasonable estimation of how much I’ve moved, stood, and worked out. My watch motives me to me more active.
Disclaimer at the bottom of the article:
"The evaluations presented are based on general observations and should not be interpreted as definitive assessments of specific brands or their features."
The most recent paper used for this meta-study is still pretty old, and this technology changes. The study tells you almost nothing that could inform a buying decision regarding today's Apple Watch vs. Garmin Epix Pro (as examples). The sensors on the market today didn't exist and were not evaluated in any of the papers.
If anyone like me was wondering how accuracy of a continuous measurement can be a percentage, apparently they're just multiplying correlation by 100.
> Correlation coefficient estimates and their 95% confidence intervals were also expressed in percentages for clarity and readability.
Can someone help me understand the percentages? Does 80% accurate imply that the measurement could be 20% inflated?
I'm genuinely surprised the accuracy is that high.
67% is good enough for me. I am not using it as a medical device, and I am not an elite athlete. I just enjoy doing exercise, and every now and again I like to review the stats from my watch.
It's like the fat % on my bathroom scale. I know it's not perfect (in fact body % measuring is incredibly hard with massive error margins even in the most expensive/intrusive tests) but I'm not looking for exacts, I'm looking for trends and I've seen them where I expected to. So yes, the % may be bogus but I can use all that data to see trends which is all that matters (to me).
precision != accuracy, correlation != accuracy
let f be a continuous real valued function, let c be a fixed bias constant, let pcc be the pearson's correlation coefficient function.
pcc(f) = pcc(f + c)
f + c is precise, but not accurate
>The analysis reveals that commonly worn fitness trackers are the most accurate for measuring heart rate, providing a strong accuracy of 76.35%. However, they are only moderately accurate for tracking step count and energy expenditure, 68.75% and 56.63%, respectively.
>The highest level of accuracy is provided by the Apple Watch, which exhibits 86.31% accuracy for heart rate and 71.02% for energy expenditure. Garmin is the most accurate for tracking step count, as it offers 82.58% accuracy.
Not surprised, it’s a shame the folks who obsess over the numbers from these devices will be the ones least likely to understand what this actually means though.
Seems like HR evaluated here was just from the wrist device (like watches) and not HR strap. Is that correct?
(I only track HR with strap and power on my bike power-meter).
Why didn't they include Whoop or Oura?
I've had an iPhone for many years and just recently discovered the Health app.
My phone has a lot of my "steps", an estimate of my calories, data about my walking stability, and tons of other stuff. You don't even need a watch to get +/- 65% accuracy on anything except heart rate.
I remember one time when I got my daily step counter alert when I was eating chips on a sofa. I chuckled.
For step counting they really seem to only track certain hand motions, I though they would somehow notice how feet hits the ground and vibrations travel through the body to the wrist.
AI assisted meta analysis. Just piling on more exercise science trash.
That's better than all SEO tools and probably most analytics imo (only measured SEO tools myself) ... the value in tracking is mainly in spotting trends.
I wish they included Oura in the research.
Do they list the models of the brands they are talking about anywhere? For example, they mention Garmin but there is a huge variety of Garmin models with varying accuracy. I'm surprised I can't find it anywhere. They mention gathering data ("The majority of commercially available devices measure these basic metrics, while other metrics typically vary between different brands and models.") on different models but then group them together under a single brand? Why? Frankly, this analysis seems a bit half-assed.
Are the phones more accurate?
[flagged]
I find fitness trackers/smartwatches expensive, useless, and frankly, creepy.
Why do I need to pay $400 for an Apple Watch that sometimes doesn't have enough battery to make it through the day, and aggressively insists on tracking every movement of mine on the grounds of 'fitness'?
Seriously, that 10 hour stand goal and 'streaks' and what other achievements are just to make sure this GPS and heart rate measuring device is on your wrist every day of every week.
It's utterly incapable of measuring anything fitness related, perhaps with the exception of some cardio stuff, but it's useless for any kind of weight training.
It's baffling to me that Apple somehow got the population at large to pay for its overpriced ankle monitor and got them to upgrade to the newest model every year.