Sleep 1.4 Performance Evaluation of Consumer Devices

• The fourth category I wanted to talk about was the performance evaluation or quote unquote validation. As I understand, validation is actually not the term that you want to use more recently, but performance evaluation. And I think it would be remiss of me if I didn't address the elephant in the room, which is the low credibility that consumer devices have within the research world.
• And I think there's been good work that somehow, somewhat addresses these concerns. For instance, Chinoy's publication in 2021 found that consumer devices outperformed research grade, actigraphies. I think, the Philips ActiWatch 2 was used, in sleep wake classification when measured against gold standard PSGs.
• Interestingly, this did not apply to the two Garmin devices, which didn't perform as well, but I want to be clear that when using Labfront, you're not relying on the Garmin sleep algorithm, but are relying on the raw data acquired from the Accelerometer or PPG, which are indistinguishable in quality, from those used in other consumer devices.
• And so, despite this, you know, I think there are going to be persistent problems with consumer devices. There's that, always that issue of lack of transparency, the black box algorithms. We just have very little idea, for instance, how Apple or Fitbit got to their sleep reports, and then these reports can change, on a dime, the firmware and software device updates, and these updates can make any results from these studies irrelevant at the drop of a hat, just because their firmware could be completely different, and we would not know exactly what goes, went into it.
• The definitions and device outcomes are not consistent with scientific terminologies, and they're very, you know, they can vary from one device to the other. So we're hoping that the access to the raw data, the configurability of actigraphy settings, et cetera, would address these concerns. But it's also clear that most studies have been done in mostly healthy and young individuals.
• And it would be really helpful, to add to the credibility of consumer devices to really move towards, assessing, sleep in various conditions. This includes multiple days, naps, shift work, co-sleeping and pets. Apparently sleeping with pets can affect your sleep. And in fact, the dog affects the human more than the human affects the dog. Skin tone, infants, children, adolescents, elderly with possibly cognitive impairment, various disease conditions, and also socio- demographic and cultural regional differences, such as co-sleeping. Or even, you know, lack of air conditioning on a hot summer night can really affect the way people sort of sleep.
• And so we recognize that there is need for gold standard references, and for sleep and sleep staging, it is PSG. And we will work with researchers to embed wearable data into existing databases. But we will also continue to explore PSG-level devices into Labfront such as an EEG device. For circadian rhythms, the gold standard are core body temperature and DLMO, dim light melatonin onset, into the platform and we could also explore additional markers for circadian rhythms. 

We're continuing to explore ways that we could bring this into our system so that it could really make your research easier. And we would also consider adding analytical tools such as Bland Altman, tools, Bland Altman plots, the Epoch, Epoch, error, error matrices to help assess the ability or performance levels of these devices against a predetermined gold standard.