Beyond cherry-picking: making sense of mixed results in sleep research

Since we began working in the sleep space, we kept running into the same dismissal: “slow-wave enhancement doesn’t work".

‍

We took these comments very seriously. We had yet to commit ourselves down this path, and were investigating different types of sleep stimulation, and potentially other ways to improve sleep, but we kept coming back to this one known method, which, we believed, had the greatest intersection of quality peer-reviewed research and impact.

‍

My co-founder and I had been engineers at CSIRO, and I recall my boss telling me that researchers always think that when their work is done, the research is complete and the end result is what becomes the product or service. He said, it never works that way. After the research is done, it’s the engineers who make it actually work.

‍

Rather than just cherry-pick research which fits our beliefs, we post all the research we are aware of on our website.

‍

A redditor took the time to dive deep into the research, and asked “the results seem inconclusive”. Rather than cherry-picking only the most positive results, we review the entire body of research. It’s the opposite of cherry-picking, and it’s the only way to learn from both successes and failures, so we can engineer a system that works reliably across people and environments.

1. When a Null result isn’t a negative
   At first glance, I was disappointed by a study reporting no increase in growth hormone during slow-wave enhancement[1]. Then I saw that the study was conducted only in healthy men aged 18–24. These participants were already in the peak years of GH production. How much higher could their levels reasonably go? Probably not what I’d call a negative result.
   An equally puzzling study showed the enhancing slow-waves did not improve metabolic function in healthy older men[2]. If someone is metabolically healthy, how would we even judge improved metabolic function?
   
   
2. Rudimentary protocols
   When I first reviewed some of these studies, they clearly challenged the premise that slow-wave enhancement was possible. On the surface some of them seemed very similar to other studies which showed a positive result, but on closer examination, and time reading through multiple studies, a pattern emerged.
   
   The most obvious issue was volume control. Stimulation volume was fixed either to a specific decibel level[3][4], or to a personalized level when the user was awake[5]. One of the most important pieces of properly implementing slow-wave enhancement is to adapt the volume in real-time.  If the volume is too low, the brain will ignore or not even register the sound. Too loud, and the participant is aroused from sleep, defeating the purpose and actually harming their sleep. When a pre-determined volume is selected, either or both of these outcomes are possible within a single night. In contrast, adaptive protocols adjust volume dynamically, ensuring the stimulation is neither too faint nor disruptive.
   
   Once I noticed this trend in a few studies, I also noticed even more rudimentary protocols, such as measuring a slow-wave and providing stimulation at a fixed interval[4][6]. This is not responding and interacting with the brain in real-time. It’s a rough approximation that hopes the stimulation will land at the right time, not a true closed-loop protocol.
   
   However, within this segment of rudimentary protocols, there are more interesting studies which are a part of the puzzle. They show responses in some participants, but not in others. and some even divided by age, noting that younger participants responded better, or in other studies with slightly different protocols, older participants had improved response[7]. It should be noted, older participants have lower slow-wave activity, so there is a greater opportunity to increase their stimulation response. This was a huge hint to us as to how to develop a more robust system to get results across a wider audience.
   
   This class of research shows how vital real-time adaptation is, and points the way toward improving response.
   
   
3. Dosage
   This one we noticed in our early testing days, and was clearly stated in a recent Alzheimer’s study[8] as the fault which prevented participants from achieving a positive result. Slow-wave enhancement is regularly performed in an on-off process within the same night. Not every opportunity to stimulate is taken advantage of. However, missing a significant number of stimulation opportunities reduces the effectiveness of stimulation. It’s like getting a minor dose of a drug, not enough to show a difference from no stimulation at all. The dose makes the poison, and missing blocks of stimulation hampers any benefits, and in this Alzheimer’s-focused study, some participants got no stimulation at all, even though the EEG data captured by the researchers clearly showed stimulation opportunity.
   
   

I do find it interesting that null results in research are often not published, and I’m thankful that studies showing null results in slow-wave enhancement have been published. It is through the maze of understanding what worked, what didn’t, and under what circumstances that we were able to design our patent-pending Ultrasleep™ stimulation protocol.

‍

That’s why we don’t cherry-pick results. We review the positive, the negative, and the null, because only then can you see the real patterns. We’re not the researchers, we’re the engineers who make research insights work in the real world.

1. https://doi.org/10.1038/s41467-017-02170-3
2. https://doi.org/10.1016/j.psyneuen.2018.08.028
3. https://doi.org/10.1093/sleep/zsac155
4. https://doi.org/10.1016/j.nlm.2021.107482
5. https://doi.org/10.3389/frsle.2023.1294957
6. https://doi.org/10.1523/ENEURO.0306-19.2019
7. https://doi.org/10.1093/sleep/zsz315
8. https://doi.org/10.1016/j.jagp.2024.07.002
   

‍