Sleepless Nights: A Statistical Look at Modern Sleep Patterns

Author(s): Daksh Trehan

Originally published on Towards AI.

A Statistical Journey Into the Factors That Shape Our Sleep

They say the best cure for sleepless nights is to count sheeps. But for some of us, the sheeps are too busy scrolling social media to help!

This playful opening reflects a deeper truth about modern life: we’ve filled our days with far more than our ancestors ever did. Unlike them, we now juggle endless distractions, social media pressures, and other modern-life traps. These factors leave our minds overstimulated and restless, with little space for mental peace.

This clutter has tangible consequences. Overstimulated and overwhelmed, our minds struggle to find peace, resulting in sleepless nights spent doom-scrolling instead of recharging. Sleep — a basic, natural necessity — is now increasingly elusive in our ever-connected world.

In this article, we’ll explore the patterns and reasons behind sleepless nights using real-world data and advanced statistical techniques. By understanding the factors that disrupt our sleep — from stress to health and lifestyle choices.

Note: The purpose of this article is purely educational. We are analyzing available data to demonstrate statistical techniques engagingly. This article is not intended to provide medical advice, as I am neither a medical professional nor practicing in the medical field. While good sleep can enrich lives, this article simply seeks to explore the data for insights. Rest assured, there is no judgment if restful sleep isn’t always a priority for you.

Content Table:

1. What can be regarded as good sleep? And why is it important?
2. Understanding our Data.
3. Possible information insights.
4. Advanced Analytics: Validating Assumptions and Discovering Patterns

What can be regarded as good sleep? And why is it important?

Good sleep isn’t just about the number of hours you spend in bed — it’s about the quality of those hours. The National Sleep Foundation recommends 7–9 hours for adults, but what defines good sleep goes beyond duration. Key indicators of good sleep include:

1. Sleep Efficiency: This measures how much of your time in bed is spent sleeping. A high sleep efficiency means you fall asleep quickly and stay asleep.
2. Continuity: Uninterrupted sleep cycles allow your body to go through all the restorative phases of sleep, enabling physical and mental recovery.
3. Feeling Rested: Waking up feeling refreshed and alert is a sign of good sleep, regardless of duration.

The Consequences of Poor Sleep

The absence of good sleep can have a significant impact on both the body and mind, leading to short-term and long-term consequences:

• Impaired Cognitive Function: Poor sleep disrupts memory, concentration, and decision-making abilities. Over time, this can affect work performance and learning capacity.
• Emotional Instability: Sleep deprivation increases irritability, anxiety, and susceptibility to stress. It also impairs emotional regulation, which may lead to strained relationships.
• Weakened Immune System: Without adequate sleep, the body struggles to fight off infections and heal properly, making individuals more susceptible to illnesses.
• Hormonal Imbalances: Poor sleep affects hormones that regulate hunger, stress, and growth, contributing to weight gain and metabolic issues.
• Decreased Longevity: Studies have shown that chronic lack of sleep can reduce life expectancy due to its cumulative effects on health.

Sleep is a cornerstone of physical and mental well-being, and the repercussions of poor sleep should not be underestimated. In the next section, we’ll dive into the data we’ve gathered to understand modern sleep patterns and what might be disrupting our ability to achieve this fundamental need.

Understanding our Data

The data is sourced from NPHA(National Poll on Healthy Aging)[1]. It includes variables that capture individual health, lifestyle, and sleep-related habits. Here’s a snapshot of what our data contains:

1. Demographic Information: Age, gender, and employment status of individuals.
2. Health Indicators: Physical and mental health ratings, stress levels, and whether any medical conditions affect sleep.
3. Sleep-Related Factors: Duration of sleep, frequency of trouble sleeping, and the use of sleep medications.
4. Lifestyle Variables: Presence of daily habits like screen time, caffeine consumption, and exercise.

The data consists of approximately 700 observations, making it a robust sample to analyze. By combining statistical techniques with these variables, we aim to uncover patterns and relationships that reveal what factors contribute to sleepless nights.

Possible Information Insights

There are several possible statistical ways to uncover basic insights from our data:

Descriptive Statistics

These are a set of tools and techniques used to summarize, describe, and organize data in a meaningful way. They help you understand the basic features of your dataset by providing a summary of its key characteristics. Descriptive statistics are often the first step in analyzing data because they give you an overview of what the data looks like.

1. Measures of Central Tendency: These describe the center or typical value in a dataset.

• Mean: The average of all values.
• Median: The middle value when the data is ordered.
• Mode: The most frequently occurring value.

2. Measures of Dispersion(Spread): These describe how spread out or dispersed the data values are.

• Range: The difference between the maximum and minimum values.
• Variance: The average squared deviation from the mean.
• Standard Deviation: The square root of variance, indicating how far values typically are from the mean.
• Interquartile Range (IQR): The range between the 25th and 75th percentiles, showing the spread of the middle 50% of the data.

3. Frequency Distributions: Show how often each value or category occurs in the dataset.

• Frequency counts for categorical data.
• Histograms or bar charts to visualize distributions.

4. Shape of the Distribution: Describes how data is distributed.

• Skewness: Indicates whether data is symmetric or skewed to the left or right.
• Kurtosis: Measures whether the data is flat or peaked relative to a normal distribution.

Why Are Descriptive Statistics Important?

• Simplification: They summarize large datasets into manageable numbers or visuals.
• Understanding Trends: They help identify patterns, trends, and outliers in the data.
• Data Cleaning: Descriptive statistics help detect missing data, errors, or anomalies.
• Foundation for Advanced Analysis: They lay the groundwork for inferential statistics and predictive modeling by providing a clear understanding of the dataset.

Descriptive Statistics on Our Data

1. Doctors Visited Category:

• Most individuals fall into the ‘Low’ or ‘Medium’ categories, indicating infrequent doctor visits.
• Those in the ‘High’ category may represent individuals with chronic conditions or persistent sleep-related issues.

2. Age Groups (if categorized):

• Younger individuals might report fewer sleep-related issues compared to middle-aged or senior groups, who are more likely to seek medical help or face sleep troubles.

3. Stress and Trouble Sleeping:

• High-stress levels are frequently associated with trouble sleeping categories, supporting earlier observations about stress being a major sleep disruptor.

4. Usage of Sleep Aids:

• Prescription sleep medication usage is more prevalent among individuals reporting higher trouble sleeping categories, indicating dependency trends.

5. Demographics and Sleep:

• Employment status (e.g., unemployed or retired individuals) and gender distributions might reveal disparities in stress levels and sleep-related issues.

Correlation

Correlation is a statistical measure that describes the strength and direction of the relationship between two numerical variables.

Positive Correlation: As one variable increases, the other variable also increases. e.g. Stress levels and trouble sleeping.

Negative Correlation: As one variable increases, the other variable decreases. e.g. Physical health and trouble sleeping.

No Correlation: No relationship between the variables. e.g. Race and prescription sleep medication usage might have no relationship (depending on your data).

1. Descriptive Measures Help Identify Trends:

• Using descriptive statistics like the mean or standard deviation, you can understand the overall patterns in your variables (e.g., average stress level, variability in trouble sleeping).
• Correlation then allows you to see how these patterns interact between variables (e.g., does higher stress correlate with more trouble sleeping?).

2. Highlight Relationships in Group Data:

• For instance, descriptive statistics might show that individuals with poor health ratings have higher stress levels. Correlation would quantify that relationship.

3. Validate Insights from Descriptive Statistics:

• If descriptive stats suggest a trend (e.g., older individuals report more trouble sleeping), correlation can confirm if age is significantly related to trouble sleeping.

Based on our dataset, the correlation matrix looks like this:

Key Insights:

1. Stress and Sleep Trouble:

• A high positive correlation between “Stress Keeps Patient from Sleeping” and “Trouble Sleeping” suggests that stress is a major contributor to sleep problems.
• Insight: Stress management interventions could directly improve sleep quality.

2. Physical Health and Trouble Sleeping:

• A negative correlation between “Physical Health” and “Trouble Sleeping” indicates that better physical health is associated with fewer sleep-related issues.
• Insight: Encouraging physical well-being could lead to better sleep outcomes.

3. Dependence on Sleep Aids:

• A moderate positive correlation between “Trouble Sleeping” and “Prescription Sleep Medication” shows that individuals with frequent trouble sleeping are more likely to rely on medication.
• Insight: Identifying non-medical interventions might reduce dependency on prescription sleep aids.

4. Pain and Sleep:

• A moderate positive correlation between “Pain Keeps Patient from Sleeping” and “Trouble Sleeping” highlights the role of physical discomfort in disrupting sleep.
• Insight: Pain management strategies could alleviate sleep issues in affected individuals.

5. Mental Health and Stress:

• A moderate negative correlation between “Mental Health” and “Stress Keeps Patient from Sleeping” suggests that poor mental health is linked to higher stress levels, which in turn affect sleep.
• Insight: Addressing mental health concerns could reduce stress and improve sleep.

6. Bathroom Needs and Sleep Disruption:

• A smaller positive correlation between “Bathroom Needs Keeps Patient from Sleeping” and “Trouble Sleeping” indicates that frequent bathroom visits moderately affect sleep quality.
• Insight: This could highlight specific conditions (e.g., bladder or prostate issues) that require attention.

7. Age and Health:

• If a variable like “Age” correlates negatively with “Physical Health,” it suggests that older individuals may experience poorer physical health, contributing to sleep challenges.
• Insight: Targeted interventions for older populations might improve overall health and sleep.

8. Low Correlations:

• Variables like “Race” and “Gender” show weak or negligible correlations with sleep-related factors.
• Insight: These demographic factors may not significantly influence sleep patterns in this dataset.

Advanced Analytics: Validating Assumptions and Discovering Patterns

Are stress levels truly the strongest predictor of trouble sleeping? Do age and health ratings influence sleep quality as much as we suspect? To answer these questions, we turn to hypothesis testing and regression analysis.

Hypothesis Testing

Hypothesis testing allows us to validate key assumptions, such as whether high-stress levels significantly correlate with frequent trouble sleeping.

e.g., we can test the hypothesis that poor physical health increases reliance on prescription sleep medication or that age significantly impacts sleep quality. These tests help us determine which relationships are statistically significant, providing confidence in our findings.

It provides clear evidence of relationships by distinguishing between random patterns and genuine associations. This makes it particularly valuable in ensuring that our conclusions are not influenced by noise in the data. By identifying statistically significant patterns, we can confidently guide decisions based on evidence rather than assumptions.

• Null Hypothesis: This is the default assumption that there is no effect or no difference. It represents the “status quo.”
• Alternative Hypothesis: This is the claim that we are testing, which suggests that there is an effect or a difference.

The goal of hypothesis testing is to determine whether the observed data provides enough evidence to reject the null hypothesis in favor of the alternative hypothesis.

Based on the insights we uncovered in the previous paragraph, let’s try to validate a few of the claims:

1. Sleep is impacted by Stress

Assumptions:

• The data is categorical (e.g., “Yes” or “No” for stress and trouble sleeping).
• Observations are independent (e.g., each person’s response is unrelated to another’s).
• The counts in each group of the table should not be too small (ideally at least 5 in each group) for the test to work properly.

Hypothesis Testing

• The Chi-Square test is specifically designed to evaluate relationships between two categorical variables.
• Our goal is to determine whether the presence of stress is associated with trouble sleeping.

Results:

• The test statistic (χ2) was 22.18, and the p-value was 0.000002.
• Since the p-value < 0.05, we rejected the null hypothesis, concluding that stress significantly impacts trouble sleeping.

2. Do we have a relationship between bad physical health & poor sleep?

Assumptions:

• The physical health score is continuous and normally distributed in the population.
• The amount of variation in the two groups (those with and without trouble sleeping) should be about the same, or we need to adjust the test to handle differences.
• Observations are independent (no repeated measures).

Hypothesis Testing?

• The t-test is appropriate for comparing the means of a continuous variable (physical health score) between two groups.
• We want to see if individuals reporting trouble sleeping had significantly lower physical health scores than those who didn’t.

Results:

• The test statistic (t) was -4.74, and the p-value was 0.000003.
• Since the p-value < 0.05, we rejected the null hypothesis, concluding that physical health significantly correlates with trouble sleeping. Specifically, poor physical health is associated with more sleep issues.

3. Does Employment Status Impact Trouble Sleeping?

Assumptions:

• The variables are categorical: Employment status (e.g., employed, retired) and trouble sleeping (binary) are both categorical.
• Observations are independent.
• Expected frequencies in the contingency table are at least 5:
• Ensures the test is reliable.

Hypothesis Testing?

• Similar to stress, employment status and trouble sleeping are categorical variables, making the Chi-Square test suitable for checking their relationship.
• We want to see if the employment status of individuals can impact a good night's sleep.

Result:

• The test statistic (χ2) is 7.93, and the p-value was 0.0470.
• Since the p-value < 0.05, we rejected the null hypothesis, concluding that employment status can significantly correlate with trouble sleeping.

Regression Analysis

Regression analysis goes a step further by quantifying the impact of variables on sleep quality.

For instance, using regression models, we can measure how much stress contributes to trouble sleeping or how changes in physical health influence the likelihood of using sleep aids. These methods provide actionable insights, allowing us to prioritize interventions for the factors most strongly affecting sleep.

1. Define the Model

• Dependent Variable (Target): Trouble Sleeping (binary: 1 = Often/Sometimes, 0 = No)
• Independent Variables (Predictors):

Stress (binary: Yes = 1, No = 0), Employment status (categorical, one-hot encoded), Physical Health (ordinal: Poor = 0, Fair = 1, Good = 2, Very Good = 3), Age group (ordinal: 18–24 = 1, 25–44 = 2, etc.), Gender (binary: Male = 1, Female = 0), Pain and bathroom needs at night (binary: Yes = 1, No = 0)

2. Choose the Regression Type

Since trouble sleeping is a binary variable (0 or 1), we’ll use logistic regression[2], which predicts the probability of an outcome falling into one of the two categories.

3. Check Assumptions of Logistic Regression

• Independence of observations: Each row in the dataset should represent a unique individual.
• No multicollinearity: Independent variables should not be highly correlated with each other.
• Linearity of independent variables: For logistic regression, continuous predictors should have a linear relationship with the log odds of the dependent variable.

4. Fit the Model

We’ll fit a logistic regression model to the data and evaluate:

• Coefficients: Show the direction and strength of each predictor’s relationship with trouble sleeping.
• P-values: Indicate whether each predictor is statistically significant (p<0.05p < 0.05p<0.05).
• Model Performance: Evaluate metrics like accuracy, precision, recall, and the Area Under the ROC Curve (AUC).

• The blue curve represents the model’s performance, showing the trade-off between the true positive rate (sensitivity) and the false positive rate.
• The red dashed line represents a random guess (no predictive power).

The closer the blue curve is to the top-left corner, the better the model performs. The Area Under the Curve (AUC) value of 70.41% indicates a moderately good ability to distinguish between individuals with and without trouble sleeping.

Plotting Coefficient Values with Predictor Variables gives us the following insights:

5. Influential Predictors

The regression coefficients (displayed in the analysis) highlight which variables have the strongest influence on trouble sleeping:

1. Stress: A strong positive predictor, confirming that individuals experiencing stress are significantly more likely to report trouble sleeping.
2. Employment Status: Employment categories (e.g., employed, retired, unemployed) showed varying impacts, reflecting differences in stress levels and routines that affect sleep.
3. Physical Health: Poor physical health is associated with a higher likelihood of trouble sleeping, consistent with earlier analyses.
4. Pain and Bathroom Needs at Night: These also emerged as positive predictors, suggesting that physical discomfort or disruptions significantly impact sleep.
5. Gender: Gender differences were accounted for, but their impact was less pronounced than stress and physical health.
6. Age Group: Age had a relatively weaker effect, aligning with earlier findings that age alone is not a significant determinant of trouble sleeping.

Looking Ahead

The findings from this analysis can inform targeted interventions to improve sleep quality:

• Stress reduction programs and workplace wellness initiatives can mitigate sleep problems, particularly among employed individuals.
• Customized strategies for retirees and unemployed individuals could focus on maintaining consistent routines and addressing physical health challenges.
• Addressing physical discomfort and nighttime disruptions can provide immediate relief for those experiencing trouble sleeping.

As always, thank you so much for reading, and please share this article if you found it useful! 🙂

References:

[1] National Poll on Healthy Aging (NPHA) [Dataset]. (2017). UCI Machine Learning Repository. https://doi.org/10.3886/ICPSR37305.v1.

[2] Logistic Regression Explained

[3] Code: Sleepless Nights: A Statistical Look at Modern Sleep Patterns

Find me on the Web: www.dakshtrehan.com

Connect with me at LinkedIn: www.linkedin.com/in/dakshtrehan

Read my Tech blogs: www.dakshtrehan.medium.com

Cheers!

Join thousands of data leaders on the AI newsletter. Join over 80,000 subscribers and keep up to date with the latest developments in AI. From research to projects and ideas. If you are building an AI startup, an AI-related product, or a service, we invite you to consider becoming a sponsor.

Published via Towards AI