Delphi's COVID-19 Trends and Impact Surveys (CTIS)
In collaboration with Facebook, along with a consortium of universities and public health officials, the Delphi group at Carnegie Mellon University conducted the COVID-19 Trends and Impact Survey (CTIS) to monitor the spread and impact of the COVID-19 pandemic in the United States. The survey was advertised through Facebook. It ran continuously from April 6, 2020, to June 25, 2022, and about 40,000 people in the United States participated every day. Survey results are publicly available on our CTIS Results Dashboard and COVIDcast Map; aggregate data can be downloaded through our COVIDcast Exporter, from our COVIDcast API, and as contingency tables with demographic breakdowns.
Such detailed data has never before been available during a public health emergency, and it has helped public health officials understand how to save lives and how to safely reopen public life. The data also helps researchers understand the social, economic, and health effects of the COVID-19 pandemic.
An international version of the survey was conducted by the University of Maryland in collaboration with Facebook. Its data is available separately.
Detailed technical documentation about the survey is available on our site for data users.
What are the surveys for?
The survey asked respondents whether they were experiencing any symptoms, then asked a series of questions designed to help researchers understand the spread of COVID-19 and its effect on people in the United States. These included questions about COVID-19 testing, prior medical conditions, social distancing measures, mental health, demographics, and the economic effects of the pandemic.
Delphi used information from the survey as part of its public COVIDcast Map, to inform its forecasts of the pandemic’s spread, and to assist public health agency partners. By providing daily data from all parts of the United States, the survey allows comparisons between regions and allows careful tracking of changes over time. Other researchers use the survey to study factors related to the pandemic’s spread, its effects on mental health, how different demographic groups were affected by the pandemic, and numerous other important questions.
Because the survey reached thousands of respondents every day, its questions focused on what the respondents were experiencing when they took the survey. This allows us to track how conditions changed across the country every day.
Who ran these surveys?
The surveys were a collaboration between Delphi at Carnegie Mellon University, numerous universities, and Facebook. Researchers at many institutions are analyzing the survey data and assisted in the survey’s development.
The survey protocol has been reviewed by the Carnegie Mellon University Institutional Review Board. Delphi only publicly releases aggregate data; de-identified individual data is made available to research partners. Our partners are bound by data use agreements to maintain the confidentiality of individual survey responses. Facebook referred its users to the survey, but it never received any individual survey data.
How were the surveys distributed?
Participants were recruited for the surveys through an advertisement placed in their Facebook news feed. Facebook automatically selected a random sample of its users to see the advertisement each day; users who clicked on the ad were taken to a survey administered by Carnegie Mellon University, and Facebook did not receive their survey responses. The survey was available in English, Spanish, Brazilian Portuguese, Vietnamese, French, and simplified Chinese.
The survey participants were sampled from Facebook users, rather than being a random sample from the entire United States population. But unlike a traditional telephone or mail survey, distribution through Facebook allowed us to reach tens of thousands of respondents every day, permitting researchers to make comparisons between many geographic areas and to detect changes as soon as they happened.
To account for the differences between Facebook users and the United States population, Facebook included a unique identifier when it linked users to the survey. Carnegie Mellon collected these identifiers and provided Facebook with a list of identifiers that completed the survey; Facebook then calculated statistical weights indicating how representative each person is of the United States population, based on demographic data known to Facebook. Crucially, Carnegie Mellon cannot use these identifiers to identify specific Facebook users, and Facebook never received individual survey responses and cannot link them to specific users.
Where can I see the results?
Our CTIS Results Dashboard shows aggregate survey results, ranging from rates of COVID-like symptoms over mask usage to mental health. In addition, our COVIDcast dashboard shows basic aggregate survey results, revealing the rates of mask usage and COVID-like symptoms across the United States. The data used by both tools is also freely available for download through the COVIDcast API and our COVIDcast Exporter. Facebook also publishes a map based on the United States and international aggregate data.
The Symptom Data Challenge challenged participants to enable earlier detection and improved situational awareness of the COVID-19 outbreak by using data from these symptom surveys. Check out the the challenge website for more details about the Challenge and the finalists.
Publications
Our primary description of the survey and its results over the first year of operation was published as part of the special feature “Beyond Cases and Deaths: The Benefits of Auxiliary Data Streams in Tracking the COVID-19 Pandemic” in PNAS:
- J. Salomon, A. Reinhart, A. Bilinski, E. J. Chua, W. La Motte-Kerr, M. M.
Rönn, M. B. Reitsma, K. A. Morris, S. LaRocca, T. H. Farag, F. Kreuter, R.
Rosenfeld, and R. J. Tibshirani (2021). The US COVID-19 Trends and Impact
Survey: Continuous real-time measurement of COVID-19 symptoms, risks,
protective behaviors, testing, and
vaccination. Proceedings of the
National Academy of Sciences 118 (51) e2111454118.
Other research publications using the survey data include:
- J. M. Cox-Ganser, P. K. Henneberger, D. N. Weissman, G. Guthrie, and C. P. Groth (2022). COVID-19 test positivity by occupation using the Delphi US COVID-19 Trends and Impact Survey, September–November 2020. American Journal of Industrial Medicine.
- M. Jahja, A. Chin, and R.J. Tibshirani (2022). Real-Time Estimation of COVID-19 Infections: Deconvolution and Sensor Fusion. Statistical Science 37 (2), 207-228.
- Henneberger, PK, Cox-Ganser, JM, Guthrie, GM, Groth, CP (2022). Estimates of COVID-19 vaccine uptake in major occupational groups and detailed occupational categories in the United States, April–May 2021. American Journal of Industrial Medicine 65 (7), 525-536.
- K. E. Wiens, C. P. Smith, E. Badillo-Goicoechea, K. H. Grantz, M. K. Grabowski, A. S. Azman, E. A. Stuart, and J. Lessler (2022). In-person schooling and associated COVID-19 risk in the United States over spring semester 2021. Science Advances 8, eabm9128.
- F. Petersen, A. Errore, and P. Karaca-Mandic (2022). Lifting statewide mask mandates and COVID-19 cases: A synthetic control study. Medical Care 60 (7), 538-544.
- F. Pierri, B. L. Perry, M. R. DeVerna, et al. (2022). Online misinformation is linked to early COVID-19 vaccination hesitancy and refusal. Scientific Reports 12, 5966.
- A. Pilehvari, J. Ton, M. R. Mohan, A. Marathe, and A. Vullikanti (2022). Drivers and Predictors of COVID-19 Vaccine Hesitancy in Virginia. In: Yang, Z., von Briesen, E. (eds), Proceedings of the 2021 Conference of The Computational Social Science Society of the Americas. CSSSA 2021.
- L. S. Flor, J. Friedman, C. N. Spencer, et al. (2022). Quantifying the effects of the COVID-19 pandemic on gender equality on health, social, and economic indicators: a comprehensive review of data from March, 2020, to September, 2021. The Lancet.
- D. P. Do and R. Frank (2022). Prior COVID-19 infection: an underappreciated factor in vaccine hesitancy in the USA. Journal of Public Health, fdab404.
- W. C. King, M. Rubinstein, A. Reinhart, and R. J. Mejia (2021). Time trends, factors associated with, and reasons for COVID-19 vaccine hesitancy: A massive online survey of US adults from January-May 2021. PLoS ONE 16 (12), e0260731.
- C. Lupton-Smith, E. Badillo-Goicochea, T.-H. Chang, H. Maniates, K. E. Riehm, I. Schmid, and E. A. Stuart (2021). Factors associated with county-level mental health during the COVID-19 pandemic. Journal of Community Psychology 50 (5), 2431-2442.
- D. P. Do and R. Frank (2021). U.S. frontline workers and COVID-19 inequities. Preventive Medicine 153, 106833.
- W. C. King, M. Rubinstein, A. Reinhart, and R. J. Mejia (2021). COVID-19 vaccine hesitancy January-May 2021 among 18–64 year old US adults by employment and occupation. Preventive Medicine Reports 24, 101569.
- C. H. Sudre, A. Keshet, M. S. Graham, A. D. Joshi, S. Shilo, H. Rossman, B. Murray, E. Molteni, K. Klaser, L. D. Canas, M. Antonelli, L. H. Nguyen, D. A. Drew, M. Modat, J. Capdevila Pujol, S. Ganesh, J. Wolf, T. Meir, A. T. Chan, C. J. Steves, T. D. Spector, J. S. Brownstein, E. Segal, S. Ourselin, and C. M. Astley (2021). Anosmia, ageusia, and other COVID-19-like symptoms in association with a positive SARS-CoV-2 test, across six national digital surveillance platforms: an observational study. The Lancet Digital Health 3 (9), e577-e586.
- R. Sukumaran, P. Patwa, S. T V, S. Shankar, R. Kanaparti, J. Bae, Y. Mathur, A. Singh, A. Chopra, M. Kang, P. Ramaswamy, and R. Raskar (2021). COVID-19 outbreak prediction and analysis using self reported symptoms. Journal of Behavioral Data Science 1 (1), 154–169.
- D. Adjodah, K. Dinakar, M. Chinazzi, S. P. Fraiberger, A. Pentland, S. Bates, K. Staller, A. Vespignani, and D. L. Bhatt (2021). Association between COVID-19 outcomes and mask mandates, adherence, and attitudes. PLoS ONE 16 (6), e0252315.
- C. M. Zipfel, V. Colizza, and S. Bansal (2021). The missing season: The impacts of the COVID-19 pandemic on influenza. Vaccine 39 (28), 3645-3648.
- J. Lessler, M. K. Grabowski, K. H. Grantz, E. Badillo-Goicoechea, C. J. E. Metcalf, C. Lupton-Smith, A. S. Azman, and E. A. Stuart (2021). Household COVID-19 risk and in-person schooling. Science 372 (6545), 1092-1097.
- Doerr, A.J. (2021). Locked (Down) and Loaded (Language): Effect of Policy and Speech on COVID-19 Outcomes. Journal of Leadership & Organizational Studies 28 (3), 340-348.
- Fischer CB, Adrien N, Silguero JJ, Hopper JJ, Chowdhury AI, Werler MM (2021). Mask adherence and rate of COVID-19 across the United States. PLoS ONE 16 (4), e0249891.
- Bilinski, A., Emanuel, E., Salomon, J.A. and Venkataramani, A. (2021). Better Late Than Never: Trends in COVID-19 Infection Rates, Risk Perceptions, and Behavioral Responses in the USA. Journal of General Internal Medicine 36, 1825-1828.
- Molteni, E., Astley, C.M., Ma, W. et al. (2021). Symptoms and syndromes associated with SARS-CoV-2 infection and severity in pregnant women from two community cohorts. Scientific Reports 11, 6928.
- Rebeiro, P., Aronoff, D, and Smith, M.K. (2021). The Impact of State Mask-Wearing Requirements on the Growth of COVID-19 Cases, Hospitalizations, and Deaths in the United States. Clinical Infectious Diseases 73 (9), 1703–1706.
- Rader, White, Burns, Chen, Brilliant, Cohen, Shaman, Brilliant, Kraemer, Moritz, Hawkins, Scarpino, Astley, and Brownstein (2021). Mask-wearing and control of SARS-CoV-2 transmission in the USA: a cross-sectional study. Lancet Digital Health 3 (3), e148-e157.
- Flaxman AD, Henning DJ and Duber HC (2020). The relative incidence of COVID-19 in healthcare workers versus non-healthcare workers: evidence from a web-based survey of Facebook users in the United States. Gates Open Research, 4:174.
- Kreuter, F., Barkay, N., Bilinski, A., Bradford, A., Chiu, S., Eliat, R., Fan, J., Galili, T., Haimovich, D., Kim, B., LaRocca, S., Li, Y., Morris, K., Presser, S., Sarig, T., Salomon, J. A., Stewart, K., Stuart, E. A., & Tibshirani, R. J. (2020). Partnering with a global platform to inform research and public policy making. Survey Research Methods, 14 (2), 159-163.
If you have used the survey data, or the aggregate data available in the COVIDcast API, to publish research results, please contact us so we can include your work here.
Blog posts and reports
- Surgo Ventures, Resolve to Save Lives (2021). COVID-19 Vaccine Precision Response Toolkit: An End-to-End Vaccination Improvement Framework to Improve COVID-19 Vaccine Uptake.
- Alejandra Arrieta, Emmanuela Gakidou, Heidi Larson, Erin Mullany, and Christopher Troeger, April 1, 2021. Through Understanding and Empathy, We Can Convince Women to Get the COVID-19 Vaccine. Think Global Health.
- Joseph Friedman, Silvia Montoya and Emmanuela Gakidou, March 8, 2021. Gender Equality in the Global Return to School. Think Global Health.
- Leanna Morinishi, Kris Barkume, Esther Kim, and Alex Reinhart, February 2, 2021. Home for the Holidays? The Impact of US Holidays on Social Behaviors and Preventative Measures. Delphi blog.
- Alex Reinhart, Esther Kim, Andy Garcia, and Sarah LaRocca, January 28, 2021. Using the COVID-19 Symptom Survey to Track Vaccination Uptake and Sentiment in the United States. Delphi blog.
- Alex Reinhart, October 12, 2020. New and Improved COVID Symptom Survey Tracks Testing and Mask-Wearing. Delphi blog.
- Ryan Tibshirani, September 21, 2020. Can Symptom Surveys Improve COVID-19 Forecasts? Delphi blog.
- Alex Reinhart and Ryan Tibshirani, August 26, 2020. COVID-19 Symptom Surveys through Facebook. Delphi blog.
Can I use the surveys in my research?
Yes! Aggregate data is available for download through the COVIDcast API. R and Python clients are available for the API. Aggregate data does not include any individual survey responses, only averages over counties and other geographic areas, and does not include all survey questions. See the API documentation for details on how to access the data, what survey questions are available, and how our aggregate values are calculated. Contingency tables giving demographic breakdowns of key survey signals are also publicly available for download.
Access to de-identified individual survey responses is available to qualified research groups who sign Data Use Agreements protecting the confidentiality of survey responses. If you are interested in using the data for your research, you can find instructions on our documentation site. The available data and survey waves are documented here.
Getting access to survey data for research: complete this form
Stacy Kish
Associate Director, Research Communications
Dietrich College
Carnegie Mellon University
All other questions:
Alex Reinhart
Assistant Teaching Professor
Department of Statistics & Data Science
Carnegie Mellon University