World Health Organization. Mental Health and COVID-19: Early Evidence of the Pandemic’s Impact: Scientific Brief (2022).
Adair, C. E. et al. Continuity of care and health outcomes among persons with severe mental illness. Psychiatr. Serv. 56, 1061–1069 (2005).
Schulte, J., Schulz, C., Wilhelm, S. & Buhlmann, U. Treatment utilization and treatment barriers in individuals with body dysmorphic disorder. BMC Psychiatry 20, 69 (2020).
van Daele, T. et al. Online consultations in mental healthcare: modelling determinants of use and experience based on an international survey study at the onset of the pandemic. Internet Interv. 30, 100571 (2022).
Paganini, S., Teigelkötter, W., Buntrock, C. & Baumeister, H. Economic evaluations of internet- and mobile-based interventions for the treatment and prevention of depression: a systematic review. J. Affect. Disord. 225, 733–755 (2018).
Mendes-Santos, C., Nunes, F., Weiderpass, E., Santana, R. & Andersson, G. Understanding mental health professionals’ perspectives and practices regarding the implementation of digital mental health: qualitative study. JMIR Form. Res. 6, e32558 (2022).
Atik, E., Schückes, M. & Apolinário-Hagen, J. Patient and therapist expectations for a blended cognitive behavioral therapy program for depression: qualitative exploratory study. JMIR Ment. Health 9, e36806 (2022).
de Witte, N. A. J. et al. Online consultations in mental healthcare during the COVID-19 outbreak: an international survey study on professionals’ motivations and perceived barriers. Internet Interv. 25, 100405 (2021).
Smoktunowicz, E. et al. Consensus statement on the problem of terminology in psychological interventions using the internet or digital components. Internet Interv. 21, 100331 (2020).
O’Loughlin, K., Neary, M., Adkins, E. C. & Schueller, S. M. Reviewing the data security and privacy policies of mobile apps for depression. Internet Interv. 15, 110–115 (2019).
Hennemann, S., Farnsteiner, S. & Sander, L. Internet- and mobile-based aftercare and relapse prevention in mental disorders: a systematic review and recommendations for future research. Internet Interv. 14, 1–17 (2018).
Torous, J. et al. Creating a digital health smartphone app and digital phenotyping platform for mental health and diverse healthcare needs: an interdisciplinary and collaborative approach. J. Technol. Behav. Sci. 4, 73–85 (2019).
Lal, S. & Adair, C. E. E-mental health: a rapid review of the literature. Psychiatr. Serv. 65, 24–32 (2014).
Pill, J. The Delphi method: substance, context, a critique and an annotated bibliography. Socio Econ. Plan. Sci. 5, 57–71 (1971).
Bartlett Ellis, R. et al. Lessons learned: beta-testing the digital health checklist for researchers prompts a call to action by behavioral scientists. J. Med. Internet Res. 23, e25414 (2021).
Shen, F. X. et al. An ethics checklist for digital health research in psychiatry: viewpoint. J. Med. Internet Res. 24, e31146 (2022).
World Health Organization. Digital Implementation Investment Guide (DIIG): Integrating Digital Interventions into Health Programmes https://apps.who.int/iris/bitstream/handle/10665/334306/9789240010567-eng.pdf (2020)/
Unsworth, H. et al. The NICE Evidence Standards Framework for digital health and care technologies—developing and maintaining an innovative evidence framework with global impact. Digit. Health 7, 20552076211018617 (2021).
Sundareswaran, V. & Sarkar, A. Chatbots RESET: A Framework for Governing Responsible Use of Conversational AI in Healthcare https://www3.weforum.org/docs/WEF_Governance_of_Chatbots_in_Healthcare_2020.pdf (World Economic Forum, 2020).
Doraiswamy, P. M. et al. Empowering 8 billion minds: enabling better mental health for all via the ethical adoption of technologies. NAM Perspect. https://doi.org/10.31478/201910b (2019).
Hekler, E. B. et al. Agile science: creating useful products for behavior change in the real world. Transl. Behav. Med. 6, 317–328 (2016).
Fiedler, J., Seiferth, C., Eckert, T., Woll, A. & Wunsch, K. A just-in-time adaptive intervention to enhance physical activity in the SMARTFAMILY2.0 trial. Sport Exerc. Perform. Psychol. https://doi.org/10.1037/spy0000311 (2022).
Chan, A. H. Y. & Honey, M. L. L. User perceptions of mobile digital apps for mental health: acceptability and usability—an integrative review. J. Psychiatr. Ment. Health Nurs. 29, 147–168 (2022).
Vial, S. & Boudhraâ, S. in Revolutions in Product Design for Healthcare (eds Subburaj, K. et al.) 21–34 (Springer, 2022).
Narayan, S., Mok, H., Ho, K. & Kealy, D. I don’t think they’re as culturally sensitive: a mixed-method study exploring e-mental health use among culturally diverse populations. J. Ment. Health 32, 241–247 (2022).
Wright, M. T., Springett, J. & Kongats, K. in Participatory Health Research (eds Wright, M. T. & Kongats, K.) 3–15 (Springer, 2018).
Wright, M. T. Partizipative Gesundheitsforschung: Ursprünge und heutiger Stand. Bundesgesundheitsblatt Gesundheitsforsch. Gesundheitsschutz 64, 140–145 (2021).
Carman, K. L. et al. Patient and family engagement: a framework for understanding the elements and developing interventions and policies. Health Affairs 32, 223–231 (2013).
Mummah, S. A., Robinson, T. N., King, A. C., Gardner, C. D. & Sutton, S. IDEAS (Integrate, Design, Assess, and Share): a framework and toolkit of strategies for the development of more effective digital interventions to change health behavior. J. Med. Internet Res. 18, e317 (2016).
McCurdie, T. et al. mHealth consumer apps: the case for user-centered design. Biomed. Instrum. Technol. https://doi.org/10.2345/0899-8205-46.s2.49 (2012).
Knowles, S. E. et al. Qualitative meta-synthesis of user experience of computerised therapy for depression and anxiety. PLoS ONE 9, e84323 (2014).
Werner-Seidler, A. et al. A smartphone app for adolescents with sleep disturbance: development of the Sleep Ninja. JMIR Ment. Health 4, e28 (2017).
Braun, V. & Clarke, V. in APA Handbook of Research Methods in Psychology, Vol 2: Research Designs: Quantitative, Qualitative, Neuropsychological, and Biological (eds Cooper, H. et al.) 57–71 (American Psychological Association, 2012).
Dopp, A. R., Parisi, K. E., Munson, S. A. & Lyon, A. R. A glossary of user-centered design strategies for implementation experts. Transl. Behav. Med. 9, 1057–1064 (2019).
Orlowski, S. K. et al. Participatory research as one piece of the puzzle: a systematic review of consumer involvement in design of technology-based youth mental health and well-being interventions. JMIR Hum. Factors 2, e12 (2015).
Orji, R. & Moffatt, K. Persuasive technology for health and wellness: state-of-the-art and emerging trends. Health Inform. J. 24, 66–91 (2018).
Esfandiari, N. et al. A specific internet-based cognitive behavior therapy for adolescents with social anxiety disorder: three-armed randomized control trial. Preprint at Research Square https://doi.org/10.21203/rs.3.rs-2123795/v1 (2022).
Yardley, L., Bradbury, K. & Morrison, L. in Qualitative Research in Psychology: Expanding Perspectives in Methodology and Design 2nd edn (ed. Camic, P. M.) 263–282 (American Psychological Association, 2021).
Bailey, E. et al. Ethical issues and practical barriers in internet-based suicide prevention research: a review and investigator survey. BMC Med. Ethics 21, 37 (2020).
Sander, L. et al. Suicide risk management in research on internet-based interventions for depression: a synthesis of the current state and recommendations for future research. J. Affect. Disord. 263, 676–683 (2020).
Kaurin, A., Dombrovski, A. Y., Hallquist, M. N. & Wright, A. G. C. Integrating a functional view on suicide risk into idiographic statistical models. Behav. Res. Ther. 150, 104012 (2022).
Friedlander, A., Nazem, S., Fiske, A., Nadorff, M. R. & Smith, M. D. Self-concealment and suicidal behaviors. Suicide Life Threat Behav. 42, 332–340 (2012).
Franklin, J. C. et al. Risk factors for suicidal thoughts and behaviors: a meta-analysis of 50 years of research. Psychol. Bull. 143, 187–232 (2017).
Steeg, S. et al. Accuracy of risk scales for predicting repeat self-harm and suicide: a multicentre, population-level cohort study using routine clinical data. BMC Psychiatry 18, 113 (2018).
Sisti, D. A. & Joffe, S. Implications of zero suicide for suicide prevention research. JAMA 320, 1633–1634 (2018).
Büscher, R. et al. Digital cognitive-behavioural therapy to reduce suicidal ideation and behaviours: a systematic review and meta-analysis of individual participant data. Evid. Based Ment. Health 25, e8–e17 (2022).
Holmes, E. A. et al. The Lancet Psychiatry Commission on psychological treatments research in tomorrow’s science. Lancet Psychiatry 5, 237–286 (2018).
Torok, M. et al. Suicide prevention using self-guided digital interventions: a systematic review and meta-analysis of randomised controlled trials. Lancet Digit. Health 2, e25–e36 (2020).
Ferreira, T. E-health applications and data protection: a comparison of selected European Union members’ national legal systems. Bioethica 8, 74–84 (2022).
Wilkowska, W. & Ziefle, M. Privacy and data security in E-health: requirements from the user’s perspective. Health Inform. J. 18, 191–201 (2012).
Albrecht, U.-V. Chances and Risks of Mobile Health Apps (CHARISHMA) (Medizinische Hochschule Hannover, 2016).
Lablans, M., Borg, A. & Ückert, F. A RESTful interface to pseudonymization services in modern web applications. BMC Med. Inf. Decis. Making 15, 2 (2015).
Cummins, N., Schuller, B. W. & Baird, A. Speech analysis for health: current state-of-the-art and the increasing impact of deep learning. Methods 151, 41–54 (2018).
Ghosh, S., Löchner, J., Mitra, B. & De, P. in Quantifying Quality of Life (eds Wac, K. & Wulfovich, S.) 209–267 (Springer, 2022).
Garriga, R. et al. Machine learning model to predict mental health crises from electronic health records. Nat. Med. 28, 1240–1248 (2022).
Valstar, M. et al. AVEC 2013: the continuous audio/visual emotion and depression recognition challenge. In Proc. 3rd ACM International Workshop on Audio/Visual Emotion Challenge 3–10 (ACM, 2013).
Wang, R. et al. Tracking depression dynamics in college students using mobile phone and wearable sensing. Proc. ACM Interact. Mob. Wearable Ubiquitous Technol. 2, 1–26 (2018).
Dhall, A., Goecke, R., Gedeon, T. & Sebe, N. Emotion recognition in the wild. J. Multimodal User Interf. 10, 95–97 (2016).
Amin, M. M., Cambria, E. & Schuller, B. W. Will affective computing emerge from foundation models and general AI? A first evaluation on ChatGPT. IEEE Intelligent Systems 2, 15–23 (2023).
Schuller, B. W. et al. Computational charisma—a brick by brick blueprint for building charismatic artificial intelligence. Preprint at https://doi.org/10.48550/arXiv.2301.00142 (2023).
Véliz, C. Chatbots shouldn’t use emojis. Nature 615, 375 (2023).
Vazire, S. Who knows what about a person? The self-other knowledge asymmetry (SOKA) model. J. Person. Soc. Psychol. 98, 281–300 (2010).
Garatva, P. et al. in Digital Phenotyping and Mobile Sensing (eds Montag, C. & Baumeister, H.) 395–411 (Springer, 2023).
Torous, J., Kiang, M. V., Lorme, J. & Onnela, J.-P. New tools for new research in psychiatry: a scalable and customizable platform to empower data driven smartphone research. JMIR Ment. Health 3, e16 (2016).
Cornet, V. P. & Holden, R. J. Systematic review of smartphone-based passive sensing for health and wellbeing. J. Biomed. Inform. 77, 120–132 (2018).
Moshe, I. et al. Predicting symptoms of depression and anxiety using smartphone and wearable data. Front. Psychiatry 12, 625247 (2021).
Vasudevan, S., Saha, A., Tarver, M. E. & Patel, B. Digital biomarkers: convergence of digital health technologies and biomarkers. npj Digit. Med. 5, 36 (2022).
Asare, O. K. et al. Predicting depression from smartphone behavioral markers using machine learning methods, hyperparameter optimization, and feature importance analysis: exploratory study. JMIR mHealth uHealth 9, e26540 (2021).
Fried, E. I., Rieble, C. & Proppert, R. K. K. Building an early warning system for depression: rationale, objectives, and methods of the WARN-D study. Preprint at PsyArXiv https://doi.org/10.31234/osf.io/9qcvs (2022).
Lattie, E. G. et al. Digital mental health interventions for depression, anxiety, and enhancement of psychological well-being among college students: systematic review. J. Med. Internet Res. 21, e12869 (2019).
Torous, J. et al. Smartphones, sensors, and machine learning to advance real-time prediction and interventions for suicide prevention: a review of current progress and next steps. Curr. Psychiatry Rep. 20, 51 (2018).
Kargl, F., van der Heijden, R. W., Erb, B. & Bösch, C. in Digital Phenotyping and Mobile Sensing (eds Montag. C. & Baumeister, H.) 13–23 (Springer, 2023).
Larsen, M. E. et al. Using science to sell apps: evaluation of mental health app store quality claims. npj Digit. Med. 2, 18 (2019).
Klein, R. A. et al. Investigating variation in replicability. Soc. Psychol. 45, 142–152 (2014).
Simblett, S., Birch, J., Matcham, F., Yaguez, L. & Morris, R. A systematic review and meta-analysis of e-mental health interventions to treat symptoms of posttraumatic stress. JMIR Ment. Health 4, e14 (2017).
Magnusson, K., Andersson, G. & Carlbring, P. The consequences of ignoring therapist effects in trials with longitudinal data: a simulation study. J. Consult. Clin. Psychol. 86, 711–725 (2018).
Johns, R. G., Barkham, M., Kellett, S. & Saxon, D. A systematic review of therapist effects: a critical narrative update and refinement to review. Clin. Psychol. Rev. 67, 78–93 (2019).
Rozental, A., Andersson, G. & Carlbring, P. In the absence of effects: an individual patient data meta-analysis of non-response and its predictors in internet-based cognitive behavior therapy. Front. Psychol. 10, 589 (2019).
Mander, J. et al. The therapeutic alliance in different mental disorders: a comparison of patients with depression, somatoform, and eating disorders. Psychol. Psychother. 90, 649–667 (2017).
Mechler, J. et al. Sudden gains and large intersession improvements in internet-based psychodynamic treatment (IPDT) for depressed adolescents. Psychother. Res. 31, 455–467 (2021).
von Klipstein, L., Riese, H., van der Veen, D. C., Servaas, M. N. & Schoevers, R. A. Using person-specific networks in psychotherapy: challenges, limitations, and how we could use them anyway. BMC Med. 18, 345 (2020).
van den Bergh, R. et al. The content of personalised network-based case formulations. Preprint at PsyArXiv https://doi.org/10.31234/osf.io/yan4k (2022).
Olthof, M. et al. Destabilization in self-ratings of the psychotherapeutic process is associated with better treatment outcome in patients with mood disorders. Psychother. Res. 30, 520–531 (2020).
Schiepek, G. et al. Real-time monitoring of psychotherapeutic processes: concept and compliance. Front. Psychol. 7, 604 (2016).
Hasselman, F. & Bosman, A. M. T. Studying complex adaptive systems with internal states: a recurrence network approach to the analysis of multivariate time-series data representing self-reports of human experience. Front. Appl. Math. Stat. https://doi.org/10.3389/fams.2020.00009 (2020).
Wallot, S. Recurrence quantification analysis of processes and products of discourse: a tutorial in R. Discourse Process. 54, 382–405 (2017).
Myin-Germeys, I. & Kuppens, P. The Open Handbook of Sampling Methodology. A Step-by-Step Guide to Designing, Conducting, and Analyzing ESM studies (The Center for Research on Experience Sampling and Ambulatory Methods Leuven, 2021).
Trull, T. J. & Ebner-Priemer, U. Ambulatory assessment. Annu. Rev. Clin. Psychol. 9, 151–176 (2013).
Insel, T. R. Digital phenotyping: a global tool for psychiatry. World Psychiatry 17, 276–277 (2018).
Ebner-Priemer, U. W. et al. Digital phenotyping: towards replicable findings with comprehensive assessments and integrative models in bipolar disorders. Int. J. Bipolar Disord. 8, 35 (2020).
Myin-Germeys, I. et al. Experience sampling methodology in mental health research: new insights and technical developments. World Psychiatry 17, 123–132 (2018).
Fortea, L. et al. Development and validation of a smartphone-based app for the longitudinal assessment of anxiety in daily life. Assessment https://doi.org/10.1177/10731911211065166 (2021).
Wrzus, C. & Neubauer, A. B. Ecological momentary assessment: a meta-analysis on designs, samples, and compliance across research fields. Assessment https://doi.org/10.1177/10731911211067538 (2022).
Geldhof, G. J., Preacher, K. J. & Zyphur, M. J. Reliability estimation in a multilevel confirmatory factor analysis framework. Psychol. Methods 19, 72–91 (2014).
Kockler, T. D., Santangelo, P. S. & Ebner-Priemer, U. W. Investigating binge eating using ecological momentary assessment: the importance of an appropriate sampling frequency. Nutrients 10, 105 (2018).
Ottenstein, C. & Werner, L. Compliance in ambulatory assessment studies: investigating study and sample characteristics as predictors. Assessment https://doi.org/10.1177/10731911211032718 (2021).
Lecomte, T. et al. Mobile apps for mental health issues: meta-review of meta-analyses. JMIR mHealth uHealth 8, e17458 (2020).
Linardon, J., Cuijpers, P., Carlbring, P., Messer, M. & Fuller-Tyszkiewicz, M. The efficacy of app-supported smartphone interventions for mental health problems: a meta-analysis of randomized controlled trials. World Psychiatry 18, 325–336 (2019).
Fleming, T. et al. Beyond the trial: systematic review of real-world uptake and engagement with digital self-help interventions for depression, low mood, or anxiety. J. Med. Internet Res. 20, e199 (2018).
Torous, J. et al. The growing field of digital psychiatry: current evidence and the future of apps, social media, chatbots, and virtual reality. World Psychiatry 20, 318–335 (2021).
Chien, I. et al. A machine learning approach to understanding patterns of engagement with internet-delivered mental health interventions. JAMA Netw. Open 3, e2010791 (2020).
Torous, J., Lipschitz, J., Ng, M. & Firth, J. Dropout rates in clinical trials of smartphone apps for depressive symptoms: a systematic review and meta-analysis. J. Affect. Disord. 263, 413–419 (2020).
Richards, D. & Richardson, T. Computer-based psychological treatments for depression: a systematic review and meta-analysis. Clin. Psychol. Rev. 32, 329–342 (2012).
Alon, N., Stern, A. D. & Torous, J. Assessing the Food and Drug Administration’s risk-based framework for software precertification with top health apps in the United States: quality improvement study. JMIR mHealth uHealth 8, e20482 (2020).
Stern, A. D. et al. Advancing digital health applications: priorities for innovation in real-world evidence generation. Lancet Digit. Health 4, e200–e206 (2022).
Lagan, S. et al. Mental health app evaluation: updating the American Psychiatric Association’s framework through a stakeholder-engaged workshop. Psychiatr. Serv. 72, 1095–1098 (2021).
Stoyanov, S. R. et al. Mobile app rating scale: a new tool for assessing the quality of health mobile apps. JMIR mHealth uHealth 3, e27 (2015).
Ramos, G., Ponting, C., Labao, J. P. & Sobowale, K. Considerations of diversity, equity, and inclusion in mental health apps: a scoping review of evaluation frameworks. Behav. Res. Ther. 147, 103990 (2021).
Carlo, A. D., Hosseini Ghomi, R., Renn, B. N. & Areán, P. A. By the numbers: ratings and utilization of behavioral health mobile applications. npj Digit. Med. 2, 54 (2019).
Lagan, S. et al. Actionable health app evaluation: translating expert frameworks into objective metrics. npj Digit. Med. 3, 100 (2020).
Szinay, D. et al. Influences on the uptake of health and well-being apps and curated app portals: think-aloud and interview study. JMIR mHealth uHealth 9, e27173 (2021).
Roberts, A. E. et al. Evaluating the quality and safety of health-related apps and e-tools: adapting the Mobile App Rating Scale and developing a quality assurance protocol. Internet Interv. 24, 100379 (2021).
Huber, M. et al. How should we define health? BMJ 343, d4163 (2011).
Wasil, A. R., Gillespie, S., Shingleton, R., Wilks, C. R. & Weisz, J. R. Examining the reach of smartphone apps for depression and anxiety. Am. J. Psychiatry 177, 464–465 (2020).
Gentili, A. et al. The cost-effectiveness of digital health interventions: a systematic review of the literature. Front. Public Health 10, 787135 (2022).
Mitchell, L. M., Joshi, U., Patel, V., Lu, C. & Naslund, J. A. Economic evaluations of internet-based psychological interventions for anxiety disorders and depression: a systematic review. J. Affect. Disord. 284, 157–182 (2021).
Wac, K. in Digital Health (eds Rivas, H. & Wac, K.) 83–108 (Springer, 2018).