MentalBERTa

This model, MentalBERTa, was developed by the DeepLearningBrasil team and secured the first position in the DepSign-LT-EDI@RANLP-2023 shared task. The objective of the task was to classify social media texts into three distinct levels of depression: "not depressed," "moderately depressed," and "severely depressed". The accompanying code is available on GitHub.

Model Description

MentalBERTa is a RoBERTa large model from rafalposwiata/deproberta-large-depression, pre-trained on a curated Reddit dataset from mental health-related communities. This pre-training allows for an enhanced understanding of nuanced mental health discourse

The best performing version of the model was trained with Loss Sample Weights and a 50% head + 50% tail truncation method.

Training Data

The model was pre-trained on a custom dataset collected from mental health-related Subreddits, which is available on Hugging Face at dlb/mentalreddit. The full pre-training dataset comprises 3.4 million comments from mental health-related subreddits and 3.2 million comments from other subreddites, occupying approximately 1.4 GB of text on disk.

Citation

@inproceedings{garcia-etal-2023-deeplearningbrasil,
    title = "{D}eep{L}earning{B}rasil@{LT}-{EDI}-2023: Exploring Deep Learning Techniques for Detecting Depression in Social Media Text",
    author = "Garcia, Eduardo  and
      Gomes, Juliana  and
      Barbosa Junior, Adalberto  and
      Borges, Cardeque  and
      da Silva, N{\'a}dia",
    booktitle = "Proceedings of the Third Workshop on Language Technology for Equality, Diversity and Inclusion",
    month = sep,
    year = "2023",
    address = "Varna, Bulgaria",
    publisher = "INCOMA Ltd., Shoumen, Bulgaria",
    url = "https://aclanthology.org/2023.ltedi-1.42",
    pages = "272--278",
}

Acknowledgments

This work has been supported by the AI Center of Excellence (Centro de Excelência em Inteligência Artificial – CEIA) of the Institute of Informatics at the Federal University of Goiás (INF-UFG).