Natural language processing (NLP) has been revolutionised in recent years to the point where it is an inseparable part of our daily lives. The transition to transformer-based models allows us to train models on vast amounts of text efficiently, proving that scale plays a crucial role in improving performance. Unfortunately, many people worldwide are marginalised from getting access to high-quality NLP models, as the language they speak and the domains they are interested in count for only a tiny fraction of current state-of-the-art models’ training sets.

This talk will address the challenges, approaches, and opportunities for democratising NLP across different languages and domains by developing methods to improve NLP in low-resource scenarios. I will start by discussing how we can ease distribution mismatches to improve performance using representation learning. However, as NLP models become increasingly present in our lives, improving other crucial aspects beyond their performance, such as their fairness, factuality, and our ability to understand their underlying mechanisms, is essential. Therefore, I will also discuss using spectral methods to remove information from neural networks to reduce undesired attributes, such as bias, to increase fairness where sensitive data is scarce. Finally, I will explore future directions for making these models accessible toa broader audience by improving the aspects mentioned above in low-resource scenarios.