Inductive biases and equivariance in language models

Models designed to perform natural language tasks continue to push numerical performance
metrics higher on evaluation tasks that are designed to assess language understanding, such
as GLUE and SuperGLUE (Wang et al., 2018, 2019). However, ever-increasing numbers on
a leaderboard can lead to a misplaced belief that natural language models are approaching
human-like capabilities. Discussions are increasingly happening about what it means for a
model to \understand" language in a human-like way, and how we might move towards that
goal (Bender and Koller, 2020; McClelland et al., 2019).
A significant amount of research has been done with the aim of highlighting and cataloguing
ways in which state-of-the-art models fail to mimic human-like behaviour. For example, Ettinger
(2020) propose a series of tests based on psycholinguistic tests that could easily be answered
well by humans. They evaluate the popular BERT (Devlin et al., 2019) on these tests and in
particular highlight its difficulties in dealing with the impact of negation in a sentence. Models
have also been shown to have difficulty generalising in a compositional fashion in the way that
humans can. Compositionality is crucial for understanding natural language -- without the
ability to understand previously unseen combinations of seen words we would be severely limited
in our ability to understand and use language. Lake and Baroni (2018) designed the SCAN
task to assess models for this ability, and found that standard sequence-to-sequence models
performed poorly.
In the quest to improve models and bring their abilities more closely in line with those of
humans, one possible avenue of interest is to consider the inductive biases of the architectures
that we use and encourage them to align more closely with inductive biases that we observe in
human cognition (Goyal and Bengio, 2020). Inductive biases describe the preference of a model
for certain kinds of solutions, which can arise from assumptions that have been made about
the solution or from properties of the architecture. For example, convolutional neural networks
(CNNs) have an inductive bias towards translation invariance. Understanding which inductive
biases are useful for models learning natural language, and understanding how we might imbue
models with this desirable biases, could allow us to create models which can succeed where
others have failed in terms of human-like performance. Further, if a model has inductive biases
which are well-suited to the task that is being trained to perform, it can train more efficiently,
with less data.
Based on this, I have conducted work with a focus on inductive biases. Some of the work that I
have conducted considers how best to assess unintended inductive biases of language models
with respect to typological features of languages. Other work considers a possible method for
imbuing models with inductive biases to encourage them to find compositional solutions that
more closely resemble those arrived at by humans.