Natural languages are believed to be (mildly) context-sensitive. Despite
underpinning remarkably capable large language models, transformers are unable
to model many context-free language tasks. In an attempt to address this
limitation in the modeling power of transformer-based language models, we
propose augmenting them with a differentiable, stack-based attention mechanism.
Our stack-based attention mechanism can be incorporated into any
transformer-based language model and adds a level of interpretability to the
model. We show that the addition of our stack-based attention mechanism enables
the transformer to model some, but not all, deterministic context-free
languages.