This Ph.D. thesis investigates the question “How to provide and ensure faithful explanations for complex general-purpose neural NLP models?” The main thesis is that we should develop new paradigms in interpretability.

The two new paradigms explored are faithfulness measurable models (FMMs) and self-explanations. Self-explanations are when LLMs explain themselves. FMMs are when models are designed such that measuring faithfulness is cheap and precise, which makes it possible to optimize explanations toward maximum faithfulness.

All investigations of faithfulness show that explanations' faithfulness is by default model and task-dependent. However, this is not the case when using FMMs. Therefore, FMMs present a new paradigm that answers the question of how to provide and ensure faithful explanations.

Large language models are increasingly being used by the public, in the form of chat models. These chat systems often provide detailed and highly convincing explanations for their answers, even when not explicitly prompted to do so. This makes users more confident in these models. However, are the explanations true? If not true, this confidence is unsupported which can be dangerous.

We measure the truthfulness (i.e. interpretability-faithfulness) of the explanations that LLMs provide, so called self-explanations. We do so by holding the models accountable to their own explanations, using self-consistency checks. We find that the truthfulness is highly dependent on the model and the specific task. Suggesting we should not have general confidence in these explanations.

Interpretability have two paradigms, post-hoc or intrinsic explanations. This paper propose a new paradigm, where models intrinsically provides the means to measure faithfulness of any explanation. We call these inherently faithfulness measurable models (FMMs).

Because measuring faithfulness is now trivial, it is possible to optimize explanations with respect to the faithfulness. As a result, the model becomes indirectly inherently explainable and we get explanations with state-of-the-art faithfulness scores.

We demonstrate this general idea using masked language model (MLM), by simply fine-tuning an MLM such that masking any tokens are in-distribution. We thoroughly validate our claims on 16 datasets and use out-of-distribution tests.

The current paradigms of interpretability are called post-hoc and intrinsic. Intrinsic says only models designed to be explained can be explained, often leading to highly constrained models. While post-hoc says we should focus on explaining general-purpose models, even though it may be challenging.

This position paper builds the idea that both paradigms have fundamental issues with how they approach interpretability, and it would be much more productive to look for new directions, which takes the best of both worlds. The position paper identifies 3 such paradigms. 1) To design models such that faithfulness can be easily measured, 2) to optimize models such that explanations become faithful, and 3) to develop models that produce both a prediction and an explanation.

Attention is a widely used component in neural networks and is often treated as an explanation in the interpretability field. However, since 2019 there has been much discussion on whether attention is actually a valid explanation. Much of this discussion exists because it is inherently impossible to say what a correct explanation is.

This paper proposes a new indirect method for measuring if explanations are valid. It is based on a previous method called ROAR which was used for computer vision. In this paper, we adapt ROAR to natural language and solve previously known issues with a new version we call Recursive ROAR. Finally, we develop a scalar benchmark that will make it easy to compare explanations between future papers.

The Inaugural talk for the research organization Cohere for AI, a great honor! My talk is a narrative about my path from independent research to PhD in interpretability. I cover many of the lessons I learned during my time as an independent research, as well as some of the reasons I got started with interpretability.

In the second act, I discuss contemporary challenges in interpretability to motivate new researchers.

A survey on post-hoc interpretability methods for Natural Language Processing (NLP). The survey covers 19 specific interpretability methods and cites more than 100 works. Each method is categorized by how it communicates, visualized in a comparative format, and its evaluation methodology is discussed.

Beyond interpretability methods, the survey covers topics on motivation for interpretability and measures of interpretability. At last, we provide general insights, as well as our opinions on future directions and challenges.

Invited talk at LiveAI, on the importance of being able to explain natural language models. Covering legal perspectives, the social impact of machine learning, and my work on textual interpretability. From my Distill publication to my python module.

Taking part in a panel discussion, on how to navigate ML academia when you are new. The other panelists were Chelsea Finn (Stanford University), Shakir Mohamed (DeepMind), Tong Zhang (Hong Kong SciTech), Ashley Edwards (ML Collective), and Edward Raff (Booz Allen Hamilton).

Interview by TWIML, on the process of developing my paper Neural Arithmetic Units. How to work with limited resources, the importance of collaboration, and the struggles of being an independent researcher by necessity.

Proposes two new arithmetic units (addition and multiplication), that improves the state-of-the-art by 3x to 20x, over existing units such as the ``Neural Arithmetic Logic Unit'' (NALU). The improvements were achieved by rigorous theoretical analysis. The new units allow for more interpretable models and potentially perfect extrapolation.

This received a spotlight award at ICLR, as it was among the 6% best-reviewed publications.

A python library for creating the interactive textual heatmap visualization, as I demonstrated in my Distill paper. This library works with Jupyter and Google Colab making it easy for researchers to apply in their interpretability research.

Proposes a new evaluation-criteria, with special confidence intervals, for extrapolation tasks. It uses these criteria in a reproduction study of the ``Neural Arithmetic Logic Unit'' (NALU), and shows that in some cases its performance is drastically worse than previously assumed.

Creates a learning-curve plot for Jupyter/Colab notebooks that is updated in real-time. This was first developed for a workshop at NodeConfEU, I later made it into its own pip package.

Developed the hand-gesture recognition machine learning model for the IoT smartwatch badge, given out at NodeConfEU 2019. The model ran on a system-on-chip using TensorFlow Lite for Microprocessors. This was done in collaboration with the TensorFlow team.

Talked about my TensorFlow.js implementation of the special functions, and especially how to survive a really difficult programming project, with lots of unknowns.

Implementation for TensorFlow.js of the special functions used in probability, calculus, differential equations, and more. Such as the beta, gamma, zeta, and Bessel functions.

I was interviewed by PracticalAI on my Distill publication that became highly acclaimed. I discuss the importance of interpretability and visualization. Such as how we develop our intuition though interaction and the importance of testing that intuition.

Proposes a visualization method for qualitatively comparing different RNN architectures' ability to memorize and understand what parts of an input-sentence make a prediction, which is great for interpretability.

Distill is a peer-reviewed journal, chaired by Chris Olah from OpenAI, and other famous researchers.

By compiling the cephes library to WebAssembly, this module allows JavaScript developers to use mathematical special functions.

Cephes.js have become a backbone for many of the Machine Learning projects at NearForm Research.

TensorFlow.js Implementation of Hidden Markov Model, that is now used filter background noise from V8 runtime in Node.js from general CPU usage signal, leaving just the main application CPU usage.

Implemented the collection runtime and analysis backend of Clinic.js Bubbleprof. The currently most advanced tool for profiling and debugging asynchronous delays in Node.js.

Implemented the collection runtime, analysis backend, and frontend of Clinic.js Doctor. Clinic.js Doctor collects runtime usage data from the application runtime and uses machine learning and advanced non-parametric statistics to classify data into a recommendation for what tool to use next.

I was later involved in hiring and managing the team that now maintains it.

A semi-supervised neural machine translation model for small bilingual datasets. The model used the ByteNet model (Kalchbrenner, et. al.) together with a beam-search marginalization approach for semi-supervised learning.

I was a critical part of getting the async_hooks module implemented in the Node.js core runtime. This module allows users to monitor all asynchronous operations happening in the application.

Implemented the sparsemax operator in the TensorFlow core, as part of a course project. This involved Python, C++, and CUDA.

Implemented interactive profiling software for monitoring all asynchronous operations in a node.js application. This used, at the time, an internal version of async_hooks, and the tool was instrumental in debugging the async_hooks implementation in Node.js.

After having introduced statistics into the Node.JS open source project for their benchmarking suite, I was invited to speak at NodeConf EU in Ireland.

The challenge was to communicate both how a Welch's t-test works to people that often dislike mathematics, and provide the psychological background of why statistics is necessary.

Complete refactor if the benchmark pipeline and tooling used in Node.js. This was done to add proper statistics to the micro benchmarks used in Node.js. A big challenge was to communicate statistical concepts to programmers, particularly for a large-scale open-source project which gets new contributors very frequently.

A comparison of paragraph2vec (a word2vec variant) and an LSTM encoder-decoder (Sutskever et al. 2014), for generating semantic vectors that are precise enough to cluster documents according to the story.

The thesis also proposes a quasi-linear-time clustering algorithm, useful for dated documents such as new articles.

I was the main implementer of the cluster module for Node.js. This allowed developers to run a server on multiple CPU cores. Because JavaScript is single-threaded that was big news. Today this is less relevant because load balancers and containers have become the default scaling strategy.