The mathematical trick to speed up transformers — A few weeks ago researchers from Google, the University of Cambridge, DeepMind and the Alan Turing Institute released the paper Rethinking Attention with Performers, which seeks to find a solution to the softmax bottleneck problem in transformers [1]. …

Why does minimizing the norms induce regularization? — If you’ve taken an introductory Machine Learning class, you’ve certainly come across the issue of overfitting and been introduced to the concept of regularization and norm. …

Evaluating NLP models using the weighted branching factor — Perplexity is a useful metric to evaluate models in Natural Language Processing (NLP). This article will cover the two ways in which it is normally defined and the intuitions behind them. Outline A quick recap of language models Evaluating language models Perplexity as the normalised inverse probability of the test set 3.1…

How to represent and manipulate data in graph form — This is a summary of some of the key points from the paper “A Review of Relational Machine Learning for Knowledge Graphs (28 Sep 2015)” [1], which gives a nice introduction to Knowledge Graphs and some of the methods used to build and expand them. The key takeaway Information can be structured in…