Schedule

Nando de Freitas: Exploration and Imitation

Research on imitation has progressed rapidly and considerably in recent years. I will cover some of the recent advances in one-shot imitation, the imitation-meta-learning connection, the impact of generative models on imitation, hierarchical imitation, high-fidelity imitation, intentional imitation, and third-person imitation. I will provide demonstrations using simulated environments and robots. Finally, I will discuss some areas where research is needed including selective imitation (ie who and what to imitate), very-long term imitation, and the emergence of abstract cultural concepts.

Vera Demberg: Must NLP consider individual differences in language processing more?

Today, variation in language comprehension by humans (as evidenced by disagreement between annotators or observed in misunderstandings) is usually regarded as noise that can be ignored. I will argue in my talk that some of the variation that we observe may be systematic, and that it may be related to individual differences in cognitive processing of language. Being able to model the underlying individual differences and predict their effects on human language comprehension may be the key to resolving problems for machine learning that result from seemingly inconsistent labels, and may enable us to build more adaptive NLP systems.

Dirk Hovy: With more Layers comes more Responsibility

Neural Networks have revolutionized NLP: they have increased performance across the board, and enabled a number of applications that were not possible before. All this is a great opportunity, but also a new responsibility for NLP: never before was it so easy to write a powerful NLP system, just because you can. However, these systems are increasingly used in applications they were not intended for, by people who treat them as interchangeable black boxes. The results can be performance drops, but also systematic biases against various user groups.

As a consequence, we as NLP practitioners suddenly have a new role in addition to developer: considering the ethical implications of our systems, and educating the public about the possibilities and limitations of our work.

I will talk about the possibilities deep learning has opened up, and the caveats that come with it, and where this might lead NLP as a field. This includes case examples and some provocations for future directions.

Joakim Nivre: Is the End of Supervised Parsing in Sight? Twelve Years Later

At ACL in Prague in 2007, Rens Bod asked whether supervised parsing models would soon be a thing of the past, giving way to superior unsupervised models. For the next decade or so, the answer seemed to be negative, as supervised approaches to syntactic parsing continued to outperform their unsupervised counterparts by large margins. However, recent developments in our field has made the question relevant again, in at least two different ways. First, there is the question of whether the end of all parsing (supervised or other) is in sight, simply because systems that are trained end-to-end for real applications have no room (or need) for traditional linguistic representations of the kind that parsers produce, a question that I will not directly discuss in this talk. However, there is also the question of whether we need traditional supervised parsers even if we want to compute discrete syntactic representations of natural language sentences. To put this question into perspective, I will survey developments in dependency parsing since 2007, showing that most of the advanced parsing models proposed during the first half of this period have been made obsolete by advances in deep learning during the second half. Moreover, the advent of deep contextualized word embeddings appear to have eliminated the last remaining differences between different algorithmic approaches, suggesting that specialized parsing algorithms are largely superfluous in the state-of-the-art systems of today.

Bonnie Webber: Implicit Discourse Relations co-exist with Explicit Relations

It is well-known that relations between sentences or clauses can be signalled explicitly (e.g., with conjunctions, discourse adverbials, lexico-syntactic constructions, etc.) or left to hearer inference (based on what the hearer takes to be the arguments of the relation). But texts containing a sequence of adjacent explicit connectives (e.g., "but instead" / "so instead", "because otherwise" / "but otherwise", "because then" / "but then" / "so then", etc.) suggest that, even if only one discourse relation between clausal/sentential arguments is signalled explicitly, other collocated evidence might lead a hearer to infer other relations as well.

Experiments that Hannah Rohde and I and colleagues have carried out over the past three years, with support from the Nuance Foundation, have shown that this is indeed the case. It also turns out to be a way of making sense of "disagreements" between human annotators who have been asked to indicate what discourse connective "best expresses" the relation that holds between two sentences [Malmi et al, LREC 2018]. As this has implications for predicting discourse connectives and relations for language technology applications, I will briefly describe some of our experiments and their results.

Ion Androutsopoulos: Embedding Biomedical Ontologies by Jointly Encoding Network Structure and Textual Node Descriptors

Network Embedding (NE) methods, which map network nodes to low-dimensional feature vectors, have wide applications in network analysis and bioinformatics. Many existing NE methods rely only on network structure, overlooking other information associated with the nodes, e.g., text describing the nodes. Recent attempts to combine the two sources of information only consider local network structure. We extend NODE2VEC, a well-known NE method that considers broader network structure, to also consider textual node descriptors using recurrent neural encoders. Our method is evaluated on link prediction in two networks derived from UMLS. Experimental results demonstrate the effectiveness of the proposed approach compared to previous work. This is joint work with Sotiris Kotitsas, Dimitris Pappas, Ryan McDonald, and Marianna Apidianaki, which was also recently presented at BioNLP 2019.

Natalie Schluter: Neural syntactic parsing seems so simple. Is it?

Neural network approaches to syntactic parsing achieve great success with rather simple models. In this talk I will explore the power of neural networks for this task and whether these models really are so simple.

Gemma Boleda: Generic and situation-specific information in distributed representations

For both humans and computational models, it is essential to be able to abstract away information from specific instances, e.g. to build a generic concept for "bird", such that it can be applied to different individuals. It is equally essential to model specific situations, for instance to understand the sentence "that bird is about to poke you" and react accordingly. Traditional distributional semantics handles generic word information very well, but struggles to account for situation-specific information. Deep learning models represent a step forward, since they are endowed with mechanisms to process contextual aspects and integrate them with generic word representations; however, it is at present unclear to what extent they represent situation-specific knowledge, and what aspects they can account for. I will present recent work at the interface between generic and situation-specific information that suggests that there is still a long way to go.

Reut Tsarfaty: The Empty Elements Project

We know, at least since Grice's Maxim of quantity (Grice, 1975), that human speakers try to be as informative as they can. That is, they try to give as much information as is needed, but no more than that. In actuality this means that speakers drop a lot of content from their utterances, while hearers adjust their interpretation quietly and without a fuss to interpret and accommodate the missing information. For example, speakers may drop a single word, as in "I started the book, it is fascinating" (started <reading>), a phrase: "I went into the room, the walls were black." (the walls <of the room>), or an entire clause: "My wife called." (<I am married.>). Most of the NLP pipeline nowadays (tagging, parsing, NER, co-ref resolution) focus on interpreting what's in the text. However, the success of NLP resides precisely in identifying and resolving the elements that are outside the text and between the lines, yet are conceived as a natural part of the NL utterance interpretation by humans. In this talk I present the Empty Elements (EE) project, where we aim to automatically infer, expand, and resolve the elements that have been dropped yet are unambiguously inferred by humans. I start by surveying these phenomena and their importance for language technology, I then discuss the challenges in soliciting EE information from human speakers and in explicitly modeling them, and I finally sketch our approach, along with some preliminary results, towards their resolution.

Lucia Specia: Towards more holistic approaches to human-machine communication

In this talk I will provide an overview of recent work on multimodal machine learning where images or videos are used to build richer context models for natural language tasks and argue towards building more holistic approach to human-machine communication.

André Martins: Beyond Sparsemax: Adaptively Sparse Transformers

Attention mechanisms have become ubiquitous in NLP. Recent architectures, notably the Transformer, learn powerful context-aware word representations through layered, multi-headed attention. The multiple heads learn diverse types of word relationships. However, with standard softmax attention, all attention heads are dense, assigning a non-zero weight to all context words. In this talk, I will introduce the adaptively sparse Transformer, wherein attention heads have flexible, context-dependent sparsity patterns. This sparsity is accomplished by replacing softmax with alpha-entmax: a differentiable generalization of softmax that allows low-scoring words to receive precisely zero weight. Moreover, we derive a method to automatically learn the alpha parameter—which controls the shape and sparsity of alpha-entmax—allowing attention heads to choose between focused or spread-out behavior. Our adaptively sparse Transformer improves interpretability and head diversity when compared to softmax Transformers on machine translation datasets. Findings of the quantitative and qualitative analysis of our approach include that heads in different layers learn different sparsity preferences and tend to be more diverse in their attention distributions than softmax Transformers. Furthermore, at no cost in accuracy, sparsity in attention heads helps to uncover different head specializations.

Joint work with Gonçalo Correia, Ben Peters, and Vlad Niculae.

Angela Fan: Pruning Transformers at Inference Time

Overparametrized transformer networks have obtained state of the art results in various natural language processing tasks, such as machine translation, language modeling, and question answering. These models contain hundreds of millions of parameters, necessitating a large amount of computation and making them prone to overfitting. In this work, we present a pruning mechanism that allows for efficient pruning at inference time to the desired Transformer depth. We show that it is possible to select smaller sub-networks from a large network, without having to finetune them and with limited impact on performance.

Rico Sennrich: What Do Transformers Learn in NLP? Recent Insights from Model Analysis

The Transformer architecture and the BERT training scheme have enjoyed great empirical success in NLP, but their inner workings are still poorly understood. What happens in multi-head self-attention? Why does the training objective in model pre-training matter so much, and why is masked language modelling such a good choice? I will discuss recent research that sheds some light on these questions, analysing the function of attention heads in multi-head self-attention, and comparing the evolution of representations in the Transformer with different objective functions.