NLP News Cypher | 02.02.20

Author(s): Ricky Costa

Originally published on Towards AI.

Weekly Newsletter Natural Language Processing (NLP) News and Research

NLP News Cypher U+007C 02.02.20

The Die Is Cast

Today is 02.02.20 — the first global palindrome day in 909 years. U+1F440

How was your week?

Well, if you live anywhere near the NLP universe, you’ve probably stumbled on the NLP database. If you haven’t, you should!

Next, I want to give a shout-out to two database contributors from the past week: Kiril Gashteovski and Chandra Sekhar. Thank You! Thus far, we have amassed 239 NLP datasets.

If you know of a dataset that you see missing or have an edit request, please contact us on the database’s web page.

This Week:

BERTs Lingua Franca

Deep Learning Boot Camp

Meena is Perplexing

The Conscious Mind

A Token of Appreciation

S&P Global NLP White Papers

Deployment Headaches

Dataset of the Week: QA-SRL Bank

BERTs Lingua Franca

On Twitter, Sebastian Ruder shared just how many international BERT models we already have! Then Hugging Face shared some more. In total, there’s a lot of country flags on display! This is good to see for the international community!

Hugging Face:

Me:

Deep Learning Boot Camp

Beyond the footsteps of the next killer robot and Lex Fridman’s dark suits, and way beyond the deepest reaches of MIT, there lies a 1-week deep learning boot-camp. And it’s on YouTube:

Meena is Perplexing

Google created a chatbot, with a training objective to minimize perplexity. Apparently, its quality amazingly good. When reading Meena’s conversations, it seems like it’s doing a great job at something that is very difficult for most chit-chat dialogue systems: memory. To solve for this, they used 1 encoder and 13 decoder blocks. The encoder stores convo’s context and decoders help formulate higher conversational quality. This is how the bot does against the grain:

I asked Google Brain’s Thang Luong if it will be open-sourced. Apparently, they are being cautious about its release similarly to how OpenAI handled it own GPT-2 release:

Blog:

Towards a Conversational Agent that Can Chat About…Anything

Modern conversational agents (chatbots) tend to be highly specialized – they perform well as long as users don't stray…

ai.googleblog.com

The Conscious Mind

Circa seven years ago, in lower Manhattan, I randomly ran in to David Chalmers outside of a movie theater (this was when he was in his leather jacket thrash metal hair phase). As we exited the establishment, I commented on my joy for his book “The Conscious Mind”. I followed this up with a Neuroscience joke. He smirked.

Anyway, here’s Chalmers on the Fridman podcast:

A Token of Appreciation

It seems that every time I read a FloydHub article, a definitive pre-requisite prior to reading is hot cocoa and a fireplace. In a recent article, they illustrate the various kinds of tokenizers and how they differ in functionality. Here’s the tokenizers discussed (and make a smore):

Subword Tokenization
Byte Pair Encoding (BPE)
Unigram Subword Tokenization
WordPiece
SentencePiece

Tokenizers: How machines read

The world of Deep Learning (DL) Natural Language Processing (NLP) is evolving at a rapid pace. We tried to capture some…

blog.floydhub.com

S&P Global NLP White Papers

S&P Global market research firm released several white papers on the use of NLP in Finance. They also share use-cases and code! Which is rare for the private industry. Anyway, always good to keep up on the business side of things.

Part I:

Part II:

Part III:

Deployment Headaches

If you want to deploy your model, then reading this article would be of help to you. Caleb Kaiser from Cortex shows the common pitfalls when one attempts to deploy a large transformer model and simultaneously requiring it work at scale.

Too big to deploy: How GPT-2 is breaking production

A look at the bottleneck around deploying massive models to production

towardsdatascience.com

Dataset of the Week: QA-SRL Bank

What is it?

It’s a question answering dataset used for semantic-role labeling.

Sample:

QA-SRL U+007C Browse Data

Edit description

browse.qasrl.org

Where is it?

uwnlp/qasrl-bank

This repository is the reference point for QA-SRL Bank 2.0, the dataset described in the paper Large-Scale QA-SRL…

github.com

Every Sunday we do a weekly round-up of NLP news and code drops from researchers around the world.

If you enjoyed this article, help us out and share with friends or social media!

For complete coverage, follow our twitter: @Quantum_Stat

Join thousands of data leaders on the AI newsletter. Join over 80,000 subscribers and keep up to date with the latest developments in AI. From research to projects and ideas. If you are building an AI startup, an AI-related product, or a service, we invite you to consider becoming a sponsor.

Published via Towards AI