Gain insights into processing text data, creating word embeddings, and using LSTMs for semantic analysis and machine translation. Explore industry-relevant NLP techniques with Python and TensorFlow.

nlp_docker.tar.gz

mnist-fashion

In this course you'll learn techniques for processing text data, creating word embeddings, and using long short-term memory networks (LSTM) for tasks such as semantic analysis and machine translation. After completing this course, you will be able to solve the important day-to-day NLP problems faced in industry, which is incredibly useful given the prevalence of text data.

The code for this course is built around the TensorFlow framework, one of the premier frameworks for industry machine learning, and the Python pandas library for data analysis. Knowledge of Python and TensorFlow are prerequisites. 

This course was created by AdaptiLab, a company specializing in evaluating, sourcing, and upskilling enterprise machine learning talent. It is built in collaboration with industry machine learning experts from Google, Microsoft, Amazon, and Apple.

Natural Language Processing with Machine Learning

<h2 class="section-title">Chapter Goals:</h2>
<ul>
	<li>Calculate softmax probabilities to make predictions for each time step</li>
</ul>


<h2>A. Calculating probabilities</h2>

<div id="section1_l8" class="collapse show">
<p>When using a feed-forward neural network for multiclass predictions, we first need to convert the model's logits into probabilities with the softmax function. For an RNN language model, the idea is the exact same. After converting the RNN's outputs into logits, we apply the softmax function to the <i>final dimension</i> of the logits.</p>

The model we've built so far in this section predicts the likely next word given an input sentence (see the __Language Model__ chapter for a refresher)


import tensorflow as tf
# Logits with a vocab_size = 100
logits = tf.compat.v1.placeholder(tf.float32, shape=(None, 5, 100))
probabilities = tf.compat.v1.nn.softmax(logits, axis=-1)

<p>By applying the softmax function to the final dimension of the logits, we calculate probabilities for every vocabulary word at each time step in each sequence. In the example above, the first dimension of <code>probabilities</code> corresponds to each sequence in the batch, the second dimension corresponds to the time steps, and the third dimension corresponds to the vocabulary word probabilities.</p>

<h2>B. Word predictions</h2>

<p>Similar to regular multiclass predictions with an MLP (Multilayer Perceptron), we calculate the RNN's word predictions by taking the highest probability word at each time step. The nice thing about how we tokenized our text sequences is that each vocabulary word becomes a positive integer. This means that, for each time step, the index containing the highest probability will be the model's word prediction.</p>
<p>The example below shows how to calculate word predictions from probabilities. We use <code>tf.compat.v1.argmax</code> to retrieve the indexes of the final dimension that contain the highest probabilities.</p>

import tensorflow as tf
# Placeholder for the model probabilities
probabilities = tf.compat.v1.placeholder(tf.float32, shape=(None, 5, 100))

word_preds = tf.compat.v1.argmax(probabilities, axis=-1)

<h2>C. Current state of the art</h2>

LSTMs are more effective at natural language processing than conventional neural networks, but more effective models do exist.

Currently, the best performing model uses transformers, which are detailed in <a href='https://arxiv.org/abs/1706.03762'>this paper</a>.

To play around with a model that uses transformers, click <a href='https://talktotransformer.com/'>this link</a>.

<h2>Chapter Goals:</h2>
<ul>
	<li>Calculate softmax probabilities to make predictions for each time step</li>
</ul>


<h2>A. Calculating probabilities</h2>

<div id="section1_l8">
<p>When using a feed-forward neural network for multiclass predictions, we first need to convert the model's logits into probabilities with the softmax function. For an RNN language model, the idea is the exact same. After converting the RNN's outputs into logits, we apply the softmax function to the <i>final dimension</i> of the logits.</p>

The model we've built so far in this section predicts the likely next word given an input sentence (see the __Language Model__ chapter for a refresher)


Create word predictions based on the output of your LSTM model.

What you'll learn from this course

Word Embeddings

Language Model

Text Classification

Seq2Seq Model

Predictions

Chapter Goals:

A. Calculating probabilities

B. Word predictions

C. Current state of the art