Today we will practice Language Processing with Deep Learning. We will use transfer learning here, to fine tune our neural network that has been trained on Wiki Text data. This Neural network understands language constraints, and we will fine tune it with the tweeting style of Elon Musk.
We will then try to predict future Elon tweets. This should be a fun exercise.
Ethics First
It’s most important that we understand ethics and responsibilities while using predictive models that work on personal or sensitive data. While this text prediction exercise is not purely a Generative model, but we make our Neural Network understand the tweeting style of an individual, and it should be used with responsibility.
Ok lets begin
Here we will use FastAi library for creating our deep learning models. We will use Kaggle dataset : Elon Musk’s Tweets to generate a language model to predict what will be the next tweet from Elon Musk.
If you haven’t watched FastAi tutorials already, please visit this link for the awesome and free tutorials.
Lets look at the data
Let’s have a look on the overview of data.
Load and analyse data
Fast ai expects the data to be loaded as a Data Bunch and then a Fast ai Learner can use this data for the models. Fastai takes care of a lot of feature engineering for us and prepares the data in a format that the neural net can understand. We can then pass this Data Bunch to our FastAi learner for training on the data.
As we can see FastAI has tokenised the text and has also introduced new tokens in the data. These tokens track numbers, capitals, symbols etc, without removing the information. The network will be able to acknowledge this information while training on the dataset.
Fit the deep learning model
Now that we have our Data Bunch ready, we can go ahead creating our Deep Learning model. Fastai provides a bunch of learners for our convenience inspired from state-of-art papers.
This is typically composed of following steps :
- Create Learner : Create an appropriate learner for data. A learner creates a neural network for us.
- Find the learning rate : We need to find a suitable learning rate for our training
- Fit the model
Network structure
This is a great post on LSTM, if you are interested. And here is another great post on Dropout for regularising Deep neural networks.
Results
Other tweets that did not make it to the leader board –
Hope this post was helpful, stay tuned for more blogs on Deep Learning.