How to apply Deep Learning on tabular data with FastAi

It’s a common sentiment that Deep Learning is only good for images and language models. This post is about using Deep Learning on tabular data, for both Regression and Classification problems. We will use fastai library for creating our deep learning models. We will use Kaggle competitions as benchmarks to see how our solutions compares to other solutions using traditional ML models.

If you haven’t watched fastai tutorials already, please visit this link for the awesome and free tutorials.

Network architecture

Important concepts:

Categorical embeddings: Similar to latent features, embedding categories into N-dimensional features.
Continuous variables: Batch Normalisation for continuous variables
Hidden layers
Output Hidden layer

Fastai figures out the default value for N for us here but lot of these can be customised while creating the data bunch and learner. To understand deeper- the Embeding(8, 5) means that a categorical embedding was created with 8 input values and 5 output latent features.

This is the internal network structure of a 2 layer model I created for one of the challenges. This is how our model looks like with fastai defaults:

Let’s have a look at the challenges and some code now.

Part 1 – Classification Problem : Titanic challenge

Here we have used Titanic dataset from kaggle to explain how fastai works for classification problem.

Lets look at the data

Let’s have a look on the overview of data and know the data types of each features, to understand the importance of features.

Load and analyse data

Fast ai expects the data to be loaded as a Data Bunch and then a Fast ai Learner can use this data for the models. Fastai takes care of a lot of feature engineering for us and prepares the data in a format that the neural net can understand.

Few things to keep in mind here:

Data Bunch: A data format for fast ai input
Dependent variable: The variable to predict
Categorical columns: The text/label columns. Or the columns with low cardinality, eg. gender, type, year etc.
Continuous columns: Numeric value columns, usually with higher cardinality, eg. salary, price, temperature.
Transformations: Feature engineering and handling data eg. Missing values, Normalisation etc

We can then pass this Data Bunch to our Fastai learner for training on the data.

Fit the deep learning model

Now that we have our Data Bunch ready, we can go ahead creating our Deep Learning model. Fastai provides a bunch of learners for our convenience inspired from state-of-art papers.

This is typically composed of following steps :

Create Learner : Create an appropriate learner for data. A learner creates a neural network for us.
Find the learning rate : We need to find a suitable learning rate for our training
Fit the model

We will create a tabular learner for our challenge.

Get Prediction

Let’s get the prediction and create the submission file to submit in Kaggle.

I found that I was able to get upto 0.8XX with an ensembled XGB + RF based model, but that involved lot of feature engineering and cross validation + ensembling.

So there is a scope of improving the deep learning model here. However this is not bad at all, without any feature engineering and network tuning.