pytorchwidedeep, deep learning for tabular data III: the deeptabular component
a flexible package to combine tabular data with text and images using wide and deep models.
This is the third of a series of posts introducing pytorchwidedeep, a flexible package to combine tabular data with text and images (that could also be used for "standard" tabular data alone).
While writing this post I will assume that the reader is not familiar with the previous two posts. Of course, reading them would help, but in order to understand the content of this post and then being able to use pytorchwidedeep
on tabular data, is not a requirement.
To start with, as always, just install the package:
pip install pytorchwidedeep
This will install v0.4.8
, hopefully the last beta version*. Codewise I think this could be already v1
, but before that I want to try it in a few more datasets and select good default values. In addition, I also intend to implement other algorithms, in particular TabNet [1], for which a very nice implementation already exists.
Moving on, and as I mentioned earlier, pytorchwidedeep
's main goal is to facilitate the combination of images and text with tabular data via wide and deep models. To that aim, wide and deep models can be built with up to four model components: wide
, deeptabular
, deeptext
and deepimage
, that will take care of the different types of input datasets ("standard" tabular, i.e. numerical and categorical features, text and images). This post focuses only on the socalled deeptabular
component, and the 3 different models available in this library that can be used to build that component. Nonetheless, and for completion, I will briefly describe the remaining components first.
The wide
component of a wide and deep model is simply a liner model, and in pytorchwidedeep
such model can be created via the Wide
class. In the case of the deeptext
component, pytorchwidedeep
offers one model, available via the DeepText
class. DeepText
builds a simple stack of LSTMs, i.e. a standard DL text classifier or regressor, with flexibility regarding the use of pretrained word embeddings, of a Fully Connected Head (FCHead), etc. For the deepimage
component, pytorchwidedeep
includes two alternatives: a pretrained Resnet model or a "standard" stack of CNNs to be trained from scratch. The two are available via the DeepImage
class which, as in the case of DeepText
, offers some flexibility when building the architecture.
To clarify the use of the term "model" and Wide and Deep "model component" (in case there is some confusion), let's have a look to the following code:
wide_model = Wide(...)
text_model = DeepText(...)
image_model = DeepImage(...)
# we use the previous models as the wide and deep model components
wdmodel = WideDeep(wide=wide_model, deeptext=text_model, deepimage=image_model)
...
Simply, a wide and deep model has model components that are (of course) models themselves. Note that any of the four wide and deep model components can be a custom model by the user. In fact, while I recommend using the models available in pytorchwidedeep
for the wide
and deeptabular
model components, it is very likely that users will want to use their own models for the deeptext
and deepimage
components. That is perfectly possible as long as the custom models have an attribute called output_dim
with the size of the last layer of activations, so that WideDeep
can be constructed (see this example notebook in the repo). In addition, any of the four components can be used independently in isolation. For example, you might want to use just a wide
component, which is simply a linear model. To that aim, simply:
wide_model = Wide(...)
# this would not be a wide and deep model but just wide
wdmodel = WideDeep(wide=wide_model)
...
If you want to learn more about different model components and the models available in pytorchwidedeep
please, have a look to the Examples folder in the repo, the documentation or the companion posts. Let's now take a deep dive into the models available for the deeptabular
component
$^*$ check the repo or this post for a caveat in the installation if you are using Mac, python 3.8 or Pytorch 1.7+. Note that this is not directly related with the package, but the interplay between Mac and OpenMP, and the new defaults of the multiprocessing
library for Mac).
deeptabular
component
1. The As I was developing the package I realised that perhaps one of the most interesting offerings in pytorchwidedeep
was related to the models available for the deeptabular
component. Remember that each component can be used independently in isolation. Building a WideDeep
model comprised only by a deeptabular
component would be what is normally referred as DL for tabular data. Of course, such model is not a wide and deep model, is "just" deep.
Currently, pytorchwidedeep
offers three models that can be used as the deeptabular
component. In order of complexity, these are:

TabMlp
: this is very similar to the tabular model in the fantastic fastai library, and consists simply in embeddings representing the categorical features, concatenated with the continuous features, and passed then through a MLP. 
TabRenset
: This is similar to the previous model but the embeddings are passed through a series of ResNet blocks built with dense layers. 
TabTransformer
: Details on the TabTransformer can be found in: TabTransformer: Tabular Data Modeling Using Contextual Embeddings. Again, this is similar to the models before but the embeddings are passed through a series of Transformer encoder blocks.
A lot has been (and is being) written about the use of DL for tabular data, and certainly each of these models would deserve a post themselves. Here, I will try to describe them with some detail and illustrate their use within pytorchwidedeep
. A proper benchmark exercise will be carried out in a notsodistant future.
TabMlp
1.1 The following figure illustrates the TabMlp
model architecture.
Fig 1. The TabMlp
: this is the simples architecture and is very similar to the tabular model available in the fantastic fastai library. In fact, the implementation of the dense layers of the MLP is mostly identical to that in that library.
The dashedborder boxes indicate that these components are optional. For example, we could use TabMlp
without categorical components, or without continuous components, if we wanted.
Let's have a look and see how this model is used with the well known adult census dataset. I assume you have downloaded the data and place it at data/adult/adult.csv.zip
:
#collapsehide
import pandas as pd
import numpy as np
from sklearn.model_selection import train_test_split
adult = pd.read_csv("data/adult/adult.csv.zip")
adult.columns = [c.replace("", "_") for c in adult.columns]
adult["income_label"] = (adult["income"].apply(lambda x: ">50K" in x)).astype(int)
adult.drop("income", axis=1, inplace=True)
for c in adult.columns:
if adult[c].dtype == 'O':
adult[c] = adult[c].apply(lambda x: "unknown" if x == "?" else x)
adult[c] = adult[c].str.lower()
adult_train, adult_test = train_test_split(adult, test_size=0.2, stratify=adult.income_label)
adult.head()
# define the embedding and continuous columns, and target
embed_cols = [
('workclass', 6),
('education', 8),
('marital_status', 6),
('occupation',8),
('relationship', 6),
('race', 6)]
cont_cols = ["age", "hours_per_week", "fnlwgt", "educational_num"]
target = adult_train["income_label"].values
# prepare deeptabular component
from pytorch_widedeep.preprocessing import TabPreprocessor
tab_preprocessor = TabPreprocessor(embed_cols=embed_cols, continuous_cols=cont_cols)
X_tab = tab_preprocessor.fit_transform(adult_train)
Let's pause for a second, since the code up until here is going to be common to all models with some minor adaptations for the TabTransformer
. So far, we have simply defined the columns that will be represented by embeddings and the numerical (aka continuous) columns. Once they are defined the dataset is prepared with the TabPreprocessor
. Internally, the preprocessor label encodes the "embedding columns" and standardizes the numerical columns. Note that one could chose not to standardizes the numerical columns and then use a BatchNorm1D
layer when building the model. That is also a valid approach. Alternatively, one could use both, as I will.
At this stage the data is prepared and we are ready to build the model
from pytorch_widedeep.models import TabMlp, WideDeep
tabmlp = TabMlp(
mlp_hidden_dims=[200, 100],
column_idx=tab_preprocessor.column_idx,
embed_input=tab_preprocessor.embeddings_input,
continuous_cols=cont_cols,
batchnorm_cont=True,
)
Let's have a look to the model we just built and how it relates to Fig 1
tabmlp
As we can see, we have a series of columns that would be represented as embeddings. The embeddings from all these columns are concatenated, to form a tensor of dim (bsz, 40)
where bsz
is batch size. Then, the "batchnormed" continuous columns are also concatenated, resulting in a tensor of dim (bsz, 44)
, that will be passed to the 2layer MLP (200 > 100)
. In summary Embeddings
+ continuous+ MLP.
One important thing to mention, common to all models, is that pytorchwidedeep
models do not build the last connection, i.e. the connection with the output neuron or neurons depending whether this is a regression, binary or multiclass classification. Such connection is built by the WideDeep
constructor class. This means that even if we wanted to use a singlecomponent model, the model still needs to be built with the WideDeep
class.
This is because the library is, a priori, intended to build WideDeep
models (and hence its name). Once the model is built it is passed to the Trainer
(as we will see now). The Trainer
class is coded to receive a parent model of class WideDeep
with children that are the model components. This is very convenient for a number of aspects in the library.
Effectively this simply requires one extra line of code.
model = WideDeep(deeptabular=tabmlp)
model
As we can see, our model
has the final connection now and is a model of class WideDeep
formed by one single component, deeptabular
, which is a model of class TabMlp
formed mainly by the embed_layers
and an MLP very creatively called tab_mlp
.
We are now ready to train it. The code below simply runs with defaults. one could use any torch
optimizer, learning rate schedulers, etc. Just have a look to the docs or the Examples folder in the repo.
from pytorch_widedeep import Trainer
from pytorch_widedeep.metrics import Accuracy
trainer = Trainer(model, objective="binary", metrics=[(Accuracy)])
trainer.fit(X_tab=X_tab, target=target, n_epochs=5, batch_size=256, val_split=0.2)
Once we understand what TabMlp
does, TabResnet
should be pretty straightforward
TabResnet
1.2 The following figure illustrates the TabResnet
model architecture.
Fig 2. The TabResnet
: this model is similar to the TabMlp
, but the embeddings (or the concatenation of embeddings and continuous features, normalised or not) are passed through a series of Resnet blocks built with dense layers. The dashedborder boxes indicate that the component is optional and the dashed lines indicate the different paths or connections present depending on which components we decide to include.
This is probably the most flexible of the three models discussed in this post in the sense that there are many variants one can define via the parameters. For example, we could chose to concatenate the continuous features, normalized or not via a BatchNorm1d
layer, with the embeddings and then pass the result of such a concatenation trough the series of Resnet blocks. Alternatively, we might prefer to concatenate the continuous features with the results of passing the embeddings through the Resnet blocks. Another optional component is the MLP before the output neuron(s). If not MLP is present, the output from the Resnet blocks or the results of concatenating that output with the continuous features (normalised or not) will be connected directly to the output neuron(s).
Each of the Resnet block is comprised by the following operations:
Fig 3. "Dense" Resnet Block. b
is the batch size and d
the dimension of the embeddings.
Let's build a TabResnet
model:
from pytorch_widedeep.models import TabResnet
tabresnet = TabResnet(
column_idx=tab_preprocessor.column_idx,
embed_input=tab_preprocessor.embeddings_input,
continuous_cols=cont_cols,
batchnorm_cont=True,
blocks_dims=[200, 100, 100],
mlp_hidden_dims=[100, 50],
)
model = WideDeep(deeptabular=tabresnet)
model
As we did previously with the TabMlp
, let's "walk through" the model. In this case, model is an instance of a WideDeep
object formed by a single component, deeptabular
that is a TabResnetmodel
. TabResnet
is formed by a series of Embedding
layers (e.g. emb_layer_education
) a series of socalled dense Resnet blocks (tab_resnet
) and a MLP (tab_resnet_mlp
). The embeddings are concatenated themselves and then, further concatenated with the normalised continuous columns. The resulting tensor of dim (bsz, 44)
is then passed through two dense Resnet blocks. The output of one Resnet block is the input of the next. Therefore, when setting blocks_dim = [200, 100, 100]
we are generating two blocks with input/output 200/100 and 100/100 respectively. The output of the second Resnet blocks, of dim (bsz, 100)
is passed through tab_resnet_mlp
, the 2layer MLP, and finally "plugged" into the output neuron. In summary: Embeddings + continuous + dense Renset + MLP.
To run it, the code is, as one might expect identical to the one shown before for the TabMlp
.
trainer = Trainer(model, objective="binary", metrics=[(Accuracy)])
trainer.fit(X_tab=X_tab, target=target, n_epochs=5, batch_size=256, val_split=0.2)
And now, last but not least, the last addition to the library, the TabTransformer
.
TabTransformer
1.3 The TabTransformer
is described in detail in TabTransformer: Tabular Data Modeling Using Contextual Embeddings [2], by the clever guys at Amazon. Is an entertaining paper that I, of course, strongly recommend if you are going to use this model on your tabular data (and also in general if you are interested in DL for tabular data).
My implementation is not the only one available. Given that the model was conceived by the researchers at Amazon, it is also available in their fantastic autogluon
library (which you should definitely check). In addition, you can find another implementation here by Phil Wang, whose entire github is simply outstanding. My implementation is partially inspired by these but has some particularities and adaptations so that it works within the pytorchwidedeep
package.
The following figure illustrates the TabTransformer
model architecture.
Fig 4. The TabTransfomer
, described in TabTransformer: Tabular Data Modeling Using Contextual Embeddings. The dashedborder boxes indicate that the component is optional.
As in previous cases, there are a number of variants and details to consider as one builds the model. I will describe some here, but for a full view of all the possible parameters, please, have a look to the docs.
I don't want to go into the details of what is a Transformer [3] in this post. There is an overwhelming amount of literature if you wanted to learn about it, with the most popular being perhaps The Annotated Transformer. Also check this post and if you are a math "maniac" you might like this paper [4]. However, let me just briefly describe it here so I can introduce the little math we will need for this post. In one sentence, a Transformer consists of a multihead selfattention layer followed by feedforward layer, with elementwise addition and layernormalization being done after each layer.
As most of you will know, a selfattention layer comprises three matrices, Key, Query and Value. Each input categorical column, i.e. embedding, is projected onto these matrices (although see the fixed_attention
option later in the post) to generate their corresponding key, query and value vectors. Formally, let $K \in R^{e \times d}$, $Q \in R^{e \times d}$ and $V \in R^{e \times d}$ be the Key, Query and Value matrices of the embeddings where $e$ is the embeddings dimension and $d$ is the dimension of all the Key, Query and Value matrices. Then every input categorical column, i.e embedding, attends to all other categorical columns through an attention head:
where
$$ A = softmax( \frac{QK^T}{\sqrt{d}} ), \hspace{6cm}(2) $$And that is all the math we need.
As I was thinking in a figure to illustrate a transformer block, I realised that there is a chance that the reader has seen every possible representation/figure. Therefore, I decided to illustrate the transformer block in a way that relates directly to the way it is implemented.
Fig 5. The Transfomer block. The letters in parenthesis indicate the dimension of the corresponding tensor after the operation indicated in the corresponding box. For example, the tensor attn_weights
has dim (b, h, s, s)
.
As the figure shows, the input tensor ($X$) is projected onto its key, query and value matrices. These are then "rearranged into" the multihead selfattention layer where each head will attend to part of the embeddings. We then compute $A$ (Eq 2), which is then multiplied by $V$ to obtain what I refer as attn_score
(Eq 1). attn_score
is then rearranged, so that we "collect" the attention scores from all the heads, and projected again to obtain the results (attn_out
), that will be added to the input and normalised (Y
). Finally Y
goes through the FeedForward layer and a further Add + Norm.
Before moving to the code related to building the model itself, there are a couple of details in the implementation that are worth mentioning
FullEmbeddingDropout
when building a TabTransformer
model, there is the possibility of dropping entirely the embedding corresponding to a categorical column. This is set by the parameter full_embed_dropout: bool
, which points to the class FullEmbeddingDropout
.
SharedEmbeddings
when building a TabTransformer
model, it is possible for all the embeddings that represent a categorical column to share a fraction of their embeddings, or define a common separated embedding per column that will be added to the column's embeddings.
The idea behind this socalled "column embedding" is to enable the model to distinguish the classes in one column from those in the other columns. In other words, we want the model to learn representations not only of the different categorical values in the column, but also of the column itself. This is attained by the shared_embed
group of parameters: share_embed : bool
, add_shared_embed: bool
and frac_shared_embed: int
. The first simply indicates if embeddings will be shared, the second sets the sharing strategy and the third one the fraction of the embeddings that will be shared, depending on the strategy. They all relate to the class SharedEmbeddings
For example, let's say that we have a categorical column with 5 different categories that will be encoded as embeddings of dim 8. This will result in a lookup table for that column of dim (5, 8)
. The two sharing strategies are illustrated in Fig 6.
Fig 6. The two sharing embeddings strategies. Upper panel: the "column embedding" replaces embedding dim / frac_shared_embed
(4 in this case) of the total embeddings that represent the different values of the categorical column. Lower panel: the "column embedding" is added (well, technically broadcasted and added) to the original embedding lookup table. Note that n_cat
here refers to the number of different categories for this particular column.
fixed_attention
fixed_attention
: this in inspired by the implementation at the Autogluon library. When using "fixed attention", the key and query matrices are not the result of any projection of the input tensor $X$, but learnable matrices (referred as fixed_key
and fixed_query
) of dim (number of categorical columns x embeddings dim)
defined separately, as you instantiate the model. fixed_attention
does not affect how the Value matrix is computed.
Let me go through an example with numbers to clarify things. Let's assume we have a dataset with 5 categorical columns that will be encoded by embeddings of dim 4 and we use a batch size (bsz
) of 6. Figure 7 shows how the key matrix will be computed for a given batch (same applies to the query matrix) with and without fixed attention.
Fig 7. Key matrix computation for a given batch with and without fixed attention (same applies to the query matrix). The different color tones in the matrices are my attempt to illustrate that, while without fixed attention the key matrix can have different values anywhere in the matrix, with fixed attention the key matrix is the result of the repetition of the "fixedkey" bsz
times. The projectlayer is, of course, broadcasted along the bsz
dimension in the upper panel.
As I mentioned, this implementation is inspired by that at the Autogluon library. Since the guys at Amazon are the ones that came up with the TabTransformer
, is only logical to think that they found a use for this implementation of attention. However, at the time of writing such use is not 100% clear to me. It is known that, in problems like machine translation, most attention heads learn redundant patterns (see e.g. Alessandro Raganato et al., 2020 [5] and references therein). Therefore, maybe the fixed attention mechanism discussed here helps reducing redundancy for problems involving tabular data.
Overall, the way I interpret fixed_attention
in layman's terms, is the following: when using fixed attention, the Key and the Query matrices are defined as the model is instantiated, and do not know of the input until the attention weights (attn_weights
) are multiplied by the value matrix to obtain what I refer as attn_score
in figure 5. Those attention weights, which are in essence the result of a matrix multiplication between the key and the query matrices (plus softmax and normalization), are going to be the same for all the heads, for all samples in a given batch. Therefore, my interpretation is that when using fixed attention, we reduce the attention capabilities of the transformer, which will focus on less aspects of the inputs, reducing potential redundancies.
Anyway, enough speculation. Time to have a look to the code. Note that, since we are going to stack the embeddings (instead of concatenating them) they all must have the same dimensions. Such dimension is set as we build the model instead that at the preprocessing stage. To avoid input format conflicts we use the for_tabtransformer
parameter at preprocessing time.
embed_cols = ['workclass', 'education', 'marital_status', 'occupation', 'relationship', 'race']
tab_preprocessor = TabPreprocessor(
embed_cols=embed_cols,
continuous_cols=cont_cols,
for_tabtransformer=True)
X_tab = tab_preprocessor.fit_transform(adult_train)
from pytorch_widedeep.models import TabTransformer
tabtransformer = TabTransformer(
column_idx=tab_preprocessor.column_idx,
embed_input=tab_preprocessor.embeddings_input,
continuous_cols=cont_cols,
shared_embed=True,
num_blocks=3,
)
model = WideDeep(deeptabular=tabtransformer)
model
As we can see, the model is an instance of a WideDeep
object formed by a single component, deeptabular
that is TabTransformer
model. TabTransformer
is formed by a series of embedding layers (e.g. emb_layer_education
) , a series of transformer encoder blocks$^*$ (tab_transformer_blks
) and a MLP (tab_transformer_mlp
). The embeddings here are of class SharedEmbeddings
, which I described before. These embeddings are stacked and passed through three transformer blocks. The output for all the categorical columns is concatenated, resulting in a tensor of dim (bsz, 192)
where 192 is equal to the number of categorical columns (6) times the embedding dim (32). This tensor is then concatenated with the "layernormed" continuous columns, resulting in a tensor of dim (bsz, 196)
. As usual, this tensor goes through tab_transformer_mlp , which following the guidance in the paper ("The MLP layer sizes are set to {4 × l, 2 × l}, where l is the size of its input.") is [784 > 392]
, and "off we go". In summary SharedEmbeddings
+ continuous + Transformer encoder blocks + MLP.
To run it, the code is, as one might expect identical to the one shown before for the TabMlp
and TabRenset
.
trainer = Trainer(model, objective="binary", metrics=[(Accuracy)])
trainer.fit(X_tab=X_tab, target=target, n_epochs=5, batch_size=256, val_split=0.2)
$^*$ Note that there is a small inconsistency in the naming of the TabTransformer
main components relative to the other two models. If you installed the package via pypi, the transformer encoder blocks are named blks
. A name more consistent with the other models would be, for example, tab_transformer_blks
. I realised of such inconsistency just after publishing v0.4.8
to pypi. Such small issue is not worthy of another subversion. However, this is fixed if you install the package from github (as I have done for this post) and both the pypi and the github versions will be consistent in future releases
2. Conclusion and future work
In this post my intention was to illustrate how one can use pytorchwidedeep
as a library for "standard DL for tabular data", i.e. without building wide and deep models and for problems that do not involve text and/or images (if you wanted to learn more about the library please visit the repo, the documentation, or the previous posts). To that aim the only component that we need is the deeptabular
component, for which pytorchwidedeep
comes with 3 models implemented "out of the box": TabMlp
, TabResnet
and TabTransformer
. In this post I have explained their architecture in detail and how to use them within the library. In the nosodistant future I intend to implement TabNet and perhaps Node, as well as performing a proper benchmarking exercise so I can set robust defaults and then release version 1.0
. Of course, you can help me by using the package in your datasets 🙂. If you found this post useful and you like the library, please give a star to the repo. Other than that, happy coding.
3. References
[1] TabNet: Attentive Interpretable Tabular Learning, Sercan O. Arik, Tomas Pfister, arXiv:1908.07442v5
[2] TabTransformer: Tabular Data Modeling Using Contextual Embeddings. Xin Huang, Ashish Khetan, Milan Cvitkovic, Zohar Karnin, 2020. arXiv:2012.06678v1
[3] Attention Is All You Need, Ashish Vaswani, Noam Shazeer, Niki Parmar, et al., 2017. arXiv:1706.03762v5
[4] A Mathematical Theory of Attention, James Vuckovic, Aristide Baratin, Remi Tachet des Combes, 2020. arXiv:2007.02876v2
[5] Fixed Encoder SelfAttention Patterns in TransformerBased Machine Translation. Alessandro Raganato, Yves Scherrer, Jörg Tiedemann, 2020. arXiv:2002.10260v3
[6] Neural Oblivious Decision Ensembles for Deep Learning on Tabular Data. Sergei Popov, Stanislav Morozov, Artem Babenko, arXiv:1909.06312v2