# #06 | The Principal Component Analysis (PCA) & Dimensionality Reduction Techniques

## Learn how to group variables to plot seven variables in a 2-dimensional plot and why PCA is an essential technique to visualize clustering models.

**© Jesús López 2022**

Ask him any doubt on **Twitter** or **LinkedIn**

## Chapter Importance

We used just two variables out of the seven we had in the whole DataFrame.

We could have computed better cluster models by giving more information to the Machine Learning model. Nevertheless, it would have been **harder to plot seven variables with seven axes in a graph**.

Is there anything we can do to compute a clustering model with more than two variables and later represent all the points along with their variables?

- Yes, everything is possible with data. As one of my teachers told me: "you can torture the data until it gives you what you want" (sometimes it's unethical, so behave).

We'll develop the code to show you the need for **dimensionality reduction** techniques. Specifically, the Principal Component Analysis (PCA).

## Load the Data

Imagine for a second you are the president of the United States of America, and you are considering creating campaigns to reduce **car accidents**.

You won't create 51 TV campaigns, one for each of the **States of the USA** (rows). Instead, you will see which States behave similarly to cluster them into 3 groups based on the variation across their features (columns).

```
import seaborn as sns #!
df_crashes = sns.load_dataset(name='car_crashes', index_col='abbrev')
df_crashes
```

Check this website to understand the measures of the following data.

## Data Preprocessing

From the previous chapter, we should know that we need to preprocess the Data so that variables with different scales can be compared.

For example, it is not the same to increase 1kg of weight than 1m of height.

We will use `StandardScaler()`

algorithm:

```
from sklearn.preprocessing import StandardScaler
scaler = StandardScaler()
data_scaled = scaler.fit_transform(df_crashes)
data_scaled[:5]
```

```
array([[ 0.73744574, 1.1681476 , 0.43993758, 1.00230055, 0.27769155,
-0.58008306, 0.4305138 ],
[ 0.56593556, 1.2126951 , -0.21131068, 0.60853209, 0.80725756,
0.94325764, -0.02289992],
[ 0.68844283, 0.75670887, 0.18761539, 0.45935701, 1.03314134,
0.0708756 , -0.98177845],
[ 1.61949811, -0.48361373, 0.54740815, 1.67605228, 1.95169961,
-0.33770122, 0.32112519],
[-0.92865317, -0.39952407, -0.8917629 , -0.594276 , -0.89196792,
-0.04841772, 1.26617765]])
```

Let's turn the array into a DataFrame for better understanding:

```
import pandas as pd
df_scaled = pd.DataFrame(data_scaled, index=df_crashes.index, columns=df_crashes.columns)
df_scaled
```

Now we see all the variables having the same scale (i.e., around the same limits):

```
df_scaled.agg(['min', 'max'])
```

## k-Means Model in Python

We follow the usual Scikit-Learn procedure to develop Machine Learning models.

### Import the Class

```
from sklearn.cluster import KMeans
```

### Instantiate the Class

```
model_km = KMeans(n_clusters=3)
```

### Fit the Model

```
model_km.fit(X=df_scaled)
```

```
KMeans(n_clusters=3)
```

### Calculate Predictions

```
model_km.predict(X=df_scaled)
```

```
array([1, 1, 1, 1, 2, 0, 2, 1, 2, 1, 0, 1, 0, 0, 0, 0, 0, 1, 1, 0, 2, 2,
2, 2, 0, 1, 1, 0, 0, 0, 2, 0, 2, 1, 1, 0, 1, 0, 1, 2, 1, 1, 1, 1,
0, 0, 0, 0, 1, 0, 1], dtype=int32)
```

### Create a New DataFrame for the Predictions

```
df_pred = df_scaled.copy()
```

### Create a New Column for the Predictions

```
df_pred.insert(0, 'pred', model_km.predict(X=df_scaled))
df_pred
```

### Visualize the Model

Now let's visualize the clusters with a 2-axis plot:

```
sns.scatterplot(x='total', y='speeding', hue='pred',
data=df_pred, palette='Set1');
```

### Model Interpretation

Does the visualization make sense?

- No, because the clusters should separate their points from others. Nevertheless, we see some green points in the middle of the blue cluster.

Why is this happening?

- We are
**just representing 2 variables**where the model was**fitted with 7 variables**. We can't see the points separated as we miss 5 variables in the plot.

Why don't we add 5 variables to the plot then?

- We could, but it'd be a way too hard to interpret.

Then, what could we do?

- We can apply PCA, a dimensionality reduction technique. Take a look at the following video to understand this concept:

## Grouping Variables with `PCA()`

### Transform Data to Components

`PCA()`

is another technique used to transform data.

How has the data been manipulated so far?

- Original Data
`df_crashes`

```
df_crashes
```

- Normalized Data
`df_scaled`

```
df_scaled
```

- Principal Components Data
`dfpca`

(now)

```
from sklearn.decomposition import PCA
pca = PCA()
data_pca = pca.fit_transform(df_scaled)
data_pca[:5]
```

```
array([[ 1.60367129, 0.13344927, 0.31788093, -0.79529296, -0.57971878,
0.04622256, 0.21018495],
[ 1.14421188, 0.85823399, 0.73662642, 0.31898763, -0.22870123,
-1.00262531, 0.00896585],
[ 1.43217197, -0.42050562, 0.3381364 , 0.55251314, 0.16871805,
-0.80452278, -0.07610742],
[ 2.49158352, 0.34896812, -1.78874742, 0.26406388, -0.37238226,
-0.48184939, -0.14763646],
[-1.75063825, 0.63362517, -0.1361758 , -0.97491605, -0.31581147,
0.17850962, -0.06895829]])
```

```
df_pca = pd.DataFrame(data_pca)
df_pca
```

```
cols_pca = [f'PC{i}' for i in range(1, pca.n_components_+1)]
cols_pca
```

```
['PC1', 'PC2', 'PC3', 'PC4', 'PC5', 'PC6', 'PC7']
```

```
df_pca = pd.DataFrame(data_pca, columns=cols_pca, index=df_crashes.index)
df_pca
```

### Visualize Components & Clusters

Let's visualize a **scatterplot** with `PC1`

& `PC2`

and colour points by cluster:

```
import plotly.express as px
px.scatter(data_frame=df_pca, x='PC1', y='PC2', color=df_pred.pred)
```

Are they **mixed** now?

- No, they aren't.

That's because both PC1 and PC2 represent almost 80% of the variability of the original seven variables.

You can see the following array, where every element represents the amount of variability explained by every component:

```
pca.explained_variance_ratio_
```

```
array([0.57342168, 0.22543042, 0.07865743, 0.05007557, 0.04011 ,
0.02837999, 0.00392491])
```

And the accumulated variability (79.88% until PC2):

```
pca.explained_variance_ratio_.cumsum()
```

```
array([0.57342168, 0.7988521 , 0.87750953, 0.9275851 , 0.9676951 ,
0.99607509, 1. ])
```

Which variables represent these two components?

## Relationship between Original Variables & Components

### Loading Vectors

The Principal Components are produced by a **mathematical equation** (once again), which is composed of the following weights:

```
df_weights = pd.DataFrame(pca.components_.T, columns=df_pca.columns, index=df_scaled.columns)
df_weights
```

We can observe that:

- Socio-demographical features (total, speeding, alcohol, not_distracted & no_previous) have higher coefficients (higher influence) in PC1.
- Whereas insurance features (ins_premium & ins_losses) have higher coefficients in PC2.

Principal Components is a technique that gathers the maximum variability of a set of features (variables) into Components.

Therefore, the two first Principal Components accurate a good amount of common data because we see two sets of variables that are correlated with each other:

### Correlation Matrix

```
df_corr = df_scaled.corr()
sns.heatmap(df_corr, annot=True, vmin=0, vmax=1);
```

I hope that everything is making sense so far.

To ultimate the explanation, you can see below how `df_pca`

values are computed:

### Calculating One PCA Value

For example, we can multiply the weights of PC1 with the original variables for **AL**abama:

```
(df_weights['PC1']*df_scaled.loc['AL']).sum()
```

```
1.6036712920638672
```

To get the transformed value of the Principal Component 1 for **AL**abama State:

```
df_pca.head()
```

The same operation applies to any value of

`df_pca`

.

## PCA & Cluster Interpretation

Now, let's go back to the PCA plot:

```
px.scatter(data_frame=df_pca, x='PC1', y='PC2', color=df_pred.pred.astype(str))
```

How can we interpret the clusters with the components?

Let's add information to the points thanks to animated plots from `plotly`

library:

```
hover = '''
<b>%{customdata[0]}</b><br><br>
PC1: %{x}<br>
Total: %{customdata[1]}<br>
Alcohol: %{customdata[2]}<br><br>
PC2: %{y}<br>
Ins Losses: %{customdata[3]}<br>
Ins Premium: %{customdata[4]}
'''
fig = px.scatter(data_frame=df_pca, x='PC1', y='PC2',
color=df_pred.pred.astype(str),
hover_data=[df_pca.index, df_crashes.total, df_crashes.alcohol,
df_crashes.ins_losses, df_crashes.ins_premium])
fig.update_traces(hovertemplate = hover)
```

If you hover the mouse over the two most extreme points along the x-axis, you can see that their values coincide with the `min`

and `max`

values across socio-demographical features:

```
df_crashes.agg(['min', 'max'])
```

```
df_crashes.loc[['DC', 'SC'],:]
```

Apply the same reasoning over the two most extreme points along the y-axis. You will see the same for the *insurance* variables because they determine the positioning of the PC2 (y-axis).

```
df_crashes.agg(['min', 'max'])
```

```
df_crashes.loc[['ID', 'LA'],:]
```

Is there a way to represent the weights of the original data for the Principal Components and the points?

That's called a Biplot, which we will see later.

### Biplot

We can observe how we position the points along the loadings vectors. Friendly reminder: the loading vectors are the weights of the original variables in each Principal Component.

```
import numpy as np
loadings = pca.components_.T * np.sqrt(pca.explained_variance_)
evr = pca.explained_variance_ratio_.round(2)
fig = px.scatter(df_pca, x='PC1', y='PC2',
color=model_km.labels_.astype(str),
hover_name=df_pca.index,
labels={
'PC1': f'PC1 ~ {evr[0]}%',
'PC2': f'PC2 ~ {evr[1]}%'
})
for i, feature in enumerate(df_scaled.columns):
fig.add_shape(
type='line',
x0=0, y0=0,
x1=loadings[i, 0],
y1=loadings[i, 1],
line=dict(color="red",width=3)
)
fig.add_annotation(
x=loadings[i, 0],
y=loadings[i, 1],
ax=0, ay=0,
xanchor="center",
yanchor="bottom",
text=feature,
)
fig.show()
```

## Conclusion

Dimensionality Reduction techniques have many more applications, but I hope you got the essence: they are great for grouping variables that behave similarly and later visualising many variables in just one component.

In short, you are simplifying the information of the data. In this example, we simplify the data from plotting seven to only two dimensions. Although we don't get this for free because we explain around 80% of the data's original variability.

This work is licensed under a Creative Commons Attribution-NonCommercial-NoDerivatives 4.0 International License.

### Did you find this article valuable?

Support **Jesús López** by becoming a sponsor. Any amount is appreciated!