#03 | Train Test Split for Model Selection

·

Chapter Importance

Machine Learning models learn a mathematical equation from historical data.

Not all Machine Learning models predict the same way; some models are better than others.

We measure how good a model is by calculating its score (accuracy).

So far, we have calculated the model's score using the same data to fit (train) the mathematical equation. That's cheating. That's overfitting.

This tutorial compares 3 different models:

• Decision Tree
• Logistic Regression
• Support Vector Machines

We validate the models in 2 different ways:

1. Using the same data during training
2. Using 30% of the data; not used during training

To demonstrate how the selection of the best model changes if we are to validate the model with data not used during training.

For example, the image below shows the best model, when using the same data for validation, is the Decision Tree (0.86 of accuracy). Nevertheless, everything changes when the model is evaluated with data not used during training; the best model is the Logistic Regression (0.85 of accuracy). Whereas the Decision Tree only gets up to 0.80 of accuracy.

Were we a bank whose losses rank up to 1M USD due to 0.01 fail in accuracy, we would have lost 5M USD. This is something that happens in real life.

In short, banks are interested in good models to predict new potential customers. Not historical customers who have already gotten a loan and the bank knows if they were good to pay or not.

This tutorial shows you how to implement the `train_test_split` technique to reduce overfitting with a practical use case where we want to classify whether a person used the Internet or not.

Load the dataset from CIS, executing the following lines of code:

``````import pandas as pd #!

df_internet = pd.read_excel('https://github.com/jsulopzs/data/blob/main/internet_usage_spain.xlsx?raw=true', sheet_name=1, index_col=0)
df_internet
``````
• The goal of this dataset is
• To predict `internet_usage` of people (rows)
• Based on their socio-demographical characteristics (columns)

Preprocess the Data

We should already know from the previous chapter that the data might be preprocessed before passing it to the function that computes the mathematical equation.

Missing Data

The function `.fit()` all the cells in the DataFrame to contain a value. `NaN` means "Not a Number" (i.e., cell for which we don't have any information). Otherwise, it won't know how to process the row and compare it to others.

For example, if you miss John's age, you cannot place John in the space to compare with other people because the point might be anywhere.

``````df_internet.isna().sum()
``````
``````internet_usage    0
sex               0
age               0
education         0
dtype: int64
``````

Dummy Variables

The function `.fit()` needs the values to be numeric. Otherwise, Python won't know the position of the axes in which to allocate the point. For example, if you have Male and Female, at which distance do you separate them, and why? You cannot make an objective assessment unless you separate each category.

Therefore, categories of the categorical columns will be transformed into new columns (one new column per category) and contain 1s and 0s depending on whether the person is or is not in the category.

``````df_internet = pd.get_dummies(df_internet, drop_first=True)
df_internet
``````

Feature Selection

Once we have preprocessed the data, we select the column we want to predict (target) and the columns we will use to explain the prediction (features/explanatory).

``````target = df_internet.internet_usage
features = df_internet.drop(columns='internet_usage')
``````

Build & Compare Models' Scores

We should already know that the Machine Learning procedure is the same all the time:

1. Computing a mathematical equation: fit
2. To calculate predictions: predict
3. And compare them to reality: score

The only element that changes is the `Class()` that contains lines of code of a specific algorithm (DecisionTreeClassifier, SVC, LogisticRegression).

`DecisionTreeClassifier()` Model in Python

``````from sklearn.tree import DecisionTreeClassifier

model_dt = DecisionTreeClassifier()
model_dt.fit(X=features, y=target)

model_dt.score(X=features, y=target)
``````
``````0.859877800407332
``````

`SVC()` Model in Python

``````from sklearn.svm import SVC

model_svc = SVC(probability=True)
model_svc.fit(X=features, y=target)

model_svc.score(X=features, y=target)
``````
``````0.7837067209775967
``````

`LogisticRegression()` Model in Python

``````from sklearn.linear_model import LogisticRegression

model_lr = LogisticRegression(max_iter=1000)
model_lr.fit(X=features, y=target)

model_lr.score(X=features, y=target)
``````
``````0.8334012219959267
``````

Function to Automate Lines of Code

• We repeated all the time the same code:
``````model.fit()
model.score()
``````
• Why not turn the lines into a `function()` to automate the process?
``````calculate_accuracy(model_dt)
calculate_accuracy(model_sv)
calculate_accuracy(model_lr)
``````
• To calculate the `accuracy`

Make a Procedure Sample for `DecisionTreeClassifier()`

``````model_dt = DecisionTreeClassifier()
model_dt.fit(X=features, y=target)
model_dt.score(X=features, y=target)
``````
``````0.859877800407332
``````

Automate the Procedure into a `function()`

Code Thinking

1. Think of the functions `result`
2. Store that `object` to a variable
3. `return` the `result` at the end
4. Indent the body of the function to the right
5. `def`ine the `function():`
6. Think of what's gonna change when you execute the function with `different models`
7. Locate the `variable` that you will change
8. Turn it into the `parameter` of the `function()`
``````model_dt = DecisionTreeClassifier()
model_dt.fit(X=features, y=target)
model_dt.score(X=features, y=target)
``````
``````0.859877800407332
``````

Distinguish the line that gives you the `result` you want and put it into a variable

``````model_dt = DecisionTreeClassifier()
model_dt.fit(X=features, y=target)
result = model_dt.score(X=features, y=target) #new
``````

Add a line with a `return` to tell the function the object you want in the end

``````model_dt = DecisionTreeClassifier()
model_dt.fit(X=features, y=target)
result = model_dt.score(X=features, y=target)

return result #new
``````

Indent everything to the right

``````    model_dt = DecisionTreeClassifier()
model_dt.fit(X=features, y=target)
result = model_dt.score(X=features, y=target)

return result
``````

Define the function in the first line

``````def calculate_accuracy(): #new

model_dt = DecisionTreeClassifier()
model_dt.fit(X=features, y=target)
result = model_dt.score(X=features, y=target)

return result
``````

What am I gonna change every time I run the function

``````def calculate_accuracy(model_dt): #modified

model_dt.fit(X=features, y=target)
result = model_dt.score(X=features, y=target)

return result
``````

Generalize the name of the parameter

``````def calculate_accuracy(model): #modified

model.fit(X=features, y=target) #modified
result = model.score(X=features, y=target)

return result
``````

``````def calculate_accuracy(model):
"""
This function calculates the accuracy for a given model as a parameter #modified
"""

model.fit(X=features, y=target)
result = model.score(X=features, y=target)

return result
``````
``````calculate_accuracy(model_dt)
``````
``````0.859877800407332
``````

Calculate Models' Accuracies

`DecisionTreeClassifier()` Accuracy

``````calculate_accuracy(model_dt)
``````
``````0.859877800407332
``````

We shall create an empty dictionary that keeps track of every model's score to choose the best one later.

``````dic_accuracy = {}
dic_accuracy['Decision Tree'] = calculate_accuracy(model_dt)
``````

`SVC()` Accuracy

``````dic_accuracy['Support Vector Machines'] = calculate_accuracy(model_svc)
dic_accuracy
``````
``````{'Decision Tree': 0.859877800407332,
'Support Vector Machines': 0.7837067209775967}
``````

`LogisticRegression()` Accuracy

``````dic_accuracy['Logistic Regression'] = calculate_accuracy(model_lr)
dic_accuracy
``````
``````{'Decision Tree': 0.859877800407332,
'Support Vector Machines': 0.7837067209775967,
'Logistic Regression': 0.8334012219959267}
``````

Which is the Best Model?

The Decision Tree is the best model with an score of 85%:

``````sr_accuracy = pd.Series(dic_accuracy).sort_values(ascending=False)
sr_accuracy
``````
``````Decision Tree              0.859878
Logistic Regression        0.833401
Support Vector Machines    0.783707
dtype: float64
``````

Let's suppose for a moment we are a bank to understand the importance of this chapter. A bank's business is, among other things, to give loans to people who can afford it.

Although the bank may commit mistakes: giving loans to people who cannot afford it or not giving to people who can.

Let's imagine the bank losses of £1M for each 1% of misclassification. As we chose the Decision Tree, the bank lost \$15M, as the score suggests. Nevertheless, can we trust the score of 85%?

No, because we are cheating the model's evaluation; we evaluated the models with the same data used for training. In other words, the bank is not interested in evaluating the model of the historical customers; they want to know how good the model is for new customers.

They cannot create new customers. What can they do then?

They separate the data into a train set (70% of customers) used to `.fit()` the mathematical equation and a test set (30% of customers) to evaluate the mathematical equation.

You can understand the problem better with the following analogy:

University Access Exams Analogy

Let's imagine:

1. You have a `math exam` on Saturday
2. Today is Monday
3. You want to calibrate your level in case you need to study more for the math exam
4. How do you calibrate your `math level`?
5. Well, you've got 100 questions `X` with 100 solutions `y` from past years exams
6. You may study the 100 questions with 100 solutions `fit(100questions, 100solutions)`
7. Then, you may do a `mock exam` with the 100 questions `predict(100questions)`
8. And compare `your_100solutions` with the `real_100solutions`
9. You've got 90/100 correct answers `accuracy` in the mock exam
10. You think you are prepared for the maths exam
11. And when you do the real exam on Saturday, the mark is 40/100
12. Why? How could we have prevented this?
13. Solution: separate the 100 questions into `70 for train` to study & `30 for test` for the mock exam.
2. your_30solutions = predict(30questions)
3. your_30solutions ?= 30solutions

`train_test_split()` the Data

• The documentation of the function contains a typical example.
``````from sklearn.model_selection import train_test_split

X_train, X_test, y_train, y_test = train_test_split(
features, target, test_size=0.30, random_state=42)
``````

What the heck is returning the function?

From all the data:

• 2455 rows
• 8 columns
``````df_internet
``````

• 1728 rows (70% of all data) → to fit the model
• 7 columns (X: features variables)
``````X_train
``````

• 737 rows (30% of all data) → to evaluate the model
• 7 columns (X: features variables)
``````X_test
``````

• 1728 rows (70% of all data) → to fit the model
• 1 columns (y: target variable)
``````y_train
``````
``````name
Eileen     0
Lucinda    1
..
Corey      0
Robert     1
Name: internet_usage, Length: 1718, dtype: int64
``````
• 737 rows (30% of all data) → to evaluate the model
• 1 columns (y: target variable)
``````y_test
``````
``````name
Thomas     0
Pedro      1
..
William    1
Charles    1
Name: internet_usage, Length: 737, dtype: int64
``````

`fit()` the model with Train Data

``````model_dt.fit(X_train, y_train)
``````
``````DecisionTreeClassifier()
``````

Compare the predictions with the real data

``````model_dt.score(X_test, y_test)
``````
``````0.8046132971506106
``````

Optimize All Models & Compare Again

Make a Procedure Sample for `DecisionTreeClassifier()`

``````model_dt = DecisionTreeClassifier()
model_dt.fit(X_train, y_train)
model_dt.score(X_test, y_test)
``````
``````0.8032564450474898
``````

Automate the Procedure into a `function()`

Code Thinking

1. Think of the functions `result`
2. Store that `object` to a variable
3. `return` the `result` at the end
4. Indent the body of the function to the right
5. `def`ine the `function():`
6. Think of what's gonna change when you execute the function with `different models`
7. Locate the `variable` that you will change
8. Turn it into the `parameter` of the `function()`
``````def calculate_accuracy_test(model):

model.fit(X_train, y_train)
result = model.score(X_test, y_test)

return result
``````

Calculate Models' Accuracies

`DecisionTreeClassifier()` Accuracy

``````dic_accuracy_test = {}

dic_accuracy_test['Decision Tree'] = calculate_accuracy_test(model_dt)
dic_accuracy_test
``````
``````{'Decision Tree': 0.8032564450474898}
``````

`SVC()` Accuracy

``````dic_accuracy_test['Support Vector Machines'] = calculate_accuracy_test(model_svc)
dic_accuracy_test
``````
``````{'Decision Tree': 0.8032564450474898,
'Support Vector Machines': 0.7788331071913162}
``````

`LogisticRegression()` Accuracy

``````dic_accuracy_test['Logistic Regression'] = calculate_accuracy_test(model_lr)
dic_accuracy_test
``````
``````{'Decision Tree': 0.8032564450474898,
'Support Vector Machines': 0.7788331071913162,
'Logistic Regression': 0.8548168249660787}
``````

Which is the Best Model with `train_test_split()`?

The picture change quite a lot as the bank is losing £20M due to the model we chose before: the Decision Tree; the score in data that hasn't been seen during training (i.e., new customers) is 80%.

We should have chosen the Logistic Regression because it's the best model (85%) to predict new data and new customers.

In short, we lose £15M if we choose the Logistic Regression, which it's better than the Decision Tree's loss of £20M. Those £5M can make a difference in my life 👀

``````sr_accuracy_test = pd.Series(dic_accuracy_test).sort_values(ascending=False)
sr_accuracy_test
``````
``````Logistic Regression        0.854817
Decision Tree              0.803256
Support Vector Machines    0.778833
dtype: float64
``````
``````df_accuracy = pd.DataFrame({
'Same Data': sr_accuracy,
'Test Data': sr_accuracy_test
})