In several algorithms, gradient descent uses one learning rate for every weight; this means that if one feature ranges from 0 to 1 and another from 0 to 100000 their gradients can have very different size, with training that may zig-zag, diverge or require a very small learning rate.
One solution is to add standardization which means transforming each column on the training dataset so that its mean is 0 and standard deviation is 1: X_scaled = (X - mean) / scale.
The training mean and scale need to be saved and used for validation, test and future predictions too.
Before feeding the input data to the model, we will need to apply the following:
def fit_standardizer(X_train):
mean = X_train.mean(axis=0)
scale = X_train.std(axis=0)
scale = np.where(scale == 0, 1.0, scale)
return mean, scale
mean, scale = fit_standardizer(X)
X = (X - mean) / scale