REGULARIZATION IN REGRESSION

Jul 21, 20215 min read

Author – Bhavesh Kumar

The word Regularize means to make things acceptable or Regular. Regularization is technique used to overcome the error by fitting a function appropriately on the training set and to avoid overfitting by introducing additional information to the cost function. Regularization can be applied to objective functions in optimization problems. The regularization term, or penalty, imposes a cost on the optimization function to make the optimal solution unique.

Table of Content:-

Problem with Overfitting and Underfitting.
Introduction to Regularization.
How can we address and reduce Overfitting?
Different Regularization techniques.
L1 (Lasso) Regularization.
L2 (Ridge) Regularization.
Regularization in Linear Regression.
Regularization in Logistic Regression.
Github URL
References

Problem with Overfitting and Underfitting:-

This situation where any given model is performing too well on the training data but the performance drops significantly over the test set is called an overfitting model.

On the other hand, if the model is performing poorly over the test and the train set, then we call that an underfitting model.

Bias is also called “error due to squared bias,”. Bias is the amount that a model’s prediction differs from the target value, compared to the training data. Bias error results from simplifying the assumptions used in a model so the target functions are easier to approximate.

Variance is the variability of model prediction for a data point or a value which tells us spread of our data. Model with high variance pays a lot of attention to training data and does not generalize on the data which it hasn’t seen before. As a result, such models perform very well on training data but has high error rates on test data.

Figure 1

In supervised learning, underfitting happens when a model unable to capture the underlying pattern of the data. These models usually have high bias and high variance. It happens when we have very less amount of data to build an accurate model or when we try to build a linear model with a nonlinear data.

In supervised learning, overfitting happens when our model captures the noise along with the underlying pattern in data. It happens when we train our model a lot over noisy dataset. These models have low bias and high variance. These models are very complex.

To build a good model, we need to find a good balance between bias and variance such that it minimizes the total error.

Figure 2

An optimal balance of bias and variance would never overfit or underfit the model.

There are two main options to address the issue of overfitting :

1) Reduce the number of feature

2) Regularization

How can we address and reduce Overfitting?

To Understand how we address and reduce Overfitting we take a example. Let suppose we have two functions represented by a green curve and a blue curve respectively. Both the curves fit those red points so well that we can consider they both bear zero loss. From the intuition of Overfitting, we may guess that green curve is the that overfits.

Figure 3

But a question arises why the green curve (or any curve which is similar to it) is overfitting the data?

To understand that in a bit more mathematical way, let’s assume the two functions that are used to draw the graph above:

The green curve:

The red curve:

If we look at each function’s equation, we will find that the green curve has larger coefficients, and that’s the primary cause of Overfitting. As mentioned before, Overfitting can be interpreted that your model fits the dataset so well, which it seems to memorize the data we showed rather than actually learn from it. Intuitively, having large coefficients can be seen as evidence of memorizing the data

Other case is if we add too many features, we will be punished with Overfitting. So we also have to avoid too many feature selection.

Regularization Term or Cost Function:-

Here we introduce mean square error cost function or Hypothesis Which we are using through regression by adding some penalty to it for regularization to overcome overfitting.

Mathematically,

Where

is the learned prediction given on the i^th input parameter Xi.

y^i) is the original prediction.
m is the total number of input sample.

Different Regularization techniques:-

Here we have a two types of Regularization techniques as follows.

L1(Lasso) Regularization.
L2 (Ridge) Regularization.

L1 (Lasso) Regularization:-

The “LASSO” stands for Least Absolute Shrinkage and Selection Operator. It is used over regression methods for a more accurate prediction. This model uses shrinkage. Shrinkage is where data values are shrunk towards a central point as the mean

Lasso Regression uses L1 regularization technique. It is used when we have more number of features because it automatically performs feature selection.

Mathematical equation of Lasso:-

Residual sum of squares (RSS) + λ * (Sum of the absolute value of the magnitude of coefficients)

Where,

α denotes the amount of shrinkage.
α = 0 implies all features are considered and it is equivalent to the linear regression where only the residual sum of squares is considered to build a predictive model
α = ∞ implies no feature is considered i.e, as λ closes to infinity it eliminates more and more features
The bias increases with increase in α
variance increases with decrease in α

Scikit learn implementation of Lasso:-

L2 (Ridge) Regularization:-

L2 Regularization adds “squared magnitude” of coefficient as penalty term to the loss function scales them by some number 𝜆. This is called the Ridge penalty.

L2 regularization is also known as weight decay as it forces the weights to decay towards zero (but not exactly zero).

Mathematically,

where

J(θ) = Cost Function
θ = Parameter
x(i) = Features
hθ(x(i)) = Hypothesis or function
y(i) = Actual or Target values
λ = Constant Value

The difference between ridge and lasso is that it tends to make coefficients to absolute zero as compared to Ridge which never sets the value of coefficient to absolute zero

Scikit learn implementation of Ridge:-

Regularization in Linear Regression:-

First of all here we introduce Gradient descent,

Mathematically,

Repeat till convergence to get extremum

We can apply regularization to both linear regression and logistic regression. We will approach linear regression first.

We will modify our gradient descent function to separate out

From the rest of the parameters because we do not want to penalize

Here in this particular gradient descent our hypothesis function is best fit line i.e

where,

k = slope of best fit

x = actual or Target data point

c = Intercept of that best fit Line.

α = Learning Rate

The term

performs our regularization

By calculation,

The first term in the above equation

will always be less than 1.

We can see it as reducing the value of θj by some amount on every update. The second term is now exactly the same as it was before

After each iteration we converge toward our Extremum.

Regularization in Logistic Regression:-

We will regularize the logistic regression as similar we have done regularization in linear regression. In we given figure we have Underfitting , optimize and overfitting cases.

Figure 4

Here, the hypothesis is slightly different from hypothesis of linear regression because here we classify the data points in binary class or multi class. so our Hypothesis should be under value of 0 to 1 that is our necessary condition for classification in logistic regression.

Mathematically, our hypothesis will be as follows.

Here we use the "Sigmoid Function," also called the "Logistic Function"