So I am trying to fit a Lasso Regression model with Polynomial Features.
To create these Polynomial Features, I am using from sklearn.preprocessing import PolynomialFeatures
It has a parameter inclue_bias which can be set to True or False as PolynomialFeatures(degree=5, include_bias=False). This bias is nothing but the column of 1 that represents the intercept in the final equation. As explained in detail here in another answer.
The issue is arising, when I either set include_bias=True and then fit Lasso regression without an intercept as it has already been taken care of or I choose to not include bias and set fit_intercept=True in from sklearn.linear_model import Lasso. They technically should give result for the intercept coefficient right? But turns out they aren't coming out to be same.
Is there internally some bug in scikit learn or what am I missing here? Let me know if anyone wants to try this on their own with some data. I'll share some.
When PolynomialFeatures.include_bias=True but Lasso.fit_intercept=False, the lasso model thinks the all-ones column is just another feature, and so the "intercept term" is penalized with the L1 penalty. When PolynomialFeatures.include_bias=False and Lasso.fit_intercept=True, the lasso model knows not to penalize the intercept that it is fitting.