Dgdrxrt

Trying to understand entropy as a novice in thermodynamics

A proverb that is used to imply that you have unexpectedly faced a big problem

Asymptotics question

How to change the tick of the color bar legend to black

Did any compiler fully use 80-bit floating point?

what is the log of the PDF for a Normal Distribution?

Why is the change of basis formula counter-intuitive? [See details]

Did Mueller's report provide an evidentiary basis for the claim of Russian govt election interference via social media?

How can a team of shapeshifters communicate?

How to write capital alpha?

Would color changing eyes affect vision?

Monty Hall Problem-Probability Paradox

How were pictures turned from film to a big picture in a picture frame before digital scanning?

Delete free apps from library

Tannaka duality for semisimple groups

GDP with Intermediate Production

Should a wizard buy fine inks every time he want to copy spells into his spellbook?

Where is the Next Backup Size entry on iOS 12?

Found this skink in my tomato plant bucket. Is he trapped? Or could he leave if he wanted?

Is it dangerous to install hacking tools on my private linux machine?

Is openssl rand command cryptographically secure?

Getting out of while loop on console

What are the main differences between the original Stargate SG-1 and the Final Cut edition?

Is there public access to the Meteor Crater in Arizona?

regularized least squares Generalized Tikhonov Regularization on real dataset

1

$begingroup$

I am using regularized least squares more specifically Generalized Tikhonov Regularization on real dataset where rows << cols:

$$𝑥=(A^TA+lambda I)^-1(A^Tb)$$

I am implementing it using C by invoking LAPACK routines. For factoring and solving the system, I am using LU decomposition with partial pivoting by invoking DGESV.

I am trying to have different values for the regularize coefficient 𝜆 and each time I am calculating mean square error (MSE) for training set and for testing set.

Conceptually, as regularize coefficient $lambda$ got smaller $lambda to 0$, MSE becomes small and close to zero. This means that the solution X is overfitting dataset.

I don't have such behavior. For example MSE for $lambda=0.0001$ and $lambda=0.0$ are the same and it is big ($MSE=0.05$ on the training dataset, and $MSE=0.07$ on the testing dataset).

Could anyone explain for me why I have the same MSE for different regularize coefficient $lambda$? Could this because of the nonlinearty of dataset?

linear-algebra statistics machine-learning

share|cite|improve this question

edited Apr 2 at 10:20

mathreadler

15.6k72263

asked Apr 2 at 9:50

dev.robidev.robi

61

$endgroup$

• 
  
  
  

  
  

  
  
  
  
  
  

  
  $begingroup$
  Welcome to Math.SE. Please consider learning mathjax typesetting as it is used here on site. I tried doing it for you this time. There should be a crash course tutorial page somewhere.
  $endgroup$
  – mathreadler
  Apr 2 at 10:12
  
  

  
  
  
  


• 
  
  
  

  
  

  
  
  
  
  
  

  
  $begingroup$
  Here it is math.meta.stackexchange.com/questions/5020/…
  $endgroup$
  – mathreadler
  Apr 2 at 10:13
  

  
  
  
  


• 
  
  
  

  
  

  
  
  
  
  
  

  
  $begingroup$
  Sure I will thanks
  $endgroup$
  – dev.robi
  Apr 2 at 10:17
  

  
  
  
  

add a comment | 

1

$begingroup$

I am using regularized least squares more specifically Generalized Tikhonov Regularization on real dataset where rows << cols:

$$𝑥=(A^TA+lambda I)^-1(A^Tb)$$

I am implementing it using C by invoking LAPACK routines. For factoring and solving the system, I am using LU decomposition with partial pivoting by invoking DGESV.

I am trying to have different values for the regularize coefficient 𝜆 and each time I am calculating mean square error (MSE) for training set and for testing set.

Conceptually, as regularize coefficient $lambda$ got smaller $lambda to 0$, MSE becomes small and close to zero. This means that the solution X is overfitting dataset.

I don't have such behavior. For example MSE for $lambda=0.0001$ and $lambda=0.0$ are the same and it is big ($MSE=0.05$ on the training dataset, and $MSE=0.07$ on the testing dataset).

Could anyone explain for me why I have the same MSE for different regularize coefficient $lambda$? Could this because of the nonlinearty of dataset?

linear-algebra statistics machine-learning

share|cite|improve this question

edited Apr 2 at 10:20

mathreadler

15.6k72263

asked Apr 2 at 9:50

dev.robidev.robi

61

$endgroup$

• 
  
  
  

  
  

  
  
  
  
  
  

  
  $begingroup$
  Welcome to Math.SE. Please consider learning mathjax typesetting as it is used here on site. I tried doing it for you this time. There should be a crash course tutorial page somewhere.
  $endgroup$
  – mathreadler
  Apr 2 at 10:12
  
  

  
  
  
  


• 
  
  
  

  
  

  
  
  
  
  
  

  
  $begingroup$
  Here it is math.meta.stackexchange.com/questions/5020/…
  $endgroup$
  – mathreadler
  Apr 2 at 10:13
  

  
  
  
  


• 
  
  
  

  
  

  
  
  
  
  
  

  
  $begingroup$
  Sure I will thanks
  $endgroup$
  – dev.robi
  Apr 2 at 10:17
  

  
  
  
  

add a comment | 

$begingroup$

I am using regularized least squares more specifically Generalized Tikhonov Regularization on real dataset where rows << cols:

$$𝑥=(A^TA+lambda I)^-1(A^Tb)$$

I am implementing it using C by invoking LAPACK routines. For factoring and solving the system, I am using LU decomposition with partial pivoting by invoking DGESV.

I am trying to have different values for the regularize coefficient 𝜆 and each time I am calculating mean square error (MSE) for training set and for testing set.

Conceptually, as regularize coefficient $lambda$ got smaller $lambda to 0$, MSE becomes small and close to zero. This means that the solution X is overfitting dataset.

I don't have such behavior. For example MSE for $lambda=0.0001$ and $lambda=0.0$ are the same and it is big ($MSE=0.05$ on the training dataset, and $MSE=0.07$ on the testing dataset).

Could anyone explain for me why I have the same MSE for different regularize coefficient $lambda$? Could this because of the nonlinearty of dataset?

linear-algebra statistics machine-learning

share|cite|improve this question

edited Apr 2 at 10:20

mathreadler

15.6k72263

asked Apr 2 at 9:50

dev.robidev.robi

61

$endgroup$

share|cite|improve this question

edited Apr 2 at 10:20

mathreadler

15.6k72263

asked Apr 2 at 9:50

dev.robidev.robi

61

share|cite|improve this question

edited Apr 2 at 10:20

mathreadler

15.6k72263

asked Apr 2 at 9:50

dev.robidev.robi

61

edited Apr 2 at 10:20

mathreadler

15.6k72263

edited Apr 2 at 10:20

mathreadler

15.6k72263

asked Apr 2 at 9:50

dev.robidev.robi

61

asked Apr 2 at 9:50

dev.robidev.robi

61