Cross validation ppt

bioanalytical method validation guidelines usfda and bootstrap tutorial for beginners with examples pdf
Prof.KristianHardy Profile Pic
Prof.KristianHardy,Austria,Teacher
Published Date:26-07-2017
Your Website URL(Optional)
Comment
 These methods re t a model of interest to samples formed from the training set, in order to obtain additional information about the tted model.  For example, they provide estimates of test-set prediction error, and the standard deviation and bias of our parameter estimates Cross-validation and the Bootstrap  In the section we discuss two resampling methods: cross-validation and the bootstrap. 1/44 For example, they provide estimates of test-set prediction error, and the standard deviation and bias of our parameter estimates Cross-validation and the Bootstrap  In the section we discuss two resampling methods: cross-validation and the bootstrap.  These methods re t a model of interest to samples formed from the training set, in order to obtain additional information about the tted model. 1/44Cross-validation and the Bootstrap  In the section we discuss two resampling methods: cross-validation and the bootstrap.  These methods re t a model of interest to samples formed from the training set, in order to obtain additional information about the tted model.  For example, they provide estimates of test-set prediction error, and the standard deviation and bias of our parameter estimates 1/44Training Error versus Test error  Recall the distinction between the test error and the training error:  The test error is the average error that results from using a statistical learning method to predict the response on a new observation, one that was not used in training the method.  In contrast, the training error can be easily calculated by applying the statistical learning method to the observations used in its training.  But the training error rate often is quite di erent from the test error rate, and in particular the former can dramatically underestimate the latter. 2/44Training- versus Test-Set Performance High Bias Low Bias Low Variance High Variance Test Sample Training Sample Low High Model Complexity 3/44 Prediction ErrorMore on prediction-error estimates  Best solution: a large designated test set. Often not available  Some methods make a mathematical adjustment to the training error rate in order to estimate the test error rate. These include the Cp statistic, AIC and BIC. They are discussed elsewhere in this course  Here we instead consider a class of methods that estimate the test error by holding out a subset of the training observations from the tting process, and then applying the statistical learning method to those held out observations 4/44Validation-set approach  Here we randomly divide the available set of samples into two parts: a training set and a validation or hold-out set.  The model is t on the training set, and the tted model is used to predict the responses for the observations in the validation set.  The resulting validation-set error provides an estimate of the test error. This is typically assessed using MSE in the case of a quantitative response and misclassi cation rate in the case of a qualitative (discrete) response. 5/44The Validation process " %"" & A random splitting into two halves: left part is training set, right part is validation set 6/44Example: automobile data  Want to compare linear vs higher-order polynomial terms in a linear regression  We randomly split the 392 observations into two sets, a training set containing 196 of the data points, and a validation set containing the remaining 196 observations. 2 4 6 8 10 2 4 6 8 10 Degree of Polynomial Degree of Polynomial Left panel shows single split; right panel shows multiple splits 7/44 Mean Squared Error 16 18 20 22 24 26 28 Mean Squared Error 16 18 20 22 24 26 28Why? Drawbacks of validation set approach  the validation estimate of the test error can be highly variable, depending on precisely which observations are included in the training set and which observations are included in the validation set.  In the validation approach, only a subset of the observations those that are included in the training set rather than in the validation set are used to t the model.  This suggests that the validation set error may tend to overestimate the test error for the model t on the entire data set. 8/44Drawbacks of validation set approach  the validation estimate of the test error can be highly variable, depending on precisely which observations are included in the training set and which observations are included in the validation set.  In the validation approach, only a subset of the observations those that are included in the training set rather than in the validation set are used to t the model.  This suggests that the validation set error may tend to overestimate the test error for the model t on the entire data set. Why? 8/44K-fold Cross-validation  Widely used approach for estimating test error.  Estimates can be used to select best model, and to give an idea of the test error of the nal chosen model.  Idea is to randomly divide the data into K equal-sized parts. We leave out part k, t the model to the other K 1 parts (combined), and then obtain predictions for the left-out kth part.  This is done in turn for each part k = 1; 2;:::K, and then the results are combined. 9/44K-fold Cross-validation in detail Divide data into K roughly equal-sized parts (K = 5 here) 1 2 3 4 5 Validation Train Train Train Train 10/44 Setting K =n yields n-fold or leave-one out cross-validation (LOOCV). The details  Let the K parts be C ;C ;:::C , where C denotes the 1 2 K k indices of the observations in part k. There are n k observations in part k: if N is a multiple of K, then n =n=K. k  Compute K X n k CV = MSE k (K) n k=1 P 2 where MSE = (y y ) =n , and y is the t for k i i k i i2C k observation i, obtained from the data with part k removed. 11/44The details  Let the K parts be C ;C ;:::C , where C denotes the 1 2 K k indices of the observations in part k. There are n k observations in part k: if N is a multiple of K, then n =n=K. k  Compute K X n k CV = MSE k (K) n k=1 P 2 where MSE = (y y ) =n , and y is the t for k i i k i i2C k observation i, obtained from the data with part k removed.  Setting K =n yields n-fold or leave-one out cross-validation (LOOCV). 11/44 LOOCV sometimes useful, but typically doesn't shake up the data enough. The estimates from each fold are highly correlated and hence their average can have high variance.  a better choice is K = 5 or 10. A nice special case  With least-squares linear or polynomial regression, an amazing shortcut makes the cost of LOOCV the same as that of a single model t The following formula holds:   n 2 X 1 y y i i CV = ; (n) n 1h i i=1 where y is the ith tted value from the original least i squares t, and h is the leverage (diagonal of the \hat" i matrix; see book for details.) This is like the ordinary MSE, except the ith residual is divided by 1h . i 12/44A nice special case  With least-squares linear or polynomial regression, an amazing shortcut makes the cost of LOOCV the same as that of a single model t The following formula holds:   n 2 X 1 y y i i CV = ; (n) n 1h i i=1 where y is the ith tted value from the original least i squares t, and h is the leverage (diagonal of the \hat" i matrix; see book for details.) This is like the ordinary MSE, except the ith residual is divided by 1h . i  LOOCV sometimes useful, but typically doesn't shake up the data enough. The estimates from each fold are highly correlated and hence their average can have high variance.  a better choice is K = 5 or 10. 12/44Auto data revisited LOOCV 10−fold CV 2 4 6 8 10 2 4 6 8 10 Degree of Polynomial Degree of Polynomial 13/44 Mean Squared Error 16 18 20 22 24 26 28 Mean Squared Error 16 18 20 22 24 26 28True and estimated test MSE for the simulated data 2 5 10 20 2 5 10 20 2 5 10 20 Flexibility Flexibility Flexibility 14/44 Mean Squared Error 0.0 0.5 1.0 1.5 2.0 2.5 3.0 Mean Squared Error 0.0 0.5 1.0 1.5 2.0 2.5 3.0 Mean Squared Error 0 5 10 15 20Why?  This bias is minimized when K =n (LOOCV), but this estimate has high variance, as noted earlier.  K = 5 or 10 provides a good compromise for this bias-variance tradeo . Other issues with Cross-validation  Since each training set is only (K 1)=K as big as the original training set, the estimates of prediction error will typically be biased upward. 15/44

Advise: Why You Wasting Money in Costly SEO Tools, Use World's Best Free SEO Tool Ubersuggest.