<p><div><div><div><div data-show-faces="false" data-height="21" data-width="100" data-layout="button_count" data-send="true" data-href="https://www.r-bloggers.com/cubic-and-smoothing-splines-in-r/"></div></div><div></div></div><div></div></div>
<div>
(This article was first published on <strong><a href="https://datascienceplus.com/cubic-and-smoothing-splines-in-r/"> R Programming – DataScience+</a></strong>, and kindly contributed to <a rel="nofollow" href="https://www.r-bloggers.com/">R-bloggers)</a> </div>
<p>Splines are a smooth and flexible way of fitting Non linear Models and learning the Non linear interactions from the data.In most of the methods in which we fit Non linear Models to data and learn Non linearities is by transforming the data or the variables by applying a Non linear transformation.</p>
<h2>Cubic Splines</h2>
<p>Cubic Splines with <i>knots(cutpoints)</i> at \(\xi_K , \ K = 1,\ 2…\ k\) is a <i>piece-wise cubic</i> polynomial with continious derivatives upto order 2 at each knot. They have continuous 1st and 2nd derivative. The order of continuity is = \( (d – 1) \) , where \(d\) is the degree of polynomial. Now we can represent the Model with truncated power Basis function \(b(x)\). What happens is that we transform the variables \(X_i\) by applying a Basis function \(b(x)\) and fit a model using these transformed variables which adds non linearities to the model and enables the splines to fit smoother and flexible Non linear functions.</p>
<p>The Regression Equation Becomes —</p>
<p>\(f(x) = y_i = \alpha + \beta_1.b_1(x_i)\ + \beta_2.b_2(x_i)\ + \ …. \beta_{k+3}.b_{k+3}(x_i) \ + \epsilon_i\)</p>
<pre>#loading the Splines Packages
require(splines)
#ISLR contains the Dataset
require(ISLR)
attach(Wage) #attaching Wage dataset
?Wage #for more details on the dataset
agelims&lt;-range(age)
#Generating Test Data
age.grid&lt;-seq(from=agelims[1], to = agelims[2])</pre>
<p><b>Now let’s fit a Cubic Spline with 3 Knots (cutpoints) </b><br>
The idea here is to transform the variables and add a linear combination of the variables using the Basis power function to the regression function <b>f(x)</b>.The \( bs() \) function is used in R to fit a Cubic Spline.</p>
<pre>#3 cutpoints at ages 25 ,50 ,60
fit&lt;-lm(wage ~ bs(age,knots = c(25,40,60)),data = Wage )
summary(fit)
<em>## ## Call:
## lm(formula = wage ~ bs(age, knots = c(25, 40, 60)), data = Wage)
## ## Residuals:
## Min 1Q Median 3Q Max ## -98.832 -24.537 -5.049 15.209 203.207 ## ## Coefficients:
## Estimate Std. Error t value Pr(>|t|) ## (Intercept) 60.494 9.460 6.394 1.86e-10 ***
## bs(age, knots = c(25, 40, 60))1 3.980 12.538 0.317 0.750899 ## bs(age, knots = c(25, 40, 60))2 44.631 9.626 4.636 3.70e-06 ***
## bs(age, knots = c(25, 40, 60))3 62.839 10.755 5.843 5.69e-09 ***
## bs(age, knots = c(25, 40, 60))4 55.991 10.706 5.230 1.81e-07 ***
## bs(age, knots = c(25, 40, 60))5 50.688 14.402 3.520 0.000439 ***
## bs(age, knots = c(25, 40, 60))6 16.606 19.126 0.868 0.385338 ## ---
## Signif. codes: 0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1
## ## Residual standard error: 39.92 on 2993 degrees of freedom
## Multiple R-squared: 0.08642, Adjusted R-squared: 0.08459 ## F-statistic: 47.19 on 6 and 2993 DF, p-value: &lt; 2.2e-16
</em></pre>
<p>Now plotting the Regression Line</p>
<pre>
#Plotting the Regression Line to the scatterplot   
plot(age,wage,col=&quot;grey&quot;,xlab=&quot;Age&quot;,ylab=&quot;Wages&quot;)
points(age.grid,predict(fit,newdata = list(age=age.grid)),col=&quot;darkgreen&quot;,lwd=2,type=&quot;l&quot;)
#adding cutpoints
abline(v=c(25,40,60),lty=2,col=&quot;darkgreen&quot;)
</pre>
<p>Gives this plot:<br>
<a rel="nofollow" href="https://i0.wp.com/datascienceplus.com/wp-content/uploads/2017/06/cubicSplines.png?ssl=1" target="_blank"><img sizes="(max-width: 490px) 100vw, 490px" src="https://i2.wp.com/datascienceplus.com/wp-content/uploads/2017/06/cubicSplines-490x350.png?w=450&ssl=1" alt="" class="wp-image-7316" data-recalc-dims="1"></a></p>
<p>The Dashed Lines are the Cutpoints or the Knots. The above Plot shows the smoothing and local effect of Cubic Splines.</p>
<h2>Smoothing Splines</h2>
<p>These are mathematically more challenging but they are more <i>smoother and flexible</i> as well. But, it does not require the selection of the number of Knots, but require selection of only a <i>Roughness Penalty</i> which accounts for the <i>wiggliness(fluctuations)</i> and controls the roughness of the function and variance of the Model. Another important thing to remember in Smoothing Splines are that they have a <i>Knot</i> for every unique value of \((x_i)\).Our aim in Smoothing Splines is to minimize the Error function which is modified by adding a Roughness Penalty which penalizes it for Roughness(Wiggliness) and high variance. </p>
<p>\(minimize:{ g \in RSS} :\ \sum\limits_{i=1}^n ( \ y_i \ – \ g(x_i) \ )^2 + \lambda \ \int g”(t)^2 dt , \quad \lambda > 0\) .</p>
<p>\( \lambda \ is \ the \ Tuning \ Parameter \)</p>
<pre>
#fitting smoothing splines using smooth.spline(X,Y,df=...)
fit1&lt;-smooth.spline(age,wage,df=16) #16 degrees of freedom
#Plotting both cubic and Smoothing Splines 
plot(age,wage,col=&quot;grey&quot;,xlab=&quot;Age&quot;,ylab=&quot;Wages&quot;)
points(age.grid,predict(fit,newdata = list(age=age.grid)),col=&quot;darkgreen&quot;,lwd=2,type=&quot;l&quot;)
#adding cutpoints
abline(v=c(25,40,60),lty=2,col=&quot;darkgreen&quot;)
lines(fit1,col=&quot;red&quot;,lwd=2)
legend(&quot;topright&quot;,c(&quot;Smoothing Spline with 16 df&quot;,&quot;Cubic Spline&quot;),col=c(&quot;red&quot;,&quot;darkgreen&quot;),lwd=2)
</pre>
<p>Gives this plot:<br>
<a rel="nofollow" href="https://i0.wp.com/datascienceplus.com/wp-content/uploads/2017/06/smoothingSplines.png?ssl=1" target="_blank"><img sizes="(max-width: 1024px) 100vw, 1024px" src="https://i1.wp.com/datascienceplus.com/wp-content/uploads/2017/06/smoothingSplines-1024x731.png?w=450&ssl=1" alt="" class="wp-image-7321" data-recalc-dims="1"></a></p>
<p>Now as we can notice that the Red line i.e Smoothing Spline is more wiggly and fits data more flexibly.This is probably due to high degrees of freedom. The best way to select the value of \(\lambda\) and df is Cross Validation . Now we have a direct method to implement cross validation in R using smooth.spline().</p>
<p>Implementing Cross Validation to select value of λ and Implement Smoothing Splines:</p>
<pre>
fit2&lt;-smooth.spline(age,wage,cv = TRUE)
fit2
<em>## Call:
## smooth.spline(x = age, y = wage, cv = TRUE)
## ## Smoothing Parameter spar= 0.6988943 lambda= 0.02792303 (12 iterations)
## Equivalent Degrees of Freedom (Df): 6.794596
## Penalized Criterion: 75215.9
## PRESS: 1593.383
</em></pre>
<pre>
#It selects $\lambda=0.0279$ and df = 6.794596 as it is a Heuristic and can take various values for how rough the #function is
plot(age,wage,col=&quot;grey&quot;)
#Plotting Regression Line
lines(fit2,lwd=2,col=&quot;purple&quot;)
legend(&quot;topright&quot;,(&quot;Smoothing Splines with 6.78 df selected by CV&quot;),col=&quot;purple&quot;,lwd=2)
</pre>
<p>Gives this plot:<br>
<a rel="nofollow" href="https://i2.wp.com/datascienceplus.com/wp-content/uploads/2017/06/cv.png?ssl=1" target="_blank"><img sizes="(max-width: 1024px) 100vw, 1024px" src="https://i1.wp.com/datascienceplus.com/wp-content/uploads/2017/06/cv-1024x731.png?w=450&ssl=1" alt="" class="wp-image-7323" data-recalc-dims="1"></a></p>
<p>This Model is also very Smooth and Fits the data well.</p>
<h2>Conclusion</h2>
<p>Hence this was a simple overview of Cubic and Smoothing Splines and how they transform variables and add Non linearities to the Model and are more flexible and smoother than other techniques. It is better in terms of extrapolation and is more smoother.Other techniques such as Polynomial regression is very bad at extrapolation and oscillates a lot once it gets out of boundaries and it becomes very wiggly and fluctuating which shows the signs of High Variance and mostly Overfits at larger values of degree of polynomials. The main thing to remember while fitting Non linear Models to the data is that we need to do some transformations to data or the variables in order to make the model more flexible and stronger in learning Non linear Interactions between the Inputs \(X_i\) and Output \( Y \) variables.
<p>Hope you guys liked the article and make sure to like and share it. Cheers!</p>
<ul></ul>
<p><strong>
<p>Related Post</p>
<p>
<ol>
<li><a rel="nofollow" href="https://datascienceplus.com/chi-squared-test-in-r/" title="Permanent Link toChi-Squared Test – The Purpose, The Math, When and How to Implement?" target="_blank">Chi-Squared Test – The Purpose, The Math, When and How to Implement?</a></li>
<li><a rel="nofollow" href="https://datascienceplus.com/missing-value-treatment/" title="Permanent Link toMissing Value Treatment" target="_blank">Missing Value Treatment</a></li>
<li><a rel="nofollow" href="https://datascienceplus.com/r-for-publication-by-page-piccinini/" title="Permanent Link toR for Publication by Page Piccinini" target="_blank">R for Publication by Page Piccinini</a></li>
<li><a rel="nofollow" href="https://datascienceplus.com/assessing-significance-of-slopes-in-regression-models-with-interaction/" title="Permanent Link toAssessing significance of slopes in regression models with interaction" target="_blank">Assessing significance of slopes in regression models with interaction</a></li>
<li><a rel="nofollow" href="https://datascienceplus.com/first-steps-with-non-linear-regression-in-r/" title="Permanent Link toFirst steps with Non-Linear Regression in R" target="_blank">First steps with Non-Linear Regression in R</a></li>
</ol></strong>
	<p>
<div>
<div>To <strong>leave a comment</strong> for the author, please follow the link and comment on their blog: <strong><a href="https://datascienceplus.com/cubic-and-smoothing-splines-in-r/"> R Programming – DataScience+</a></strong>.</div>
<hr>
<a rel="nofollow" href="https://www.r-bloggers.com/">R-bloggers.com</a> offers <strong><a rel="nofollow" href="https://feedburner.google.com/fb/a/mailverify?uri=RBloggers">daily e-mail updates</a></strong> about <a rel="nofollow" href="https://www.r-project.org/" title="The R Project for Statistical Computing">R</a> news and <a rel="nofollow" href="https://www.r-bloggers.com/search/tutorial" title="R tutorials">tutorials</a> on topics such as: <a rel="nofollow" href="https://www.r-bloggers.com/search/data%20science" title="Data science">Data science</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/Big%20Data" title="Big Data">Big Data, <a rel="nofollow" href="https://www.r-users.com/" title="R jobs">R jobs</a>, visualization (<a rel="nofollow" href="https://www.r-bloggers.com/search/ggplot2" title="ggplot and ggplot2 tutorials">ggplot2</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/boxplot" title="Boxplots using lattice and ggplot2 tutorials">Boxplots</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/map" title="Maps and gis">maps</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/animation" title="Animation in R">animation</a>), programming (<a rel="nofollow" href="https://www.r-bloggers.com/search/RStudio" title="RStudio IDE for R">RStudio</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/sweave" title="Sweave and literate programming">Sweave</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/LaTeX" title="LaTeX in R">LaTeX</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/SQL" title="SQL and databases">SQL</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/eclipse" title="Eclipse IDE for R">Eclipse</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/git" title="git and github, Version Control System">git</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/hadoop" title="Large data in R using Hadoop">hadoop</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/Web+Scraping" title="Web Scraping of google, facebook, yahoo, twitter and more using R">Web Scraping</a>) statistics (<a rel="nofollow" href="https://www.r-bloggers.com/search/regression" title="Regressions and ANOVA analysis tutorials">regression</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/PCA" title="principal component analysis tutorial">PCA</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/time+series" title="Time series">time series</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/trading" title="finance trading">trading</a>) and more...
<img width="1" alt="" src="http://feeds.feedburner.com/~r/RBloggers/~4/XCmknAvbGFU" height="1"></a></div>