<p><div><div><div><div data-width="100" data-show-faces="false" data-send="true" data-layout="button_count" data-height="21" data-href="https://www.r-bloggers.com/trading-strategy-making-the-most-of-the-out-of-sample-data/"></div></div><div><a data-url="https://www.r-bloggers.com/trading-strategy-making-the-most-of-the-out-of-sample-data/" data-counturl="https://www.r-bloggers.com/trading-strategy-making-the-most-of-the-out-of-sample-data/" data-count="horizontal" data-text="Trading strategy: Making the most of the out of sample data" class="twitter-share-button" data-via="rbloggers" href="https://twitter.com/share"></a></div><div></div></div><div></div></div>
<div>
(This article was first published on <strong><a href="http://www.thertrader.com/2016/08/19/trading-strategy-making-the-most-of-the-out-of-sample-data/"> R – The R Trader</a></strong>, and kindly contributed to <a rel="nofollow" href="https://www.r-bloggers.com/">R-bloggers)</a> </div>
<p>When testing trading strategies a common approach is to divide the initial data set into <strong>in sample</strong> data: the part of the data designed to calibrate the model and <strong>out of sample</strong> data: the part of the data used to validate the calibration and ensure that the performance created in sample will be reflected in the real world. As a rule of thumb around 70% of the initial data can be used for calibration (i.e. in sample) and 30% for validation (i.e. out of sample). Then a comparison of the in and out of sample data help to decide whether the model is robust enough. This post aims at going a step further and provides a statistical method to decide whether the out of sample data is in line with what was created in sample.</p>
<p>In the chart below the blue area represents the out of sample performance for one of my strategies.</p>
<p><a target="_blank" rel="attachment wp-att-1393" href="https://i0.wp.com/www.thertrader.com/wp-content/uploads/2016/05/system1Performance.png"><img sizes="(max-width: 326px) 100vw, 326px" alt="system1Performance" class="wp-image-1393" src="https://i0.wp.com/www.thertrader.com/wp-content/uploads/2016/05/system1Performance-300x182.png?resize=326%2C198" data-recalc-dims="1"></a></p>
<p>A simple visual inspection reveals a good fit between the in and out of sample performance but what degree of confidence do I have in this? At this stage not much and this is the issue. What is truly needed is a measure of similarity between the in and out of sample data sets. In statistical terms this could be translated as the likelihood that the in and out of sample performance figures coming from the same distribution. There is a non-parametric statistical test that does exactly this: the <strong>Kruskall-Wallis Test</strong>. A good definition of this test could be found on <a target="_blank" href="http://www.r-tutor.com/elementary-statistics/non-parametric-methods/kruskal-wallis-test">R-Tutor</a> “A collection of data samples are <span>independent </span>if they come from unrelated populations and the samples do not affect each other. Using the <span>Kruskal-Wallis</span> <span>Test</span>, we can decide whether the population distributions are identical <span>without</span> assuming them to follow the normal distribution.” The added benefit of this test is not assuming a normal distribution.</p>
<p>It exists other tests of the same nature that could fit into that framework. The <strong>Mann-Whitney-Wilcoxon</strong> test or the <strong>Kolmogorov-Smirnov</strong> tests would perfectly suits the framework describes here however this is beyond the scope of this article to discuss the pros and cons of each of these tests. A good description along with R examples can be found <a target="_blank" href="http://www.r-tutor.com/elementary-statistics/non-parametric-methods/mann-whitney-wilcoxon-test">here</a>.</p>
<p>Here’s the code used to generate the chart above and the analysis:</p>
<pre>
################################################
## Making the most of the OOS data
##
## thertrader@gmail.com - Aug. 2016
################################################
library(xts)
library(PerformanceAnalytics)

thePath &lt;- &quot;myPath&quot; #change this
theFile &lt;- &quot;data.csv&quot; 
data &lt;- read.csv(paste0(thePath,theFile),header=TRUE,sep=&quot;,&quot;)
data &lt;- xts(data[,2],order.by=as.Date(as.character(data[,1]),format = &quot;%d/%m/%Y&quot;))

##----- Strategy's Chart
par(mex=0.8,cex=1)
thePeriod &lt;- c(&quot;2012-02/2016-05&quot;)
chart.TimeSeries(cumsum(data),
 main = &quot;System 1&quot;,
 ylab=&quot;&quot;,
 period.areas = thePeriod,
 grid.color = &quot;lightgray&quot;,
 period.color = &quot;slategray1&quot;)

##----- Kruskal tests
pValue &lt;- NULL
i &lt;- 1
while (i &lt; 1000){
 isSample &lt;- sample(isData,length(osData))
 pValue &lt;- rbind(pValue,kruskal.test(list(osData, isSample))$p.value)
 i &lt;- i + 1
}

##----- Mean of p-values 
mean(pValue)
</pre>
<p>In the example above the in sample period is longer than the out of sample period therefore I randomly created 1000 subsets of the in sample data each of them having the same length as the out of sample data. Then I tested each in sample subset against the out of sample data and I recorded the p-values. This process creates not a single p-value for the Kruskall-Wallis test but a distribution making the analysis more robust.<b> </b>In this example the mean of the p-values is well above zero (0.478) indicating that the null hypothesis should be accepted: there are strong evidences that the in and out of sample data is coming from the same distribution.</p>
<p>As usual what is presented in this post is a toy example that only scratches the surface of the problem and should be tailored to individual needs. However I think it proposes an interesting and rational statistical framework to evaluate out of sample results.</p>
<p><div><div><div><div data-width="55" data-show-faces="false" data-send="true" data-layout="box_count" data-height="62" data-href="https://www.r-bloggers.com/trading-strategy-making-the-most-of-the-out-of-sample-data/"></div></div><div><a data-url="https://www.r-bloggers.com/trading-strategy-making-the-most-of-the-out-of-sample-data/" data-counturl="https://www.r-bloggers.com/trading-strategy-making-the-most-of-the-out-of-sample-data/" data-count="vertical" data-text="Trading strategy: Making the most of the out of sample data" class="twitter-share-button" data-via="rbloggers" href="https://twitter.com/share"></a></div><div></div></div><div></div></div>
<div>
<div>To <strong>leave a comment</strong> for the author, please follow the link and comment on their blog: <strong><a href="http://www.thertrader.com/2016/08/19/trading-strategy-making-the-most-of-the-out-of-sample-data/"> R – The R Trader</a></strong>.</div>
<hr>
<a rel="nofollow" href="https://www.r-bloggers.com/">R-bloggers.com</a> offers <strong><a rel="nofollow" href="https://feedburner.google.com/fb/a/mailverify?uri=RBloggers">daily e-mail updates</a></strong> about <a title="The R Project for Statistical Computing" rel="nofollow" href="https://www.r-project.org/">R</a> news and <a title="R tutorials" rel="nofollow" href="https://www.r-bloggers.com/?s=tutorial">tutorials</a> on topics such as: <a title="Data science" rel="nofollow" href="https://www.r-bloggers.com/?s=data%20science">Data science</a>, <a title="Big Data" rel="nofollow" href="https://www.r-bloggers.com/?s=Big%20Data">Big Data, <a title="R jobs" rel="nofollow" href="https://www.r-users.com/">R jobs</a>, visualization (<a title="ggplot and ggplot2 tutorials" rel="nofollow" href="https://www.r-bloggers.com/?s=ggplot2">ggplot2</a>, <a title="Boxplots using lattice and ggplot2 tutorials" rel="nofollow" href="https://www.r-bloggers.com/?s=boxplot">Boxplots</a>, <a title="Maps and gis" rel="nofollow" href="https://www.r-bloggers.com/?s=map">maps</a>, <a title="Animation in R" rel="nofollow" href="https://www.r-bloggers.com/?s=animation">animation</a>), programming (<a title="RStudio IDE for R" rel="nofollow" href="https://www.r-bloggers.com/?s=RStudio">RStudio</a>, <a title="Sweave and literate programming" rel="nofollow" href="https://www.r-bloggers.com/?s=sweave">Sweave</a>, <a title="LaTeX in R" rel="nofollow" href="https://www.r-bloggers.com/?s=LaTeX">LaTeX</a>, <a title="SQL and databases" rel="nofollow" href="https://www.r-bloggers.com/?s=SQL">SQL</a>, <a title="Eclipse IDE for R" rel="nofollow" href="https://www.r-bloggers.com/?s=eclipse">Eclipse</a>, <a title="git and github, Version Control System" rel="nofollow" href="https://www.r-bloggers.com/?s=git">git</a>, <a title="Large data in R using Hadoop" rel="nofollow" href="https://www.r-bloggers.com/?s=hadoop">hadoop</a>, <a title="Web Scraping of google, facebook, yahoo, twitter and more using R" rel="nofollow" href="https://www.r-bloggers.com/?s=Web+Scraping">Web Scraping</a>) statistics (<a title="Regressions and ANOVA analysis tutorials" rel="nofollow" href="https://www.r-bloggers.com/?s=regression">regression</a>, <a title="principal component analysis tutorial" rel="nofollow" href="https://www.r-bloggers.com/?s=PCA">PCA</a>, <a title="Time series" rel="nofollow" href="https://www.r-bloggers.com/?s=time+series">time series</a>, <a title="finance trading" rel="nofollow" href="https://www.r-bloggers.com/?s=trading">trading</a>) and more...
<img height="1" alt="" width="1" src="http://feeds.feedburner.com/~r/RBloggers/~4/W7eQUFUpaGs"></a></div>