<p><div><div><div><div data-show-faces="false" data-height="21" data-width="100" data-layout="button_count" data-send="true" data-href="https://www.r-bloggers.com/hacking-statistics-or-how-i-learned-to-stop-worrying-about-calculus-and-love-stats-exercises-part-1/"></div></div><div></div></div><div></div></div>
<div>
(This article was first published on <strong><a href="http://www.r-exercises.com/2017/07/01/hacking-statistics-exercises-part-1/?utm_source=rss&utm_medium=rss&utm_campaign=hacking-statistics-exercises-part-1"> R-exercises</a></strong>, and kindly contributed to <a rel="nofollow" href="https://www.r-bloggers.com/">R-bloggers)</a> </div>
<p><a rel="nofollow" href="https://i0.wp.com/www.r-exercises.com/wp-content/uploads/2017/06/HS_part1-20.png" target="_blank"><img sizes="(max-width: 300px) 100vw, 300px" src="https://i0.wp.com/www.r-exercises.com/wp-content/uploads/2017/06/HS_part1-20-300x300.png?resize=300%2C300" alt="" class="wp-image-6728" data-recalc-dims="1"></a>Statistics are often taught in school by and for people who like Mathematics. As a consequence, in those class emphasis is put on leaning equations, solving calculus problems and creating mathematics models instead of building an intuition for probabilistic problems. But, if you read this, you know a bit of R programming and have access to a computer that is really good at computing stuff! So let’s learn how we can tackle useful statistic problems by writing simple R query and how to think in probabilistic terms.</p>
<p>This exercise set will introduce you to common distributions and simple sampling related concepts which will be useful when we’ll see more advance concept like bootstrapping and A/B testing in some future post in this series.</p>
<p>Answers to the exercises are available <a rel="nofollow" href="http://r-exercises.com/2017/07/01/hacking-statistics-solutions-part-1/" target="_blank">here</a>.</p>
<p><strong>Exercise 1</strong><br>
Use <code>rnorm()</code> to generate 100 points in the interval [0,1], then plot those points in an histogram.</p>
<p><strong>Exercise 2</strong><br>
Repeat exercise 1, but this time with, 500, 1000 and 10000 points.</p>
<p><strong>Exercise 3</strong><br>
We can see that the more points are generated, the more the histogram become symmetric and centered around 0. The reason for this is that <code>rnorm()</code> generate the point based on a function which dictate precisely what should be the proportion of points in each subinterval of [0,1] and that function has for characteristics to be symmetric, centered around 0 with two inflection points which make his shape look like a bell. That density function is called a Normal distribution and a lot of practical application use it.</p>
<p>Use the <code>dnorm()</code> function to plot the density function of a normal distribution of mean 0 and standard deviation of 1 and add it to the last histogram you plot.</p>
<p>The histograms we plotted before where discrete approximation of this continuous function. Since we deal with a random process, each bin of the histogram doesn’t fill up with the correct frequency evenly. As a consequence, it can take a lot of observation before the histogram represent the underling distribution of the random process. Here lies the biggest problem that statisticians face: can we make decisions based on a sample of size n or does a bigger sample would reveal that the random process is distributed under another density function.<br>
<strong>Exercise 4</strong><br>
We can use this shape to verify if a random process is a normal process. Another useful plot is the empirical cumulative distribution function (ECDF) which represent visually the probability that an observation is smaller than a certain value. Plot the cumulative histogram of 10000 points from a standard normal distribution, then add the ECDF curve to the plot by using the <code>pnorm()</code> function.</p>
<p><strong>Exercise 5</strong><br>
There’s a lot of distribution other than the standard normal distribution that you can find in practice. To familiarize with the shape of those function, plot the density function of those common functions:</p>
<ul>
<li>Exponential with a rate of 0.5</li>
<li>Exponential with a rate of 1</li>
<li>Exponential with a rate of 2</li>
<li>Exponential with a rate of 10</li>
<li>Gamma with a shape of 1 and a scale equal to 2</li>
<li>Gamma with a shape of 2 and a scale equal to 2</li>
<li>Gamma with a shape of 5 and a scale equal to 2</li>
<li>Gamma with a shape of 5 and a scale equal to 0.5</li>
<li>Student with 10 degree of freedom</li>
<li>Student with 5 degree of freedom</li>
<li>Student with 2 degree of freedom</li>
<li>Student with 1 degree of freedom</li>
</ul>
<p>For reference you can visit this <a rel="nofollow" href="https://en.wikibooks.org/wiki/R_Programming/Probability_Distributions" target="_blank"> page.</a></p>
<p><strong>Exercise 6</strong><br>
Repeat the steps of exercise 5, but plot the ECDF instead.</p>
<p><strong>Exercise 7</strong><br>
Now it’s time to put what we learn to test! Download <a rel="nofollow" href="http://www.r-exercises.com/wp-content/uploads/2017/06/Norm.t.data_.csv" target="_blank">this dataset</a> and try to find if those observations have the same distribution. Start by looking at the histogram of both variables in this dataset.</p>
<p><strong>Exercise 8</strong><br>
Both dataset seems symmetric and to have the same domain.</p>
<p><strong>Exercise 9</strong><br>
Use the <code>ecdf()</code> function to plot the empirical cumulative distribution function of both sample.</p>
<p><strong>Exercise 10</strong><br>
The plots indicate that there’s little difference between the distribution of both sample. Using the Kolmogorov-Smirnov test is a good way to determine if two sample share the same distribution. This test measure the maximum difference between the ecdf of both samples and compute the probability of such difference to appear when the ecdf are the same.</p>
<p>Use the <code>ks.test()</code> function to run the Kolmogorov-Smirnov test on both samples.</p>
<p>The first sample in the dataset was sampled from a Student distribution with 10 degrees of freedom, while the second was sampled from a standart normal distribution. Both density functions are quite similar and in practice using one over the other won’t make a huge difference, but some function have a heavy tail, meaning that they can create some rare events who take huge value. Those events usually won’t appear in a small sample and failing to differentiate such function for other can generate huge estimation errors. In the next post, we’ll see method to distinguish between two similar distributions.</p>
<div>
<h3>Related exercise sets:</h3>
<ol>
<li><a rel="nofollow" href="http://www.r-exercises.com/2017/06/10/density-based-clustering-exercises/" title="Density-Based Clustering Exercises" target="_blank">Density-Based Clustering Exercises </a></li>
<li><a rel="nofollow" href="http://www.r-exercises.com/2017/02/13/data-science-for-doctors-part-3-distributions/" title="Data Science for Doctors – Part 3 : Distributions" target="_blank">Data Science for Doctors – Part 3 : Distributions </a></li>
<li><a rel="nofollow" href="http://www.r-exercises.com/2017/06/16/forecasting-for-small-business-exercises-part-2/" title="Forecasting for small business Exercises (Part-2)" target="_blank">Forecasting for small business Exercises (Part-2) </a></li>
<li><a rel="nofollow" href="http://www.r-exercises.com/start-here-to-learn-r/" target="_blank">Explore all our (>1000) R exercises</a></li>
<li><a rel="nofollow" href="http://www.r-exercises.com/r-courses/" target="_blank">Find an R course using our R Course Finder</a> directory</li>
</ol>
</div>
	<p>
<div>
<div>To <strong>leave a comment</strong> for the author, please follow the link and comment on their blog: <strong><a href="http://www.r-exercises.com/2017/07/01/hacking-statistics-exercises-part-1/?utm_source=rss&utm_medium=rss&utm_campaign=hacking-statistics-exercises-part-1"> R-exercises</a></strong>.</div>
<hr>
<a rel="nofollow" href="https://www.r-bloggers.com/">R-bloggers.com</a> offers <strong><a rel="nofollow" href="https://feedburner.google.com/fb/a/mailverify?uri=RBloggers">daily e-mail updates</a></strong> about <a rel="nofollow" href="https://www.r-project.org/" title="The R Project for Statistical Computing">R</a> news and <a rel="nofollow" href="https://www.r-bloggers.com/search/tutorial" title="R tutorials">tutorials</a> on topics such as: <a rel="nofollow" href="https://www.r-bloggers.com/search/data%20science" title="Data science">Data science</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/Big%20Data" title="Big Data">Big Data, <a rel="nofollow" href="https://www.r-users.com/" title="R jobs">R jobs</a>, visualization (<a rel="nofollow" href="https://www.r-bloggers.com/search/ggplot2" title="ggplot and ggplot2 tutorials">ggplot2</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/boxplot" title="Boxplots using lattice and ggplot2 tutorials">Boxplots</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/map" title="Maps and gis">maps</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/animation" title="Animation in R">animation</a>), programming (<a rel="nofollow" href="https://www.r-bloggers.com/search/RStudio" title="RStudio IDE for R">RStudio</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/sweave" title="Sweave and literate programming">Sweave</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/LaTeX" title="LaTeX in R">LaTeX</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/SQL" title="SQL and databases">SQL</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/eclipse" title="Eclipse IDE for R">Eclipse</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/git" title="git and github, Version Control System">git</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/hadoop" title="Large data in R using Hadoop">hadoop</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/Web+Scraping" title="Web Scraping of google, facebook, yahoo, twitter and more using R">Web Scraping</a>) statistics (<a rel="nofollow" href="https://www.r-bloggers.com/search/regression" title="Regressions and ANOVA analysis tutorials">regression</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/PCA" title="principal component analysis tutorial">PCA</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/time+series" title="Time series">time series</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/trading" title="finance trading">trading</a>) and more...
<img width="1" alt="" src="http://feeds.feedburner.com/~r/RBloggers/~4/2ldGFatQm8k" height="1"></a></div>