<p><div><div><div><div data-width="100" data-show-faces="false" data-send="true" data-layout="button_count" data-height="21" data-href="https://www.r-bloggers.com/r-for-beginners-basic-graphics-code-to-produce-informative-graphs-part-two-working-with-big-data/"></div></div><div></div></div><div></div></div>
<div>
(This article was first published on <strong><a href="https://dmwiig.net/2017/01/16/r-for-beginners-basic-graphics-code-to-produce-informative-graphs-part-two-working-with-big-data/"> r – R Statistics and Programming</a></strong>, and kindly contributed to <a rel="nofollow" href="https://www.r-bloggers.com/">R-bloggers)</a> </div>
<p><strong>R for beginners: Some basic graphics code to produce informative graphs, part two, working with big data</strong></p>
<p>A tutorial by D. M. Wiig</p>
<p>In part one of this tutorial I discussed the use of R code to produce 3d scatterplots. This is a useful way to produce visual results of multi- variate linear regression models. While visual displays using scatterplots is a useful tool when using most datasets it becomes much more of a challenge when analyzing big data. These types of databases can contain tens of thousands or even millions of cases and hundreds of variables.</p>
<p>Working with these types of data sets involves a number of challenges. If a researcher is interested in using visual presentations such as scatterplots this can be a daunting task. I will start by discussing how scatterplots can be used to provide meaningful visual representation of the relationship between two variables in a simple bivariate model.</p>
<p>To start I will construct a theoretical data set that consists of ten thousand x and y pairs of observations. One method that can be used to accomplish this is to use the R <em>rnorm()</em> function to generate a set of random integers with a specified mean and standard deviation. I will use this function to generate both the x and y variable.</p>
<p>Before starting this tutorial make sure that R is running and that the <em>datasets</em>, <em>LSD,</em> and <em>stats</em> packages have been installed. Use the following code to generate the x and y values such that the mean of x= 10 with a standard deviation of 7, and the mean of y=7 with a standard deviation of 3:</p>
<h6><strong>##############################################</strong><br>
<strong> ## make sure package LSD is loaded</strong><br>
<strong> ##</strong><br>
<strong> library(LSD)</strong><br>
<strong> x &lt;- rnorm(50000, mean=10, sd=15)   # # generates x values #stores results in variable x</strong><br>
<strong> y &lt;- rnorm(50000, mean=7, sd=3)    ## generates y values #stores results in variable y</strong><br>
<strong> ####################################################</strong></h6>
<p>Now the scatterplot can be created using the code:</p>
<h6><strong>##############################################</strong><br>
<strong> ## plot randomly generated x and y values</strong><br>
<strong> ##</strong><br>
<strong> plot(x,y, main=”Scatterplot of 50,000 points”)</strong><br>
<strong> ####################################################</strong></h6>
<p><a target="_blank" rel="nofollow" href="https://i1.wp.com/dmwiig.files.wordpress.com/2017/01/screenshot-graphics-device-number-2-active-e28093-27rkward271.png?ssl=1"><img sizes="(max-width: 300px) 100vw, 300px" data-image-description="" data-orig-file="https://dmwiig.files.wordpress.com/2017/01/screenshot-graphics-device-number-2-active-e28093-27rkward271.png" data-orig-size="771,668" data-image-title="screenshot-graphics-device-number-2-active-%27rkward%27" class="wp-image-284" data-comments-opened="1" data-recalc-dims="1" data-attachment-id="284" data-permalink="https://dmwiig.net/2017/01/16/r-for-beginners-basic-graphics-code-to-produce-informative-graphs-part-two-working-with-big-data/screenshot-graphics-device-number-2-active-%27rkward%27-3/" alt="screenshot-graphics-device-number-2-active-%27rkward%27" data-medium-file="https://i1.wp.com/dmwiig.files.wordpress.com/2017/01/screenshot-graphics-device-number-2-active-e28093-27rkward271.png?resize=300%2C260&amp;ssl=1" data-large-file="https://dmwiig.files.wordpress.com/2017/01/screenshot-graphics-device-number-2-active-e28093-27rkward271.png?w=474" data-image-meta="{&quot;aperture&quot;:&quot;0&quot;,&quot;credit&quot;:&quot;&quot;,&quot;camera&quot;:&quot;&quot;,&quot;caption&quot;:&quot;&quot;,&quot;created_timestamp&quot;:&quot;0&quot;,&quot;copyright&quot;:&quot;&quot;,&quot;focal_length&quot;:&quot;0&quot;,&quot;iso&quot;:&quot;0&quot;,&quot;shutter_speed&quot;:&quot;0&quot;,&quot;title&quot;:&quot;&quot;,&quot;orientation&quot;:&quot;0&quot;}" src="https://i1.wp.com/dmwiig.files.wordpress.com/2017/01/screenshot-graphics-device-number-2-active-e28093-27rkward271.png?resize=300%2C260&ssl=1"></a></p>
<p>As can be seen the resulting plot is mostly a mass of black with relatively few individual x and y points shown other than the outliers.  We can do a quick histogram on the x values and the y values to check the normality of the resulting distribution. This shown in the code below:<br>
<strong>####################################################</strong><br>
<strong> ## show histogram of x and y distribution</strong><br>
<strong> ####################################################</strong><br>
<strong> hist(x)   ## histogram for x mean=10; sd=15; n=50,000</strong><br>
<strong> ##</strong><br>
<strong> hist(y)   ## histogram for y mean=7; sd=3; n-50,000</strong><br>
<strong> ####################################################</strong></p>
<p><a target="_blank" rel="nofollow" href="https://i2.wp.com/dmwiig.files.wordpress.com/2017/01/screenshot-graphics-device-number-2-active-e28093-27rkward27-5.png?ssl=1"><img sizes="(max-width: 300px) 100vw, 300px" data-image-description="" data-orig-file="https://dmwiig.files.wordpress.com/2017/01/screenshot-graphics-device-number-2-active-e28093-27rkward27-5.png" data-orig-size="771,668" data-image-title="screenshot-graphics-device-number-2-active-%27rkward%27-5" data-comments-opened="1" class="wp-image-285" data-recalc-dims="1" data-attachment-id="285" data-permalink="https://dmwiig.net/2017/01/16/r-for-beginners-basic-graphics-code-to-produce-informative-graphs-part-two-working-with-big-data/screenshot-graphics-device-number-2-active-%27rkward%27-5/" alt="screenshot-graphics-device-number-2-active-%27rkward%27-5" data-medium-file="https://i2.wp.com/dmwiig.files.wordpress.com/2017/01/screenshot-graphics-device-number-2-active-e28093-27rkward27-5.png?resize=300%2C260&amp;ssl=1" data-large-file="https://dmwiig.files.wordpress.com/2017/01/screenshot-graphics-device-number-2-active-e28093-27rkward27-5.png?w=474" data-image-meta="{&quot;aperture&quot;:&quot;0&quot;,&quot;credit&quot;:&quot;&quot;,&quot;camera&quot;:&quot;&quot;,&quot;caption&quot;:&quot;&quot;,&quot;created_timestamp&quot;:&quot;0&quot;,&quot;copyright&quot;:&quot;&quot;,&quot;focal_length&quot;:&quot;0&quot;,&quot;iso&quot;:&quot;0&quot;,&quot;shutter_speed&quot;:&quot;0&quot;,&quot;title&quot;:&quot;&quot;,&quot;orientation&quot;:&quot;0&quot;}" src="https://i2.wp.com/dmwiig.files.wordpress.com/2017/01/screenshot-graphics-device-number-2-active-e28093-27rkward27-5.png?resize=300%2C260&ssl=1"></a></p>
<p><a target="_blank" rel="nofollow" href="https://i0.wp.com/dmwiig.files.wordpress.com/2017/01/screenshot-graphics-device-number-2-active-e28093-27rkward27-4.png?ssl=1"><img sizes="(max-width: 300px) 100vw, 300px" data-image-description="" data-orig-file="https://dmwiig.files.wordpress.com/2017/01/screenshot-graphics-device-number-2-active-e28093-27rkward27-4.png" data-orig-size="771,668" data-image-title="screenshot-graphics-device-number-2-active-%27rkward%27-4" data-comments-opened="1" class="wp-image-286" data-recalc-dims="1" data-attachment-id="286" data-permalink="https://dmwiig.net/2017/01/16/r-for-beginners-basic-graphics-code-to-produce-informative-graphs-part-two-working-with-big-data/screenshot-graphics-device-number-2-active-%27rkward%27-4/" alt="screenshot-graphics-device-number-2-active-%27rkward%27-4" data-medium-file="https://i0.wp.com/dmwiig.files.wordpress.com/2017/01/screenshot-graphics-device-number-2-active-e28093-27rkward27-4.png?resize=300%2C260&amp;ssl=1" data-large-file="https://dmwiig.files.wordpress.com/2017/01/screenshot-graphics-device-number-2-active-e28093-27rkward27-4.png?w=474" data-image-meta="{&quot;aperture&quot;:&quot;0&quot;,&quot;credit&quot;:&quot;&quot;,&quot;camera&quot;:&quot;&quot;,&quot;caption&quot;:&quot;&quot;,&quot;created_timestamp&quot;:&quot;0&quot;,&quot;copyright&quot;:&quot;&quot;,&quot;focal_length&quot;:&quot;0&quot;,&quot;iso&quot;:&quot;0&quot;,&quot;shutter_speed&quot;:&quot;0&quot;,&quot;title&quot;:&quot;&quot;,&quot;orientation&quot;:&quot;0&quot;}" src="https://i0.wp.com/dmwiig.files.wordpress.com/2017/01/screenshot-graphics-device-number-2-active-e28093-27rkward27-4.png?resize=300%2C260&ssl=1"></a></p>
<p>The histogram shows a normal distribution for both variables. As is expected, in the x vs. y scatterplot the center mass of points is located at the x = 10; y=7 coordinate of the graph as this coordinate contains the mean of each distribution. A more meaningful scatterplot of the dataset can be generated using a the R functions <em>smoothScatter()</em> and <em>heatscatter().</em> The <em>smoothScatter()</em> function is located in the <em>graphics</em> package and the <em>heatscatter()</em> function is located in the <em>LSD</em> package.</p>
<p>The<em> smoothScatter() </em>function creates a smoothed color density representation of a scatterplot. This allows for a better visual representation of the density of individual values for the x and y pairs. To use the <em>smoothScatter()</em> function with the large dataset created above use the following code:</p>
<h6><strong>##############################################</strong><br>
<strong> ## use smoothScatter function to visualize the scatterplot of #50,000 x ## and y values</strong><br>
<strong> ## the x and y values should still be in the workspace as #created  above with the rnorm() function</strong><br>
<strong> ##</strong><br>
<strong> smoothScatter(x, y, main = “Smoothed Color Density Representation of 50,000 (x,y) Coordinates”)</strong><br>
<strong> ##</strong><br>
<strong> ####################################################</strong></h6>
<p><a target="_blank" rel="nofollow" href="https://i0.wp.com/dmwiig.files.wordpress.com/2017/01/screenshot-graphics-device-number-2-active-e28093-27rkward27-6.png?ssl=1"><img sizes="(max-width: 300px) 100vw, 300px" data-image-description="" data-orig-file="https://dmwiig.files.wordpress.com/2017/01/screenshot-graphics-device-number-2-active-e28093-27rkward27-6.png" data-orig-size="771,668" data-image-title="screenshot-graphics-device-number-2-active-%27rkward%27-6" data-comments-opened="1" class="wp-image-287" data-recalc-dims="1" data-attachment-id="287" data-permalink="https://dmwiig.net/2017/01/16/r-for-beginners-basic-graphics-code-to-produce-informative-graphs-part-two-working-with-big-data/screenshot-graphics-device-number-2-active-%27rkward%27-6/" alt="screenshot-graphics-device-number-2-active-%27rkward%27-6" data-medium-file="https://i0.wp.com/dmwiig.files.wordpress.com/2017/01/screenshot-graphics-device-number-2-active-e28093-27rkward27-6.png?resize=300%2C260&amp;ssl=1" data-large-file="https://dmwiig.files.wordpress.com/2017/01/screenshot-graphics-device-number-2-active-e28093-27rkward27-6.png?w=474" data-image-meta="{&quot;aperture&quot;:&quot;0&quot;,&quot;credit&quot;:&quot;&quot;,&quot;camera&quot;:&quot;&quot;,&quot;caption&quot;:&quot;&quot;,&quot;created_timestamp&quot;:&quot;0&quot;,&quot;copyright&quot;:&quot;&quot;,&quot;focal_length&quot;:&quot;0&quot;,&quot;iso&quot;:&quot;0&quot;,&quot;shutter_speed&quot;:&quot;0&quot;,&quot;title&quot;:&quot;&quot;,&quot;orientation&quot;:&quot;0&quot;}" src="https://i0.wp.com/dmwiig.files.wordpress.com/2017/01/screenshot-graphics-device-number-2-active-e28093-27rkward27-6.png?resize=300%2C260&ssl=1"></a></p>
<p>The resulting plot shows several bands of density surrounding the coordinates x=10, y=7 which are the means of the two distributions rather than an indistinguishable mass of dark points.</p>
<p>Similar results can be obtained using the <em>heatscatter()</em> function. This function produces a similar visual based on densities that are represented as color bands. As indicated above, the LSD package should be installed and loaded to access the <em>heatscatter()</em> function. The resulting code is:</p>
<h6><strong>##############################################</strong><br>
<strong> ## produce a heatscatter plot of x and y</strong><br>
<strong> ##</strong><br>
<strong> library(LSD)</strong><br>
<strong> heatscatter(x,y, main=”Heat Color Density Representation of 50,000 (x, y) Coordinates”) ## function heatscatter() with #n=50,000</strong><br>
<strong> ####################################################</strong></h6>
<p><a target="_blank" rel="nofollow" href="https://i2.wp.com/dmwiig.files.wordpress.com/2017/01/screenshot-graphics-device-number-2-active-e28093-27rkward27-7.png?ssl=1"><img sizes="(max-width: 300px) 100vw, 300px" data-image-description="" data-orig-file="https://dmwiig.files.wordpress.com/2017/01/screenshot-graphics-device-number-2-active-e28093-27rkward27-7.png" data-orig-size="771,668" data-image-title="screenshot-graphics-device-number-2-active-%27rkward%27-7" data-comments-opened="1" class="wp-image-288" data-recalc-dims="1" data-attachment-id="288" data-permalink="https://dmwiig.net/2017/01/16/r-for-beginners-basic-graphics-code-to-produce-informative-graphs-part-two-working-with-big-data/screenshot-graphics-device-number-2-active-%27rkward%27-7/" alt="screenshot-graphics-device-number-2-active-%27rkward%27-7" data-medium-file="https://i2.wp.com/dmwiig.files.wordpress.com/2017/01/screenshot-graphics-device-number-2-active-e28093-27rkward27-7.png?resize=300%2C260&amp;ssl=1" data-large-file="https://dmwiig.files.wordpress.com/2017/01/screenshot-graphics-device-number-2-active-e28093-27rkward27-7.png?w=474" data-image-meta="{&quot;aperture&quot;:&quot;0&quot;,&quot;credit&quot;:&quot;&quot;,&quot;camera&quot;:&quot;&quot;,&quot;caption&quot;:&quot;&quot;,&quot;created_timestamp&quot;:&quot;0&quot;,&quot;copyright&quot;:&quot;&quot;,&quot;focal_length&quot;:&quot;0&quot;,&quot;iso&quot;:&quot;0&quot;,&quot;shutter_speed&quot;:&quot;0&quot;,&quot;title&quot;:&quot;&quot;,&quot;orientation&quot;:&quot;0&quot;}" src="https://i2.wp.com/dmwiig.files.wordpress.com/2017/01/screenshot-graphics-device-number-2-active-e28093-27rkward27-7.png?resize=300%2C260&ssl=1"></a></p>
<p>In comparing this plot with the <em>smoothScatter()</em> plot one can more clearly see the distinctive density bands surrounding the coordinates x=10, y=7. You may also notice depending on the computer you are using that there is a noticeably longer processing time required to produce the <em>heatscatter()</em> plot.</p>
<p>This tutorial has hopefully provided some useful information relative to visual displays of large data sets. In the next segment I will discuss how these techniques can be used on a live database containing millions of cases.</p>
<p> <a target="_blank" rel="nofollow" href="http://feeds.wordpress.com/1.0/gocomments/dmwiig.wordpress.com/281/"><img alt="" src="http://feeds.wordpress.com/1.0/comments/dmwiig.wordpress.com/281/" border="0"></a> <img alt="" src="https://i1.wp.com/pixel.wp.com/b.gif?resize=1%2C1&ssl=1" border="0" data-recalc-dims="1"></p>
<p>
<div>
<div>To <strong>leave a comment</strong> for the author, please follow the link and comment on their blog: <strong><a href="https://dmwiig.net/2017/01/16/r-for-beginners-basic-graphics-code-to-produce-informative-graphs-part-two-working-with-big-data/"> r – R Statistics and Programming</a></strong>.</div>
<hr>
<a rel="nofollow" href="https://www.r-bloggers.com/">R-bloggers.com</a> offers <strong><a rel="nofollow" href="https://feedburner.google.com/fb/a/mailverify?uri=RBloggers">daily e-mail updates</a></strong> about <a title="The R Project for Statistical Computing" rel="nofollow" href="https://www.r-project.org/">R</a> news and <a title="R tutorials" rel="nofollow" href="https://www.r-bloggers.com/search/tutorial">tutorials</a> on topics such as: <a title="Data science" rel="nofollow" href="https://www.r-bloggers.com/search/data%20science">Data science</a>, <a title="Big Data" rel="nofollow" href="https://www.r-bloggers.com/search/Big%20Data">Big Data, <a title="R jobs" rel="nofollow" href="https://www.r-users.com/">R jobs</a>, visualization (<a title="ggplot and ggplot2 tutorials" rel="nofollow" href="https://www.r-bloggers.com/search/ggplot2">ggplot2</a>, <a title="Boxplots using lattice and ggplot2 tutorials" rel="nofollow" href="https://www.r-bloggers.com/search/boxplot">Boxplots</a>, <a title="Maps and gis" rel="nofollow" href="https://www.r-bloggers.com/search/map">maps</a>, <a title="Animation in R" rel="nofollow" href="https://www.r-bloggers.com/search/animation">animation</a>), programming (<a title="RStudio IDE for R" rel="nofollow" href="https://www.r-bloggers.com/search/RStudio">RStudio</a>, <a title="Sweave and literate programming" rel="nofollow" href="https://www.r-bloggers.com/search/sweave">Sweave</a>, <a title="LaTeX in R" rel="nofollow" href="https://www.r-bloggers.com/search/LaTeX">LaTeX</a>, <a title="SQL and databases" rel="nofollow" href="https://www.r-bloggers.com/search/SQL">SQL</a>, <a title="Eclipse IDE for R" rel="nofollow" href="https://www.r-bloggers.com/search/eclipse">Eclipse</a>, <a title="git and github, Version Control System" rel="nofollow" href="https://www.r-bloggers.com/search/git">git</a>, <a title="Large data in R using Hadoop" rel="nofollow" href="https://www.r-bloggers.com/search/hadoop">hadoop</a>, <a title="Web Scraping of google, facebook, yahoo, twitter and more using R" rel="nofollow" href="https://www.r-bloggers.com/search/Web+Scraping">Web Scraping</a>) statistics (<a title="Regressions and ANOVA analysis tutorials" rel="nofollow" href="https://www.r-bloggers.com/search/regression">regression</a>, <a title="principal component analysis tutorial" rel="nofollow" href="https://www.r-bloggers.com/search/PCA">PCA</a>, <a title="Time series" rel="nofollow" href="https://www.r-bloggers.com/search/time+series">time series</a>, <a title="finance trading" rel="nofollow" href="https://www.r-bloggers.com/search/trading">trading</a>) and more...
<img height="1" alt="" width="1" src="http://feeds.feedburner.com/~r/RBloggers/~4/GJTKrvVfu9s"></a></div>