<p><div><div><div><div data-width="100" data-show-faces="false" data-send="true" data-layout="button_count" data-height="21" data-href="http://www.r-bloggers.com/calculating-memory-requirements/"></div></div><div><a data-url="http://www.r-bloggers.com/calculating-memory-requirements/" data-counturl="http://www.r-bloggers.com/calculating-memory-requirements/" data-count="horizontal" data-text="Calculating Memory Requirements" class="twitter-share-button" data-via="rbloggers" href="https://twitter.com/share"></a></div><div></div></div><div></div></div>
<div>
(This article was first published on <strong><a href="http://danielmarcelino.github.io/blog/2016/calculating-memory-requirements.html"> Daniel's Blog</a></strong>, and kindly contributed to <a rel="nofollow" href="http://www.r-bloggers.com/">R-bloggers)</a> </div>
<p>I had a conversation with people at the office about size and memory requirements that a computer must have to deal with a data frame. It started like this: suppose you have a data frame with 2,000,000 rows and 250 columns, all of which are numeric data (2,000,000 × 250 × 8 bytes/numeric). Roughly saying, how much memory is required to store this data?</p>
<p>We can calculate the proximate memory usage using number of rows and columns within the data:</p>
<figure>
<pre><code data-lang="r"><span># bytes
</span><span>></span><span> </span><span>2000000</span><span>*</span><span>250</span><span>*</span><span>8</span><span> </span><span>[</span><span>1</span><span>]</span><span> </span><span>4000000000</span></code></pre>
</figure>
<figure>
<pre><code data-lang="r"><span>></span><span> </span><span># bytes/MB
</span><span>></span><span> </span><span>2000000</span><span>*</span><span>250</span><span>*</span><span>8</span><span>/</span><span>2</span><span>^</span><span>{</span><span>20</span><span>}</span><span>
</span><span>[</span><span>1</span><span>]</span><span> </span><span>3814.697</span></code></pre>
</figure>
<figure>
<pre><code data-lang="r"><span>></span><span> </span><span># MB
</span><span>></span><span> </span><span>round</span><span>(</span><span>2000000</span><span>*</span><span>250</span><span>*</span><span>8</span><span>/</span><span>2</span><span>^</span><span>{</span><span>20</span><span>},</span><span>2</span><span>)</span><span>
</span><span>[</span><span>1</span><span>]</span><span> </span><span>3814.7</span></code></pre>
</figure>
<figure>
<pre><code data-lang="r"><span>></span><span> </span><span># GB
</span><span>></span><span> </span><span>round</span><span>(</span><span>2000000</span><span>*</span><span>250</span><span>*</span><span>8</span><span>/</span><span>2</span><span>^</span><span>{</span><span>20</span><span>}</span><span>/</span><span>1024</span><span>,</span><span> </span><span>2</span><span>)</span><span>
</span><span>[</span><span>1</span><span>]</span><span> </span><span>3.73</span></code></pre>
</figure>
<p>We also can apply the same approach on other types of data with attention to number of bytes used to store different data types, as character, factors etc. We also can use 10 rather than 8 as the bytes/numeric coefficient, and triple that to figure out how much contiguous space is needed. If the data frame is just character data, the ratio is going to be around 1.1, and reading it in as factor data might be even below this.</p>
<p>Finally, if you want to get a solid understanding of R’s memory management, you will find a section in the Hadley Wickham’s <a target="_blank" href="http://adv-r.had.co.nz/memory.html">book</a> very useful.</p>
<p><div><div><div><div data-width="55" data-show-faces="false" data-send="true" data-layout="box_count" data-height="62" data-href="http://www.r-bloggers.com/calculating-memory-requirements/"></div></div><div><a data-url="http://www.r-bloggers.com/calculating-memory-requirements/" data-counturl="http://www.r-bloggers.com/calculating-memory-requirements/" data-count="vertical" data-text="Calculating Memory Requirements" class="twitter-share-button" data-via="rbloggers" href="https://twitter.com/share"></a></div><div></div></div><div></div></div>
<div>
<div>To <strong>leave a comment</strong> for the author, please follow the link and comment on their blog: <strong><a href="http://danielmarcelino.github.io/blog/2016/calculating-memory-requirements.html"> Daniel's Blog</a></strong>.</div>
<hr>
<a rel="nofollow" href="http://www.r-bloggers.com/">R-bloggers.com</a> offers <strong><a rel="nofollow" href="http://feedburner.google.com/fb/a/mailverify?uri=RBloggers">daily e-mail updates</a></strong> about <a title="The R Project for Statistical Computing" rel="nofollow" href="http://www.r-project.org/">R</a> news and <a title="R tutorials" rel="nofollow" href="http://www.r-bloggers.com/?s=tutorial">tutorials</a> on topics such as: <a title="Data science" rel="nofollow" href="http://www.r-bloggers.com/?s=data%20science">Data science</a>, <a title="Big Data" rel="nofollow" href="http://www.r-bloggers.com/?s=Big%20Data">Big Data, <a title="R jobs" rel="nofollow" href="http://www.r-users.com/">R jobs</a>, visualization (<a title="ggplot and ggplot2 tutorials" rel="nofollow" href="http://www.r-bloggers.com/?s=ggplot2">ggplot2</a>, <a title="Boxplots using lattice and ggplot2 tutorials" rel="nofollow" href="http://www.r-bloggers.com/?s=boxplot">Boxplots</a>, <a title="Maps and gis" rel="nofollow" href="http://www.r-bloggers.com/?s=map">maps</a>, <a title="Animation in R" rel="nofollow" href="http://www.r-bloggers.com/?s=animation">animation</a>), programming (<a title="RStudio IDE for R" rel="nofollow" href="http://www.r-bloggers.com/?s=RStudio">RStudio</a>, <a title="Sweave and literate programming" rel="nofollow" href="http://www.r-bloggers.com/?s=sweave">Sweave</a>, <a title="LaTeX in R" rel="nofollow" href="http://www.r-bloggers.com/?s=LaTeX">LaTeX</a>, <a title="SQL and databases" rel="nofollow" href="http://www.r-bloggers.com/?s=SQL">SQL</a>, <a title="Eclipse IDE for R" rel="nofollow" href="http://www.r-bloggers.com/?s=eclipse">Eclipse</a>, <a title="git and github, Version Control System" rel="nofollow" href="http://www.r-bloggers.com/?s=git">git</a>, <a title="Large data in R using Hadoop" rel="nofollow" href="http://www.r-bloggers.com/?s=hadoop">hadoop</a>, <a title="Web Scraping of google, facebook, yahoo, twitter and more using R" rel="nofollow" href="http://www.r-bloggers.com/?s=Web+Scraping">Web Scraping</a>) statistics (<a title="Regressions and ANOVA analysis tutorials" rel="nofollow" href="http://www.r-bloggers.com/?s=regression">regression</a>, <a title="principal component analysis tutorial" rel="nofollow" href="http://www.r-bloggers.com/?s=PCA">PCA</a>, <a title="Time series" rel="nofollow" href="http://www.r-bloggers.com/?s=time+series">time series</a>, <a title="finance trading" rel="nofollow" href="http://www.r-bloggers.com/?s=trading">trading</a>) and more...
<img height="1" alt="" width="1" src="http://feeds.feedburner.com/~r/RBloggers/~4/hkN76YHtudY"></a></div>