<p><div><div><div><div data-width="100" data-show-faces="false" data-send="true" data-layout="button_count" data-height="21" data-href="http://www.r-bloggers.com/red-herring-bites/"></div></div><div><a data-url="http://www.r-bloggers.com/red-herring-bites/" data-counturl="http://www.r-bloggers.com/red-herring-bites/" data-count="horizontal" data-text="Red herring bites" class="twitter-share-button" data-via="rbloggers" href="https://twitter.com/share"></a></div><div></div></div><div></div></div>
<div>
(This article was first published on <strong><a href="http://blog.h2o.ai/2016/05/red-herring-bites/"> R-Bloggers – H2O blog</a></strong>, and kindly contributed to <a rel="nofollow" href="http://www.r-bloggers.com/">R-bloggers)</a> </div>
<p>At the <a target="_blank" href="http://blog.revolutionanalytics.com/2016/02/bay-area-user-group-lightning-talks.html">Bay Area R User Group in February</a> I presented progress in big-join in H2O which is based on the algorithm in R’s <a target="_blank" href="https://github.com/Rdatatable/data.table/wiki">data.table</a> package. The <a target="_blank" href="https://github.com/h2oai/h2o-meetups/raw/master/2016_02_09_BigJoin_BARUG/BigJoin.pdf">presentation</a> had two goals: i) describe one test in great detail so everyone understands what is being tested so they can judge if it is relevant to them or not; and ii) show how it scales with data size and number of nodes.</p>
<p>These were the final two slides :</p>
<p><a target="_blank" rel="attachment wp-att-631" href="https://github.com/h2oai/h2o-meetups/raw/master/2016_02_09_BigJoin_BARUG/BigJoin.pdf"><img sizes="(max-width: 300px) 100vw, 300px" srcset="http://i1.wp.com/blog.h2o.ai/wp-content/uploads/2016/04/Slide14-300x225.png?resize=300%2C225 300w, http://blog.h2o.ai/wp-content/uploads/2016/04/Slide14-768x576.png 768w, http://blog.h2o.ai/wp-content/uploads/2016/04/Slide14.png 1024w" alt="Slide14" class="wp-image-631" src="http://i1.wp.com/blog.h2o.ai/wp-content/uploads/2016/04/Slide14-300x225.png?resize=300%2C225" data-recalc-dims="1"></a>     <a target="_blank" rel="attachment wp-att-632" href="https://www.youtube.com/watch?v=8VpzNibOme0"><img sizes="(max-width: 300px) 100vw, 300px" srcset="http://i0.wp.com/blog.h2o.ai/wp-content/uploads/2016/04/Slide15-300x225.png?resize=300%2C225 300w, http://blog.h2o.ai/wp-content/uploads/2016/04/Slide15-768x576.png 768w, http://blog.h2o.ai/wp-content/uploads/2016/04/Slide15.png 1024w" alt="Slide15" class="wp-image-632" src="http://i0.wp.com/blog.h2o.ai/wp-content/uploads/2016/04/Slide15-300x225.png?resize=300%2C225" data-recalc-dims="1"></a></p>
<p>I left a blank for 1e10 (10 billion high cardinality rows joined with 10 billion high cardinality rows returning 10 billion high cardinality rows) because it didn’t work at that time. Although each node has 256GB RAM (and 32 cores) the 10 billion row test involves joining two 10 billion row tables (each 200GB) and returning a third table (also ~10 billion rows) of 300GB, total 700GB. I was giving 200GB to each of the 4 H2O nodes (to leave 50GB on each node for the operating system and my forgiving colleagues) which meant the H2O cluster had 800GB RAM. The join algorithm needed more than a mere 100GB to complete the task and hence failed. Given that a common rule of thumb is “3x data size” for working memory, to fail with 0.1x data size as working memory is very reasonable. Internally we refer to H2O as a fast <em>calculator</em>. Like R and Python it is in-memory. Unlike R and Python a single data frame can be bigger than a single node.</p>
<p>So I scaled up to 10 nodes: 2TB RAM and 320 cores.</p>
<p>But it still didn’t work. It ran for 30 minutes and then failed with out-of-memory. I felt that 2TB of RAM really should be enough to complete this task: 200GB joined with 200GB returning 300GB. 2TB-700GB = 1.3TB; that amount of working memory should be enough. It pointed to something being wrong somewhere. Indeed with help from colleagues we identified one point where data was being duplicated and shouldn’t have been. This reduced the working memory needed and then it worked.</p>
<p>But it took 20 minutes.</p>
<p>This didn’t fit with my gut feel given I knew how the algorithm works (or was supposed to work). It should be nearer to linear scaling. 1 billion rows took under a minute so 10 billion should take 10 minutes. But it was taking twice that: 20 minutes. More to the point I’m not just having a cup of coffee or lunch while it’s running; I’m watching the CPUs and the network traffic between the nodes. The puzzling thing was that the network wasn’t saturated and the CPU’s were often cold. <em>Something</em> was <em>wrong</em>.</p>
<p>There ensued many proposals and discussions as to what it might be. Focusing on the algorithm and its implementation.</p>
<p>Until one day I went back to the 1 billion row test and ran it on 4 nodes, but a <em>different</em> set of 4 nodes. I usually run on servers 6-9. There was no reason for picking those servers. The first time I did it I asked which ones I could use and then never changed it. This time I ran on servers 1-4 instead. It was 3 times slower. At first I thought there must be a difference in the libraries on the servers or my code. After ruling out many things I rubbed my eyes and ran it again a few times again and then again on servers 6-9. It was repeatable and confirmed. How <em>on earth</em> could it make a difference <em>which</em> servers I ran on? All 10 servers are identical spec in the same rack connected to the same switch. Here’s a photo :</p>
<p><a target="_blank" rel="attachment wp-att-639" href="http://i1.wp.com/blog.h2o.ai/wp-content/uploads/2016/05/rackFront-1.jpg"><img sizes="(max-width: 300px) 100vw, 300px" srcset="http://i1.wp.com/blog.h2o.ai/wp-content/uploads/2016/05/rackFront-1-300x294.jpg?resize=300%2C294 300w, http://blog.h2o.ai/wp-content/uploads/2016/05/rackFront-1-768x753.jpg 768w, http://blog.h2o.ai/wp-content/uploads/2016/05/rackFront-1.jpg 1000w" alt="rackFront" class="wp-image-639" src="http://i1.wp.com/blog.h2o.ai/wp-content/uploads/2016/05/rackFront-1-300x294.jpg?resize=300%2C294" data-recalc-dims="1"></a></p>
<p>I was monitoring CPU and network usage. Nothing else (either human or artificial) was using the cluster. I was the only one. I had this physical cluster to myself. For sure.</p>
<p>I realized we might have been barking up the wrong tree when we were looking at the algorithm and its implementation. When I scaled up to 10bn rows and 10 nodes, perhaps I didn’t <em>just</em> scale up as intended, but <em>perhaps</em> I included a server that was somehow, <em>faulty</em>?!</p>
<p>I reran on servers 2-5 and it was 3 times faster than on servers 1-4. The culprit appears to be server 1, then. We have a network tester in H2O that I had already run but ran it again. Our server names correspond to the last digit of their IP addresses: server 1 = .181, server 2 = .182, server 3 = .183, etc. Here was the image :</p>
<p><a target="_blank" rel="attachment wp-att-627" href="http://i1.wp.com/blog.h2o.ai/wp-content/uploads/2016/04/NetworkTest.png"><img sizes="(max-width: 799px) 100vw, 799px" srcset="http://blog.h2o.ai/wp-content/uploads/2016/04/NetworkTest-300x109.png 300w, http://blog.h2o.ai/wp-content/uploads/2016/04/NetworkTest-768x278.png 768w, http://i1.wp.com/blog.h2o.ai/wp-content/uploads/2016/04/NetworkTest.png?resize=456%2C165 799w" alt="NetworkTest" class="wp-image-627" src="http://i1.wp.com/blog.h2o.ai/wp-content/uploads/2016/04/NetworkTest.png?resize=456%2C165" data-recalc-dims="1"></a></p>
<p>There doesn’t appear to be much wrong here. I asked colleagues and they advised to more thoroughly check the network speeds as they’d seen problems in the past. I Googled for how to test network speed which quickly returned iperf. I knew the problem might be server 1 so I chose server 3 to be the server and compared speeds from servers 1, 2 and 4 as clients to server 3 as the server. Here’s the result :</p>
<p><a target="_blank" rel="attachment wp-att-629" href="http://i1.wp.com/blog.h2o.ai/wp-content/uploads/2016/04/iperf-1.png"><img sizes="(max-width: 1024px) 100vw, 1024px" srcset="http://blog.h2o.ai/wp-content/uploads/2016/04/iperf-1-300x113.png 300w, http://blog.h2o.ai/wp-content/uploads/2016/04/iperf-1-768x288.png 768w, http://i1.wp.com/blog.h2o.ai/wp-content/uploads/2016/04/iperf-1.png?resize=456%2C171 1024w" alt="iperf" class="wp-image-629" src="http://i1.wp.com/blog.h2o.ai/wp-content/uploads/2016/04/iperf-1.png?resize=456%2C171" data-recalc-dims="1"></a></p>
<p>So server 1 is more than 10 times slower the others. I trotted over to our server room again and I had a look at the back of server 1.</p>
<p><a target="_blank" rel="attachment wp-att-641" href="http://i2.wp.com/blog.h2o.ai/wp-content/uploads/2016/05/rackBack.jpg"><img alt="rackBack" class="wp-image-641" src="http://i0.wp.com/blog.h2o.ai/wp-content/uploads/2016/05/rackBack-170x300.jpg?resize=170%2C300" data-recalc-dims="1"></a>   <a target="_blank" rel="attachment wp-att-642" href="http://i2.wp.com/blog.h2o.ai/wp-content/uploads/2016/05/rackBackOrange-1.jpg"><img alt="rackBackOrange" class="wp-image-642" src="http://i0.wp.com/blog.h2o.ai/wp-content/uploads/2016/05/rackBackOrange-1-170x300.jpg?resize=170%2C300" data-recalc-dims="1"></a></p>
<p>See that orange light? That’s what was wrong. Either the switch or the network card had auto negotiated itself down to 1G speed when all its friends in the rack are happy at 10G speed. Despite it being up 57 days, it hadn’t auto-negotiated itself back up to 10G speed. Or something like that. We think.</p>
<p>What’s the solution? Old school: I unplugged the ethernet cable and plugged it back in. The orange light turned green. I went back to my laptop and tested again with iperf. This time iperf reported 10G speed for server 1 consistent with the other servers. The non-physical way to do this is to use <a target="_blank" href="http://www.thegeekstuff.com/2010/10/ethtool-command/">ethtool</a>. Next time a problem occurs I’ll try it to save some foot steps to the server room.</p>
<p>Rerunning the 10 billion row to 10 billion row high cardinality join test now comes in twice as fast: 10 minutes instead of 20 minutes. I’m not really sure why that made such a big difference since the network wasn’t saturated enough for it to be a pure data transfer speed issue. I’ll chalk it up to <em>something</em> to do with that network card or switch and move on. I’ll ensure that iperf reports 10G speed between all nodes going forward.</p>
<p>In my talk in Chicago this week I connected to these servers and gave a <a target="_blank" href="http://library.fora.tv/2016/05/03/big_joins_scalable_data_munging_and_datatable">live demo</a>. That video along with all our other presentations that day are available <a target="_blank" href="http://library.fora.tv/conference/h2o_open_tour_chicago">here</a>.</p>
<p>The event itself was quite packed as seen in the picture below:<br>
<img sizes="(max-width: 600px) 100vw, 600px" srcset="http://blog.h2o.ai/wp-content/uploads/2016/05/h2o-chicago-300x170.jpeg 300w, http://i0.wp.com/blog.h2o.ai/wp-content/uploads/2016/05/h2o-chicago.jpeg?resize=456%2C258 600w" alt="h2o-chicago" class="wp-image-644" src="http://i0.wp.com/blog.h2o.ai/wp-content/uploads/2016/05/h2o-chicago.jpeg?resize=456%2C258" data-recalc-dims="1"></p>
<blockquote data-card-key="050ed2d5bc95488f84ab6fa397e146a4" data-card-branding="0" data-card-type="article-full"><h4><a target="_blank" href="http://library.fora.tv/2016/05/03/big_joins_scalable_data_munging_and_datatable">Big Joins, Scalable Data Munging and Data.Table</a></h4>
<p>Matt Dowle, Hacker, H2O.ai</p>
</blockquote>
<p></p>
<p>I’ll be presenting on the same topic at <a target="_blank" href="http://databythebay2016.sched.org/event/6EQ7/parallel-and-distributed-big-joins-in-h2o/">Data by the Bay</a> on Monday 16 May.</p>
<p><div><div><div><div data-width="55" data-show-faces="false" data-send="true" data-layout="box_count" data-height="62" data-href="http://www.r-bloggers.com/red-herring-bites/"></div></div><div><a data-url="http://www.r-bloggers.com/red-herring-bites/" data-counturl="http://www.r-bloggers.com/red-herring-bites/" data-count="vertical" data-text="Red herring bites" class="twitter-share-button" data-via="rbloggers" href="https://twitter.com/share"></a></div><div></div></div><div></div></div>
<div>
<div>To <strong>leave a comment</strong> for the author, please follow the link and comment on their blog: <strong><a href="http://blog.h2o.ai/2016/05/red-herring-bites/"> R-Bloggers – H2O blog</a></strong>.</div>
<hr>
<a rel="nofollow" href="http://www.r-bloggers.com/">R-bloggers.com</a> offers <strong><a rel="nofollow" href="http://feedburner.google.com/fb/a/mailverify?uri=RBloggers">daily e-mail updates</a></strong> about <a title="The R Project for Statistical Computing" rel="nofollow" href="http://www.r-project.org/">R</a> news and <a title="R tutorials" rel="nofollow" href="http://www.r-bloggers.com/?s=tutorial">tutorials</a> on topics such as: <a title="Data science" rel="nofollow" href="http://www.r-bloggers.com/?s=data%20science">Data science</a>, <a title="Big Data" rel="nofollow" href="http://www.r-bloggers.com/?s=Big%20Data">Big Data, <a title="R jobs" rel="nofollow" href="http://www.r-users.com/">R jobs</a>, visualization (<a title="ggplot and ggplot2 tutorials" rel="nofollow" href="http://www.r-bloggers.com/?s=ggplot2">ggplot2</a>, <a title="Boxplots using lattice and ggplot2 tutorials" rel="nofollow" href="http://www.r-bloggers.com/?s=boxplot">Boxplots</a>, <a title="Maps and gis" rel="nofollow" href="http://www.r-bloggers.com/?s=map">maps</a>, <a title="Animation in R" rel="nofollow" href="http://www.r-bloggers.com/?s=animation">animation</a>), programming (<a title="RStudio IDE for R" rel="nofollow" href="http://www.r-bloggers.com/?s=RStudio">RStudio</a>, <a title="Sweave and literate programming" rel="nofollow" href="http://www.r-bloggers.com/?s=sweave">Sweave</a>, <a title="LaTeX in R" rel="nofollow" href="http://www.r-bloggers.com/?s=LaTeX">LaTeX</a>, <a title="SQL and databases" rel="nofollow" href="http://www.r-bloggers.com/?s=SQL">SQL</a>, <a title="Eclipse IDE for R" rel="nofollow" href="http://www.r-bloggers.com/?s=eclipse">Eclipse</a>, <a title="git and github, Version Control System" rel="nofollow" href="http://www.r-bloggers.com/?s=git">git</a>, <a title="Large data in R using Hadoop" rel="nofollow" href="http://www.r-bloggers.com/?s=hadoop">hadoop</a>, <a title="Web Scraping of google, facebook, yahoo, twitter and more using R" rel="nofollow" href="http://www.r-bloggers.com/?s=Web+Scraping">Web Scraping</a>) statistics (<a title="Regressions and ANOVA analysis tutorials" rel="nofollow" href="http://www.r-bloggers.com/?s=regression">regression</a>, <a title="principal component analysis tutorial" rel="nofollow" href="http://www.r-bloggers.com/?s=PCA">PCA</a>, <a title="Time series" rel="nofollow" href="http://www.r-bloggers.com/?s=time+series">time series</a>, <a title="finance trading" rel="nofollow" href="http://www.r-bloggers.com/?s=trading">trading</a>) and more...
<img height="1" alt="" width="1" src="http://feeds.feedburner.com/~r/RBloggers/~4/I-M_-X35Lts"></a></div>