<p><div><div><div><div data-show-faces="false" data-height="21" data-width="100" data-layout="button_count" data-send="true" data-href="https://www.r-bloggers.com/r%e2%81%b6-reticulating-parquet-files/"></div></div><div></div></div><div></div></div>
<div>
(This article was first published on <strong><a href="https://rud.is/b/2017/08/01/r%E2%81%B6-reticulating-parquet-files/"> R – rud.is</a></strong>, and kindly contributed to <a rel="nofollow" href="https://www.r-bloggers.com/">R-bloggers)</a> </div>
<p>The <code>reticulate</code> package provides a very clean &amp; concise interface bridge between R and Python which makes it handy to work with modules that have yet to be ported to R (going native is always better when you can do it). This post shows how to use <code>reticulate</code> to create parquet files directly from R using <code>reticulate</code> as a bridge to the <code>pyarrow</code> module, which has the ability to natively create parquet files.</p>
<p>Now, you can <a rel="nofollow" href="https://rud.is/b/2017/01/22/create-parquet-files-from-r-data-frames-with-sergeant-apache-drill-a-k-a-make-parquet-files-great-again-in-r/" target="_blank">create parquet files through R with Apache Drill</a> — and, I’ll provide another example for that here — but, you may have need to generate such files and not have the ability to run Drill.</p>
<p>The Python parquet process is pretty simple since you can convert a <code>pandas</code> <code>DataFrame</code> directly to a <code>pyarrow</code> <code>Table</code> which can be written out in parquet format with <code>pyarrow.parquet</code>. We just need to follow this process through <code>reticulate</code> in R:</p>
<pre><code>library(reticulate)

pd &lt;- import(&quot;pandas&quot;, &quot;pd&quot;)
pa &lt;- import(&quot;pyarrow&quot;, &quot;pa&quot;)
pq &lt;- import(&quot;pyarrow.parquet&quot;, &quot;pq&quot;)

mtcars_py &lt;- r_to_py(mtcars)
mtcars_df &lt;- pd$DataFrame$from_dict(mtcars_py)
mtcars_tab &lt;- pa$Table$from_pandas(mtcars_df)

pq$write_table(mtcars_tab, path.expand(&quot;~/Data/mtcars_python.parquet&quot;))</code></pre>
<p>I wouldn’t want to do that for ginormous data frames, but it should work pretty well for modest use cases (you’re likely using Spark, Drill, Presto or other “big data” platforms for creation of larger parquet structures). Here’s how we’d do that with Drill via the <code>sergeant</code> package:</p>
<pre><code>readr::write_csv(mtcars, &quot;~/Data/mtcars_r.csvh&quot;)
dc &lt;- drill_connection(&quot;localhost&quot;)
drill_query(dc, &quot;CREATE TABLE dfs.tmp.`/mtcars_r.parquet` AS SELECT * FROM dfs.root.`/Users/bob/Data/mtcars_r.csvh`&quot;)</code></pre>
<p>Without additional configuration parameters, the reticulated-Python version (above) generates larger parquet files and also has an index column since they’re needed in Python <code>DataFrame</code>s (ugh), but small-ish data frames will end up in a single file whereas the Drill created ones will be in a directory with an additional CRC file (and, much smaller by default). NOTE: You <em>can</em> use <code>preserve_index=False</code> on the call to <code>Table.from_pandas</code> to get rid of that icky index.</p>
<p>It’s fairly efficient even for something like <code>nycflights13::flights</code> which has ~330K rows and 19 columns:</p>
<pre><code>system.time(
  r_to_py(nycflights13::flights) %>% 
  pd$DataFrame$from_dict() %>% 
  pa$Table$from_pandas() %>% 
  pq$write_table(where = &quot;/tmp/flights.parquet&quot;)
)
##    user  system elapsed 
##   1.285   0.108   1.398 </code></pre>
<p>If you need to generate parquet files in a pinch, <code>reticulate</code> seems to be a good way to go.</p>
	<p>
<div>
<div>To <strong>leave a comment</strong> for the author, please follow the link and comment on their blog: <strong><a href="https://rud.is/b/2017/08/01/r%E2%81%B6-reticulating-parquet-files/"> R – rud.is</a></strong>.</div>
<hr>
<a rel="nofollow" href="https://www.r-bloggers.com/">R-bloggers.com</a> offers <strong><a rel="nofollow" href="https://feedburner.google.com/fb/a/mailverify?uri=RBloggers">daily e-mail updates</a></strong> about <a rel="nofollow" href="https://www.r-project.org/" title="The R Project for Statistical Computing">R</a> news and <a rel="nofollow" href="https://www.r-bloggers.com/search/tutorial" title="R tutorials">tutorials</a> on topics such as: <a rel="nofollow" href="https://www.r-bloggers.com/search/data%20science" title="Data science">Data science</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/Big%20Data" title="Big Data">Big Data, <a rel="nofollow" href="https://www.r-users.com/" title="R jobs">R jobs</a>, visualization (<a rel="nofollow" href="https://www.r-bloggers.com/search/ggplot2" title="ggplot and ggplot2 tutorials">ggplot2</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/boxplot" title="Boxplots using lattice and ggplot2 tutorials">Boxplots</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/map" title="Maps and gis">maps</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/animation" title="Animation in R">animation</a>), programming (<a rel="nofollow" href="https://www.r-bloggers.com/search/RStudio" title="RStudio IDE for R">RStudio</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/sweave" title="Sweave and literate programming">Sweave</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/LaTeX" title="LaTeX in R">LaTeX</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/SQL" title="SQL and databases">SQL</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/eclipse" title="Eclipse IDE for R">Eclipse</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/git" title="git and github, Version Control System">git</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/hadoop" title="Large data in R using Hadoop">hadoop</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/Web+Scraping" title="Web Scraping of google, facebook, yahoo, twitter and more using R">Web Scraping</a>) statistics (<a rel="nofollow" href="https://www.r-bloggers.com/search/regression" title="Regressions and ANOVA analysis tutorials">regression</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/PCA" title="principal component analysis tutorial">PCA</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/time+series" title="Time series">time series</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/trading" title="finance trading">trading</a>) and more...
<img width="1" alt="" src="http://feeds.feedburner.com/~r/RBloggers/~4/lmRu_59sw90" height="1"></a></div>