<p><div><div><div><div data-width="100" data-show-faces="false" data-send="true" data-layout="button_count" data-height="21" data-href="http://www.r-bloggers.com/loading-data-with-pandas/"></div></div><div><a data-url="http://www.r-bloggers.com/loading-data-with-pandas/" data-counturl="http://www.r-bloggers.com/loading-data-with-pandas/" data-count="horizontal" data-text="Loading Data with Pandas" class="twitter-share-button" data-via="rbloggers" href="https://twitter.com/share"></a></div><div></div></div><div></div></div>
<div>
(This article was first published on <strong><a href="http://www.quintuitive.com/2016/06/27/loading-data-pandas/"> R – Quintuitive</a></strong>, and kindly contributed to <a rel="nofollow" href="http://www.r-bloggers.com/">R-bloggers)</a> </div>
<p>On at least a couple of occasions lately, I realized that I may need Python in the near future. While I have amassed some <a target="_blank" href="http://www.quintuitive.com/2015/05/27/moved-quantscript-to-github/">limited experience</a> with the language over the years, I never spent the time to understand <a target="_blank" href="http://pandas.pydata.org/">Pandas</a>, its de-facto standard data-frame library.</p>
<p><span></span></p>
<p>Where does one start? For me its usually with the data. Simple stuff, loading, wrangling, etc. Re-writing <a target="_blank" href="http://www.quintuitive.com/2015/05/20/databases-an-ideal-application-for-r6/">my little R6 helper class</a> to load future’s data looked like a perfect candidate.</p>
<p>There was some frustration, totally expected after years of experience with R. Some things were less intuitive, however, surprisingly pretty much nothing was straight ugly. <img alt="🙂" class="wp-smiley" src="http://i2.wp.com/s.w.org/images/core/emoji/72x72/1f642.png?w=456&ssl=1" data-recalc-dims="1"> And when it comes to code, I am not easy to please. The end result is available <a target="_blank" href="https://gist.github.com/ivannp/1b4475f214cf246e246ee3e7d143d843">here</a>.</p>
<p>Here is a little example how to use the code, although one can’t do much without the data, which I can’t distribute:</p>
<pre>
import pandas as pd
import instrumentdb as idb

def main():
    # Crate the object for the database
    db = idb.CsiDb()

    # Load the data for three elements
    all = db.mload_bars([&quot;HO2&quot;, &quot;RB2&quot;, &quot;CL2&quot;])
    print(all['HO2'].head())
    print(all['RB2'].head())

    # Build an array of the closing prices for each series
    closes = []
    for ss in all.keys():
        closes.append(all[ss]['close'])

    # Create a single data frame using these series
    all_df = pd.concat(closes, join='inner', axis=1)
    all_df.columns = [xx.lower() for xx in all.keys()]

    print(all_df.tail())

    # That's the only line that would work without the data.
    print(db.future_list())

if __name__ == &quot;__main__&quot;:
    main()
</pre>
<p>The structure of the database is available from <a target="_blank" href="https://github.com/ivannp/tradelib">Tradelib’s source code</a> (I am using the <a target="_blank" href="https://github.com/ivannp/tradelib/blob/master/src/main/scripts/sql/data.sqlite.sql">SQLite’s version</a> for this test). To bootstrap (create) the database I use <em>sqlite3.exe’s read</em> command, to which I pass <em>data.sqlite.sql</em> as a parameter. To be used via the <em>CsiDb</em> class, the database is configured using a <a target="_blank" href="https://github.com/toml-lang/toml">TOML configuration file</a>.</p>
<pre>
flavor = &quot;SQLite&quot;
db = &quot;sqlite:///C:/Users/qmoron/Documents/csidata.sqlite&quot;
bars_table = &quot;csi_bars&quot;
</pre>
<p>Now a little rant: In the above code, I tried to create a module, <em>instrumentdb</em>, to keep the source code in it. This created some problems while developing the module. Apparently, once loaded, it’s pretty hard to re-load the module properly within the same REPL interpreter. From R’s perspective, where I am used to re-loading files, or even packages, as my development goes, that seemed quite an obstacle. After straggling with the issue for a while, the best I was able to come up with, is the above approach of using a full-blown “main” file to drive the execution and some tests. This is unlikely to scale (in the sense of using it in a rapid REPL prototyping) – I am open to suggestions.</p>
<p>The post <a target="_blank" rel="nofollow" href="http://www.quintuitive.com/2016/06/27/loading-data-pandas/">Loading Data with Pandas</a> appeared first on <a target="_blank" rel="nofollow" href="http://www.quintuitive.com/">Quintuitive</a>.</p>
<p><div><div><div><div data-width="55" data-show-faces="false" data-send="true" data-layout="box_count" data-height="62" data-href="http://www.r-bloggers.com/loading-data-with-pandas/"></div></div><div><a data-url="http://www.r-bloggers.com/loading-data-with-pandas/" data-counturl="http://www.r-bloggers.com/loading-data-with-pandas/" data-count="vertical" data-text="Loading Data with Pandas" class="twitter-share-button" data-via="rbloggers" href="https://twitter.com/share"></a></div><div></div></div><div></div></div>
<div>
<div>To <strong>leave a comment</strong> for the author, please follow the link and comment on their blog: <strong><a href="http://www.quintuitive.com/2016/06/27/loading-data-pandas/"> R – Quintuitive</a></strong>.</div>
<hr>
<a rel="nofollow" href="http://www.r-bloggers.com/">R-bloggers.com</a> offers <strong><a rel="nofollow" href="http://feedburner.google.com/fb/a/mailverify?uri=RBloggers">daily e-mail updates</a></strong> about <a title="The R Project for Statistical Computing" rel="nofollow" href="http://www.r-project.org/">R</a> news and <a title="R tutorials" rel="nofollow" href="http://www.r-bloggers.com/?s=tutorial">tutorials</a> on topics such as: <a title="Data science" rel="nofollow" href="http://www.r-bloggers.com/?s=data%20science">Data science</a>, <a title="Big Data" rel="nofollow" href="http://www.r-bloggers.com/?s=Big%20Data">Big Data, <a title="R jobs" rel="nofollow" href="http://www.r-users.com/">R jobs</a>, visualization (<a title="ggplot and ggplot2 tutorials" rel="nofollow" href="http://www.r-bloggers.com/?s=ggplot2">ggplot2</a>, <a title="Boxplots using lattice and ggplot2 tutorials" rel="nofollow" href="http://www.r-bloggers.com/?s=boxplot">Boxplots</a>, <a title="Maps and gis" rel="nofollow" href="http://www.r-bloggers.com/?s=map">maps</a>, <a title="Animation in R" rel="nofollow" href="http://www.r-bloggers.com/?s=animation">animation</a>), programming (<a title="RStudio IDE for R" rel="nofollow" href="http://www.r-bloggers.com/?s=RStudio">RStudio</a>, <a title="Sweave and literate programming" rel="nofollow" href="http://www.r-bloggers.com/?s=sweave">Sweave</a>, <a title="LaTeX in R" rel="nofollow" href="http://www.r-bloggers.com/?s=LaTeX">LaTeX</a>, <a title="SQL and databases" rel="nofollow" href="http://www.r-bloggers.com/?s=SQL">SQL</a>, <a title="Eclipse IDE for R" rel="nofollow" href="http://www.r-bloggers.com/?s=eclipse">Eclipse</a>, <a title="git and github, Version Control System" rel="nofollow" href="http://www.r-bloggers.com/?s=git">git</a>, <a title="Large data in R using Hadoop" rel="nofollow" href="http://www.r-bloggers.com/?s=hadoop">hadoop</a>, <a title="Web Scraping of google, facebook, yahoo, twitter and more using R" rel="nofollow" href="http://www.r-bloggers.com/?s=Web+Scraping">Web Scraping</a>) statistics (<a title="Regressions and ANOVA analysis tutorials" rel="nofollow" href="http://www.r-bloggers.com/?s=regression">regression</a>, <a title="principal component analysis tutorial" rel="nofollow" href="http://www.r-bloggers.com/?s=PCA">PCA</a>, <a title="Time series" rel="nofollow" href="http://www.r-bloggers.com/?s=time+series">time series</a>, <a title="finance trading" rel="nofollow" href="http://www.r-bloggers.com/?s=trading">trading</a>) and more...
<img height="1" alt="" width="1" src="http://feeds.feedburner.com/~r/RBloggers/~4/xoFU-KOxFgk"></a></div>