<p><div><div><div><div data-show-faces="false" data-height="21" data-width="100" data-layout="button_count" data-send="true" data-href="https://www.r-bloggers.com/my-data-science-tool-box/"></div></div><div></div></div><div></div></div>
<div>
(This article was first published on <strong><a href="http://people.fas.harvard.edu/~izahn/posts/my-toolbox/"> Ista Zahn (Posts about R)</a></strong>, and kindly contributed to <a rel="nofollow" href="https://www.r-bloggers.com/">R-bloggers)</a> </div>
<p>
This post describes the tools I currently use for working with data.<br>
People often ask me to recommend specific tools, and I always<br>
hesitate, because so much boils down to personal preference. I<br>
recently added a workshop to the <a rel="nofollow" href="https://dss.iq.harvard.edu/" target="_blank">DSS lineup</a> providing an overview of<br>
popular <a rel="nofollow" href="http://tutorials.iq.harvard.edu/DataScienceTools/DataScienceTools.html" target="_blank">tools for working with data</a>. The core idea is that researchers<br>
have a lot of choices available when it comes to choosing tools to<br>
implement a reproducible workflow. For example, it doesn’t really<br>
matter whether you choose to learn <a rel="nofollow" href="https://r-project.org/" target="_blank">R</a> or <a rel="nofollow" href="https://python.org/" target="_blank">Python</a>; the important thing is<br>
that you write and document code of some kind so that your analysis<br>
can be reproduced. Similarly, it doesn’t matter much whether you<br>
choose to use <a rel="nofollow" href="https://www.rstudio.com/products/rstudio" target="_blank">RStudio</a> or <a rel="nofollow" href="https://jupyter.org/" target="_blank">Jupyter notebooks</a>; the important thing is<br>
that you have a development and authoring environment that encourages<br>
good research practices. Still, inquiring minds want to know, <i>what do<br>
you use?</i>
</p>
<p>
The short answer is as follows:
</p>
<dl>
<dt> Operating system </dt>
<dd><a rel="nofollow" href="https://archlinux.org/" target="_blank">Arch Linux</a>
</dd>
<dt> Programming language </dt>
<dd><a rel="nofollow" href="https://r-project.org/" target="_blank">R</a> and <a rel="nofollow" href="https://python.org/" target="_blank">Python</a>
</dd>
<dt> Editor / IDE </dt>
<dd><a rel="nofollow" href="https://www.gnu.org/software/emacs/" target="_blank">Emacs</a>
</dd>
<dt> Markup language </dt>
<dd><a rel="nofollow" href="https://orgmode.org/" target="_blank">Org Mode</a> and <a rel="nofollow" href="https://www.latex-project.org/" target="_blank">LaTeX</a>
</dd>
<dt> Revision control system </dt>
<dd><a rel="nofollow" href="https://git-scm.com/" target="_blank">git</a>
</dd>
<dt> Shell </dt>
<dd><a rel="nofollow" href="https://fishshell.org/" target="_blank">fish</a>
</dd>
</dl>
<p>
Those curious to know <i>why</i> I prefer these tools and how I’ve<br>
customized them to suit my needs and preferences can read on.
</p>
<p></p>
<div>
<h2>I use the Arch Linux operating system</h2>
<div>
<p>
Operating systems are often excluded from “toolkit” discussions,<br>
presumably because most popular tools are cross-platform and abstract<br>
away a lot of the differences across operating systems. Nevertheless,<br>
operating systems are not all created equal, and in my opinion<br>
Linux-based operating systems are currently the best option for<br>
working with data.
</p>
<p>
Linux is similar in many ways to OS X (they are both UNIX-like), but<br>
without the annoying restrictions and limitations. Linux gives you a<br>
package manager and the freedom to install whatever tools you need,<br>
and to configure them however you like. All this freedom can make it<br>
easier to shoot yourself in the foot, but I find that preferable to<br>
being restricted by technical limitations (Windows) or greedy<br>
corporate policy (Apple OS X).
</p>
<p>
It is worth noting that the world of Linux is diverse and varied, and<br>
that there can be significant differences among different Linux<br>
distributions. One frequent issue is that many Linux distributions are<br>
very conservative about releasing software updates. This might make<br>
sense for embedded devices or servers (though I have my doubts) but it<br>
makes absolutely no sense on a laptop or personal machine. Generally<br>
speaking, I want to run the latest stable release of all the programs<br>
installed on my computers. Arch Linux is one of the few Linux<br>
distributions that makes it easy to keep your applications up-to-date.<br>
If you are new to Linux you may wish to start with Manjaro Linux, a<br>
pre-configured Arch Linux derivative.
</p>
</div>
</div>
<div>
<h2>I use the Emacs text editor</h2>
<div>
<p>
I use the Emacs text editor because I haven’t yet found anything that<br>
I like better. It has a lot of legacy baggage that makes it feel alien<br>
and intimidating at first, but I got used the these quirks after a<br>
using it for a week or two. The defining feature of Emacs is that it<br>
can be customized and extended without limit. It is both a text editor<br>
and a toolkit to building your own editing environment. A large and<br>
robust ecosystem of community developed packages makes it easy to use<br>
Emacs not only as a text editor, but also as a Git and GitHub<br>
front-end, an email client, and StackOverflow browser (to give but a<br>
few examples).
</p>
</div>
<div>
<h3>Emacs customization</h3>
<div>
<p>
I <a rel="nofollow" href="http://iqss.github.io/IQSS.emacs/" target="_blank">customize Emacs extensively</a> to improve the user interface, provide<br>
better support for specific programming and markup languages, and to<br>
provide front-ends for reading email and managing Git repositories.
</p>
<p>
Highlights of my Emacs configuration include
</p>
<ul>
<li>improved support for running R, python, or other programming<br>
languages inside,
</li>
<li>support for LaTeX and other markup languages,
</li>
<li>support for literate programming using org-mode or R markdown,
</li>
<li>consistent and familiar code evaluation using <code>CTRL-RETURN</code>,
</li>
<li>consistent and familiar indentation and code completion using the<br>
<code>TAB</code> key,
</li>
<li>powerful and simple search-based tools for finding commands, files<br>
and buffers, inserting citations etc.
</li>
<li>more standard select/copy/paste keys and right-click behavior makes<br>
it more familiar to those new to Emacs,
</li>
<li>more powerful and convenient window management.
</li>
</ul>
<p>
If you are interested in giving Emacs a try take a look at the<br>
<a rel="nofollow" href="http://iqss.github.io/IQSS.emacs/" target="_blank">instructions</a> and <a rel="nofollow" href="http://iqss.github.io/IQSS.emacs/issues/" target="_blank">report any problems</a> you may encounter.
</p>
</div>
</div>
</div>
<div>
<h2>I write documents using the Org Mode, Markdown, and LaTeX markup languages</h2>
<div>
<p>
Most of the documents I produces these days are technical or training<br>
materials that include lots of example code. I often use<br>
<a rel="nofollow" href="https://en.wikipedia.org/wiki/Literate_programming" target="_blank">literate programming</a> techniques to keep the examples and the output<br>
produced by those examples together in a single document. The markup<br>
language I use for this purpose depends on the complexity of the<br>
document.
</p>
<p>
<i>Markdown</i> is the simplest and most ubiquitous of the markup languages<br>
I use. I often use it for simple documents, or those on which<br>
non-emacs users are collaborating. <i>Org mode</i> is a more powerful<br>
markup language for which adequate support is available only in Emacs.<br>
I use it for many things, including most of my <a rel="nofollow" href="https://github.com/IQSS/workshops" target="_blank">workshop notes</a>. Finally<br>
<i>LaTeX</i> is the most powerful, complex, and verbose of the markup<br>
languages I use. It is useful when you need more control over the<br>
appearance of the resulting document.
</p>
</div>
<div>
<h3>Markdown customization</h3>
<div>
<p>
I write markdown using <a rel="nofollow" href="https://jblevins.org/projects/markdown-mode/" target="_blank">markdown-mode</a> in Emacs. To “typeset” the<br>
documents for printing or posting on line I use <a rel="nofollow" href="https://pandoc.org/" target="_blank">pandoc</a> to convert<br>
markdown to <code>.pdf</code> files (via LaTeX), <code>.html</code>, or <code>.ipynb</code> (jupyter<br>
notebook) format. I have written a couple of scripts to make this<br>
process easier in specific cases, e.g. <a rel="nofollow" href="https://github.com/IQSS/workshops/tree/master/build_scripts" target="_blank">these scripts convert markdown<br>
documents to jupyter notebooks</a> and <a rel="nofollow" href="https://github.com/IQSS/IQSS.emacs/blob/master/tohtml.sh" target="_blank">this one</a> converts markdown to<br>
<code>.html</code> using a <a rel="nofollow" href="https://github.com/IQSS/IQSS.emacs/blob/master/template.html" target="_blank">custom template</a>.
</p>
</div>
</div>
<div>
<h3>Org mode customization</h3>
<div>
<p>
Since many of the documents I prepare in org-mode include code<br>
examples, I’ve configured <a rel="nofollow" href="https://iqss.github.io/IQSS.emacs/init.html#note_taking_and_outlining_(org-mode)" target="_blank">org-mode support for bash, R, Python and<br>
Matlab</a>. I also use a <a rel="nofollow" href="https://github.com/IQSS/workshops/blob/master/R/workshopPreamble.org" target="_blank">custom template</a> to export R workshop notes for<br>
publication on <a rel="nofollow" href="https://tutorials.iq.harvard.edu/" target="_blank">https://tutorials.iq.harvard.edu</a>.
</p>
</div>
</div>
<div>
<h3>LaTeX customization</h3>
<div>
<p>
Although I rarely write directly in LaTeX these days (Markdown and Org<br>
mode and much simpler and I strongly prefer them) LaTeX remains<br>
important as a <i>backend</i>. For example, I may prepare notes for a<br>
presentation in Markdown and then export to LaTeX in order to typeset<br>
the slides using <a rel="nofollow" href="https://ctan.org/pkg/beamer" target="_blank">beamer</a>.
</p>
<p>
I have developed a <a rel="nofollow" href="https://github.com/IQSS/iqss-beamer-theme" target="_blank">custom beamer theme</a> using <a rel="nofollow" href="https://iq.harvard.edu/" target="_blank">IQSS</a> colors that might be<br>
useful to you, especially if you are an IQSS affiliate. Highlights<br>
include
</p>
<ul>
<li>a <a rel="nofollow" href="https://github.com/khaledhosny/libertinus" target="_blank">modern font</a> that includes math symbols,
</li>
<li>simple and clean layout (e.g., only section same and page numbers in<br>
the footer),
</li>
<li>plenty of IQSS orange!
</li>
</ul>
<p>
If you have any questions or difficulties using this theme please <a rel="nofollow" href="https://github.com/IQSS/iqss-beamer-theme" target="_blank">open<br>
an issue</a>.
</p>
</div>
</div>
</div>
<div>
<h2>I use the R and Python programming languages</h2>
<div>
<p>
I use R and Python for working with data because they provide a good<br>
balance of flexibility and convenience. I prefer them to statistics<br>
packages like SPSS or SAS because both R and Python are full-fledged<br>
programming languages that give me the power and flexibility I need to<br>
address unusual or complicated tasks. At the same time, their<br>
substantial standard libraries and huge package repositories make it<br>
easy to accomplish standard or common data management and analysis<br>
tasks.
</p>
<p>
Notably, I do not choose these languages because the core language<br>
design – in both cases I don’t particularly <i>like</i> the languages. It<br>
really is the ecosystem of packages that keeps me using these tools<br>
instead of running off to a shiny new thing like <a rel="nofollow" href="https://julialang.org/" target="_blank">Julia</a> or <a rel="nofollow" href="https://golang.org/" target="_blank">Go</a>, or even<br>
from wandering off to an old but more interesting environment like<br>
<a rel="nofollow" href="https://haskell.org/" target="_blank">Haskell</a>.
</p>
</div>
<div>
<h3>R customization</h3>
<div>
<p>
My customization of R is mostly limited to the installation of<br>
packages. My <a rel="nofollow" href="https://cran.r-project.org/doc/manuals/r-release/R-intro.html#Customizing-the-environment" target="_blank">.Rprofile</a> just sets a default CRAN repository and <a rel="nofollow" href="https://people.fas.harvard.edu/~izahn/posts/useless-but-fun-r-packages/" target="_blank">prints<br>
an amusing quote</a>.
</p>
<p>
I find the following R packages especially useful:
</p>
<dl>
<dt> ggplot2 </dt>
<dd>robust graphics package
</dd>
<dt> lvaan </dt>
<dd>structural equation models
</dd>
<dt> lme4 </dt>
<dd>mixed effects modes in R
</dd>
<dt> Amelia, mitools, mice </dt>
<dd>multiple imputation
</dd>
<dt> purrr </dt>
<dd>consistent and clean functional programming tools
</dd>
<dt> stringi </dt>
<dd>powerful text manipulation tools
</dd>
<dt> xml2, jsonlite </dt>
<dd>powerful tools for manipulating and converting XML and JSON data
</dd>
<dt> httr </dt>
<dd>a web client written in R
</dd>
</dl>
<p>
Other than installing these packages I mostly use the default R configuration.
</p>
</div>
</div>
<div>
<h3>Python customization</h3>
<div>
<p>
As with R I don’t really customize Python much. I mostly <a rel="nofollow" href="https://iqss.github.io/IQSS.emacs/init.html#run_python_in_emacs_(python-mode)" target="_blank">use python<br>
from Emacs</a>, sometimes using <a rel="nofollow" href="https://iqss.github.io/IQSS.emacs/init.html#note_taking_and_outlining_(org-mode)" target="_blank">Org mode with python code blocks</a> for<br>
literate programming.
</p>
</div>
</div>
</div>
<div>
<h2>I use the fish shell in the Terminology emulator</h2>
<div>
<p>
It has been my experience that kids these days don’t really like<br>
shells. Honestly I don’t blame theme. Shell technology has been stuck<br>
in the 80’s for far too long. That situation is <a rel="nofollow" href="https://github.com/oilshell/oil/wiki/ExternalResources" target="_blank">starting to change</a>,<br>
but bash (a shell from the 80’s!) is still by far the most commonly<br>
used. I think it is time for a change, and <a rel="nofollow" href="https://fishshell.com/" target="_blank">fish</a> is the most<br>
well-developed at the moment. It doesn’t make using the command line<br>
<i>fun</i> exactly, but it feels a lot less like being forced to<br>
time-travel 40 years into the past.
</p>
<p>
I use the <a rel="nofollow" href="https://www.enlightenment.org/about-terminology" target="_blank">Terminology</a> emulator because unlike other terminal emulators<br>
it can display images and video directly in your terminal. This also<br>
helps avoid the forced-time-travel feeling commonly induced by using<br>
<i>bash</i> in a typical terminal emulator.
</p>
</div>
<div>
<h3>Fish customization</h3>
<div>
<p>
One of the nice things about the fish sell is that it has all the<br>
bells and whistles turned on by default. Very little configuration is<br>
needed to have a pleasant environment. My fish configuration is<br>
limited to a handful of convenience functions (AKA <i>aliases</i>) e.g., to<br>
update my system or <i>ssh</i> to a particular computer.
</p>
</div>
</div>
</div>
<div>
<h2>I use Git and GitHub for revision control</h2>
<div>
<p>
I don’t particularly like <a rel="nofollow" href="https://git-scm.com/" target="_blank">git</a>, but since everyone uses it I don’t feel<br>
like I have much of a choice. If it were up to me I would use<br>
something simpler like <a rel="nofollow" href="https://www.mercurial-scm.org/" target="_blank">mercurial</a> but it’s not so I use git. It is much<br>
more complicated and frustrating than it needs to be, but it doesn’t<br>
suck once you get the hang of it.
</p>
</div>
<div>
<h3>Git customization</h3>
<div>
<p>
I mostly use git from a terminal, but I often launch graphical tools<br>
from the command line, e.g., <a rel="nofollow" href="https://wiki.gnome.org/Apps/Gitg/" target="_blank">gitg</a> for viewing history and <a rel="nofollow" href="http://meldmerge.org/" target="_blank">meld</a> for<br>
viewing and merging diffs.
</p>
</div>
</div>
</div>
<div>
<h2>Commonalities and alternatives</h2>
<div>
<p>
If you’ve read this far you must be really interested in tools for<br>
working with data! While I hope it was interesting to read about my<br>
choices, I encourage you to try out some alternatives and pick a set<br>
of tools that works well for you.
</p>
<p>
In reflecting on my own tool choices I notice that customization and<br>
community activity are key values for me. For example, I like R not so<br>
much because of the design of the language, but because it is flexible<br>
and has an active community building and sharing tools in the form of<br>
<a rel="nofollow" href="https://r-pkg.org/" target="_blank">R packages</a>. Similarly, I value Emacs because it is easy to configure<br>
and because there is an active community developing <a rel="nofollow" href="https://melpa.org/" target="_blank">Emacs packages</a>. I<br>
value these tools not because of their design per se., but because<br>
they are actually <i>platforms</i> that their user and developer<br>
communities have built tools on top of. The downside of this<br>
preference for power and flexibility is that these tools are often<br>
complex. Some people prefer simpler tools, e.g., <i>Stata</i> instead of<br>
<i>R</i> or <i>Jupyter Notebooks</i> instead of <i>Emacs with Org mode</i>, and that<br>
is perfectly reasonable.
</p>
<p>
My <a rel="nofollow" href="http://tutorials.iq.harvard.edu/DataScienceTools/DataScienceTools.html" target="_blank">data science tools</a> workshop notes describe some alternative tools<br>
and is a good place to start if you’re not sure what you should use<br>
for a particular task.
</p>
</div>
</div>
	<p>
<div>
<div>To <strong>leave a comment</strong> for the author, please follow the link and comment on their blog: <strong><a href="http://people.fas.harvard.edu/~izahn/posts/my-toolbox/"> Ista Zahn (Posts about R)</a></strong>.</div>
<hr>
<a rel="nofollow" href="https://www.r-bloggers.com/">R-bloggers.com</a> offers <strong><a rel="nofollow" href="https://feedburner.google.com/fb/a/mailverify?uri=RBloggers">daily e-mail updates</a></strong> about <a rel="nofollow" href="https://www.r-project.org/" title="The R Project for Statistical Computing">R</a> news and <a rel="nofollow" href="https://www.r-bloggers.com/search/tutorial" title="R tutorials">tutorials</a> on topics such as: <a rel="nofollow" href="https://www.r-bloggers.com/search/data%20science" title="Data science">Data science</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/Big%20Data" title="Big Data">Big Data, <a rel="nofollow" href="https://www.r-users.com/" title="R jobs">R jobs</a>, visualization (<a rel="nofollow" href="https://www.r-bloggers.com/search/ggplot2" title="ggplot and ggplot2 tutorials">ggplot2</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/boxplot" title="Boxplots using lattice and ggplot2 tutorials">Boxplots</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/map" title="Maps and gis">maps</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/animation" title="Animation in R">animation</a>), programming (<a rel="nofollow" href="https://www.r-bloggers.com/search/RStudio" title="RStudio IDE for R">RStudio</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/sweave" title="Sweave and literate programming">Sweave</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/LaTeX" title="LaTeX in R">LaTeX</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/SQL" title="SQL and databases">SQL</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/eclipse" title="Eclipse IDE for R">Eclipse</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/git" title="git and github, Version Control System">git</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/hadoop" title="Large data in R using Hadoop">hadoop</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/Web+Scraping" title="Web Scraping of google, facebook, yahoo, twitter and more using R">Web Scraping</a>) statistics (<a rel="nofollow" href="https://www.r-bloggers.com/search/regression" title="Regressions and ANOVA analysis tutorials">regression</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/PCA" title="principal component analysis tutorial">PCA</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/time+series" title="Time series">time series</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/trading" title="finance trading">trading</a>) and more...
<img width="1" alt="" src="http://feeds.feedburner.com/~r/RBloggers/~4/C3PADyxx_Co" height="1"></a></div>