<p><div><div><div><div data-show-faces="false" data-height="21" data-width="100" data-layout="button_count" data-send="true" data-href="https://www.r-bloggers.com/onboarding-visdat-a-tool-for-preliminary-visualisation-of-whole-dataframes/"></div></div><div></div></div><div></div></div>
<div>
(This article was first published on <strong><a href="http://ropensci.org/blog/blog/2017/08/22/visdat"> rOpenSci Blog</a></strong>, and kindly contributed to <a rel="nofollow" href="https://www.r-bloggers.com/">R-bloggers)</a> </div>
<blockquote>
<p>Take a look at the data</p>
</blockquote>
<p>This is a phrase that comes up when you first get a dataset.</p>
<p>It is also ambiguous. Does it mean to do some exploratory modelling? Or make some histograms, scatterplots, and boxplots? Is it both?</p>
<p>Starting down either path, you often encounter the non-trivial growing pains of working with a new dataset. The mix ups of data types – height in cm coded as a factor, categories are numerics with decimals, strings are datetimes, and somehow datetime is one long number. And let's not forget everyone's favourite: missing data.</p>
<p>These growing pains often get in the way of your basic modelling or graphical exploration. So, sometimes you can't even start to take a look at the data, and that is frustrating.</p>
<p>The <a rel="nofollow" href="https://github.com/ropensci/visdat" target="_blank"><code>visdat</code></a> package aims to make this preliminary part of analysis easier. It focuses on creating visualisations of whole dataframes, to make it easy and fun for you to &quot;get a look at the data&quot;.</p>
<p>Making <code>visdat</code> was fun, and it was easy to use. But I couldn't help but think that maybe <code>visdat</code> could be more.</p>
<ul>
<li> I felt like the code was a little sloppy, and that it could be better.</li>
<li> I wanted to know whether others found it useful.</li>
</ul>
<p>What I needed was someone to sit down and read over it, and tell me what they thought. And hey, a publication out of this would certainly be great.</p>
<p>Too much to ask, perhaps? No. Turns out, not at all. This is what the rOpenSci <a rel="nofollow" href="https://github.com/ropensci/onboarding" target="_blank">onboarding process</a> provides.</p>
<h3>rOpenSci onboarding basics</h3>
<p>Onboarding a package onto rOpenSci is an open peer review of an R package. If successful, the package is migrated to rOpenSci, with the option of putting it through an accelerated publication with <a rel="nofollow" href="http://joss.theoj.org/" target="_blank">JOSS</a>.</p>
<p><strong>What's in it for the author?</strong></p>
<ul>
<li>Feedback on your package</li>
<li>Support from rOpenSci members</li>
<li>Maintain ownership of your package</li>
<li>Publicity from it being under rOpenSci</li>
<li>Contribute something to rOpenSci</li>
<li>Potentially a publication</li>
</ul>
<p><strong>What can rOpenSci do that CRAN cannot?</strong></p>
<p>The rOpenSci onboarding process provides a stamp of quality on a package that you do not necessarily get when a package is on CRAN <sup><a rel="nofollow" href="http://ropensci.org/blog/blog/2017/08/22/visdat#fn1" target="_blank">1</a></sup>. Here's what rOpenSci does that CRAN cannot:</p>
<ul>
<li>Assess documentation readability / usability</li>
<li>Provide a code review to find weak points / points of improvement</li>
<li>Determine whether a package is overlapping with another.</li>
</ul>
<p>So I submitted <code>visdat</code> to the onboarding process. For me, I did this for three reasons.</p>
<ol>
<li>So <code>visdat</code> could become a better package</li>
<li>Pending acceptance, I would get a publication in JOSS</li>
<li>I get to contribute back to rOpenSci</li>
</ol>
<p>Submitting the package was actually quite easy – you go to <a rel="nofollow" href="https://github.com/ropensci/onboarding/issues/new" target="_blank">submit an issue</a> on the onboarding page on GitHub, and it provides a magical template for you to fill out <sup><a rel="nofollow" href="http://ropensci.org/blog/blog/2017/08/22/visdat#fn2" target="_blank">2</a></sup>, with no submission gotchas – this could be the future <sup><a rel="nofollow" href="http://ropensci.org/blog/blog/2017/08/22/visdat#fn3" target="_blank">3</a></sup>. Within 2 days of submitting the issue, I had a response from the editor, <a rel="nofollow" href="https://github.com/noamross" target="_blank">Noam Ross</a>, and two reviewers assigned, <a rel="nofollow" href="https://github.com/batpigandme" target="_blank">Mara Averick</a>, and <a rel="nofollow" href="https://github.com/seaaan" target="_blank">Sean Hughes</a>.</p>
<p>I <a rel="nofollow" href="https://github.com/ropensci/onboarding/issues/87" target="_blank">submitted <strong>visdat</strong></a> and waited, somewhat apprehensively. <em>What would the reviewers think?</em></p>
<p>In fact, Mara Averick wrote a post: &quot;So you (don't) think you can review a package&quot; about her experience evaluating <code>visdat</code> as a first-time reviewer.</p>
<h1>Getting feedback</h1>
<h3>Unexpected extras from the review</h3>
<p>Even before the review started officially, I got some great concrete feedback from Noam Ross, the editor for the <code>visdat</code> submission.</p>
<ul>
<li>Noam used the <a rel="nofollow" href="https://github.com/MangoTheCat/goodpractice" target="_blank">goodpractice</a> package, to identify bad code patterns and other places to immediately improve upon in a concrete way. This resulted in me:
<ul>
<li>Fixing error prone code such as using <code>1:length(...)</code>, or <code>1:nrow(...)</code></li>
<li>Improving testing using the visualisation testing software <a rel="nofollow" href="https://github.com/lionel-/vdiffr" target="_blank"><code>vdiffr</code></a>)</li>
<li>Reducing long code lines to improve readability</li>
<li>Defining global variables to avoid a NOTE (&quot;no visible binding for global variable&quot;)</li>
</ul>
</li>
</ul>
<p>So before the review even started, <code>visdat</code> is in better shape, with 99% test coverage, and clearance from <code>goodpractice</code>.</p>
<h3>The feedback from reviewers</h3>
<p>I received prompt replies from the reviewers, and I got to hear really nice things like &quot;I think <code>visdat</code> is a very worthwhile project and have already started using it in my own work.&quot;, and &quot;Having now put it to use in a few of my own projects, I can confidently say that it is an incredibly useful early step in the data analysis workflow. <code>vis_miss()</code>, in particular, is helpful for scoping the task at hand …&quot;. In addition to these nice things, there was also great critical feedback from Sean and Mara.</p>
<p>A common thread in both reviews was that the way I initially had <code>visdat</code> set up was to have the first row of the dataset at the bottom left, and the variable names at the bottom. However, this doesn't reflect what a dataframe typically looks like – with the names of the variables at the top, and the first row also at the top. There was also suggestions to add the percentage of missing data in each column.</p>
<p><img src="https://i1.wp.com/ropensci.org/assets/blog-images/2017-08-22-visdat/blog-vis-dat-compare-1.png?w=50%25" data-recalc-dims="1"><img src="https://i2.wp.com/ropensci.org/assets/blog-images/2017-08-22-visdat/blog-vis-dat-compare-2.png?w=50%25" data-recalc-dims="1"><img src="https://i0.wp.com/ropensci.org/assets/blog-images/2017-08-22-visdat/blog-vis-dat-compare-3.png?w=50%25" data-recalc-dims="1"><img src="https://i0.wp.com/ropensci.org/assets/blog-images/2017-08-22-visdat/blog-vis-dat-compare-4.png?w=50%25" data-recalc-dims="1"></p>
<p><em>On the left are the old <code>visdat</code> and vismiss plots, and on the right are the new <code>visdat</code> and vismiss plots.</em></p>
<p>Changing this makes the plots make a lot more sense, and read better.</p>
<p>Mara made me aware of the warning and error messages that I had let crop up in the package. This was something I had grown to accept – the plot worked, right? But Mara pointed out that from a user perspective, seeing these warnings and messages can be a negative experience for the user, and something that might stop them from using it – how do they know if their plot is accurate with all these warnings? Are they using it wrong?</p>
<p>Sean gave practical advice on reducing code duplication, explaining how to write general construction method to prepare the data for the plots. Sean also explained how to write C++ code to improve the speed of <code>vis_guess()</code>.</p>
<p>From both reviewers I got nitty gritty feedback about my writing – places where documentation that was just a bunch of notes I made, or where I had reversed the order of a statement.</p>
<h1>What did I think?</h1>
<p>I think that getting feedback in general on your own work can be a bit hard to take sometimes. We get attached to our ideas, we've seen them grow from little thought bubbles all the way to &quot;all growed up&quot; R packages. I was apprehensive about getting feedback on <code>visdat</code>. But the feedback process from rOpenSci was, as Tina Turner put it, <a rel="nofollow" href="https://www.youtube.com/watch?v=mNU3aIJs88g" target="_blank">&quot;simply the best&quot;</a>.</p>
<p>Boiling down the onboarding review process down to a few key points, I would say it is <strong>transparent</strong>, <strong>friendly</strong>, and <strong>thorough</strong>.</p>
<p>Having the entire review process on GitHub means that everyone is accountable for what they say, and means that you can track exactly what everyone said about it <em>in one place</em>. No email chain hell with (mis)attached documents, accidental reply-alls or single replies. The whole internet is cc'd in on this discussion.</p>
<p>Being an rOpenSci initiative, the process is incredibly <strong>friendly</strong> and respectful of everyone involved. Comments are upbeat, but are also, importantly <strong>thorough</strong>, providing constructive feedback.</p>
<h1>So what does <code>visdat</code> look like?</h1>
<div>
<pre><code data-lang="r"><span></span><span>library</span><span>(</span>visdat<span>)</span>

vis_dat<span>(</span>airquality<span>)</span>
</code></pre>
</div>
<p><img alt="visdat-example" src="https://i1.wp.com/ropensci.org/assets/blog-images/2017-08-22-visdat/blog-vis-dat-example-1.png?w=456" data-recalc-dims="1"></p>
<p>This shows us a visual analogue of our data, the variable names are shown on the top, and the class of each variable is shown, along with where missing data.</p>
<p>You can focus in on missing data with <code>vis_miss()</code></p>
<div>
<pre><code data-lang="r"><span></span>vis_miss<span>(</span>airquality<span>)</span>
</code></pre>
</div>
<p><img alt="vis-miss-example" src="https://i2.wp.com/ropensci.org/assets/blog-images/2017-08-22-visdat/blog-vis-miss-aq-1.png?w=456" data-recalc-dims="1"></p>
<p>This shows only missing and present information in the data. In addition to <code>vis_dat()</code> it shows the percentage of missing data for each variable and also the overall amount of missing data. <code>vis_miss()</code> will also indicate when a dataset has no missing data at all, or a very small percentage.</p>
<h1>The future of <code>visdat</code></h1>
<p>There are some really exciting changes coming up for <code>visdat</code>. The first is making a plotly version of all of the figures that provides useful tooltips and interactivity. The second and third changes to bring in later down the track are to include the idea of visualising expectations, where the user can search their data for particular things, such as particular characters like &quot;~&quot; or values like -99, or -0, or conditions &quot;x > 101&quot;, and visualise them. Another final idea is to make it easy to visually compare two dataframes of differing size. We also want to work on providing consistent palettes for particular datatypes. For example, character, numerics, integers, and datetime would all have different (and consistently different) colours.</p>
<p>I am very interested to hear how people use <code>visdat</code> in their work, so if you have suggestions or feedback I would love to hear from you! The best way to leave feedback is by <a rel="nofollow" href="https://github.com/ropensci/visdat/issues/new" target="_blank">filing an issue</a>, or perhaps sending me an email at nicholas [dot] tierney [at] gmail [dot] com.</p>
<h1>The future of your R package?</h1>
<p>If you have an R package you should give some serious thought about submitting it to the rOpenSci through their onboarding process. There are very clear guidelines on their <a rel="nofollow" href="https://github.com/ropensci/onboarding" target="_blank">onboarding GitHub page</a>. If you aren't sure about <a rel="nofollow" href="https://github.com/ropensci/onboarding/blob/master/policies.md" target="_blank">package fit</a>, you can submit a <a rel="nofollow" href="https://github.com/ropensci/onboarding/issues?q=is%3Aissue+label%3A0%2Fpresubmission" target="_blank">pre-submission enquiry</a> – the editors are nice and friendly, and a positive experience awaits you!</p>
<div>
<hr>
<ol>
<li>
<p>CRAN is an essential part of what makes the r-project successful and certainly without CRAN R simply would not be the language that it is today. The tasks provided by the rOpenSci onboarding require human hours, and there just isn't enough spare time and energy amongst CRAN managers. <a rel="nofollow" rev="footnote" href="http://ropensci.org/blog/blog/2017/08/22/visdat#fnref1" target="_blank">↩</a></p>
</li>
<li>
<p>Never used GitHub? Don't worry, creating an account is easy, and the template is all there for you. You provide very straightforward information, and it's all there at once. <a rel="nofollow" rev="footnote" href="http://ropensci.org/blog/blog/2017/08/22/visdat#fnref2" target="_blank">↩</a></p>
</li>
<li>
<p>With some journals, the submission process means you aren't always clear what information you need ahead of time. Gotchas include things like &quot;what is the residential address of every co-author&quot;, or getting everyone to sign a copyright notice. <a rel="nofollow" rev="footnote" href="http://ropensci.org/blog/blog/2017/08/22/visdat#fnref3" target="_blank">↩</a></p>
</li>
</ol>
</div>
	<p>
<div>
<div>To <strong>leave a comment</strong> for the author, please follow the link and comment on their blog: <strong><a href="http://ropensci.org/blog/blog/2017/08/22/visdat"> rOpenSci Blog</a></strong>.</div>
<hr>
<a rel="nofollow" href="https://www.r-bloggers.com/">R-bloggers.com</a> offers <strong><a rel="nofollow" href="https://feedburner.google.com/fb/a/mailverify?uri=RBloggers">daily e-mail updates</a></strong> about <a rel="nofollow" href="https://www.r-project.org/" title="The R Project for Statistical Computing">R</a> news and <a rel="nofollow" href="https://www.r-bloggers.com/search/tutorial" title="R tutorials">tutorials</a> on topics such as: <a rel="nofollow" href="https://www.r-bloggers.com/search/data%20science" title="Data science">Data science</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/Big%20Data" title="Big Data">Big Data, <a rel="nofollow" href="https://www.r-users.com/" title="R jobs">R jobs</a>, visualization (<a rel="nofollow" href="https://www.r-bloggers.com/search/ggplot2" title="ggplot and ggplot2 tutorials">ggplot2</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/boxplot" title="Boxplots using lattice and ggplot2 tutorials">Boxplots</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/map" title="Maps and gis">maps</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/animation" title="Animation in R">animation</a>), programming (<a rel="nofollow" href="https://www.r-bloggers.com/search/RStudio" title="RStudio IDE for R">RStudio</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/sweave" title="Sweave and literate programming">Sweave</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/LaTeX" title="LaTeX in R">LaTeX</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/SQL" title="SQL and databases">SQL</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/eclipse" title="Eclipse IDE for R">Eclipse</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/git" title="git and github, Version Control System">git</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/hadoop" title="Large data in R using Hadoop">hadoop</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/Web+Scraping" title="Web Scraping of google, facebook, yahoo, twitter and more using R">Web Scraping</a>) statistics (<a rel="nofollow" href="https://www.r-bloggers.com/search/regression" title="Regressions and ANOVA analysis tutorials">regression</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/PCA" title="principal component analysis tutorial">PCA</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/time+series" title="Time series">time series</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/trading" title="finance trading">trading</a>) and more...
<img width="1" alt="" src="http://feeds.feedburner.com/~r/RBloggers/~4/0-3G5CbJbEE" height="1"></a></div>