<p><div><div><div><div data-show-faces="false" data-height="21" data-width="100" data-layout="button_count" data-send="true" data-href="https://www.r-bloggers.com/rtimicropem-using-an-r-package-as-platform-for-harmonized-cleaning-of-data-from-rti-micropem-air-quality-sensors/"></div></div><div></div></div><div></div></div>
<div>
(This article was first published on <strong><a href="http://ropensci.org/blog/blog/2017/08/29/rtimicropem"> rOpenSci Blog</a></strong>, and kindly contributed to <a rel="nofollow" href="https://www.r-bloggers.com/">R-bloggers)</a> </div>
<p>As you might remember from <a rel="nofollow" href="https://ropensci.org/blog/blog/2017/02/21/ropenaq" target="_blank">my blog post about <code>ropenaq</code></a>, I work as a data manager and statistician for an <a rel="nofollow" href="http://www.chaiproject.org/" target="_blank">epidemiology project called CHAI</a> for Cardio-vascular health effects of air pollution in Telangana, India. One of our interests in CHAI is determining exposure, and sources of exposure, to PM2.5 which are very small particles in the air that have diverse adverse health effects. You can find more details about CHAI <a rel="nofollow" href="https://www.ncbi.nlm.nih.gov/pubmed/28606699" target="_blank">in our recently published protocol paper</a>. In this blog post that partly corresponds to the content of <a rel="nofollow" href="http://sched.co/AxrS" target="_blank">my useR! 2017 lightning talk</a>, I'll present a package we wrote for dealing with the output of a scientific device, which might remind you of similar issues in your experimental work.</p>
<h1>Why write the <code>rtimicropem</code> package?</h1>
<p><img alt="" src="https://i0.wp.com/ropensci.org/assets/blog-images/2017-08-29-rtimicropem/Allequipment_Frontview_cropped.jpg?w=456" data-recalc-dims="1"></p>
<p>Part of the CHAI project is a panel study involving about 40 people wearing several devices, as you see above. The devices include a GPS, an accelerometer, a wearable camera, and a PM2.5 monitor outputting time-resolved data (the grey box on the left). Basically, with this device, the RTI MicroPEM, we get one PM2.5 exposure value every 10 seconds. This is quite exciting, right? Except that we have two main issues with it…</p>
<p><img alt="" src="https://i2.wp.com/ropensci.org/assets/blog-images/2017-08-29-rtimicropem/screenshot_output2.png?w=456" data-recalc-dims="1"></p>
<p>First of all, the output of the device, a file with a &quot;.csv&quot; extension corresponding to a session of measurements, in our case 24 hours of measurements, is not really a csv. The header contains information about settings of the device for that session, and then comes the actual table with measurements.</p>
<p>Second, since the RTI MicroPEMs are nice devices but also a work-in-progress, we had some problems with the data, such as negative relative humidity. Because of these issues, we decided to write an R package whose three goals were to:</p>
<ul>
<li>
<p>Transform the output of the device into something more usable.</p>
</li>
<li>
<p>Allow the exploration of individual files after a day in the field.</p>
</li>
<li>
<p>Document our data cleaning process.</p>
</li>
</ul>
<p>We chose R because everything else in our project, well data processing, documentation and analysis, was to be implemented in R, and because we wanted other teams to be able to use our package.</p>
<h1>Features of <code>rtimicropem</code>: transform, explore and learn about data cleaning</h1>
<p>First things first, our package lives <a rel="nofollow" href="https://github.com/ropensci/rtimicropem" target="_blank">here</a> and is <a rel="nofollow" href="https://cran.r-project.org/web/packages/rtimicropem/index.html" target="_blank">on CRAN</a>. It has a <a rel="nofollow" href="http://ropensci.github.io/rtimicropem/" target="_blank">nice documentation website</a> thanks to <a rel="nofollow" href="https://github.com/hadley/pkgdown" target="_blank"><code>pkgdown</code></a>.</p>
<h2>Transform and explore single files</h2>
<p>In <code>rtimicropem</code> after the use of the <code>convert_output</code> function, one gets an object of the R6 class <code>micropem</code> class. Its fields include the settings and measurements as two <code>data.frames</code>, and it has methods such as <code>summary</code> and <code>plot</code> for which you see the static output below (no unit on this exploratory plot).</p>
<p><img alt="" src="https://i2.wp.com/ropensci.org/assets/blog-images/2017-08-29-rtimicropem/plotexample.png?w=456" data-recalc-dims="1"></p>
<p>The plot method can also outputs an interactive graph thanks to <a rel="nofollow" href="http://hafen.github.io/rbokeh/" target="_blank"><code>rbokeh</code></a>.</p>
<p>While these methods can be quite helpful to explore single files as an R user, they don't help non R users a lot. Because we wanted members of our team working in the field to be able to explore and check files with no R knowledge, we created a Shiny app that allows to upload individual files and then look at different tabs, including one with a plot, one with the summary of measurements, etc. This way, it was easy to spot a device failure for instance, and to plan a new measurement session with the corresponding participant.</p>
<h2>Transform a bunch of files</h2>
<p>At the end of the CHAI data collection, we had more than 250 MicroPEM files. In order to prepare them for further processing we wrote the <code>batch_convert</code> function that saves the content of any number of MicroPEM files as two (real!) csv, one with the measurements, one with the settings.</p>
<h2>Learn about data cleaning</h2>
<p>As mentioned previously, we experienced issues with MicroPEM data quality. Although we had heard other teams complain of similar problems, in the literature there were very few details about data cleaning. We decided to gather information from other teams and the manufacturer and to document our own decisions, e.g. remove entire files based on some criteria, in a <a rel="nofollow" href="http://ropensci.github.io/rtimicropem/articles/chai_data_cleaning.html" target="_blank">vignette of the package</a>. This is our transparent answer to the question &quot;What was your experience with MicroPEMs?&quot; which we get often enough from other scientists interested in PM2.5 exposure.</p>
<h1>Place of <code>rtimicropem</code> in the R package ecosystem</h1>
<p>When preparing <code>rtimicropem</code> submission to rOpenSci, I started wondering whether one would like to have one R package for each scientific device out there. In our case, having the weird output to deal with, and the lack of a central data issues documentation place, were enough of a motivation. But maybe one could hope that manufacturers of scientific devices would focus a bit more on making the output format analysis-friendly, and that the open documentation of data issues would be language-agnostic and managed by the manufacturers themselves. In the meantime, we're quite proud to have taken the time to create and share our experience with <code>rtimicropem</code>, and have already heard back from a few users, including one who found the package via googling &quot;RTI MicroPEM data&quot;! Another argument I in particular have to write R packages for dealing with scientific data is that it might motivate people to learn R, but this is maybe a bit evil.</p>
<p>What about the place of <code>rtimicropem</code> in the rOpenSci package collection? After <a rel="nofollow" href="https://github.com/ropensci/onboarding/issues/126" target="_blank">very useful reviews</a> by <a rel="nofollow" href="https://github.com/LucyMcGowan" target="_blank">Lucy D'Agostino McGowan</a> and <a rel="nofollow" href="https://github.com/karawoo" target="_blank">Kara Woo</a> our package got onboarded which we were really thankful for and happy about. Another package I can think off the top of my head to deal with the output of a scientific tool is <a rel="nofollow" href="https://ropensci.org/blog/blog/2017/02/06/plater-blog-post" target="_blank"><code>plater</code></a>. Let me switch roles from CHAI team member to rOpenSci onboarding co-editor here and do some advertisement… Such packages are unlikely to become the new <code>ggplot2</code> but their specialization doesn't make them less useful and they fit very well in the &quot;data extraction&quot; of the <a rel="nofollow" href="https://github.com/ropensci/onboarding/blob/master/policies.md" target="_blank">onboarding categories</a>. So if you have written such a package, please consider submitting it! It'll get better thanks to review and might get more publicity as part of a larger software ecosystem. For the <code>rtimicropem</code> submission we took advantage of the joint submission process of rOpenSci and the Journal of Open Source Software, JOSS, so now our piece of software has <a rel="nofollow" href="http://joss.theoj.org/papers/7ead5e9a445da3e885d99247c5d6e58e" target="_blank">its JOSS paper with a DOI</a>. And hopefully, having more submissions of packages for scientific hardware might inspire R users to package up the code they wrote to use the output of their scientific tools!</p>
	<p>
<div>
<div>To <strong>leave a comment</strong> for the author, please follow the link and comment on their blog: <strong><a href="http://ropensci.org/blog/blog/2017/08/29/rtimicropem"> rOpenSci Blog</a></strong>.</div>
<hr>
<a rel="nofollow" href="https://www.r-bloggers.com/">R-bloggers.com</a> offers <strong><a rel="nofollow" href="https://feedburner.google.com/fb/a/mailverify?uri=RBloggers">daily e-mail updates</a></strong> about <a rel="nofollow" href="https://www.r-project.org/" title="The R Project for Statistical Computing">R</a> news and <a rel="nofollow" href="https://www.r-bloggers.com/search/tutorial" title="R tutorials">tutorials</a> on topics such as: <a rel="nofollow" href="https://www.r-bloggers.com/search/data%20science" title="Data science">Data science</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/Big%20Data" title="Big Data">Big Data, <a rel="nofollow" href="https://www.r-users.com/" title="R jobs">R jobs</a>, visualization (<a rel="nofollow" href="https://www.r-bloggers.com/search/ggplot2" title="ggplot and ggplot2 tutorials">ggplot2</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/boxplot" title="Boxplots using lattice and ggplot2 tutorials">Boxplots</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/map" title="Maps and gis">maps</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/animation" title="Animation in R">animation</a>), programming (<a rel="nofollow" href="https://www.r-bloggers.com/search/RStudio" title="RStudio IDE for R">RStudio</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/sweave" title="Sweave and literate programming">Sweave</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/LaTeX" title="LaTeX in R">LaTeX</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/SQL" title="SQL and databases">SQL</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/eclipse" title="Eclipse IDE for R">Eclipse</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/git" title="git and github, Version Control System">git</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/hadoop" title="Large data in R using Hadoop">hadoop</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/Web+Scraping" title="Web Scraping of google, facebook, yahoo, twitter and more using R">Web Scraping</a>) statistics (<a rel="nofollow" href="https://www.r-bloggers.com/search/regression" title="Regressions and ANOVA analysis tutorials">regression</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/PCA" title="principal component analysis tutorial">PCA</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/time+series" title="Time series">time series</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/trading" title="finance trading">trading</a>) and more...
<img width="1" alt="" src="http://feeds.feedburner.com/~r/RBloggers/~4/QoFFGUiMuKg" height="1"></a></div>