At the end of March, Jonah Gabry and I released
bayesplot 1.5.0. The major additions
to the package were visualizations using ridgelines and a new plot for PIT
diagnostics from LOO validation. I don’t know what that LOO PIT thing is yet, so
I’ll ta...
Who says there's no art in mathematics? I've long admired the generative art that Thomas Lin Peterson occasionally posts (and that you can see on Instagram), and though he's a prolific R user I'm not quite sure how he makes his art. Marcus Volz has another beautiful portfolio of generative art, and has also created an R package you can use to create your own designs: the mathart package. Generative art uses mathematical equations and standard graphical rendering tools (point and lines, color and transparency) to create designs. The mathart package provides a number of R functions to create some...
In the last months, I started increasingly using Rmd documents for preparing scientific reports, blog posts, etcetera. While I really like the flexibility offered by the system, one thing that I thought could be improved is the support for easily inserting tables. So, “inspired” also by the recent addition of the excellent insert image addin in blogdown, I decided to give it a go and try to implement some kind of addin to facilitate table insertion in Rmd documents.
After struggling a bit due to my rather nonexistent shiny skills, in the end I managed to obtain a “basic but useful” (IMO) addin.
See full post on my new blogdown page here: https://lbusett.netlify.com/post/a-new-rstudio-addin-to-facilitate-inserting-tables-in-rmarkdown-documents/
This post describes the tools I currently use for working with data.
People often ask me to recommend specific tools, and I always
hesitate, because so much boils down to personal preference. I
recently added a workshop to the DSS lineup providing an ...
I’ve often said that the foundation of data science is data analysis. That’s not to say that data analysis and data science are 100% synonymous (they’re not), but rather that you need to master the basic tools of data analysis before you can get into the more advanced skills that make up data science. Said […]
The post Why you should master the small multiple chart appeared first on SHARP SIGHT.
In this post we’re going to explore the Chi Squared Goodness of Fit test using M&amp;M’s as our subject material. From there we’ll take a look at simultaneous confidence intervals a.k.a. multiple comparisons. On the R side of things we’ll make use of some old friends like ggplot2 and dplyr but we’ll also make use of two packages that were new to me scimp and ggimage. We’ll also make heavy use of the kable package to make our output tables look nicer.
In this blogpost, we will show 6 keyword extraction techniques which allow to find keywords in plain text. Keywords are frequently occuring words which occur somehow together in plain text. Common examples are New York, Monte Carlo, Mixed Models, Brussels Hoofdstedelijk Gewest, Public Transport, Central Station, p-values, ...
If you master these techniques, it will allow you to easily step away from doing simple word frequency statistics to more business-relevant text summarisation. For this, we will use the udpipe R package (docs at https://CRAN.R-project.org/package=udpipe or https://bnosac.github.io/udpipe/en) which is the core R package you need for doing this type of t ext processing.We'll basically show how to easily extract keywords as follows:
1. Find keywords by doing Parts of Speech tagging in order to identify nouns2. Find keywords based on Collocations and Co-occurrences3. Find keywords based on the Textrank algorithm4. Find keywords based on RAKE (rapid automatic keyword extraction)5. Find keywords by looking for Phrases (noun phrases / verb phrases)6. Find keywords based on results of dependency parsing (getting the subject of the text)
These techniques will allow you to move away from showing silly word graphs to more relevant graphs containing keywords.
Example
As an example we are going to use feedback in Spanish of customers going to an AirBnB appartment in Brussels. This data is part of the udpipe R package. We extract the Spanish text and annotate it using the udpipe R package. Annotation performs tokenisation, parts of speech tagging, lemmatisation and dependency parsing.
library(udpipe)library(textrank)## First step: Take the Spanish udpipe model and annotate the text. Note: this takes about 3 minutesdata(brussels_reviews)comments
The Reliability chapter of my Empirical software engineering book has been added to the draft pdf (download here). I have been working on this draft for four months and it still needs lots of work; time to move on and let it stew for a while. Part of the problem is lack of public data; […]
Need help with R, data visualization, and/or statistics? Hire me or attend my 2 day workshop! As the maintainer of the R package plotly, I’m certainly aware that ggplotly() is not perfect.1 And, even when conversion from ggplot2 to plotly ‘works’ it can leave some things to be desired. For example, it might take a […]
x[idxs + 1] or x[idxs + 1L]? That is the question.
Assume that we have a vector $x$ of $n = 100,000$ random values, e.g.
__ n x idxs y y typeof(idxs)
[1] &quot;integer&quot;
__ typeof(idxs + 1)
[1] &quot;double&quot;
__ typeof(1)
[1] &quot;double&quot;
Note also that doubles (aka “numerics” in R) take up twice the amount of memory:
__ object.size(idxs)
400040 bytes
__ object.size(idxs + 1)
800032 bytes
which is because integers are stored as 4 bytes and doubles as 8 bytes.
By using 1L instead, we can avoid this coercion from integers to doubles:
__ typeof(idxs)
[1] &quot;integer&quot;
__ typeof(idxs + 1L)
[1] &quot;integer&quot;
__ typeof(1L)
[1] &quot;integer&quot;
and we save some, otherwise wasted, memory;
__ object.size(idxs + 1L)
400040 bytes
Does it really matter for the overall performance? It should because less memory is allocated which always comes with some overhead. Possibly more importantly, by using objects that are smaller in memory, the more likely it is that elements can be found in the memory cache rather than in the RAM itself, i.e. the chance for cache hits increases. Accessing data in the cache is orders of magnitute faster than in RAM. Furthermore, we also avoid coercion/casting of doubles to integers when R adds one to each element, which may add some extra CPU overhead.
The performance gain is confirmed by running microbenchmark on the two alternatives:
__ microbenchmark::microbenchmark(
+ y
We are very pleased to announce the availability of an RStudio TensorFlow template for the Paperspace cloud desktop service.
If you don’t have local access to a modern NVIDIA GPU, your best bet is typically to run GPU intensive training jobs...
Six months ago we considered the question: Is the Government realising its ambition for SMEs on G-Cloud? This article reviews progress six months later, additionally considering the Digital Outcomes &amp; Specialists (DOS) framework.
The post Digital Marketplace. Six months later. appeared first on thinkr.
Need help with R, data viz, and/or stats? Work with me or attend my 2 day workshop! In my last post, we explored interactive visualizations of simple features (i.e., interactive maps) via ggplot2’s geom_sf() and plotly’s ggplotly(). This time we’ll make similar visualizations using plotly’s “non-ggplot2” mapping interfaces (namely plot_ly(), plot_geo(), and plot_mapbox()). Here’s a […]
R tip: first organize your tasks in terms of data, values, and desired transformation of values, not initially in terms of concrete functions or code. I know I write a lot about coding in R. But it is in the service of supporting statistics, analysis, predictive analytics, and data science. R without data is like … Continue reading R Tip: Think in Terms of Values
In my last two posts I have been writing about the task of using R to “drive” MS Excel. The first post focused on just the basic mechanics of getting my colleague what she needed. The second post picked up with some ugly inefficient code and made it better using lapply and a for loop, just good old fashioned automation (the thing that computers excel at). Today I’ll take it another step and show how to produce the same graphs in R using ggplot2 as well as how to write some simple functions to make your programming life easier.
You have to have been living under a rock to not know about Cloudflare’s new 1.1.1.1 DNS offering. I won’t go into “privacy”, “security” or “speed” concepts in this post since that’s a pretty huge topic to distill for folks given the, now, plethora of confusing (and pretty technical) options that exist to support one... Continue reading →
I am happy to introduce oneliner, a package that implements the one-line-style as a third-party style guide ready to use with styler. Hence, after the tidyverse style guide, this is the first third-party style guide for styler I am aware of - and a particularly useful one.
The fs package makes it super quick and easy to find out just how much “package hoarding” you’ve been doing: While you can modify the above and peruse the list of packages/directories in tabular format or programmatically, you can also do a bit more work to get a visual overview of package size (click/tap the... Continue reading →
Track performance over time - A Bump Chart is a special form of a line plot. This kind of plot is designed for exploring changes in rank over time. The focus here is usually on comparing the position or performance of multiple observation...
An arithmetic Le Monde mathematical puzzle: A magical integer m is such that the remainder of the division of any prime number p by m is either a prime number or 1. What is the unique magical integer between 25 and 100? And is there any less than 25? The question is dead easy to […]
Caution - this blog post discusses homicide and suicide statistics and could be upsetting. If you feel disturbed or unhappy then here is a list of ways to get some emergency counselling help around the world.
A grim research question
In my last post...
Recently, while working on the Azure Data Lake R extension, I had to figure out a good way to create a zip file containing a package together with all its dependencies. This came down to understanding where does R store and search for packages. Despite the documentation, it did require additional reading and experimentation. First, […]
The post Package Paths in R appeared first on Quintuitive.
 It’s the ten crack commandments, what? homie can’t tell me nothing about this code Can’t tell me nothing about these #rstats Number 1, make a function from a script. Everyone knows we’re to busy to be copy/pasting shit http://adv-r.had.co.nz/Functions.html Number 2, never let ’em know your data manipulation moves. Don’t you know Bad Boys move in silence and […]
Discover who is coming to eRum 2018 in Budapest while playing with an interactive data visualization in D3.js. Vist visualization’s page.
You can meet me (Olga) at the conference where I’ll be speaking about cool and open source packages ...
Since its inception over 40 years ago, when S (R's predecessor) was just a sketch on John Chambers' wall at Bell Labs, R has always been a language for providing interfaces. I was reminded of this during Dirk Eddelbuettel's presentation at the Chicago R User Group meetup last night, where he enumerated Chambers' three principles behind its design (from his 2016 book, Extending R): Object: Everything that exists in R is an object Function: Everything that happens in R is a function call Interface: Interfaces to other software are a part of R The third principle &quot;Interface&quot; is demonstrated by...
Remember my blog post about automatic tools for improving R packages? One of these tools is Jim Hester’s lintr, a package that performs static code analysis. In my experience it mostly helps identifying too long code lines and missing space, although...
Introduction
I have received a few queries recently that can be categorized as “How do I collapse a list of categories or values into a shorter list of category or values?” For example, one user wanted to collapse species of fish into ...
Opening day is on the way Time to set up a persistent database to collect every pitch thrown in this year’s baseball season.
The mlbgameday package is designed to facilitate extract, transform and load for MLBAM “Gameday” data. The package is op...
Excited to see our new Hangul/Korean edition of “Practical Data Science with R” by Nina Zumel, John Mount, translated by Daekyoung Lim. Thank you for producing a handsome edition, Manning and JPub.kr!
Twitter's bot problem is well documented, influencing discourse on divisive topics like politics and civil rights. But it's getting harder and harder to spot such nefarious bots, who often borrow biographies and tweets from real (and often stolen) profiles to evade detection. (The New York Times recently published an outstanding feature on bots and follower factories.) Can we distinguish bots from real users using data science? Journalism professor Mike Kearney has developed an R package and Shiny web application that analyzes the tweets of a given user to calculate the probability that user is a bot. For example, it correctly...
In this post, I come back to my first experience teaching an advanced R course over the past month.
Content
This course was programmed for 10 sessions (3 hours each) and I initially wanted to talk about the following subjects:
R programming and g...
We’ve released the newest version of NIMBLE on CRAN and on our website. Version 0.6-10 primarily contains updates to the NIMBLE internals that may speed up building and compilation of models and algorithms, as well as a few bug fixes. Changes include: some steps of model and algorithm building and compilation are faster; compiled execution […]
After a lot of slacking around, I finally got to finishing the upgraded version of the inegiR package on CRAN. This version combines quite a few changes that I will explain further in this post.
New language
The biggest change upfront is the migration to english in both function names and documentation. The rationale behind this is to make it more accessible to developers around the world (I have recieved a few emails asking for translations). Also, the non-ASCII characters were not helpful. For the Mexican users, I assume that if you know R, you can probably find yourself around an english document.
To avoid crashing workflows, I left the legacy functions intact except for a warning to use the english version instead. An example of this is the commercial growth rate functions, which are:
# english
rate_commerce()
# spanish (old version)
tasa_comercio()Route API
With some help from Arturo Cárdenas and a revamp of the Sákbe API in INEGI, I was able to add functions to access route information.
The two main ones are:
# to search for a destiny id
inegi_destiny()
# to get route information
inegi_route()The first thing to understand is that INEGI has categorized sites in Mexico according to a “destiny id”. For example, the International Airport in Mexico city is destiny id #57. The inegi_destiny() function will help you find a destiny id based on a text criteria, sort of like googling the place and getting an address. Here is an example with a plaza in Monterrey:
# download on CRAN or newest dev version (if not accepted yet)
# install.packages(&quot;inegiR&quot;)
# or... # devtools::install_github(&quot;eflores89/inegiR&quot;)
library(inegiR)
library(knitr)
# to search for Macroplaza destiny id
token
(R⁶ == brief, low-expository posts) @yoniceedee suggested I look at the Cambridge Analytics “whistleblower” testimony proceedings: @hrbrmstr giving the term &quot;improving r&amp;d&quot; a whole new meaning … https://t.co/f1KA8U3htT — yoni sidi (@yoniceedee) March 29, 2018 I value the resources @yoniceedee tosses my way (they often end me down twisted paths like this one, though :-)... Continue reading →
You’ve heard me say it a thousand times: to master data science, you need to practice. You need to “practice small” by practicing individual techniques and functions. But you also need to “practice big” by working on larger projects. To get some practice, my recommendation is to find reasonably sized datasets online and plot them. […]
The post How to map public debt data with ggplot2 appeared first on SHARP SIGHT.
Hello everyone, welcome back! This post marks the beginning, hopefully, of your foray into the wonderful world of R and RStudio…and my delve into the odd vlog to go with the blog! I’ve brushed my hair for you, and I don’t do that for just anyone, so you’d better watch it at least once! So without further ado.
R R is an open source language released in 2001 that’s ideal for data wrangling and data science.
A Simple Method for Purging Mediation Effects among Independent Variables Mediation can occur when one independent variable swamps the effect of another, suggesting high correlation between the two variables. Though there are some great packages for mediation analysis out there, the simple intuition of its need is often ambiguous, especially for younger graduate students. Thus, … Continue reading Introducing purging: An R package for addressing mediation effects
In yesterday’s post I focused on the task of using R to “drive” MS Excel. I deliberately ended the post with a fully functioning (pun intended) but very ugly set of code. Why “ugly”? Well, because the last set of code wound up repeating 4 lines of code 12 times!
Beth Ashlee, Data Scientist
After the success of rstudio::conf 2017 this year the conference was back and bigger and better than ever with 1000+ attendees in sunny San D...
I’ll be talking about Deep Learning with Keras in R and Python at the following upcoming meetup:
Ruhr.Py 2018 on Wednesday, April 4th
Introducing Deep Learning with Keras and Python Keras is a high-level API written in Python for building and prot...
Here is an R tip. Want to re-map a column of values? Use a named vector as the mapping. Example: library(&quot;dplyr&quot;) library(&quot;wrapr&quot;) head(starwars[, qc(name, gender)]) # # A tibble: 6 x 2 # name gender # # 1 Luke Skywalker male # 2 C-3PO NA # 3 R2-D2 NA # 4 Darth Vader … Continue reading R Tip: Use Named Vectors to Re-Map Values
Newly published postcode open data!
Time is running out for you to send in your abstract submissions for this year’s EARL Conferences. If you have an R story you’d like to share then make sure you send...
In our next MünsteR R-user group meetup on Tuesday, April 17th, 2018 Kai Lichtenberg will talk about deep learning with Keras. You can RSVP here: http://meetu.ps/e/DDY1B/w54bW/f
Although neural networks have been around for quite a while now, dee...
Cats are great. Perhaps Hadley Wickham and Lionel Henry think so too given the wonderful choice of name for their purrr package. Hadley Wickham has also created a superb package called forcats, likely  an abbreviation of “for categoricals” but wittingly cat-themed, which is very, very useful to the data scientist. In the data science profession,… Continue reading Cats are great and so is the forcats R package →
This article shows you how to enter data so that you can easily open in statistics packages such as R, SAS, SPSS, or jamovi (code or GUI steps below). Excel has some statistical analysis capabilities, but they often provide incorrect answers. For … Continue reading →
Discriminant analysis is used when the variable to be predicted is categorical in nature. This analysis requires that the way to define data points to the respective categories is known which makes it different from cluster analysis where the classification criteria is not know. It works by calculating a score based on all the predictor … Continue reading Discriminant Analysis: Statistics All The Way
I have until recently made it a habit to draw a clear distinction about using R for data analysis and Microsoft Excel for other office productivity tasks. I know there are people who use Excel to process data and even (gasp) to teach statistics with it. But I’m a bit snobbish that way and to date all my efforts have been in getting data out of Excel and into R, either by simple methods like read.csv or if the task was more meaty by using Hadley Wickham’s marvelous readxl package.
The Azure Computer Vision API can extract all sorts of interesting information from images — tags describing objects found in the images, locations of detected faces, and more — but today I want to play around with just one: caption generation. I was inspired by @picdescbot on Twitter, which selects random images from Wikimedia Commons and generates a caption using the API. The results are sometimes impressive, sometimes funny, and sometimes bizarre. On the bizarre side, recurring motifs include &quot;lush green field&quot;, and &quot;a clock tower on a building&quot;, but the bot doesn't discriminate when the confidence score returned by...
When checking the Python t distribution random generator, np.random.standard_t(), I came upon this manual page, which actually does not explain how the random generator works but spends instead the whole page to recall Gosset’t test, illustrating its use on an energy intake of 11 women, but ending up misleading the readers by interpreting a .009 […]
This article explains how to call or run python from R. Both the tools have its own advantages and disadvantages. It's always a good idea to use the best packages and functions from both the tools and combine it. In data science world, these tools have a good market share in terms of usage. R is mainly known for data analysis, statistical modeling and visualization. While python is popular for deep learning and natural language processing.In recent KDnuggets Analytics software survey poll, Python and R were ranked top 2 tools for data science and machine learning. If you really want to boost your career in data science world, these are the languages you need to focus on.
Combine Python and R
RStudio developed a package called reticulate which provides a medium to run Python packages and functions from R.Install and Load Reticulate PackageRun the command below to get this package installed and imported to your system.# Install reticulate packageinstall.packages(&quot;reticulate&quot;)# Load reticulate packagelibrary(reticulate)
Check whether Python is available on your system
py_available()
It returns TRUE/FALSE. If it is TRUE, it means python is installed on your system.Import a python module within RYou can use the function import( ) to import a particular package or module.os
The abstract submission deadline for the Insurance Data Science conference at Cass Business School on 16 July 2018 is closing soon. You have until the 9th of April to submit your abstract.
Please send your abstract to info@insurancedatascience.org.
...
To post your R job on the next post Just visit  this link and post a new R job  to the R community. You can post a job for  free  (and there are also “featured job” options available for extra exposure). Current R jobs Job seekers:  please follow the links below to learn more and apply for your R job of interest: Featured Jobs Full-Time Director, Data Labs Pew Research Center – Posted by oneclickorders@jobtarget.com Washington District of Columbia, United States 12 Mar 2018 Full-Time Customer Success Rep RStudio Inc. – Posted by jclemens1 Anywhere 26 Feb 2018 More New R Jobs Full-Time Post-Doctoral Fellow – Data Scientist @ CIMMYT (eastern Africa) International Maize and Wheat Improvement Center – Posted by cimmyt-jobs Marsabit County Kenya 27 Mar 2018 Freelance Statistician / R Developer – for Academic Statistical Research Academic Research – Posted by empiricus Anywhere 27 Mar 2018 Full-Time Graduate Data Scientist JamieAi – Posted by JamieAi Anywhere 27 Mar 2018 Freelance Backtesting stock-trading strategy termod Anywhere 14 Mar 2018 Full-Time Director, Data Labs Pew Research Center – Posted by oneclickorders@jobtarget.com Washington District of Columbia, United States 12 Mar 2018 Freelance R Report Builder DDI – Posted by survey.support@ddiworld.com Anywhere 10 Mar 2018 Freelance Statistician/Econometrician – R Programmer for Academic Statistical Research (for 2-3 weeks) Academic […]
Another way of analysis? When there are so many tools and techniques of prediction modelling, why do we have another field known as survival analysis? As one of the most popular branch of statistics, Survival analysis is a way of prediction at various points in time. This is to say, while other prediction models make … Continue reading Steps to Perform Survival Analysis in R
Another R tip. Need to replace a name in some R code or make R code re-usable? Use wrapr::let(). Here is an example involving dplyr. Let’s look at some example data: library(&quot;dplyr&quot;) library(&quot;wrapr&quot;) starwars %__% select(., name, homeworld, species) %__% head(.) # # A tibble: 6 x 3 # name homeworld species # … Continue reading R Tip: Use let() to Re-Map Names
Quantide has been a sponsor for MilanoR for a long time. Let's have a look at their collaborations and Milano's amazing contributions and events of 2017.
The post Quantide and MilanoR: a year-long experience appeared first on Quantide - R training &amp; consulting.
https://twitter.com/PennCausal/status/977329657610584064 I wanted to share the news that registration is open for this summer institute (see tweet above for link to registration/schedule). Along with lectures on theory/methods, we will be having interactive R sessions. Attendees will bring their laptops and we'll go through live R examples of things like Estimating treatment effects (matching, propensity score estimation, … Continue reading Registration Open: Causal Inference and Big Data Summer Institute at UPenn →
I’m pleased to announce an update of my sjstats-package. New features are specifically implemented for the Anova and Bayesian statistic and summary functions. Here’s a short overview of what’s new… Anova statistics Beside the already implemented functions to calculate eta-squared, partial eta-squared and omega-squared, it is now also possible to calculate partial omega-squared statistics for … Weiterlesen Anova-Freak and Bayesian Hipster #rstats →
Third post from the short series about black-box explainers implemented in the DALEX package. Learn more about DALEX at SER (Warsaw, April 2018), eRum (Budapest, May 2018), WhyR (Wroclaw, June 2018) or UseR (Brisbane, July 2018). Two weeks ago I wrote about single variable conditional responses and last week I wrote about decompositions of a … Czytaj dalej DALEX: which variables are really important? Ask your black box model!
The M4 forecasting competition is well under-way, and a few of my PhD students have been working on submissions.
Pablo Montero-Manso, Carla Netto, and Thiyanga Talagala have made an R package containing all of the data (100,000 time series), which shou...
There are only a few well-known signals which I consider reliable. One of them is the Dow Theory. According to it, or at least to some interpretations of it, the bull market cycle almost ended this Friday. This time I am adding something new. I recorded myself while doing some of the analysis: Different people […]
The post The Bull Survived on Friday, but Barely appeared first on Quintuitive.
My second Shiny app is dedicated to data visualization.Here users can simply upload any csv or txt file and create several plots:Histograms (with option for faceting)
Barchart (with error bars, and option for color with dodging and faceting)
BoxPlots (with option for faceting)
Scatterplots (with options for color, size and faceting)
TimeSeries
Error bars in barcharts are computed with the mean_se function in ggplot2, which computes error bars as mean ± standard error. When the color option is set, barcharts are plotted one next to the other for each color (option dodging).
For scatterplots, if the option for faceting is provided each plot will include a linear regression lines.
Some examples are below:
For the time being there is no option for saving plots, apart from saving the image from the screen. However, I would like to implement an option to have plots in tiff at 300dpi, but all the code I tried so far did not work. I will keep trying.
The app can be accessed here: https://fveronesi.shinyapps.io/DataViz/
The R code is available here: https://github.com/fveronesi/Shiny_DataViz
One week ago I was in Cape Town for the local satRday conference, where I had the honor to be one of the two keynote speakers, the other one being sports analytics extraordinaire Stephanie Kovalchik (You can read Stephanie Kovalchik’s account of the conference in this blog post). It was a fantastic experience! The event was very well organized, and 100% corresponds to its description as a “one day conference packed with R goodness”. You can watch all talks on Youtube. In my talk, I presented rOpenSci onboarding system of packages and… wore a hard hat!
It’d be a bit hard for me to really write a good recap of satRday that’d do it justice! Instead, I’ll use rtweet and a bit of html hacking to storrrify it (like Storify, but in R) using my live tweets!
Getting my satRday tweets
To get my own satRday Cape Town tweets, I’ll first get my timeline and then filter tweets containing the official conference hashtag, “#SatRdayCapeTown”. I shall only keep my tweets.
my_tweets
It’s Happening, Folks!Whew! Less than a month left and I still haven’t publicised this on my blog. People, we are hosting the 1st Computational Social Science Hackathon at the London School of Economics and you are invited!It’s completely free an...
A quick Friday post to let folks know about three in-development R packages that you’re encouraged to poke the tyres o[fn] and also jump in and file issues or PRs for. Alleviating aversion to versions I introduced a “version chart” in a recent post and one key element of tagging years (which are really helpful... Continue reading →
Should I learn sf or sp for spatial R programming?
I recently received an email about some of the short-courses on my webpage. The correspondant was asked whether my course on Maps and GIS in R was still relevant, it being written in 2014.
The R env...
It’s clear that there are some economic shifts happening in the world, if not the US itself. In light of this, I decided to do some simple investigation into the economic performance of US cities. This is, by the way, one of the critical reasons to master data science. One you know a few critical […]
The post Plotting US Metro Area GDP with ggplot appeared first on SHARP SIGHT.
Introduction a. A robot may not injure a human being or, through inaction, allow a human being to come to harm. b. A robot must obey orders given it by human beings except where such orders would conflict with the First Law. c. A robot must protect its own existence as long as such protection … Continue reading Deep Learning from first principles in Python, R and Octave – Part 5
Another minor maintenance release of the RcppCNPy package arrived on CRAN this evening.
RcppCNPy provides R with read and write access to NumPy files thanks to the cnpy library by Carl Rogers.
There is only small code change: a path is now checked be...
I came across this excellent article lately “Machine learning at central banks” which I decided to use as a basis for a new cheat sheet called Machine Learning Modelling in R. The cheat sheet can be downloaded from RStudio cheat sheets repository. As the R ecosystem is now far too rich to present all available packages and […]
During a discussion with some other members of the R Consortium, the question came up: who maintains the most packages on CRAN? DataCamp maintains a list of most active maintainers by downloads, but in this case we were interested in the total number of packages by maintainer. Fortunately, this is pretty easy to figure thanks to the CRAN repository tools now included in R, and a little dplyr (see the code below) gives the answer quickly[*]. And the answer? The most prolific maintainer is Scott Chamberlain from ROpenSci, who is currently the maintainer of 77 packages. Here's a list of...
One of the basic things the students are taught in statistics classes is that the comparison of models using information criteria can only be done when the models have the same response variable. This means, for example, that when you have \(\log(y_t)\) and calculate AIC, then this value is not comparable with AIC from a […]
While in Brussels last week I noticed an interesting question on X validated that I considered in the train back home and then more over the weekend. This is a question about spacings, namely how long on average does it take to cover an interval of length L when drawing unit intervals at random (with […]
There are a number of easy ways to avoid illegible code nesting problems in R. In this R tip we will expand upon the above statement with a simple example. At some point it becomes illegible and undesirable to compose operations by nesting them, such as in the following code. head(mtcars[with(mtcars, cyl == 8), c(&quot;mpg&quot;, … Continue reading R Tip: Break up Function Nesting for Legibility
Last week we updated the cronR R package and released it to CRAN allowing you to schedule any R code on whichever timepoint you like. The package was updated in order to comply to more stricter CRAN policies regarding writing to folders. Along the lines, the RStudio add-in of the package was also updated. It now looks as shown below and is tailored to Data Scientists that want to automate basic R scripts.
 
The cronR (https://github.com/bnosac/cronR) and taskscheduleR (https://github.com/bnosac/taskscheduleR) R packages are distributed on CRAN and provide the basic functionalities to schedule R code at your specific timepoints of interest. The taskscheduleR R package is designed to schedule processes on Windows, the cronR R package allows to schedule your jobs on Linux or Mac. Hope you enjoy the packages.
If you need support in automating and integrating more complex R flows in your current architecture, feel free to get into contact here.
In this post I have a few goals: 1. Become (re-)familiar with available geoms 2. Become (re-)familiar with aesthetic mappings in geoms (stroke who knew?) 3. Answer these questions: How often do various geoms appear and how often do they … Continue reading →
My R packge powerlmm 0.2.0 is now out on CRAN. It can be installed from CRAN https://cran.r-project.org/package=powerlmm or GitHub https://github.com/rpsychologist/powerlmm.
Changes in version 0.2.0
New features
Analytical power calculations now suppo...
If you regularly have to deal with specific versions of R, or different package combinations, or getting R set up to work with other databases or applications then, well, it can be a pain. You could dedicate a special machine for each configuration you need, I guess, but that's expensive and impractical. You could set up virtual machines in the cloud which works well for one-off situations, but gets tedious having to re-configure a new VM each time. Or, you could use Docker containers, which were expressly designed to make it quick easy to configure and launch an independent and...
We’ll be presenting a webinar on NIMBLE, hosted by the Eastern North America Region of the International Biometric Society. Details are as follows. Programming with hierarchical statistical models: An introduction to the BUGS-compatible NIMBLE system for MCMC and more Friday, April 13, 2018 11:00 a.m. – 1:00 p.m. EST Must register before April 12. You […]
Last week I did an analysis of gold diggers at the Olympics. Here I take you along how I scraped the data from wikepedia and subsequently used the tools of the tidyverse to get the data in a format in which they can be analyzed. You are invited to copy-paste the code and see how the data is gradually getting into a shape in which we can analyze it. I will often not show the data in the intermediate steps, because this will clog the blog. Copy the code to find out for yourself.
This post introduces a new R package I am working on called tabr for creating guitar tablature (“tabs”) from R code. The tabr package provides programmatic music notation and a wrapper around LilyPond for creating quality guitar tablature. [Click here for original post] (in process of switching where I post to R-Bloggers from) tabr offers […]
I am happy to say that the latest issue of The R Journal includes a paper
describing rentrez,
the rOpenSci package for retrieving data from the National Center for Biotechnology Information
(NCBI).
The NCBI is one of the most important sources of biological data. The centre
provides access to information on 28 million scholarly articles through PubMed and 250
million DNA sequences through GenBank. More importantly, records in the 50 public
databases maintained by the NCBI are strongly cross-referenced. As a result, it is
possible to pinpoint searches using almost 2 million taxonomic names or a
controlled vocabulary with 270,000 terms.
rentrez has been designed to make it easy to search for and download NCBI
records and download them from within an R session.
The paper and the package vignette
both describe typical usages of rentrez. I though it might be fun to use this
post to find out where papers describing R packages are published these days.
Although PubMed only covers journals in the biological sciences, searching that
database will at least give us an idea of which journals like to publish these
sorts of papers. Here we use the entrez_search and entrez_summary functions
to get some information on all of the papers published in 2017 with the term
‘R package’ in their title:
library(rentrez)
pkg_search
Data that’s been only partially observed
I’ve been updating my skills in fitting models to truncated data and was pleased to find that, like so much else in statistics, it’s easier than it used to be.
First, some definitions:
censored data ...
Kids today are so sophisticated (at least they are in New York City, where I live). While I didn’t hear about the chi-square test of independence until my first stint in graduate school, they’re already talking about it in high school. When my kids came home and started talking about it, I did what I usually do when they come home asking about a new statistical concept. I opened up R and started generating some data. Of course, they rolled their eyes, but when the evening was done, I had something that might illuminate some of what underlies the theory of this ubiquitous test.
Actually, I created enough simulations to justify two posts - so this is just part 1, focusing on the \(\chi^2\) distribution and its relationship to the Poisson distribution. Part 2 will consider contingency tables, where we are often interested in understanding the nature of the relationship between two categorical variables. More on that the next time.
The chi-square distribution
The chi-square (or \(\chi^2\)) distribution can be described in many ways (for example as a special case of the Gamma distribution), but it is most intuitively characterized in relation to the standard normal distribution, \(N(0,1)\). The \(\chi^2_k\) distribution has a single parameter \(k\) which represents the degrees of freedom. If \(U\) is standard normal, (i.e \(U \sim N(0,1)\)), then \(U^2\) has a \(\chi^2_1\) distribution. If \(V\) is also standard normal, then \((U^2 + V^2) \sim \chi^2_2\). That is, if we add two squared standard normal random variables, the distribution of the sum is chi-squared with 2 degrees of freedom. More generally, \[\sum_{j=1}^k X^2_j \sim \chi^2_k,\]
where each \(X_j \sim N(0,1)\).
The following code defines a data set with two standard normal random variables and their sum:
library(simstudy)
def
This xts release contains mostly bugfixes, but there are a few noteworthy features. Some of these features were added in version 0.10-1, but I forgot to blog about it. Anyway, in no particular order:
endpoints() gained sub-second accuracy on Windows (#202)!
na.locf.xts() now honors 'x' and 'xout' arguments by dispatching to the next method (#215). Thanks to Morten Grum for the report.
na.locf.xts() and na.omit.xts() now support character xts objects. Thanks to Ken Williams and Samo Pahor for the reports (#42).
Many of the bug fixes were related to the new plot.xts() introduced in 0.10-0. And a handful of bugfixes were to make xts more consistent with zoo in some edge cases.As always, I'm looking forward to your questions and feedback!  If you have a question, please ask on Stack Overflow and use the [r] and [xts] tags.  Or you can send an email to the R-SIG-Finance mailing list (you must subscribe to post).  Open an issue on GitHub if you find a bug or want to request a feature, but please read the contributing guide first!
A new release, now at 0.2.1, of the RcppSMC package arrived on CRAN earlier this afternoon (and once again as a very quick pretest-publish within minutes of submission).
RcppSMC provides Rcpp-based bindings to R for the Sequential Monte Carlo Templat...
R tip: use stringsAsFactors = FALSE. R often uses a concept of factors to re-encode strings. This can be too early and too aggressive. Sometimes a string is just a string. Sigmund Freud, it is often claimed, said: “Sometimes a cigar is just a cigar.” To avoid problems delay re-encoding of strings by using stringsAsFactors … Continue reading R Tip: Use stringsAsFactors = FALSE
Per a CRAN email sent to 300+ maintainers, this package (just like many others) was asked to please register its S3 method. So we did, and also overhauled a few other packagaging standards which changed since the last upload in 2014.
No NEWS.Rd file ...
Previous posts in this series on MCMC samplers for Bayesian inference (in order of publication): Bayesian Simple Linear Regression with Gibbs Sampling in R Blocked Gibbs Sampling in R for Bayesian Multiple Linear Regression Metropolis-in-Gibbs Sampling and Runtime Analysis with Profviz The code for all of these posts can be found in my BayesianTutorials GitHub … Continue reading Speeding up Metropolis-Hastings with Rcpp →
Per a CRAN email sent to 300+ maintainers, this package (just like many others) was asked to please register its S3 method. So we did, and also overhauled a few other packagaging standards which changed since the previous uploads in December of 2012 ...
Take care if trying the new RPostgres database connection package. By default it returns some non-standard types that code developed against other database drivers may not expect, and may not be ready to defend against. Danger, Will Robinson! Trying the new package One can try the newer RPostgres as a drop-in replacement for the usual … Continue reading Take Care If Trying the RPostgres Package
Some advices from a referee - Back in 2007 I wrote a Matlab package for estimating regime switching models. I was just starting to learn to code and this project was my way of doing it. After publishing it in FEX (Matlab file exch...
This will be my last post for a while (I promise!).  After today I’ll be taking a rest from all this, until at least the start of April.  Hopefully all this USS stuff will be resolved by then, though! In yesterday’s post I showed a graph, followed by some comments to suggest that future USS […]
R 3.4.4 has been released, and binaries for Windows, Mac, Linux and now available for download on CRAN. This update (codenamed &quot;Someone to Lean On&quot; — likely a Peanuts reference, though I couldn't find which one with a quick search) is a minor bugfix release, and shouldn't cause any compatibility issues with scripts or packages written for prior versions of R in the 3.4.x series. This update improves automatic timezone detection on some systems, and adds fixes for a some unusual corner cases in the statistics library. For a complete list of the changes, check the NEWS file for R...
In response to my previous post, “Latest USS proposal: Who would lose most?“, someone asked me about doing the same calculation for the USS JNC-supported proposals from January.  For a summary of those January proposals and my comments about their fairness, please see my earlier post “USS pension scheme and fairness“. Anyway, the calculation is […]
Simulation studies are used in a wide range of areas from risk management, to epidemiology, and of course in statistics. The MonteCarlo package provides tools to automatize the design of these kind of simulation studies in R. The user only has to specify the random experiment he or she wants to conduct and to specify … Continue reading Visualizing MonteCarlo Simulation Results: Mean vs Median
In this post you will learn: How to create expressions that have mixed (1) strings, (2) expressions, &amp; (3) numbers How to pass in values as variables to an expression I wanted to name this post “Ahhhhhhhhhhh #$@%&amp;!!!!” but SEO isn’t … Continue reading →
PubMed Commons, the NCBI’s experiment in comments for PubMed articles, has been discontinued. Thoroughly too, with all traces of it expunged from the NCBI website. Last time I wrote about the service, I concluded “all it needs now is more active users, more comments per user and a real API.” None of those things happened. … Continue reading Farewell then, PubMed Commons
   
   Last March 8, millions of women in more than 170 countries around the world joined street protests calling for &quot;a society free of sexist oppression, exploitation and violence&quot;. Spanish strike was one of the most numerous, where around 5.3 million women joined the strike. Today, I will use some data (download) regarding the scientific career in Spain (2016) to make a plot using ggplot2 and discuss about it. Here you have the graph and the code:
library(ggplot2) library(scales) dat
Some days I see R as an eclectic programming language preferred by scientists. “Programming languages as people.” From Leftover Salad (David Marino). Other days I see it more like the following. “Statistical tools as cars.” Shared previously by Darren L. Dhaly and brought to my attention by post by Dimitri Liakhovitski, not sure of original … Continue reading The Many Faces of R
In case you missed them, here are some articles from February of particular interest to R users. The R Consortium opens a new round of grant applications for R-related user groups and projects, and has issued US$0.5M in grants to date for R-related projects and events. Microsoft R Client 3.4.3 and Microsoft ML Server 9.3, both built with R 3.4.3, have been released. An 8-step, 5-minute tutorial for setting up a cluster in Azure for use with the sparklyr package. A smartphone app uses R and the keras package to identify &quot;spells&quot; using accelerometer data. &quot;Machine Learning with R and...
“Darwin Core (DwC) is a standard maintained by the Darwin Core maintenance group. It includes a glossary of terms (in other contexts these might be called properties, elements, fields, columns, attributes, or concepts) intended to facilitate the sharing of information about biological diversity by providing identifiers, labels, and definitions. Darwin Core is primarily based on […]
With pretty regular frequency I get emails asking if RSiteCatalyst can be used with Microsoft Power BI. While admittedly I’m not a frequent user of the Windows operating system (nor dashboarding tools like Tableau or Power BI), I am pleased to report...
The problem
I am in the process of migrating my (rather ugly) small blog from “Bloggers” to `blogdown` and, as several others, I choose to use the hugo-academic theme due to its good looks, simplicity, and “focus” towards researchers.
One nice feature of hugo-academic is that it includes out-of-the-box a “Publications” section, allowing researchers to easily create a list of their publication as a section of the website.
Unfortunately, in order to populate that list, users have to manually create one different .md file for each publication, by cutting and pasting several different info (e.g., title, authors, etc.) in a &quot;simple&quot; template like this one.
THIS IS BORING!
Since I was not in the mood of doing that, and no automatic solutions could be found (well, there appears to be a python one, but we are speaking R, here…), I decided to try and develop some script to automatically create the required md files starting from a BibTex list of my publications. Here are the results of that effort.
A possible solution
Preparing the BibTex file
To automatically create the publications md files, all you need is a (properly formatted) BibTex file. Since I did not have one ready, I created mine by exporting my publications list from Scopus, but you could use any valid BibTeX file.
One important thing, though, is that you have to be sure that the file is saved with UTF-8 encoding. If you are not sure, you can open it in RStudio (or any decent text editor), and then re-save it with a forced encoding (in RStudio, you can use File--__Save with Encoding)
Creating an import script
Now, you need a script that reads the BibTex entries and use the data to populate one different mdfile for each publication. Below you can find my attempt at this.
bibtex_2academic
The sixteenth update the 0.12.* series of Rcpp landed on CRAN earlier this evening after a few days of gestation in incoming/ at CRAN.
Once again, this release follows the 0.12.0 release from July 2016, the 0.12.1 release in September 2016, the 0.12....
Being a self learner and explorative by nature, I love to try out new technology and keep learning something all the time. Especially, working in a Biotech organization as a data scientist, and being the only one in the region, &quot;technical&quot; talks and discussions are very limited from the programming world perspective. So I spend my evenings experimenting with technology and reading tech blogs, websites and papers.One visualization tool I am in love with is Tableau. I find it very cool to be able to quickly build a dashboard of graphs, with all the interactivity, in few minutes. Of course data needs to be &quot;prepared&quot; first like any other data science.I installed the Tableau app from Tableau public website. Now the data, there is a plethora of data for budding data scientists to play with. I got this data set from https://data.gov.in/ which provides data to the public as a part of the open data initiative from the Govt.I wanted to see how much the various minerals were produced and what were their values were. Also see the states on the map of India and in parallel visualize the numbers. Another interesting bit is to know what type of mineral each is - fuel, metallic or non metallic. After a few iterations, this is how my dashboard finally looks.
R project has been participating in Google Summer of Code (GSoC) since 2008. We have numerous packages which have received code contributions from GSoC participants. This year too, R is participating and there is already an impressive list of potential projects listed at [https://github.com/rstats-gsoc/gsoc2018/wiki/table-of-proposed-coding-projects] and the project mentors are seeking students to work on these […]
Many companies are still largely managed with spreadsheets. While working with data we have seen the full spectrum of use cases of Excel, including some truly scary ones. There is, however, a clear trend of companies switching to alternatives af...
In the previous blog post, I showed you usage of my TSrepr package. There was shown what kind of time series representations are implemented and what are they good for.
In this tutorial, I will show you one use case how to use time series representati...
In the latest Redmonk language rankings, R has risen to the #12 position, up from #14 in the June 2017 rankings. (Python remains steady in the #3 position.) The Redmonk rankings are based on activity in StackOverflow (as a proxy for user engagement) and Github (as a proxy for developer engagement). Here's the chart from January 2018 of Github popularity ranking versus StackOverflow popularity ranking. Here's what Redmonk analyst Stephen O'Grady had to say about Powershell, R and Typescript. (R isn't a Microsoft property, but Microsoft is a founding member of the R Consortium and incorporates R into several products...
Yesterday the UUK/UCU negotiations at ACAS concluded with an agreement document. In this post I’ll look at the numbers in those proposed interim changes to the Universities Superannuation Scheme, to work out how much money would effectively be lost by USS members at each salary level. This is inevitably a fairly rough calculation, but its […]
(No statistical graphs in this one. This is what my dog Artemis looks like when she wants my attention during work hours.)
Mindy L. Mallory (@ace_prof) wrote a blog post on Machine...
Often overlooked, testing is a critical process that saves time over the long term and enables building complex systems. Unit …Continue reading →
The R-Lab#6 is ready to go, and on the occasion, we're also discussing the outcomes and contents of the precedent R-Lab, about personalized medicine.
The post R-Lab#5: the report &amp; R-Lab#6 ready to go! appeared first on MilanoR.
(FWIW I think I even caused myself pain due to the title of this blog post). Kaiser Fung (@junkcharts) did a makeover post on this chart about U.S. steel tariffs: Kaiser’s makeover is good (Note: just because I said “good” does not mean I’m endorsing the use of pie charts): But, I’m curious as to... Continue reading →
Not all data analysis tools are created equal. Recently, I started looking into data sets to compete in Go Code Colorado (check it out if you live in CO). The problem with such diversity in data sets is finding a way to quickly visualize the data and do exploratory analysis. While tools like Tableau make data […]
In the digital age of today, data comes in many forms. Many of the more common file types like CSV, XLSX, and plain text (TXT) are easy to access and manage. Yet, sometimes, the data we need is locked away in a file format that is less accessible such ...
Prologue In 2013, I had worked on developing a Music Recommendation System as a pilot project, which had not even completed. There were lots of reasons, but the critical one was the choice and concentration strategy of my division. Since then, I keep these passion in my deep mind for a long time, but could […]
tldr: I created an open dataset for the explicit practice of data munging.
Feel free to use it in assignments,
but do mention where you got it from (CC-by-4.0). Also unicorns are awesome.
Find the dataset at: https://github.com/RMHogervorst/unico...
It came to a close two weeks ago. Over two weeks of obsessive watching everything that moves on skis, skates or boards. My tiny country of the Netherlands won about half of all the medals at the speed skating events. Since there are many medals to be won here this made it to end up high in the medal table. This makes it look as winter sports nation, where all we skating buffs. (Sadly, we can’t hardly skate anymore last years. Stop global warming everyone, the Dutch need their ice!) To me, this begged the question, are other top nations also in the high ranks because they dominate a single sport. Or are they all round snow and ice eating badasses?
Apriétame bien la mano, que un lucero se me escapa entre los dedos (Coda Flamenca, Extremoduro) I have the privilege of being teacher at ESTALMAT, a project run by Spanish Royal Academy of Sciences that tries to detect, guide and stimulate in a continuous way, along two courses, the exceptional mathematical talent of students of … Continue reading Mandalas Colored →
If you are working with predictive modeling or machine learning in R this is the R tip that is going to save you the most time and deliver the biggest improvement in your results. R Tip: Use the vtreat package for data preparation in predictive analytics and machine learning projects. When attempting predictive modeling with … Continue reading R Tip: Use the vtreat Package For Data Preparation
When I first encountered R, I learned to use the levels() function to find the possible values of a categorical variable.  However, I recently noticed something very strange about this function. Consider the built-in data set “iris” and its character variable “Species”.  Here are the possible values of “Species”, as shown by the levels() function. __ levels(iris$Species) […]
March Madness is officially upon us as College Basketball teams across the US try to get a few more signature wins on their resume during conference tournaments before Selection Sunday. With only 30 or so games in a given season and around 350 teams in...
styler 1.0.1 is now awailable on CRAN. It’s mainly a maintenance release and contains bug fixes and speed improvements. The highlights are:
This is guest post contributed by Thomas Lin Pedersen, creator of the lime package. The post originally appeared on Thomas’ blog at https://www.data-imaginist.com/2018/lime-v0-4-the-kitten-picture-edition/.
I’m happy to report a new maj...
Last semester I taught an R programming at MIMUW. My lectures are project oriented, the second project was related to package development. The idea was straightforward: each team of students shall create a package that produces IFS fractals (based on iterated function systems). Each package shall have two generic functions: create() and plot(), documentation and … Czytaj dalej How fractals helped my students to master package development in R
Here is an R tip. Use loop indices to avoid for()-loops damaging classes. Below is an R annoyance that occurs again and again: vectors lose class attributes when you iterate over them in a for()-loop. d [1] &quot;2018-02-18 10:16:16 PST&quot; &quot;2018-02-18 10:16:16 PST&quot; for(di in d) { print(di) } #__ … Continue reading R Tip: Introduce Indices to Avoid for() Class Loss Issues
I have been searching for various ways to find information about R packages for some time now, but I only recently learned about the CRAN_package_db() function in the base tools package. If a colleague hadn’t pointed it out to me, I am sure I would never have found it on my own.
pdb = 3.2.0)&quot;, &quot;R (...
## $ Imp NA, &quot;httr, XML, curl, readr, plyr, progress&quot;, NA, NA, ...
## $ Aut &quot;Scott Fortmann-Roe&quot;, &quot;Gaurav Sood [aut, cre]&quot;, &quot;Csill...
## $ RD NA, NA, &quot;abctools, EasyABC&quot;, &quot;abc&quot;, NA, NA, NA, NA, NA...
## $ RI NA, NA, &quot;ecolottery&quot;, NA, NA, NA, NA, NA, NA, NA, NA, ...
Note that in this post, I am using data from a file I pulled from CRAN just before my talk, but you should not have any trouble working with CRAN_package_db() to get an updated data set.
The next thing I did was to add two new variables to the data set: DepImp, the union of the dependent packages and imported packages, and RDRI, the union of the reverse depends and reverse imports packages.
library(stringr)
# A helper function to unlist and string split.
fcn %
mutate(Package = meta_data$Package[.x])
}) %__%
select(Package,Aut, Dep, Imp, RD, RI)
#head(c_dat1)
# Process the fields DepImp RDRI
c_dat2 %
map_df(~{
meta_data[.x, ] %__%
select(-Package, -Aut, -Dep, -Imp, -RD, -RI) %__%
map_df(~ifelse(is.na(.x), 0, length(rm_na(.x)[[1]])))
}) %__%
select(DepImp, RDRI)
#head(c_dat2)
c_dat
by Antony Unwin, University of Augsburg, Germany There are many different methods for identifying outliers and a lot of them are available in R. But are outliers a matter of opinion? Do all methods give the same results? Articles on outlier methods use a mixture of theory and practice. Theory is all very well, but outliers are outliers because they don’t follow theory. Practice involves testing methods on data, sometimes with data simulated based on theory, better with `real’ datasets. A method can be considered successful if it finds the outliers we all agree on, but do we all agree...
Getting out of the office and making time to share our R stories and ideas can often slip down our to-do lists when we’re busy, but taking the opportunity to meet othe...
Recently, I have written and hosted several multi-user Shiny apps. One example, is Taddle that runs on my server at http://taddle.mathematik.uni-ulm.de. Taddle is a multi-user shiny app that offers a free and simple way to allocate seminar topics. (Nex...
In a recent blog post I wrote about having a template for blogdown posts. I wanted to know if it was possible to do this and make my life (and others hopefully) easier for writing new blog posts that are ready to go with the features I frequently re-us...
As emailed to me by Aki Vehtari, the next StanCon will take place this summer in the wonderful city of Helsinki, at the end of August. On Aalto University Töölö Campus precisely. The list of speakers and tutorial teachers is available on the webpage. (The only “negative point” is that the conference does not include […]
The intrval R package is lightweight (~11K), standalone (apart from importing from graphics, has exactly 0 non-base dependency), and it has a very narrow scope: it implements relational operators for intervals — very well aligned with the tiny manifesto. In this post we will explore the use of the package in two shiny apps with sliders.
While looking at X validated questions, I came upon this comment that simulating a uniform distribution on a d-dimensional unit sphere does not proceed from generating angles at random on (0,2π) and computing spherical coordinates… Which I must confess would have been my initial suggestion! This is obvious, nonetheless, when computing the Jacobian of the […]
In an effort to streamline product development, maintenance, and support to ensure the best experience for our users, we have created a strategy for operating system and browser deprecation. This will allow us to focus our work on modern platf...
In R, we often need to get values or perform calculations from information not on the same row. We need to either retrieve specific values or we need to produce some sort of aggregation. This post explores some of the options and explains the weird (to...
Excel users starting to use R likely have some established concepts about data: where it's stored, how functions apply to data, etc. In general, R does things differently to Excel (or any spreadsheet, in fact). In a useful guide, Steph de Silva from Rex Analytics explains the concepts of data management in R and how they differ from Excel, which provides a useful mental model for those making the transition or working in both. Click the image above for the complete guide, but in summary the differences are: Excel stores data in the grid structure of the worksheet, whereas R...
I work with internet-scale data and do my fair share of macro-analyses on vulnerabilities. I use the R semver package for most of my work and wanted to blather on a bit about it since it’s super-helpful for this work and doesn’t get the attention it deserves. semver makes it possible to create charts like... Continue reading →
by Subhadeep (Deep) Mukhopadhyay and Douglas Fletcher, Department of Statistical Science, Temple University Bayesians and Frequentists have long been ambivalent toward each other. The concept of “Prior” remains the center of this 250 years old tug-of-war: frequentists view prior as a weakness that can cloud the final inference, whereas Bayesians view it as a strength to incorporate expert knowledge into the data analysis. So, the question naturally arises, how can we develop a Bayes-frequentist consolidated data analysis workflow that enjoys the best of both worlds? To develop a “defendable and defensible” Bayesian learning model, we have to go beyond blindly...
Another R tip. Use vector(mode = &quot;list&quot;) to pre-allocate lists. result [[1]] #__ NULL #__ #__ [[2]] #__ NULL #__ #__ [[3]] #__ NULL The above used to be critical for writing performant R code (R seems to have greatly improved incremental list growth over the years). It … Continue reading R Tip: Use vector(mode = &quot;list&quot;) to Pre-Allocate Lists
I recently submitted a Proposal to the R Consortium. I decided to share this proposal on my blog for two reasons: To raise awareness of...
The post My Proposal to the R Consortium appeared first on AriLamstein.com.
If that’s not a great 1960’s band name then I don’t know what is (hint: I don’t know what is). At the start of the year I set out my ‘goals for 2018’ just like many of us do; an...Continue Reading →
This afternoon, I took a quick look at the current Riddler puzzle, which sums up as, given three points A, B, C, arbitrarily moving on a plane with a one-shot view of their respective locations, find a moving rule to bring the three together at the same point at the same time. And could not […]
SQL Server Reporting services (SSRS) is an outstanding tool for creating, deploying and managing paginated, mobile, KPI reports as well as Power BI reports. Tool provides simple way to share and get data insights in your corporate environment. (Photo source: Microsoft docs) Using the privileges of R language to enrich your data, your statistical analysis … Continue reading Using R in SQL Server Reporting Services (SSRS)
A. INTRODUCTION When building statistical models, the goal is to define a compact and parsimonious mathematical representation of some data generating process. Many of these techniques require that one make assumptions about the data or how the analysis is specified. For example, Auto Regressive Integrated Moving Average (ARIMA) models require that the time series is … Continue reading Semiparametric Regression in R →
I posted a while back about using joy plots for heart rate data. Over the past couple of months, I grew tired of opening RStudio every time I wanted to look through my fitness tracker data. I decided to create a shiny dashboard that I can load via web browser. This involved setting up a shiny server in a dockerized container that runs an Rmd file. I’ve had issues getting plots to size themselves appropriately using base shiny, so I used the flexdashboard package to created a dashboard that will automatically resize. I wanted to be able to look at individual workouts as …
a faceting function with bare variable names in ggplot2 - I’m working on a package of dplyr and ggplot2 functions that I can use on a day to day basis to plot data over time. I’ve done some basic experimenting with my preferred ggplot2 th...
We've released our videos of the talks at the 2018 RStudio conference. My talk was Modeling in the Tidyverse (video) and I was also in the Tidyverse fireside chat (video). There ar...
Exploratory multivariate data analysis is studied and has been taught in a “French-way” for a long time in France. You can enroll in a MOOC (completely free) on Exploratory Multivariate Data Analysis. The MOOC will start the 5th of March 2018. This MOOC focuses on 5 essential and basic methods, those with the largest potential in terms of […]
R tip: get out of the habit of calling View() directly. View() only works correctly in interactive environments, not currently in RMarkdown contexts. It is better to call something else that safely dispatches to View(), or to something else depending if you are in an interactive or non-interactive session. The following code will work interactively, … Continue reading R Tip: Get Out of the Habit of Calling View() Directly
A codebook is a technical document that provides an
overview of and information about the variables in a dataset. The
codebook ensures that the statistician has the complete background
information necessary to undertake the analysis, and a codebook
do...
I would like to thank LinkedIn for letting me speak with some of their data scientists and analysts. John Mount discussing rquery SQL generation at LinkedIn. If you have a group using R at database or Spark scale, please reach out ( jmount at win-vector.com ). We (Win-Vector LLC) have some great new tools I’d … Continue reading Speaking on New Tools for R at Big Data Scale
The 3.7.0 release of simmer, the Discrete-Event Simulator for R, is on CRAN. It includes several API improvements and bug fixes. Among the former, the new timeout_from_attribute() activity makes it easier and more efficient the common task of placing a timeout based on a previously set attribute. Another common task is to increment or decrement a given attribute.… Continuar leyendo simmer 3.7.0 →
The R programming language is a powerful tool used in data science for business (DS4B), but R can be unnecessarily challenging to learn. We believe you can learn R quickly by taking an 80/20 approach to learning the most in-demand functions and package...
Tis the season for finding out how well Maine fisherfolk did last year; specifically, Maine lobsterfolk. Most of the news sites in Maine do a feature on the annual landings (here’s one from Bangor Daily News). There was a marked decline — the largest ever — in both poundage and revenue in 2017 and many... Continue reading →
A while back I got interested in (baby) surnames. It is interesting to follow how friends name their newly born. There are various naming “strategies”, a) pick a name that has been used in previous generations but now uncommon (e.g. in Germany tha...
-- title: &quot;2015 US Fatal Vehicle Accidents Analysis&quot; author: &quot;Matthew Sun&quot; date: &quot;2/01/2018&quot; output: html_document: default --- #### Introduction   There are over 5 millions of vehicle accidents reported in the US in 2015, according to data from National Highway Traffic Safety Administration (NHTSA). About 7% of them are fatal accidents. As the […]
RcppArmadillo release 0.8.400.0.0, originally prepared and uploaded on February 19, finally hit CRAN today (after having been available via the RcppCore drat repo for a number of days). A corresponding Debian release was prepared and uploaded as wel...
We told you EARL 2018 was going to be awesome!
We're excited to announce that Hadley Wickham will be the Keynote Speaker at our EARL Houston event on 9 November 2018. T...
I think this is the R Tip that is going to be the most controversial yet. Its potential pitfalls include: it is a style prescription (which makes it different than and less immediately useful than something of the nature of R Tip: Force Named Arguments), and it is heterodox (this is not how magrittr/dplyr is … Continue reading R Tip: Make Arguments Explicit in magrittr/dplyr Pipelines
There's a pay-off from an exploration of multiple supervised machine learning models. We'll explore five approaches whilst predicting criminal goings-on in London.
The post Criminal goings-on in a random forest appeared first on thinkr.
With ggimage, we are able to plot images using grammar of graphics. The layers defined in ggimage can be directly applied to ggtree to annotate phylogenetic tree using local/online image files.
ggtree seamlessly work with ggimage. The geom_tiplab and geom_nodelab can accept parameter of geom=&quot;image&quot; to parse taxa labels as image files and use them to “label” the taxa using images instead of text strings. Here are some examples for demonstration.
Annotate tips
library(ggimage)
library(ggtree)
url
This is a very short post with example code
Over time when you move your shiny app from your computer to a server, you
want to add some logging. Generally logging is defined in levels : INFO (everything you want to print),
WARNING (it does not sto...
The R Consortium passed a significant milestone this month: since its inception, the non-profit body has provided more than US$500,000 in grant funding to project proposed by the R Community. The R Consortium uses the dues from its member organizations to fund grant proposals, which are reviewed twice a year by its Infrastructure Steering Committee. (If you'd like to propose a project, proposals for the next round are being accepted through April 1.) New projects funded in this round include: Creating a new data type for R to unify the &quot;units&quot; and &quot;errors&quot; packages Updating the R module for the...
The mlr: Machine Learning in R package provides a generic, object-oriented and extensible framework for classification, regression, survival analysis and clustering for the statistical programming language R. The package targets practitioners who want to quickly apply machine learning algorithms, as well as researchers who want to implement, benchmark, and compare their new methods in a structured environment.
We are happy to announce that we now offer training courses specialized on mlr:
The Munich R Courses already offer the three-day course “Machine Learning and Data Mining in R”.
Introduction
In the quest for ever faster code, one generally begins exploring ways to integrate C++
with R using Rcpp. This post provides an example of multiple
implementations of a European Put Option pricer. The implementations are done in pure R,
pure Rcpp using some Rcpp sugar functions,
and then in Rcpp using
RcppArmadillo, which exposes the
incredibly powerful linear algebra library, Armadillo.
Under the Black-Scholes model The value of a European put option has the closed form solution:
where
and
Armed with the formulas, we can create the pricer using just R.
put_option_pricer
This report describes different methods for correctly rendering macrons in Māori text within R plots. The topics covered will also have relevance to rendering other special characters in R graphics and possibly to rendering macrons in other software. ...
I’m running a one day course with Timberlake, Stata’s UK distributors on this topic. We’ll run it next Friday the … More
In this latest blog, I’m responding to a cry for help. Someone got in touch with us recently asking for some advice on image detection algorithms, so let’s see what we can do!
They already know what algorithms they want to use, so let’s start with those. Hang on no, for the uninitiated, let’s start with what even is an image detection algorithm?
“An image detection algorithm takes an image, or piece of an image as an input, and outputs what it thinks the image contains.
This past week I was preparing slides for a reading assessment class with a lecture focus on the Visual Word Form Area [VWFA] (Cohen, et al., 2000). This is an area of the brain that is hypothesized to be able … Continue reading →
Experiences with using R/exams for multiple automated and randomized exams in an applied statistics course for environmental scientists at Universität Koblenz-Landau.
Guest post by Ralf B. Schäfer (Universität Koble...
The 2013 force majeur is a big news for Cartier: they introduced their first internal timing movement, the 1904-CH MC, on the Cartier chronograph. Self-winding vertical clutches Column wheel motors are brand-based Cartier campaigns. We got it today and it impressed me in a few ways. The vertical clutch cylinder rollers are known for their … Continue reading Introducing The Calibre De Cartier Chronograph: An Impressive New Interior Movement
An update to Microsoft R Client, Microsoft's distribution of open source R with additional proprietary packages — including RevoScaleR (for data analysis at scale) and MicrosoftML (for machine learning) — is now available. Microsoft R Client 3.4.3 updates the R engine to R 3.4.3, and (on Linux) now supports deploying computations to a remote SQL Server with the sqlrutils package. Microsoft R Client 3.4.3 is free to download and use, and as the video above shows is designed for developing analytical applications that will be deployed to production servers. It works with Microsoft Machine Learning Server 9.3, also released today....
This may be Cartier Replica, many of us are waiting. Earlier today, in fact, when CNN asked every hodinkee team member to pick and see, they really liked the performance of the day, and this is the new form of men’s wristwatches Cartier-driven – I chose. why? Because I’ve been eager to find the right … Continue reading Cartire Drive, A New Man’s Wear Watch With Past Eyes
Just In Time For Baseball
The mlbgameday package has just reached the milestone of version 0.1.0.
Designed to facilitate extract, transform and load for MLBAM “Gameday” data. The package is optimized for parallel processing of data that may be la...
The nmslibR package is a wrapper of NMSLIB, which according to the authors “… is a similarity search library and a toolkit for evaluation of similarity search methods. The goal of the project is to create an effective and comprehensive to...
Another R tip. Get in the habit of using drop = FALSE when indexing (using [ , ] on) data.frames. Prince Rupert’s drops (img: Wikimedia Commons) In R, single column data.frames are often converted to vectors when manipulated. For example: d x #__ 1 1 #__ 2 2 #__ … Continue reading R Tip: Use drop = FALSE with data.frames
Data analytics is rapidly changing the face of manufacturing as we know it. At Mango, we’re seeing companies using their data effectively to gain an advantage over com...
In this 4th post of my series on Deep Learning from first principles in Python, R and Octave – Part 4, I explore the details of creating a multi-class classifier using the Softmax activation unit in a neural network. The earlier posts in this series were 1. Deep Learning from first principles in Python, R … Continue reading Deep Learning from first principles in Python, R and Octave – Part 4
I’ve just updated The Popularity of Data Science Software to reflect my take on Gartner’s 2018 report, Magic Quadrant for Data Science and Machine Learning Platforms. To save you the trouble of digging though all 40+ pages of my report, … Continue reading →
Last week, my brand new DataCamp course Shiny Case Studies was finally released after being in the works for a full year!
Check out the course
This is an online course consisting of a series of short videos, mulitple choice exercises, a...
Last week I wrote about single variable explainers implemented in the DALEX package. They are useful to plot relation between a model output and a single variable. But sometimes we are more focused on a single model prediction. If our model predicts possible drug response for a patient, we really need to know which factors … Czytaj dalej DALEX: how would you explain this prediction?
Consider you have a server that is running behind a firewall and, for security
reasons, cannot make external http(s) requests. Further, you have R running on
this server and you need to install a set of packages. The simple approach of
install.packa...
An updated tutorial to set up a blogdown blog    Git &amp; GitHub repos RStudio and blogdown setup Customising paths and styles Modify config.toml file Custom css Custom javascript Site build with blogdown Deploying the site Updating the local git repository Pushing to GitHub Add R code A new … Sigue leyendo →
To post your R job on the next post Just visit  this link and post a new R job  to the R community. You can post a job for  free  (and there are also “featured job” options available for extra exposure). Current R jobs Job seekers:  please follow the links below to learn more and apply for your R job of interest: Featured Jobs Freelance Short-term consultancy – Biostatistics Ausvet Pty Ltd – Posted by Ausvet Bruce Australian Capital Territory, Australia 11 Feb 2018 Full-Time Customer Success Representative RStudio, Inc. – Posted by jclemens1 Anywhere 8 Jan 2018   More New R Jobs Freelance R + SQL + Google Cloud Platform + Microsoft PowerBi + Multiplatform APIs Automated Reporting Project @ London, U.K. Greenlight Digital – Posted by Hend Erigat London England, United Kingdom 21 Feb 2018 Full-Time Data Scientist and R Programmer @ Cambridge, Massachusetts, U.S. Metrum Research Group, LLC – Posted by michellej Cambridge Massachusetts, United States 18 Feb 2018 Full-Time Data Scientist and R Programmer @ Simsbury, Connecticut, U.S. Metrum Research Group, LLC – Posted by michellej Simsbury Connecticut, United States 18 Feb 2018 Freelance R Programmer for Statistical Research Data Processing (for $350) Academic Research – Posted by empiricus Anywhere 18 Feb 2018 Part-Time Garch(1,1) forecasting – fast job – Quick money […]
I need a few volunteers to please “test pilot” the development version of the R package cdata, please. Jacqueline Cochran: at the time of her death, no other pilot held more speed, distance, or altitude records in aviation history than Cochran. Our cdata package has an upcoming new feature called “build_frame()” that allows for the … Continue reading Wanted: cdata Test Pilots
When I learned and experimented a new model, I always like to start with its likelihood function in order to gain a better understanding about the statistical nature. That’s why I extensively used the SAS/NLMIXED procedure that gives me more flexibility. Today, I spent a couple hours playing the optim() function and its wrappers, e.g. […]
If you've ever wanted to play around with big data sets in a Spark cluster from R with the sparklyr package, but haven't gotten started because setting up a Spark cluster is too hard, well ... rest easy. You can get up and running in about 5 minutes using the guide SparklyR on Azure with AZTK, and you don't even have to install anything yourself. I'll summarize the steps below, but basically you'll run a command-line utility to launch a cluster in Azure with everything you need already installed, and then connect to RStudio Server using your browser to analyze...
Branko Milanovic asked on Twitter:
Idea for a paper: “Homogamy” on Twitter. Do people with more followers follow people with more followers?
I don’t have time to write a paper but I was sufficiently interested to want to blog about it. Th...
Just like your mom's chippity chipper recipe - In Data Science, there are numerous instances where different techniques call for the use of different tools. For me, this means hopping between R and python on a weekly basis. I’ve been fortun...
Is R base::subset() really that bad? Notes discussing subset() often refer to the following text (from help(subset), referred to in examples: 1, 2): Warning This is a convenience function intended for use interactively. For programming it is better to use the standard sub-setting functions like [, and in particular the non-standard evaluation of argument subset … Continue reading Is R base::subset() really that bad?
Yesterday a friend told me, “Yihui has written the most remarkably open blog post, and you’ve got to read it.” I did and it was. Though my post here is not about R per se, it is about a great contributor to R, our Yihui, Dr. of Statistics and (according to him) Master of Procrastination. … Continue reading Xie Yihui, R Superstar and Mensch →
By J Espasandin, O Lado, A Bouzas, A Baluja. You can also check this post, written in #blogdown, here: intro-joint-modeling-r. These days, between the 19th and 21st of February, has taken place the learning activity titled “An Introduction to the … Sigue leyendo →
Using your smartphone (any modern phone with a built-in accelerometer should work), visit the Cast Your Spell page created by Nick Strayer. (If you need to type it to your phone browser directly, here's a shortlink: bit.ly/castspell .) Scroll down and click the &quot;Press To Cast!&quot; button, and then wave your phone like a wand using one of the shapes shown. The app will attempt to detect which of the four &quot;spells&quot; you gestured. It was pretty confident in its detection when I cast &quot;Incendio&quot;, but your mileage may vary depending on your wizarding ability and the underlying categorization model....
Yesterday I wrote that I’ve received a number of choroplethr bug reports recently that I simply cannot reproduce. Due to a large number of people...
The post Update: “Difficult to Reproduce Choroplethr Bug” Isolated and Fixed appeared first on AriLamstein.com.
Hex stickers remind me of Pogs, except they’re cooler because you can combine them together! Some people do that very smartly.
Now when I forget how to do an #rstats analysis I can just check the back of my laptop #runconf16 pic.twitter.com/DY7UgeBzYV— David Robinson (@drob) 1 d’abril de 2016
I’ve got a pretty random hex stickers combination on my laptop, but after all it could be worse…
My boyfriend decided he is post-hex / post-#tidyverse pic.twitter.com/ADtyORMxHL— Hilary Parker (@hspter) 13 de setembre de 2017
Now since I’m a magick/collage fan, you can bet I’ve wondered how to use R in order to combine stickers automatically! Say I have a bunch of sticker PNGs, how could I produce a map to design my laptop style? Read to find out more…
Getting some hex stickers to play with
I do agree that in real life you’ll most often not have PNGs of the stickers but hey I can dream… especially since I’m sure that one will soon be able to identify stickers on a conference haul pic in the blink of an eye with the Rvision package. I got stickers from hex.bin whose existence I learnt from this tweet:
A plea to my favorite #rstats 📦 developers: please submit your BIG hex sticker image files (with vector versions 🙏) to https://t.co/7KWg3d2fZA! @old_man_chester @nj_tierney @samfirke they will be right at home 👇 pic.twitter.com/AIqgnBIx1X— Alison Hill (@apreshill) 21 de febrer de 2018
And well if more R packages are added to it my Rvision dream could come true.
Note: When I say I got them from hex.bin, I mean I lazily downloaded the “hexagons/” folder of this GitHub repository by using a link as explained in this SO answer.
Combining a few of them at random
I’ll sample 42 stickers from my nice little internet haul.
library(&quot;magrittr&quot;)
set.seed(42)
ultimate_sample %
sample(size = 42)
After that I’ll create rows which is pretty straightforward.
no_row
R tip: force the use of named arguments when designing function signatures. R’s named function argument binding is a great aid in writing correct programs. It is a good idea, if practical, to force optional arguments to only be usable by name. To do this declare the additional arguments after “...” and enforce that none … Continue reading R Tip: Force Named Arguments
Introduction to Looping system Imagine you were to perform a simple task, let’s say calculating sum of columns for 3X3 matrix, what do you think is the best way? Calculating it directly using traditional methods such as calculator or even pen and paper doesn’t sound like a bad approach. A lot of us may prefer … Continue reading Whys and Hows of Apply Family of Functions in R
R-Ladies Remote is kicking off and we want YOU! Do you want to be part of the R community but can’t attend meetups? There are many R-Ladies across the globe who love the idea of the organisation, but aren’t able to connect with it easily due to their distance, their work or their caring responsibilities. … Continue reading Introducing R-Ladies Remote Chapter
About a year and a half back I was working in Python a bit and became accustomed to the explicit importing of modules (akin to R packages) and functions. Python imports packages like this: import tidyr import dplyr as dp … Continue reading →
Topology is the silly putty of mathematics. One of the strangest shapes in topology is the Möbius strip. This strip is a surface with only one side. This article shows how the visualise this shape in R using the RGL package. Continue reading →
The post Topological Tomfoolery in R: Plotting a Möbius Strip appeared first on The Devil is in the Data.
Modern machine learning platforms like Tensorflow have to date been used mainly by the computer science crowd, for applications like computer vision and language understanding. But as JJ Allaire pointed out in his keynote at the RStudio conference earlier this month (embedded below), there's a wealth of applications in the data science domain that have yet to be widely explored using these techniques. This includes things like time series forecasting, logistic regression, latent variable models, and censored data analysis (including survival analysis and failure data analysis). The keras package for R provides a flexible, high-level interface for specifying machine learning...
R tip: use [[ ]] wherever you can. In R the [[ ]] is the operator that (when supplied a scalar argument) pulls a single element out of lists (and the [ ] operator pulls out sub-lists). For vectors [[ ]] and [ ] appear to be synonyms. However, when writing reusable code you may … Continue reading R Tip: Use [[ ]] Wherever You Can
Locke Data’s great blog is Markdown-based. What this means is that all blog posts exist as Markdown files: you can see all of them here. They then get rendered to html by some sort of magic cough blogdown cough we don’t need to fully understand here. For marketing efforts, I needed a census of existing blog posts along with some precious information. Here is how I got it, in other words here is how I rectangled the website GitHub repo and live version to serve our needs.
Since November 2017 I have received three bug reports from users who see this error when running any command in the choroplethr package: Theme element panel.border missing   Error in if (theme$panel.ontop)...
The post Difficult to Reproduce Choroplethr Bug: Can You Help? appeared first on AriLamstein.com.
I have to admit my initial thoughts of deep learning were pessimistic and in order to not succumb to impostor syndrome, I put off learning any new techniques in the growing sub field of machine learning, until recently. After attending &amp; ...
Great effort has recently been made to encourage also the not-so-experienced to jump into the water and blog about data science. Some of the community’s hot shots gracefully draw attention to blogs of first-times on twitter to give them extra exposure. Most of the blogger newbies write about an analysis they did on a topic they care about. That sounds like the obvious thing to do. However, I think there is a second option that is not considered by many. That is writing about a topic you just learned. This might seem strange, why would you want to tell the world about something you are by no means an expert on? Won’t you just make a fool of yourself by pretending you know stuff you just picked up? I don’t think so, let me give you four reasons why I think this is actually a good idea.
This post will cover two aspects: the first will be a function to convert daily returns into a table of … Continue reading →
The problem
Last week, I replied to this interesting question posted by @Tim_K over stackoverflow. He was seeking efficient solutions to identify all points falling within a maximum distance of xx meters with respect to each single point in a spatial points dataset.
If you have a look at the thread, you will see that a simple solution based on creating a “buffered” polygon dataset beforehand and then intersecting it with the original points is quite fast for “reasonably sized” datasets, thanks to sf spatial indexing capabilities which reduce the number of the required comparisons to be done (See http://r-spatial.org/r/2017/06/22/spatial-index.html). In practice, something like this:
# create test data: 50000 uniformly distributed points on a &quot;square&quot; of 100000# metresmaxdist
Hi everyone,
In this blog post, I will be short and I will introduce our shiny application on bike self-service stations. Your browser does not support the video tag. The code is in 2 parts, the ui.R file for the interface and the server.R file for the backend. You can check the code on GitHub if you want to download the code and run it on your computer. JCDecaux provides an API that gives us real time information on each bike self-service station. This infomation is: •Station id •Station name •Address •Position latitude/longitude •Presence of a payment terminal •Presence of a bonus station •If the station is open •How many bike stands are in the station •how many bike stands are available in the station (no bikes on the stand) •How many bikes are available •Time of the last api update The JCDecaux API gives the data under the following format:
{
&quot;number&quot;: 123,
&quot;contract_name&quot; : &quot;Paris&quot;,
&quot;name&quot;: &quot;stations name&quot;,
&quot;address&quot;: &quot;address of the station&quot;,
&quot;position&quot;: {
&quot;lat&quot;: 48.862993,
&quot;lng&quot;: 2.344294
},
&quot;banking&quot;: true,
&quot;bonus&quot;: false,
&quot;status&quot;: &quot;OPEN&quot;,
&quot;bike_stands&quot;: 20,
&quot;available_bike_stands&quot;: 15,
&quot;available_bikes&quot;: 5,
&quot;last_update&quot;: }
Hence, our shiny application gets real time information on bike stations in 27 cities.
We also used the modern open-source library leaflet to display city map (open street map). This application works better on computer than on smartphone because shiny is not fully smartphone friendly. However, Shiny has a user-friendly interface.
If you want to try your own Shiny app, I advice you to check this gallery. It contains a lot of examples that will serve as a good introduction.
Don’t hesitate to follow us on twitter @rdata_lu and to subscribe to our youtube channel. You can also contact us if you have any comments or suggestions. See you for the next post!
Getting our feet wet with dplyr programming - I’ve been putting off trying to suss out dplyr’s tidy evaluation for a long time, but I’ve been experimenting a bit with it so just making some notes here. It’s hard to talk about tidy e...
Another R tip. Use seq_len() to avoid The backwards seqeunce bug. Many R users use the “colon sequence” notation to build sequences. For example: for(i in 1:5) { print(paste(i, i*i)) } #__ [1] &quot;1 1&quot; #__ [1] &quot;2 4&quot; #__ [1] &quot;3 9&quot; #__ [1] &quot;4 16&quot; However, the colon notation can be unsafe as … Continue reading R Tip: Use seq_len() to Avoid The Backwards Sequence Bug
Black-box models, like random forest model or gradient boosting model, are commonly used in predictive modelling due to their elasticity and high accuracy. The problem is, that it is hard to understand how a single variable affects model predictions. As a remedy one can use excellent tools like pdp package (Brandon Greenwell, pdp: An R … Czytaj dalej DALEX: understand a black box model – conditional responses for a single variable
These are my sketchnotes for Sam Charrington’s podcast This Week in Machine Learning and AI about Learning “Common Sense” and Physical Concepts with Roland Memisevic:
Sketchnotes from TWiMLAI talk #111: Learning “Common Sense” and Physical ...
1. Introduction and the Business Problem In the sneaker world, the two big names &quot;Nike&quot; and &quot;Adidas&quot; are cutthroat rivals. The business problem for both brands is to figure out how to sell more products and add to the bottom line. Based on the trends in sneaker resales, we can infer preferences of sneaker buyers. […]
A few people have recently asked me about the SwissID, as SwissPost has just been sending spam emails out to people telling them &quot;Link your Swiss Post user account to SwissID&quot;.
SwissID is not the only digital identity solution in Switzerland but as it ...
Want to know whether lapply is quicker than a for-loop? Is there a faster solution? This is the place!
A. Motivation During the recent RStudio Conference, an attendee asked the panel about the lack of support provided by the tidyverse in relation to time series data. As someone who has spent the majority of their career on time series problems, this was somewhat surprising because R already has a great suite of tools for … Continue reading Packages for Getting Started with Time Series Analysis in R →
Here is an R tip. Need to quote a lot of names at once? Use qc(). This is particularly useful in selecting columns from data.frames: library(&quot;wrapr&quot;) # get qc() definition head(mtcars[, qc(mpg, cyl, wt)]) # mpg cyl wt # Mazda RX4 21.0 6 2.620 # Mazda RX4 Wag 21.0 6 2.875 # Datsun 710 22.8 … Continue reading R Tip: Use qc() For Fast Legible Quoting
This blog post is mostly for myself but maybe it’s useful to others. It contains my current R markdown blog template. I initially posted this as a question at StackOverflow. Then I read how much a burden we put in Yihui Xie and decided that my curren...
I last blogged about books in September 2016 and it’s still in my 10 most popular posts. I thought I’d update it with some of the data-relevant books I read in 2017. Mostly these are books that were published before 2017 - it’s just that it was...
Like the last several updates, this blog post will be fairly short, given only a single bug fix was added.
Thanks again to GitHub user leocwlau who reported that the GetReportSuiteGroups function added an additional field AND provided the solution. No...
What’s Up? The NPR Visuals Team created and maintains a javascript library that makes it super easy to embed iframes on web pages and have said documents still be responsive. The widgetframe R htmlwidget uses pym.js to bring this (much needed) functionality into widgets and (eventually) shiny apps. NPR reported a critical vulnerability in this... Continue reading →
Summary
Running an application on Spark with external dependencies, such as R and python packages, requires the installation of these dependencies on all the workers. To automate this tedious process, a continuous deployment workflow has been developed...
Make memorable plots with memery. memery is an R package that generates internet memes including superimposed inset graphs and other atypical features, combining the visual impact of an attention-grabbing meme with graphic results of data analysis. Version 0.4.2 of memery is now on CRAN. The latest development version and a package vignette are available on […]
While ggplot2 (and its various extensions) is often the go-to package for graphics in R these days, if you need to step outside the boundaries of what ggplot2 can do, you can always step back to base R graphics (and the built-in lattice package) and customize to your hearts content. The problem is that (unlike for ggplot2) the default look for base graphics is kinda ... meh. That being said, the base graphics system offers almost unlimited flexibility, both via function options and via the par system for modifying layouts and graphic defaults. As Colin Gallespie explains in a recent...
Many data scientists (and even statisticians) often suffer under one of the following misapprehensions: They believe a technique doesn’t work in their current situation (when in fact it does), leading to useless precautions and missed opportunities. They believe a technique does work in their current situation (when in fact it does not), leading to failed … Continue reading We Want to be Playing with a Moderate Number of Powerful Blocks
In November I announced a new collaboration with the Census Bureau around data science training. Today I’d like to provide an update on this project....
The post Update on My Collaboration with the Census Bureau appeared first on AriLamstein.com.
MEDIA RELEASE
14 February 2018
Mango Solutions are delighted to announce that loyalty programme pioneer and data science innovator, Edwina Dunn, will keynote at the 2018...
This blog post is about my newly released RGF package (the blog post consists mainly of the package Vignette). The RGF package is a wrapper of the Regularized Greedy Forest python package, which also includes a Multi-core implementation (Fast...
Somebody recently shared this with me, this is what happens when you attempt to access Parship, an online dating site, from the anonymous Tor Browser.
Experian is basically a private spy agency. Their website boasts about how they can:
Know who your ...
This is a quick script showing how to make multi-tabbed .xlsx files. I recently had the need to do this and used the flexible openxlsx package maintained by : Alexander Walker. The package is described by the author like this: … Continue reading →
Mathematics is a place where you can do things which you can’t do in the real world (Marcus Du Sautoy, mathematician) From time to time I have a look to some of my previous posts: it’s like seeing them through another’s eyes. One of my first posts was this one, where I draw fractals using the … Continue reading Mandalas →
Outcome Supervision
Yesterday I was part of an introductory session on machine learning and unsurprisingly, the issue of supervised vs. unsupervised learning came up. In social sciences, there is a definite tendency for the former; there is more or less always a target outcome or measure that we want to optimise the performance of our models for. This reminded me of a draft that I had written the code a couple of months ago but for some reason never converted into a blog post until now. This will also allow me to take a break from conflict forecasting for a bit and go back to my usual topic of UK. My frequent usage of all things UK is at such a level that my Google Search Console insights lists r/CasualUK as the top referrer. Cheers, mates!
UK General Elections and Brexit
I think I have exploited rvest enough recently, so I will rely on a couple of good old csv’s this time. The Electoral Commission website provides the EU referendum voting totals by region. I want to combine electoral data with other socio-economic indicators so I will subset the data to London only (this is how privilege perpetuates itself). Within that subset, I will only keep the area code (for matching purposes later), and the raw number of votes for each option in the referendum. I will also create a variable indicating ‘the outcome’ based on which side got more votes:
##EU Referendum data for London
eu
A new release of the BH package arrived on CRAN a little earlier: now at release 1.66.0-1. BH provides a sizeable portion of the Boost C++ libraries as a set of template headers for use by R, possibly with Rcpp as well as other packages.
This release...
jamovi is software that aims to simplify two aspects of using R. It offers a point-and-click graphical user interface (GUI). It also provides functions that combines the capabilities of many others, bringing a more SPSS- or SAS-like method of programming … Continue reading →
How we taught a computer to understand Luxembourguish
Today we reveal a project that Kevin and myself have been working on for the past 2 months, Liss. Liss is a sentiment analysis artificial intelligence; you can let Liss read single words or who...
Our paper, weakly informative reparameterisations of location-scale mixtures, with Kaniav Kamary and Kate Lee, got accepted by JCGS! Great news, which comes in perfect timing for Kaniav as she is currently applying for positions. The paper proposes a unidimensional mixture Bayesian modelling based on the first and second moment constraints, since these turn the remainder […]
I was genuinely chuffed to get a shout-out in the most recent episode of Not So Standard Deviations, the awesome statistics-and-R themed podcast hosted by Hilary Parker and Roger Peng. In that episode, Roger recounts his recent discovery of the Microsoft ecosystem of tools for R, which he (jokingly) dubbed the &quot;Microsoft-verse&quot;. While we're flattered by the allusion to the tidyverse, in general Microsoft's developments with R are designed to work with the entire R ecosystem rather than be distinct from it. Here's a quick overview of what Microsoft has developed around R. It's in three sections: the first two...
Earlier this month we released a new version of the tesseract package to CRAN. This package provides R bindings to Google’s open source optical character recognition (OCR) engine Tesseract.
Two major new features are support for HOCR and support for the upcoming Tesseract 4.
hOCR output
Support for HOCR output was requested by one of our users on Github. The ocr() function gains a parameter HOCR which allows for returning results in hOCR format:
library(tesseract)
# Text output
text
A brand new shiny package has entered the world yesterday: shinyalert. It does only one thing, but does it well: show a message to the user in a modal (aka popup, dialog, or alert box). Actually, it can do one more thing: shinyalert can also ...
One of the greatest things about the R community is its diversity. This is largely thanks to organizations like Forwards and R-Ladies, who have been instrumental in welcoming women and other under-represented groups to the world of R. Likewise, conferences in the R community encourage diversity, with open codes of conduct, facilitations like on-site child-care, and by offering scholarships for travel and lodging to encourage attendees from diverse backgrounds. Here are three upcoming R community events that are offering diversity scholarships: The rOpenSci Unconference (May 21-22, Seattle) is now accepting self-nominations to participate. You can request travel support as part...
Trick question: is a 10,000 cell numeric data.frame big or small? In the era of &quot;big data&quot; 10,000 cells is minuscule. Such data could be fit on fewer than 1,000 punched cards (or less than half a box). The joking answer is: it is small when they are selling you the system, but can be … Continue reading Is 10,000 Cells Big?
In my last posts (here and here) I explored how to use embeddings to represent categorical variables. Furthermore, I showed how to represent categorical variables with embeddings and add other variable to create a more complex model. Both posts focused...
Udemy is offering readers of R-bloggers access to its global online learning marketplace for $15 per course! This deal (offering over 50%-90% discount) is for hundreds of their courses – including many R-Programming, data science, machine learning etc. Click here to browse ALL (R and non-R) courses Advanced R courses:  Regression modelling Comprehensive Linear Modeling with R (15 Hours of video) Linear regression in R for Data Scientists (7 hours) Linear Mixed-Effects Models with R (11 Hours of video) Linear Regression, GLMs and GAMs with R (8 hours) Forecasting Models with R (5 hours) Time Series Analysis and Forecasting in R (3 Hours of video) Regression, Data Mining, Text Mining, Forecasting using R (32 hours) Statistical/machine learning Bayesian Computational Analyses with R (12 Hours of video) Data Science and Machine Learning Bootcamp with R (17 hours) Decision Tree – Theory, Application and Modeling using R (5 hours) Data Mining with R: Go from Beginner to Advanced! (12 hours) Machine Learning A-Z™: Hands-On Python &amp; R In Data Science (40 hours) Applied Multivariate Analysis with R (13 Hours of video)  Visualization Graphs in R (ggplot2, plotrix, base R) – Data Visualization with R Programming Language (5 Hours of video) Multivariate Data Visualization with R (7 Hours of video) Other topics Create Interactive Web […]
Have you spent hours, pulling your hair out trying to figure out how to access datasets in R? Once imported to a variable, columns from a dataset (eg: CSV) can be very tricky to access. Sometimes columns contain spaces, funky characters or other incosistencies. Here are some examples on how to access the data from CSV and JSON datasets.
CSV
We’ll use this CSV dataset since it has *very* long column names.Read File: csvDataset
Owen Jones, Placement Student
Songwriting is a very mysterious process. It feels like creating something from nothing. It’s something I don’t feel like I really con...
Steph is currently out of the office, teaching people cool Data Science stuff on a cruise at Tech Outbound. She counts on her team to keep the company’s Twitter account afloat in the meantime, so I had to think of a way to contribute. What about advertising existing content from her blog in the style of her Twitter role model Mara Averick, i.e. an informative tweet accompanied by appealing screenshots?
By Gabriel Vasconcelos Overview There are several ways to do portfolio optimization out there, each with its advantages and disadvantages. We already discussed some techniques here. Today I am going to show another method to perform portfolio optimization that works … Continue reading →
Keras is a popular Python package to do the prototyping for deep neural networks with multiple backends, including TensorFlow, CNTK, and Theano. Currently, there are two R interfaces that allow us to use Keras from R through the reticulate package. While the keras R package is able to provide a flexible and feature-rich API, the […]
Previously we’ve covered the basics of exogenous variables in smooth functions. Today we will go slightly crazy and discuss automatic variables selection. But before we do that, we need to look at a Santa’s little helper function implemented in [crayon-5a7f874ccc437104037960-i/]. It is called [crayon-5a7f874ccc443987759959-i/]. It is useful in cases when you think that your exogenous […]
Official statistical classifications can be big, complex things. For example, the complete version of the “International Standard Industrial Classification of All Economic Activities, Rev.4” comes as either a 300 page PDF or a 1.3GB Microsoft Acce...
It all started with this paper in Methods in Ecol. Evol. where we looked at
detectability of many species. So we wanted to use life history
traits to validate our results. But we had to cut the manuscript,
and there was this leftover with some neat patterns, but without much focus.
It took a few years, and the most positive peer-review experience ever,
and the paper is now early view in Ecography. This post is a quick summary of the goodies stuffed inside the lhreg R package that makes the whole analysis reproducible, and provides some functions for similar PGLMM models.
We’ve released the newest version of NIMBLE on CRAN and on our website. Version 0.6-9 is primarily a maintenance release with various bug fixes and fixes for CRAN packaging issues. New features include: dimensions in a model will now be determined from either ‘inits’ or ‘data’ if not otherwise available; one can now specify “nBootReps […]
For a fifth year running, we are excited to announce the rOpenSci unconference, our annual event loosely modeled on Foo Camp. rOpenSci unconferences have a rich history. You can get a feel for them by reading collected stories about people and projects from unconf17.
We’re organizing unconf18 to bring together scientists, developers, and open data enthusiasts from academia, industry, government, and non-profits to get together for a couple of days to hack on various projects and generally enrich our community. The agenda is mostly decided during the unconference itself. Past projects have related to open data, data visualization, data publication and open science using R. This event is unlike many other unconferences in that it is primarily invite-only, with a few spots set aside for self-nominations from the community at large. That’s you!
rOpenSci is committed to providing a welcoming community for all participants. One of our priorities is to increase and retain the diversity of contributors to open research by helping people “see what they can be”. Therefore, we strongly encourage applications from women and other underrepresented genders, people of color, people who are LGBTQ, people with disabilities or any other underrepresented minorities in research. To ensure a safe, enjoyable, and friendly experience for everyone who participates, we follow a code of conduct.
Where: Downtown Seattle, WA
When: May 21-22, 2018
Apply: If you’d like to attend, it only takes a few minutes to nominate yourself. Submissions close Thursday, March 8, 2018 at midnight Pacific Time.
We’re grateful to our sponsors for making #runconf18 possible.
Questions? Contact us
by Boxuan Cui, Data Scientist at Smarter Travel Once upon a time, there was a joke: In Data Science, 80% of time spent prepare data, 20% of time spent complain about need for prepare data. — Big Data Borat (@BigDataBorat) February 27, 2013 According to a Forbes article, cleaning and organizing data is the most time-consuming and least enjoyable data science task. Of all the resources out there, DataExplorer is one of them, with its sole mission to minimize the 80%, and make it enjoyable. As a result, one fundamental design principle is to be extremely user-friendly. Most of the...
We get a lot of questions about the usage of R libraries. The most common question is “can I use all the R libraries in your notebooks/consoles?”
Yes! You can use any of the libraries that have been published to the R package repository (CRAN).
Open up your notebook/console:
1. Install
install.packages(&quot;ggplot2&quot;) this will install the package if it hasn’t already been installed.
2. Include
library(ggplot2) this will call/include the library into your session.
3. Use
ggplot(data=iris,aes(x=Sepal.Length,y=Petal.Length))+geom_point()
Remember, when installing packages, you have to use quote marks inside the install.packages(&quot;&quot;) function but not when including them using the library().
Once libraries are installed, you don’t have to install them again. You can just call them with library() and then use them as you wish.
Link to notebook: https://www.datazar.com/file/f105230b5-0a2f-4976-af08-db1b00f793a4
How to Install and Include an R Package was originally published in Datazar Blog on Medium, where people are continuing the conversation by highlighting and responding to this story.
Many water utilities are implementing or considering digital metering. This article describes analysing digital water meter data using the data science Tidyverse library. Continue reading →
The post Analysing Digital Water Meter Data using the Tidyverse appeared first on The Devil is in the Data.
A new minor release 0.3.3.4.0 of RcppEigen hit CRAN earlier today, and just went to Debian as well. It brings Eigen 3.3.4 to R.
Yixuan once again did the leg-work of bringing the most recent Eigen release in along with the small set of patches we hav...
This report describes the development of a colour list and colour metric for the ‘roloc’ package that is based on the ISCC-NBS System of Color Designation. Paul Murrell Download
Data science tells a story through visualisation. Both story and visualisation rely on a good plot. And an abundance of those has evolved over time. Many have their own dedicated Wikipedia page! Which generate the most interest? How is the interest in each trending over time?
The post The plots thicken appeared first on thinkr.
In case you missed them, here are some articles from January of particular interest to R users. Josh Katz and Peter Aldhous used R to analyze the content and presentation of the most recent State of the Union speech from the US president. Slides for my presentation &quot;Speeding up R with Parallel Processing in the Cloud&quot;, with applications of the doAzureParallel and sparklyr packages. An example of using the doAzureParallel package to speed up a statistical simulation. 5 lines of R code to create a list of US Representatives from a Wikipedia table. A package to visualize routes from activities...
If you have ever developed or used an open-source R package, you’re likely familiar with continuous integration. By automating the process of testing each proposed change in the source code, you can reduce the risk of errors, avoid unnecessary...
In my previous post (https://statcompute.wordpress.com/2018/01/28/modeling-lgd-with-proportional-odds-model), I’ve discussed how to use Proportional Odds Models in the LGD model development. In particular, I specifically mentioned that we would estimate a sub-model, which can be Gamma or Simplex regression, to project the conditional mean for LGD values in the (0, 1) range. However, it is worth pointing out […]
I’m pleased to announce an update for the sjmisc-package, which was just released on CRAN. Here I want to point out two important changes in the package. New default option for recoding and transformation functions First, a small change in the code with major impact on the workflow, as it affects argument defaults and is … Weiterlesen Data transformation in #tidyverse style: package sjmisc updated #rstats →
The Data Science Virtual Machine was featured on a recent episode of the AI Show with Seth Juarez and Gopi Kumar. If you want a quick and easy way to spin up a virtual machine with all of the data science tools you'll ever need — including R and RStudio — already installed and ready to go, this video explains what the Data Science Virtual Machine is used for and (at 21:00) how to launch one in the Azure portal. The Data Science Virtual Machine is available in both Windows and Linux flavors. Even if you don't yet have an...
Leeds (Predictive Analytics in R)
London (Tensorflow)
Birmingham (Python &amp; Machine Learning)
Hi there!
We’re running some courses on R, Python and Tensorflow around the UK that you might be interested in! All courses are spearheaded with lec...
Much of what I need to do for work-work involves using tools that are (for the moment) not in R. Today, I needed to test the validity of (and other processing on) DMARC records and I’m loathe to either reinvent the wheel or reticulate bits from a fragmented programming language ecosystem unless absolutely necessary. Thankfully,... Continue reading →
Adding a file to your R notebook is a simple 2 step process.1. “Import” file to notebook session.This makes sure your dataset is isolated in one state so that it doesn’t change as you analyze it. Remember, you can always load the latest content...
Last week I set up a shiny server, it was relatively easy! But I wanted something more, a way to make changes on my local computer and push it to the server.
Shiny server (I used the open source version) has multiple installers provided by RStudi...
We’re delighted to announce that Julia Silge will be joining us on 7 November in Seattle as our Keynote speaker.
Julia is a Data Scientist at Stack Overflow, has a PhD...
A friend and colleague of mine, Péter Batáry
has circulated news from Nature
magazine about the EU freezing innovation funds to Bulgaria.
The article had a figure about publication trends for
Bulgaria, compared with Romania and Hungary.
As I have blogged about such trends in ecology before
(here and here), I felt the need to update my PVA models
with two years worth of data from WoS.
The 2018 edition of the long-running Statistical Practice in Epidemiology using R (SPE) course will take place in Lyon, Fance on 20-24 June 2018 at the International Agency for Research on Cancer. The deadline for applications is 1 April.
Project Objective
Untappd has some usage restrictions for their API namely not allowing any exploratory of analytics uses, so I’m going to explore tweets of beer and brewery check-ins from the Untappd app to find some implicit trends in how users share their activity.
Exploratory Analysis
library(tidyverse)
library(rtweet)
library(stringr)
library(wesanderson)
library(maps)
library(tidytext)
library(dumas) # http://jasdumas.github.io/dumas/
library(wordcloud)
All social media shares from the Untappd app include their own short URL ‘untp.beer’, which makes the search query criteria identifiable using the search_tweets() function.
untp
Registration is now open for my 1.5-day workshop on deep learning with Keras and TensorFlow using R.
It will take place on April 12th and 13th in Hamburg, Germany.
In my workshop, you will learn
the basics of deep learning
what cross-entropy and loss ...
This is my Shiny Dashboard for stocks listed on S&amp;P500. Using R shiny to give a visualized general insight of market and using ARIMA model to analyze risk for a single stock. Background When I worked as a trader. I found that it was really inconvenient to do the daily market research because the information on […]
The Charges Data Set The NYC Open Data project provides the Department of Consumer Affairs data from 2016-2017.  The Department of Consumer Affairs was started in 1969 with the goal to &quot;ensure compliance with local consumer protection and licensing laws, and State and federal regulations.&quot;  They oversee almost every industry in NYC.  This data set describes the charges […]
Abstract Background: This blog post is going to show you how to write a reproducible article in the field of clinical chemistry using R Mardkown. The only thing that will change for journal to journal will be the reference fomating and perhaps section numbering. The source code itself will be provided so that you can … Continue reading Reproducible Research: Write your Clinical Chemistry paper using R Markdown →
My new office introduced me to a betting game I wasn’t previously familiar with: Super Bowl squares. It’s played with a ten-by-ten grid, like this one from printyourbrackets.com:
Each row and column gets an assortment of digits from 0-9 represen...
Background Do you like earning money? And, at the same time contribute to saving the planet? What if I prove to you Solar Business is the next booming field having plenty of opportunities in coming decades, with plus point of reducing our carbon footprint. You'd argue...Really? But solar installations are so expensive! They are not […]
In my last post, I explored how to use embeddings to represent categorical variables. Furthermore, I showed how to extract the embeddings weights to use them in another model.
While the concept of embedding representation has been used in NLP for quit...
Yet more Metalliplots
I’m just going to tidy up some issues from my last post where I did some tidytext analysis on the legendary rockers - Metallica.
It was a fun way to pass some time and keep my hand in with text mining, something I st...
 In this tutorial, we will explore the characterisitcs of projects on Kickstarter and try to understand what separates the winners from the projects that failed to reach their funding goals. ( This post is mirrored from our main blog site blog.journeyofanalytics.com . The code can be downloaded or run “LIVE” from Kaggle using this link.) Qs … Continue reading How to raise money on Kickstarter – tutorial with EDA and predictions
As the reader may know, R objects are documented in files written in “R documentation” (Rd) format, a simple markup language much of which closely resembles (La)TeX, which can be processed into a variety of formats, including LaTeX, HTML and plain text. This LaTeX-like syntax, combined with the fact that the actual R objects live… Continuar leyendo Documenting R packages: roxygen2 vs. direct Rd input →
And yet another boring little RVowpalWabbit package update, now to version 0.0.12, and still in response to the CRAN request of not writing files where we should not (as caught by new tests added by Kurt). I had misinterpreted one flag and actually i...
A puzzle
In my last blog post, I gave a hypothetical test exercise for candidates in a recruitment process, involving the International Visitor Survey (IVS) from the Ministry of Business, Innovation and Employment (MBIE). I once managed the team resp...
Date conversion in R can be a real pain. However, it is a very important initial step when you first get your data into R...
The Main Competition
The Secondary Competition
So… big news.
Jumping Rivers is sponsoring eRum 2018 and in light of this news we are giving away a free place at the conference!
(Not to mention our very own lead consultant, Colin Gillespie, i...
Here at Win-Vector LLC we like permutation tests. Our team has written on them (for example: How Do You Know if Your Data Has Signal?) and they are used to estimate significances in our sigr and WVPlots R packages. For example permutation methods are used to estimate the significance reported in the following ROC plot. … Continue reading Why No Exact Permutation Tests at Scale?
For those of you out there who speak German:
I was interviewed for a tech podcast where I talked about machine learning, neural nets, why I love R and Rstudio and how I became a Data Scientist.
You can download and listen to the podcast here: https://m...
On Wednesday, April 25th 2018 I am going to talk about explainability of machine learning models at the Minds Mastering Machines conference in Cologne. The conference will be in German, though.
ERKLÄRBARKEIT VON MACHINE LEARNING: WIE KÖNNEN WIR ...
This article simulates water consumption to assist with developing leak detection algorithms. Simulating water consumption helps to develop business tools. Continue reading →
The post Simulating Water Consumption to Develop Analysis and Reporting appeared first on The Devil is in the Data.
President Trump's State of the Union speech was last night, and it seemed to me it dragged on a bit. That's because it was apparently the slowest SOTU speech in history, based on average number of words spoken per minute. The chart below, based on these data, was created by Josh Katz. (The R code for the chart is here. Note the use of his needs package to load the required R packages.) Peter Aldhous at Buzzfeed analyzed last night's State of the Union speech as well. Not only was the speech delivered slowly, the script was at a 9-year-old's...
In another continuation of the summer of Bayesian conferences in Europe, the Universidat de Valencià is organising a summer school on Bayesian statistics, from 16 July till 20 July, 2018. Which thus comes right after our summer school on computational statistics at Warwick. With a basic course on Bayesian learning (2 days). And a more […]
If you use SQL Server (or Azure SQL DB) as your data store and you need to connect to the databasse from shinyapps.io, you’re presently stuck with FreeTDS. If you have any control over infrastructure I cannot recommend highly enough the actual ODBC Driver on Linux for ease. Alas, shinyapps.io does not let you control the infrastructure. We have to make do with with FreeTDS and it can be pretty painful to get right.
Introduction Not long ago I was reading on t-Distributed Stochastic Neighbor Embedding (t-sne), a very interesting dimension reduction technique, and on Mel frequency cepstrum a sound processing technique. Details of both techniques can be found here and here. Can we combine … Continue reading →
Last call for the course on Advanced R programming scheduled in Leuven, Belgium on Febuary 20-21 2018. Register at: https://lstat.kuleuven.be/training/coursedescriptions/AdvancedprogramminginR.html You'll learn during that course:
The apply family of functions, basic parallel programming for these functions and commonly needed data manipulation skills
Making a basic reproducible report using Sweave and knitr including tables, graphs and literate programming
How to create an R package
Understand how S3 programming works, generics, environments, namespaces.
Basic tips on how to organise and develop R code and test it.
Need other training: visit http://bnosac.be/index.php/training
 
So, Strava's heatmap made quite a stir the last few weeks. I decided to give it a try myself. I wanted to create some kind of &quot;personal heatmap&quot; of my runs, using Strava's API. Also, combining the data with Leaflet maps allows us to make use of the beautiful map tiles supported by Leaflet and to zoom and move the maps around - with the runs on it, of course.So, let's get started. First, you will need an access token for Strava's API. I found all the necessary information for this in this helpful &quot;Getting started&quot; post. As soon as you have the token, you have access to your own data.Now, let's load some packages and define functions for getting and handling the data. For the get.activities() function, I adapted code from here.library(httr)library(rjson)library(OpenStreetMap)library(leaflet)library(scales)library(dplyr)token %
  fitBounds(lng1 = min(lons.range), lat1 = max(lats.range), lng2
When dealing with Machine Learning problems in R, most of the time you rely on already existing libraries. This fastens the analysis process, but do you really understand what is behind the algorithms? Could you implement a logistic regression from scratch with R? The goal of this post is to create our own basic machine […]
The post Create your Machine Learning library from scratch with R ! (1/3) appeared first on Enhance Data Science.
[…]
I was pleased to read Shirish Agarwal's blog in reply to the blog I posted last week Do the little things matter?
Given the militaristic theme used in my own post, I was also somewhat amused to see news this week of the Strava app leaking locations and...
“Once upon a time, I, Chuang Tzu, dreamt I was a butterfly, fluttering hither and thither, to all intents and purposes a butterfly. I was conscious only of following my fancies as a butterfly, and was unconscious of my individuality as a man. Suddenly, I awoke, and there I lay, myself again. Now I do … Continue reading Deep Learning from first principles in Python, R and Octave – Part 3
Get WikiTables from Lists
Recently I was asked to submit a short take-home challenge and I thought what better excuse for writing a quick blog post! It was on short notice so initially I stayed within the confines of my comfort zone and went for something safe and bland. However, I alleviated that rather fast; I guess you want to stand out a bit in a competitive setting. Note that it was a visualisation task, so the data scraping was just a necessary evil. On that note.
I resorted to using Wikipedia as I was asked to visualise change in a certain x going back about 500 hundred years. Not many academic datasets go that far, so Wiki will have to do for our purposes. And once you are there, why only visualise half a millennium, let’s go from 1 AD to present day. First, we need a subject. I went with my PhD topic, war duration. So let’s get the list of lists first using rvest:
#Get wikipedia time periods for List of Wars
url %
html_nodes(&quot;ul&quot;) %__%
html_text(trim = TRUE) %__%
strsplit(split = &quot;\n&quot;) %__%
unlist()
periods
I am happy to announce that on Tuesday, April 24th 2018 Uwe Friedrichsen and I will give a talk about Deep Learning - a Primer at the JAX conference in Mainz, Germany.
Deep Learning is one of the “hot” topics in the AI area – a lot of hype, a lo...
Song lyric and sentiment analysis for all
So - a while back I did a tidy text analysis on Faith No More lyrics. I had thought about doing this with Metallica album lyrics, as they have had a long career, spanning thier late teens/twenties to...
We are super pumped to be launching a weekly data science podcast called DataFramed, in which Hugo Bowne-Anderson (me), a data scientist and educator at DataCamp, speaks with industry experts about what data science is, what it's capable of, what it looks like in practice and the direction it is heading over the next decade … Continue reading Introducing DataFramed, a Data Science Podcast
This is a guest post from Leon Eyrich Jessen, a postdoctoral researcher in the Immunoinformatics and Machine Learning Group at the Technical University of Denmark.
Introduction
Simon Caulton, Adoptive T-cell therapy, CC BY-SA 3.0
In my...
This past weekend I attended the R User Day at Data Day Texas in Austin. It was a great event, mainly because so many awesome people from the R community came to give some really interesting talks. Lucy D’Agostino McGowan has kindly provided a list of the talks and links to slides, and I thoroughly recommend checking it out: you're sure to find a talk (or two, or five or ten) that interests you. My own talk was on Speeding up R with Parallel Programming in the Cloud, where I talked about using the doAzureParallel package to launch clusters for...
At Mango, we’re passionate about R and promoting its use in enterprise – it’s why we created the EARL Conferences. We understand the importance of sharing knowledg...
Hey folks, I wanted to give y’all an update about Locke Data one year on from when I started it up.
In the past year, I’ve delivered more than 32 days of training, wrote and published 2 books, worked with 3 clients, and generally whimpered at my schedule. It has been amazing how much support the community has given me, and I’ve tried to give back where possible by giving away books each month, doing the usual presenting, holding free office hours, and offering community workshops.
Another boring little RVowpalWabbit package update to version 0.0.11 came in response to another CRAN request: We were writing temporary output (a cache file for the fit/prediction, to be precise) to a non-temporary directory, which is now being caug...
Unsupervised machine learning looks for hidden structure in &quot;unlabeled&quot; data, i.e. a classification or categorisation that is not included in the observations. Hierarchical clustering will reveal a &quot;cluster of six&quot; Labour MPs as being the most &quot;distant&quot; from the wider party.
The post The “cluster of six” appeared first on thinkr.
I love making maps, I also love making gifs.
In this short post I make an animated gif of parts of a map moving. In this case the parts of the map only move in the xy direction, but you can also turn them, and make them bigger or smaller.
Today I ...
And yet another small maintenance release, now at version 0.6.15, of the digest package arrived on CRAN and in Debian today.
digest creates hash digests of arbitrary R objects (using the 'md5', 'sha-1', 'sha-256', 'sha-512', 'crc32', 'xxhash32', 'xxh...
What do you do when analyzing data is fun, but you don't have any new data? You make it up. This simulation tests the type I error rates of two-sample t-test in R and SAS. It demonstrates efficient methods for simulation, and it reminders the reader not to take the result of any single hypothesis test as gospel truth. That is, there is always a risk of a false positive (or false negative), so determining truth requires more than one research study.
A type I error is a false positive. That is, it happens when a hypothesis test rejects the null hypothesis when in fact it is not true. In this simulation the null hypothesis is true by design, though in the real world we cannot be sure the null hypothesis is true. This is why we write that we &quot;fail to reject the null hypothesis&quot; rather than &quot;we accept it.&quot; If there were no errors in the hypothesis tests in this simulation, we would never reject the null hypothesis, but by design it is normal to reject it according to alpha, the significance level. The de facto standard for alpha is 0.05. R
First, we run a simulation in R by repeatedly comparing randomly-generated sets of normally-distributed values using the two-sample t-test. Notice the simulation is vectorized: there are no &quot;for&quot; loops that clutter the code and slow the simulation. Read more »For more posts like this, see Heuristic Andrew.
If you work on applications for clients or have open sourced some shiny apps, a question that arises is how is your application being used. What you can do in order to find out how your hard work is being consumed is putting your code in logs and then ...
In order to stay up to date, I try to follow Jeremy Howard on a regular basis. In one of his recent videos, he shows how to use embeddings for categorical variables (e.g. weekdays).
First off; what are embeddings?
An embedding is a mapping of a catego...
On every SQL community event, where there could be a cluster of sessions dedicated to BI or analytics, I would have people asking me, “which one would you recommend?” or “which one I  prefer?” So, questions about recommendation and preferences are in my opinion the hardest one. And not that I would know my preferences … Continue reading R or Python? Python or R? The ongoing debate.
These are my sketchnotes for Sam Charrington’s podcast This Week in Machine Learning and AI about Neuroevolution: Evolving Novel Neural Network Architectures with Kenneth Stanley:
Sketchnotes from TWiMLAI talk #94: Neuroevolution: Evolving Novel Ne...
The LGD model is an important component in the expected loss calculation. In https://statcompute.wordpress.com/2015/11/01/quasi-binomial-model-in-sas, I discussed how to model LGD with the quasi-binomial regression that is simple and makes no distributional assumption. In the real-world LGD data, we usually would observe 3 ordered categories of values, including 0, 1, and in-betweens. In cases with a […]
I would like to demonstrate some helpful wrapr R notation tools that really neaten up your R code. Img: Christopher Ziemnowicz. Named Map Builder First I will demonstrate wrapr‘s &quot;named map builder&quot;: :=. The named map builder adds names to vectors and lists by nice &quot;names on the left and values on the right&quot; notation. … Continue reading Supercharge your R code with wrapr
Remember the nascent series of blog posts about Parks and recreation? Well, we’re still at one post, but don’t worry, here is a new one, and I’m sure the series will eventually be a real one. I’m looking at you, my R-Ladies friends. That said, ...
I wrote a quick function for simulating a Gaussian process given only its kernel function and used it to simulate processes with different kernels and very different properties.
I read this tweet thread yesterday, and one of the great things in it was discovering that the Department for Transport release traffic count data for Great Britain. When you download individual observation… Continue reading →
Time for another look at what’s new and interesting in the #rstats world with the help of Peter Meissner’s (@marvin_dpr) crossword.r🔗. The answers to last week’s puzzle have been posted (it seemed to make more sense posting the answers a week later vs the Monday after). There is a dedicated category — puzzler — to... Continue reading →
The prrd package was introduced recently, and made it to CRAN shortly thereafter. The idea of prrd is simple, and described in some more detail on its webpage and its GitHub repo. Reverse dependency checks are an important part of package development...
This blog post series is on machine learning with R. We will use the Caret package in R. In this part, we will first perform exploratory Data Analysis (EDA) on a real-world dataset, and then apply non-regularized linear regression to solve a supervised regression problem on the dataset. We will predict power output given a […]
Related Post
How to perform Logistic Regression, LDA, &amp; QDA in R
How to apply Linear Regression in R
Linear Regression in Python; Predict The Bay Area’s Home Prices
Building A Logistic Regression in Python, Step by Step
Multicollinearity in R
The keynotes for eRum 2018 are announced! Check them out and every other information about the international conference taking place in Budapest, this year! ---- The eRum conferences are particularly thought for the many Europeans that can’t manage to take part in the use!R conferences when they are based in USA – or, just like this year, in Australia. The 2018 eRum edition will take place in Budapest, Hungary, from the 14th to the 16th of May. Today we are particularly happy to announce the conference keynotes speakers, and what a list of names!
We will have with us Achim Zeileis, Stefano Iacus, Martin Mächler and Nathalie Villa-Vialaneix.
The post eRum 2018: Keynotes announced! appeared first on Quantide - R training &amp; consulting.
In a widely shared video, US Admiral McRaven addressing University of Texas at Austin's Class of 2014 chooses to deliver a simple message: make your bed every day.
A highlight of this talk is the quote The little things in life matter. If you can't do...
I'm a big fan using R to simulate data. When I'm trying to understand a data set, my first step is sometimes to simulate data from a model and compare the results to the data, before I go down the path of fitting an analytical model directly. Simulations are easy to code in R, but they can sometimes take a while to run — especially if there are a bunch of parameters you want to explore, which in turn requires a bunch of simulations. When your pc only has four core processors and your parallel processing across three of the.#DataScience...
In a recent release of tidytext, we added tidiers and support for building Structural Topic Models from the stm package. This is my current favorite implementation of topic modeling in R, so let’s walk through an example of how to get started with th...
We are pleased to announce the release of XLConnect 0.2-14, now available from CRAN and from our GitHub repository.
This release is mainly a maintenance release and comes with the following changes:
Upgrade to Apache POI 3.17
Upgraded XLCo...
Unfortunately this was not taught in any of my statistics or data analysis classes at university (wtf it so needs to be :scream_cat:).
So it took me some until I learned that the AUC has a nice probabilistic meaning.
What’s AUC anyway?
Consider:
A dataset : , where
is a vector of features collected for the th subject,
is the th subject’s label (binary outcome variable of interest, like a disease status, class membership, or whatever binary label).
A classification algorithm (like logistic regression, SVM, deep neural net, or whatever you like), trained on , that assigns a score (or probability) to any new observation signifying how likely its label is .
Then:
A decision threshold (or operating point) can be chosen to assign a class label ( or ) to based on the value of .
The chosen threshold determines the balance between how many false positives and false negatives will result from this classification.
Plotting the true positive rate (TPR) against the false positive rate (FPR) as the operating point changes from its minimum to its maximum value yields the receiver operating characteristic (ROC) curve. Check the confusion matrix if you are not sure what TPR and FPR refer to.
The area under the ROC curve, or AUC, is used as a measure of classifier performance.
Here is some R code for clarification (not even using tidyverse :stuck_out_tongue:):
# load some data, fit a logistic regression classifier
data(iris)
versicolor_virginica
This report introduces the ‘roloc’ package for R, which provides functions for converting colour specifications into colour names. Paul Murrell Download
In what is rapidly becoming a series — cool things you can do with R in a tweet — Julia Silge demonstrates scraping the list of members of the US house of representatives on Wikipedia in just 5 R statements: library(rvest) library(tidyverse) h % html_node(&quot;#mw-content-text __ div __ table:nth-child(18)&quot;) %__% html_table() reps % as_tibble() pic.twitter.com/25ANm7BHkj — Julia Silge (@juliasilge) January 12, 2018 Since Twitter munges the URL in the third line when you cut-and-paste, here's a plain-text version of Julia's code: library(rvest) library(tidyverse) h % html_node(&quot;#mw-content-text __ div...
This is guest post contributed by Pete Mohanty, creator of the kerasformula package.
Overview
The kerasformula package offers a high-level interface for the R interface to Keras. It’s main interface is the kms function, a regression-style...
Exploratory data analysis, data preparation and model performance using 'funModeling' R package.
This post will be about comparing a volatility signal using three different variations of implied volatility indices to predict when … Continue reading →
There is a new version of the R package vtreat now up on CRAN. vtreat is an essential data preparation system for predictive modeling that helps defend your predictive modeling work against real world data issues including: High cardinality categorical variables Rare levels (including new or novel levels during application) in categorical variables Missing data … Continue reading Latest vtreat up on CRAN
To post your R job on the next post Just visit  this link and post a new R job  to the R community. You can post a job for  free  (and there are also “featured job” options available for extra exposure). Current R jobs Job seekers:  please follow the links below to learn more and apply for your R job of interest: Featured Jobs Freelance R Shiny Developer – Work From Home Data Science Talent – Posted by Data Science Talent Frankfurt am Main Hessen, Germany 16 Jan 2018 Full-Time Customer Success Representative RStudio, Inc. – Posted by jclemens1 Anywhere 8 Jan 2018 Full-Time Core Data Science @Facebook – PhD Intern (London 2018) Tal Galili London England, United Kingdom 21 Dec 2017 Full-Time R Shiny Dashboard Engineer in Health Tech Castor EDC – Posted by Castor EDC Amsterdam-Zuidoost Noord-Holland, Netherlands 12 Dec 2017 Part-Time Data Scientist Alchemy – Posted by Andreas Voniatis Anywhere 8 Dec 2017 More New R Jobs Full-Time Senior Data Scientist @ Manchester, UK Hello Soda – Posted by LeanneFitzpatrick Manchester England, United Kingdom 24 Jan 2018 Full-Time Staff Scientist – Imaging, Statistics, AI @ Maryland, U.S. US FDA, Center for Devices and Radiological Health – Posted by brandon.gallas Colesville Maryland, United States 24 Jan 2018 Freelance R shiny application developer (freelance) World […]
Pretty recently I found a paper with the title “Why Should I Trust You?”: Explaining the Predictions of Any Classifier. The topic of interpretability is very important in the times of complex machine learning models and it’s also related to my PhD topic (reliability of machine learning models). Therefore I wanted to play around with … Continue reading Exploring lime on the house prices dataset
The post Exploring lime on the house prices dataset appeared first on verenahaunschmid.
We’re proud to be part of the Data Science and R communities.
We recognise the importance of knowledge sharing across industries, helping people with their personal an...
In our next MünsteR R-user group meetup on March 5th, 2018 Frank Rühle will talk about bioinformatics and how to analyse genome data.
You can RSVP here: http://meetu.ps/e/DDY1B/w54bW/f
Next-Generation sequencing and array-based technologies prov...
A purely geometric Le Monde mathematical puzzle this (or two independent ones, rather): Find whether or not there are inscribed and circumscribed circles to a convex polygon with 2018 sides of lengths ranging 1,2,…,2018. In the first (or rather second) case, the circle of radius R that is tangential to the polygon and going through […]
Highlights from StanCon 2018
This year we had the privelage of sponsoring StanCon. Unfortunately, we weren’t able to actually attend the conference. Rather than let our ticket go to waste, we ran a small competition, which Ignacio Martinez won ...
The pbapply R package that adds progress bar to vectorized functions has been know to accumulate overhead when calling parallel::mclapply with forking (see this post for more background on the issue). Strangely enough, a GitHub issue held the key to the solution that I am going to outline below. Long story short: forking is no longer expensive with pbapply, and as it turns out, it never was.
Strava is a fitness app that records you activities, including the routes of your walks, rides and runs. The service also provides an API that allows you to extract all of your data for analysis. University of Melbourne research fellow Marcus Volz created an R package to download and visualize Strava data, and created a chart to visualize all of his runs over six years as a small multiple. Inspired by his work (and the availability of the R package he created), others also visualized bike rides, activity calendars, and aggregated route maps with elevation data. (You can see several...
Of course, I’m paraphrasing Dirk’s fifteenth post in the rarely rational R rambling series: #15: Tidyverse and data.table, sitting side by side … (Part 1). I very much liked it, because, although I’m a happy tidyverse user, I’m always trying not to be tied into that verse too much by replicating certain tasks with other tools (and languages) as an… Continuar leyendo Tidyverse and data.table, sitting side by side… and then base R walks in →
Before we start…
We hope you found the first half of this post useful and interesting. Before we dive into the code, I want to explain a few important aspects of data science. Firstly, implementing data science in practice is always a researc...
Introduction Services that deliver the ingredients and recipes you need to make fresh, home-cooked meals, like Instacart have grown in popularity as people are turning away from low-cost, fast food in favor of healthier alternatives. However, those subscription services are still luxury for many so I decided to create a interactive dashboard to help the […]
A management challenge
Between 2011 and 2017 I selected somewhere between 20 and 30 staff and contractors, for New Zealand public sector roles with titles like Analyst, Senior Analyst and Principal Analyst. Alternative names for these roles could h...
Making it even easier to download and organize stock prices from Yahoo Finance - I just released a long due update to package BatchGetSymbols. The
files are under review in CRAN and you should get the update soon.
Meanwhile, you c...
The Best Algorithms are the Simplest The field of data science has progressed from simple linear regression models to complex ensembling techniques but the most preferred models are still the simplest and most interpretable. Among them are regression, logistic, trees and naive bayes techniques. Naive Bayes algorithm, in particular is a logic based technique which … Continue reading Understanding Naïve Bayes Classifier Using R
In any model development exercise, a considerable amount of time is spent in understanding the underlying data, visualizing relationships and validating preliminary hypothesis (broadly categorized as Exploratory data Analysis). A key element of EDA involves visually analyzing the data to glean valuable insights and understand underlying relationships &amp; patterns in the data. While EDA is […]
Related Post
Building a Daily Bitcoin Price Tracker with Coindeskr and Shiny in R
Getting started with dplyr in R using Titanic Dataset
How to apply Monte Carlo simulation to forecast Stock prices using Python
Analysing iOS App Store iTunes Reviews in R
Handling ‘Happy’ vs ‘Not Happy’: Better sentiment analysis with sentimentr in R
Another Rblpapi release, now at version 0.3.8, arrived on CRAN yesterday. Rblpapi provides a direct interface between R and the Bloomberg Terminal via the C++ API provided by Bloomberg Labs (but note that a valid Bloomberg license and installation is...
In war, truth is the first casualty (Aeschylus) I am not a pessimistic person. On the contrary, I always try to look at the bright side of life. I also believe that living conditions are now better than years ago as these plots show. But reducing the complexity of our world to just six graphs … Continue reading Fatal Journeys: Visualizing the Horror →
One of the great features of R is the possibility to quickly access web-services. While some companies have the habit and policy to document their APIs, there is still a large chunk of undocumented but great web-services that help the regular data scie...
I currently advise R dplyr users to take care when using multiple assignment dplyr::mutate() commands on databases. (image: Kingroyos, Creative Commons Attribution-Share Alike 3.0 Unported License) In this note I exhibit a troublesome example, and a systematic solution. First let’s set up dplyr, our database, and some example data. library(&quot;dplyr&quot;) ## ## Attaching package: 'dplyr' … Continue reading Advisory on Multiple Assignment dplyr::mutate() on Databases
I revisited my previous post on creating beautiful time series calendar heatmaps in ggplot, moving the code into the tidyverse. To obtain following example:Simply use the following code: I hope the commented code is self-explanatory - enjoy :-)
In this tutorial, we will explore the open roles at Google, and try to see what common attributes Google is looking for, in future employees. This dataset is a compilation of job descriptions of 1200+ open roles at Google offices across the world. This dataset is available for download from the Kaggle website, and contains … Continue reading Who wants to work at Google?
This week’s R bulletin covers some interesting ways to list functions, to list files and illustrates the use of double colon operator. Hope you like this R weekly bulletin. Enjoy reading!   Shortcut Keys 1. New document – Ctrl+Shift+N 2. Close active document – Ctrl+W 3. Close all open documents – Ctrl+Shift+W Problem Solving Ideas...
The post R Weekly Bulletin Vol – XIV appeared first on .
Statisfix - Which fixing should I buy?
I have a bathroom cabinet to put up.
It needs to go onto a tiled plasterboard (drywall) wall. Because of the tiles, I can’t use the fixings I normally use to keep heavy objects fixed to the wall. A...
ggtern version 2.2.2 has just been submitted to CRAN, and it includes a number of new features. This time around, I have adapted the hexbin geometry (and stat), and additionally, created an almost equivalent geometry which operates on a triangular mesh rather than a hexagonal mesh. There are some subtle differences which give some added […]
The post Version 2.2.2 Released appeared first on ggtern: ternary diagrams in R.
Recently, there has been a lot of attention for the imposter syndrome. Even seasoned programmers admit they suffer from feelings of anxiety and low self-esteem. Some share their personal stories, which can be comforting for those suffering in silence. I focus on a method that helped me grow confidence in recent years. It is a simple, yet very effective way to deal with being overwhelmed by the many things a data scientis can acquaint him or herself with.
General Linear Models: The Basics
General linear models are one of the most widely used statistical tool
in the biological sciences. This may be because they are so flexible and
they can address many different problems, that they provide useful
output...
Peter Meissner (@marvin_dpr) released crossword.r🔗 to CRAN today. It’s a spiffy package that makes it dead simple to generate crossword puzzles. He also made a super spiffy javascript library to pair with it, which can turn crossword model output into an interactive puzzle. I thought I’d combine those two creations with a way to highlight... Continue reading →
This week I had the opportunity to teach R at my workplace, again. This course was the “advanced R” course, and unlike the one I taught at the end of last year, I had one more day (so 3 days in total) where I could show my colleagues the joys of th...
These are my sketchnotes for Sam Charrington’s podcast This Week in Machine Learning and AI about Learning State Representations with Yael Niv: https://twimlai.com/twiml-talk-92-learning-state-representations-yael-niv/
Sketchnotes from TWiMLAI talk...
Registration is now open for useR! 2018, the official R user conference to be held in Brisbane, Australia July 10-13. If you haven't been to a useR! conference before, it's a fantastic opportunity to meet and mingle with other R users from around the world, see talks on R packages and applications, and attend tutorials for deep dives on R-related topics. This year's conference will also feature keynotes from Jenny Bryan, Steph De Silva, Heike Hofmann, Thomas Lin Pedersen, Roger Peng and Bill Venables. It's my favourite conference of the year, and I'm particularly looking forward to this one. This...
What is SatRday?
SatRday in Cape Town
Be in it to win it
Jumping Rivers is proud to be sponsoring the upcoming SatRday conference in Cape Town, South Africa on 17th March 2018.
What is SatRday?
SatRdays are a collection of free/cheap accessible ...
When fitting a Bayesian model using MCMC (often via JAGS/BUGS/Stan), a number of checks are typically performed to make sure your model is worth interpreting without further manipulation (remember: all models are wrong, some are useful!): R-hat (AKA Gelman-Rubin statistic) … Continue reading →
We’re doing some interesting studies (cybersecurity-wise, not finance-wise) on digital currency networks at work-work and — while I’m loathe to create a geo-map from IPv4 geolocation data — we: do get (often, woefully inaccurate) latitude &amp; longitude data from our geolocation service (I won’t name-and-shame here); and, there are definite geo-aspects to the prevalence of... Continue reading →
Slides from January 2018 EdinbR session on spatial data in R.
After last year’s successful premiere, the eoda R-Akademie on the Road will be back in Hamburg this year for four inspiring days of professional R training for data scientists, programmers and R enthusiasts. The program for German-speaking users will start with a two-day introduction to R on 17-18 April. This course provides insights into the … „R-Akademie back in town – eoda R training returns to Hamburg in April“ weiterlesen
caret has been able to utilize parallel processing for some time (before it was on CRAN in October 2007) using slightly different versions of the package. Around September of 2011, caret started using the foreach package was used to &quot;harmonize&quot; the par...
Bioinformatics platform at Berlin Institute for Medical Systems Biology has been organizing computational genomics courses in Berlin since 2015. This year we will focus on bisulfite sequencing analysis and genomics data integration.The course will cover the following:1) bisulfite-seq analysis (alignment, QC, differential methylation, segmentation)2) integrating and visualizing processed genomics data. These methods can be used for any genomics data set not necessarily BS-seq data. There will be theoretical lectures followed by practical sessions where students directly apply what they have learned. The programming will be mainly done in R and unix shell.The deadline for application is March 1st! You can get more info at http://compgen.mdc-berlin.de computational genomics course 2018 berlin from Altuna Akalin
Microsoft R Open (MRO), Microsoft's enhanced distribution of open source R, has been upgraded to version 3.4.3 and is now available for download for Windows, Mac, and Linux. This update upgrades the R language engine to the latest R (version 3.4.3) and updates the bundled packages (specifically: checkpoint, curl, doParallel, foreach, and iterators) to new versions. MRO is 100% compatible with all R packages. MRO 3.4.3 points to a fixed CRAN snapshot taken on January 1 2018, and you can see some highlights of new packages released since the prior version of MRO on the Spotlights page. As always, you...
I’ve just shared a short webcast on data reshaping in R using the cdata package. (link) We also have two really nifty articles on the theory and methods: Fluid data reshaping with cdata Coordinatized Data: A Fluid Data Specification Please give it a try! This is the material I recently presented at the January 2017 … Continue reading Data Reshaping with cdata
Mazama Science has released the first official version (1.0) of the PWFSLSmoke R package for working with PM2.5 monitoring data. A beta version was released last year, along with an accompanying blog post. In this post, we discuss the purpose …   read more ...
I have created a new data package for R to help users obtain national population statistics of Mexico.
The conapomx package contains the mxpopulation data set, which is a tidy data.frame containing population estimates from the CONAPO (National Population Commission) official agency. The estimates are divided by age groups, gender, municipality and year.
To install, just call CRAN (or github if you are reading this before it is accepted).
install.packages(&quot;conapomx&quot;) # or... library(devtools)
install_github(&quot;eflores89/conapomx&quot;)
# explore the dataset
head(mxpopulation)
y
st
id_st
mun
id_mun
geoid
gender
age
population
2010
Aguascalientes
1
Aguascalientes
1
1001
0
0-14
124263.71
2010
Aguascalientes
1
Aguascalientes
1
1001
0
15-29
106695.14
2010
Aguascalientes
1
Aguascalientes
1
1001
0
30-44
81088.65
2010
Aguascalientes
1
Aguascalientes
1
1001
0
45-64
60379.29
2010
Aguascalientes
1
Aguascalientes
1
1001
0
65+
17679.26
2010
Aguascalientes
1
Aguascalientes
1
1001
1
0-14
119535.77
Here is the data for 2018, just out of curiosity:
library(dplyr)
mxpopulation %__% filter(y == &quot;2018&quot;) %__%
group_by(st) %__% summarize(&quot;Population&quot; = sum(population))
st
Population
Aguascalientes
1337792.5
Baja California
3633772.2
Baja California Sur
832827.2
Campeche
948459.3
Chiapas
5445232.7
Chihuahua
3816865.4
Coahuila
3063662.5
Colima
759686.5
Distrito Federal
8788140.6
Durango
1815965.6
Guanajuato
5952086.5
Guerrero
3625040.0
Hidalgo
2980532.2
Jalisco
8197483.1
Mexico
17604619.1
Michoacan
4687210.5
Morelos
1987595.8
Nayarit
1290518.8
Nuevo Leon
5300618.6
Oaxaca
4084674.0
Puebla
6371380.8
Queretaro
2091823.2
Quintana Roo
1709478.7
San Luis Potosi
2824976.0
Sinaloa
3059321.7
Sonora
3050472.7
Tabasco
2454294.5
Tamaulipas
3661161.7
Tlaxcala
1330142.6
Veracruz
8220321.9
Yucatan
2199617.6
Zacatecas
1612014.2
Hopefully this helps to avoid the notoriously bad datos.gob webpage! Happy data wrangling!
The new package simmer.bricks has found its way to CRAN. The simmer package provides a rich and flexible API to build discrete-event simulations. However, there are certain recurring patterns that are typed over and over again, higher-level tasks which can be conceptualised in concrete activity sequences. This new package is intended to capture this kind of patterns into usable bricks,… Continuar leyendo simmer.bricks 0.1.0: new add-on for simmer →
Bob Rudis (@hrbrmstr) is a famed expert, author and developer in Data Security and the Chief Security Data Scientist at Rapid7. Bob also creates the most deliciously vivid images of his meals documented by the #nom hashtag. I’m going to use a similar method used in my previous projects (Hipster Veggies &amp; Machine Learning Flashcards) to wrangle all those images into a nice collection - mostly for me to look at for inspiration in recipe planning.
Yum! Have you ever thought about collecting all these recipes &amp; images into a cookbook?!— Jasmine Dumas (@jasdumas) January 15, 2018
Source Repository: jasdumas/bobs-noms
Analysis
library(rtweet) # devtools::install_github(&quot;mkearney/rtweet&quot;)
library(tidyverse)
library(dplyr)
library(stringr)
library(magick)
library(knitr)
library(kableExtra)
# get all of bob's recent tweets
bobs_tweets
Am update of RcppMsgPack got onto CRAN today. It contains a number of enhancements Travers had been working on, as well as one thing CRAN asked us to do in making a suggested package optional.
MessagePack itself is an efficient binary serialization f...
The SparklyR package from RStudio provides a high-level interface to Spark from R. This means you can create R objects that point to data frames stored in the Spark cluster and apply some familiar R paradigms (like dplyr) to the data, all the while leveraging Spark's distributed architecture without having to worry about memory limitations in R. You can also access the distributed machine-learning algorithms included in Spark directly from R functions. If you don't happen to have a cluster of Spark-enabled machines set up in a nearby well-ventilated closet, you can easily set one up in your favorite cloud...
I have created a new data package for R to help users obtain national population statistics of Mexico.
The conapomx package contains the mxpopulation data set, which is a tidy data.frame containing population estimates from the CONAPO (National Population Commission) official agency. The estimates are divided by age groups, gender, municipality and year.
To install, just call CRAN (or github if you are reading this before it is accepted).
install.packages(&quot;conapomx&quot;) # or... library(devtools)
install_github(&quot;eflores89/conapomx&quot;)
# explore the dataset
head(mxpopulation)
y
st
id_st
mun
id_mun
geoid
gender
age
population
2010
Aguascalientes
1
Aguascalientes
1
1001
0
0-14
124263.71
2010
Aguascalientes
1
Aguascalientes
1
1001
0
15-29
106695.14
2010
Aguascalientes
1
Aguascalientes
1
1001
0
30-44
81088.65
2010
Aguascalientes
1
Aguascalientes
1
1001
0
45-64
60379.29
2010
Aguascalientes
1
Aguascalientes
1
1001
0
65+
17679.26
2010
Aguascalientes
1
Aguascalientes
1
1001
1
0-14
119535.77
Here is the data for 2018, just out of curiosity:
library(dplyr)
mxpopulation %__% filter(y == &quot;2018&quot;) %__%
group_by(st) %__% summarize(&quot;Population&quot; = sum(population))
st
Population
Aguascalientes
1337792.5
Baja California
3633772.2
Baja California Sur
832827.2
Campeche
948459.3
Chiapas
5445232.7
Chihuahua
3816865.4
Coahuila
3063662.5
Colima
759686.5
Distrito Federal
8788140.6
Durango
1815965.6
Guanajuato
5952086.5
Guerrero
3625040.0
Hidalgo
2980532.2
Jalisco
8197483.1
Mexico
17604619.1
Michoacan
4687210.5
Morelos
1987595.8
Nayarit
1290518.8
Nuevo Leon
5300618.6
Oaxaca
4084674.0
Puebla
6371380.8
Queretaro
2091823.2
Quintana Roo
1709478.7
San Luis Potosi
2824976.0
Sinaloa
3059321.7
Sonora
3050472.7
Tabasco
2454294.5
Tamaulipas
3661161.7
Tlaxcala
1330142.6
Veracruz
8220321.9
Yucatan
2199617.6
Zacatecas
1612014.2
Hopefully this helps to avoid the notoriously bad datos.gob webpage! Happy data wrangling!
Problem overview
This article will be divided into a series of posts. In this post, I will explain the basic concepts behind convolutional neural networks and how to build them using Keras. In the next post, I will focus on improving the perfor...
Mark Sellors, Head of Data Engineering
I started working with R around about 5 years ago. Parts of the R world have changed substantially over that time, while other par...
We're very excited to announce a new format for the Enterprise Applications of the R Language Conference (EARL) in the US - a three-stop roadshow! We'll be visiting thr...
NOTE: The likelihood of this recipe being added to the recent practice bookdown book is slim, but I’ll try to keep the same format for the blog post. Problem You want to collect all the tweets in a Twitter tweet thread Solution Use a few key functions in rtweet to piece the thread elements back... Continue reading →
“Base R” (call it “Pure R”, “Good Old R”, just don’t call it “Old R” or late for dinner) can be fast for in-memory tasks. This is despite the commonly repeated claim that: “packages written in C/C++ are faster than R code.” The benchmark results of “rquery: Fast Data Manipulation in R” really called out … Continue reading Base R can be Fast
One of the features of the functions in smooth package is the ability to use exogenous (aka “external”) variables. This potentially leads to the increase in the forecasting accuracy (given that you have a good estimate of the future exogenous variable). For example, in retail this can be a binary variable for promotions and we […]
Owen Jones, Placement Student
Prelude
Maybe you're looking for a change of scene. Maybe you're looking for your first job. Maybe you're stuck in conversation with a rela...
Another small maintenance release, version 0.6.14, of the digest package arrived on CRAN and in Debian today.
digest creates hash digests of arbitrary R objects (using the 'md5', 'sha-1', 'sha-256', 'crc32', 'xxhash' and 'murmurhash' algorithms) perm...
One of the highest pleasures for an academic teacher is to be surprised by an extraordinary student’s project or homework. Something that greatly exceeds expectations. I’ve reoriented my courses in a way to make such surprises frequent. The second project in my Data Visualisation classes was related to interactive graphics. The task was to create … Czytaj dalej Top interactive visualizations of movie scripts
These are my sketchnotes for Sam Charrington’s podcast This Week in Machine Learning and AI about Philosophy of Intelligence with Matthew Crosby:
You can listen to the podcast here.
This week on the podcast we’re featuring a series of conversa...
[crayon-5a5ad6bfdb9ea208679487-i/] is a useful frequentist approach to hierarchical/multilevel linear regression modelling. For good reason, the model output only includes t-values and doesn’t include p-values (partly due to the difficulty in estimating the degrees of freedom, as discussed here). Yes, p-values are evil and we should continue to try and expunge them from our analyses. But I […]
I have recently been working on projects using Amazon EC2 (elastic compute cloud), and RStudio Server. I thought I would share some of my working notes. Amazon EC2 supplies near instant access to on-demand disposable computing in a variety of sizes (billed in hours). RStudio Server supplies an interactive user interface to your remote R … Continue reading Setting up RStudio Server quickly on Amazon EC2
This post will demonstrate how to replicate the volatility ETNs (XIV, VXX, ZIV, VXZ) from CBOE futures, thereby allowing any … Continue reading →
In a recent post, I mentioned three avenues for working with TensorFlow from R:
* The keras package, which uses the Keras API for building scaleable, deep learning models * The tfestimators package, which wraps Google’s Estimators API for fitting models with pre-built estimators
* The tensorflow package, which provides an interface to Google’s low-level TensorFlow API
In this post, Edgar and I use the linear_classifier() function, one of six pre-built models currently in the tfestimators package, to train a linear classifier using data from the titanic package.
library(tfestimators)
library(tensorflow)
library(tidyverse)
library(titanic)
The titanic_train data set contains 12 fields of information on 891 passengers from the Titanic. First, we load the data, split it into training and test sets, and have a look at it.
titanic_set % filter(!is.na(Age))
# Split the data into training and test data sets
indices
by Le Zhang (Data Scientist, Microsoft) and Graham Williams (Director of Data Science, Microsoft) As an in-memory application, R is sometimes thought to be constrained in performance or scalability for enterprise-grade applications. But by deploying R in a high-performance cloud environment, and by leveraging the scale of parallel architectures and dedicated big-data technologies, you can build applications using R that provide the necessary computational efficiency, scale, and cost-effectiveness. We identify four application areas and associated applications and Azure services that you can use to deploy R in enterprise applications. They cover the tasks required to prototype, build, and operationalize an...
Summary: in this post, I implemenent an R function for computing \( P(\theta_1 __ \theta2) \), where \( \theta_1 \) and \( \theta_2 \) are beta-distributed random variables. This is useful for estimating the probability that one binomial proportion is greater than another.
I am working on a project in which I need to compare two binomial proportions to see if one is likely to be greater than the other. I wanted to do this in a Bayesian framework; fortunately, estimating binomial proportions is one of the first subjects taught to introduce Bayesian statistics (and statistics in general, for that matter) so the method is well established. Rasmus Bååth has a very nice article describing the Bayesian binomial test, and an estimation approach using JAGS.
I made the problem even simpler by using the fact that the beta distribution is the conjugate prior to the binomial distribution. That is, if the prior is \( \mathrm{beta}(\alpha, \beta) \) distributed, then the posterior after observing \( k \) successes in \( n \) trials is
If you want to start with a noninformative prior, then you can use the beta distribution with \( \alpha = \beta = 1\), which collapses to the uniform distribution. This makes the posterior
What I am interested is in comparing two binomial proportions, \( \theta_1 \) and \( \theta_2 \). I want to calculate
We observe \( k_1 \) successes out of \( n_1 \) trials and estimate a posterior distribution for \( \theta_1 \), and \( k_2 \) successes out of \( n_2 \) trials for \( \theta_2 \). Then the probability in question is
So far, so textbook. At this point, the simplest way to get an estimate is to draw two sets of random numbers, distributed by the two respective beta distributions, and see how many times the first random numbers is greater than the second:
set.seed(1)
k1
If you've ever wondered how neural networks work behind the scenes, check out this guide to implementing neural networks in scratch with R, by David Selby. You may be surprised how with just a little linear algebra and a few R functions, you can train a function that classifies the red dots from the blue dots in a complex pattern like this: David also includes some elegant R code that implements neural networks using R6 classes. For a similar implementation using base R function, you may also want to check out this guide to implementing neural networks in R by...
“What does the world outside your head really ‘look’ like? Not only is there no color, there’s also no sound: the compression and expansion of air is picked up by the ears, and turned into electrical signals. The brain then presents these signals to us as mellifluous tones and swishes and clatters and jangles. Reality … Continue reading Deep Learning from first principles in Python, R and Octave – Part 2
I use RStudio every day at work. And every day my eyes say thank you to the developers for implementing themes support for the editor, and more recently a complete dark skin for the GUI. Dark themes are not only important for your health and the health of the planet. They are also absolutely essential to look cool in the office and make your colleagues believe you’re working on very complicated projects. 😀 However, one little thing could dispel the...Lire la suite Lire la suite
This is guest post contributed by Matt Dancho, CEO of Business Science. The post was originally published on the Business Science blog.
Customer churn is a problem that all companies need to monitor, especially those that depend on subscript...
The R language surged to 8th place in the 2017 TIOBE language rankings, up 8 places from a year before. Fellow data science language language Python also saw an increase in rankings, taking the 4th spot (one ahead of its January 2016 ranking). (Click the table for the current top 20 rankings.) TIOBE ranks programming languages according to their search engine rankings, and R has been steadily climbing since the rankings began: You can find the current TIOBE language rankings, updated monthly, at the link below. TIOBE: TIOBE Index
Udemy is offering readers of R-bloggers access to its global online learning marketplace for only $10 per course! This deal (offering over 50%-90% discount) is for hundreds of their courses – including many R-Programming, data science, machine learning etc. Click here to browse ALL (R and non-R) courses Advanced R courses:  Regression modelling Comprehensive Linear Modeling with R (15 Hours of video) Linear regression in R for Data Scientists (7 hours) Linear Mixed-Effects Models with R (11 Hours of video) Linear Regression, GLMs and GAMs with R (8 hours) Forecasting Models with R (5 hours) Time Series Analysis and Forecasting in R (3 Hours of video) Regression, Data Mining, Text Mining, Forecasting using R (32 hours) Statistical/machine learning Bayesian Computational Analyses with R (12 Hours of video) Data Science and Machine Learning Bootcamp with R (17 hours) Decision Tree – Theory, Application and Modeling using R (5 hours) Data Mining with R: Go from Beginner to Advanced! (12 hours) Machine Learning A-Z™: Hands-On Python &amp; R In Data Science (40 hours) Applied Multivariate Analysis with R (13 Hours of video)  Visualization Graphs in R (ggplot2, plotrix, base R) – Data Visualization with R Programming Language (5 Hours of video) Multivariate Data Visualization with R (7 Hours of video) Other topics Create Interactive […]
- While I would like to post a success story about the Brazilian post office using R to analyze its internal data, the real story is the use of R’s logo for its Registered mail :). Have a look:
My best guess is that someone...
Overview
We are excited to announce the availability of the cloudml package, which provides an R interface to Google Cloud Machine Learning Engine. CloudML provides a number of services including:
Scalable training of models built with th...
The property market in London SW10 is weak. But planning applications are in the ascendency. Is this how home owners are adding space?
The post SW10 digs deep appeared first on thinkr.
By Gabriel Vasconcelos When dealing with forecasting models there is an issue that generates a lot of confusion, which is the difference between direct and recursive forecasts. I believe most people are more used to recursive forecasts because they are … Continue reading →
These are my sketchnotes taken from the “This week in Machine Learning &amp; AI” podcast number 88 about Using Deep Learning and Google Street View to Estimate Demographics with Timnit Gebru:
Sketchnotes from TWiMLAI talk #88: Using Deep Learning...
LSE IR Gender and Diversity ProjectTwo shinydashboard posts in a row, that’s a first. As I mentioned on Twitter, I’m not really this productive; rather, the apps had been on the proverbial shelf for a while and I’m just releasing them now. In fac...
Let’s admit it. The whole world has been going crazy with Bitcoin. Bitcoin (BTC), the first cryptocurrency (in fact, the first digital currency to solve the double-spend problem) introduced by Satoshi Nakamoto has become bigger than well-established firms (even a few countries). So, a lot of Bitcoin Enthusiasts and Investors are looking to keep a […]
Related Post
Getting started with dplyr in R using Titanic Dataset
How to apply Monte Carlo simulation to forecast Stock prices using Python
Analysing iOS App Store iTunes Reviews in R
Handling ‘Happy’ vs ‘Not Happy’: Better sentiment analysis with sentimentr in R
Creating Reporting Template with Glue in R
With the release of the new version of mlrMBO we added some minor fixes and added a practical feature called Human-in-the-loop MBO.
It enables you to sequentially
visualize the state of the surrogate model,
obtain the suggested parameter configuration for the next iteration and
update the surrogate model with arbitrary evaluations.
In the following we will demonstrate this feature on a simple example.
I have written another blogpost about Looking beyond accuracy to improve trust in machine learning at my company codecentric’s blog:
Traditional machine learning workflows focus heavily on model training and optimization; the best model is usually chosen via performance measures like accuracy or error and we tend to assume that a model is good enough for deployment if it passes certain thresholds of these performance criteria. Why a model makes the predictions it makes, however, is generally neglected. But being able to understand and interpret such models can be immensely important for improving model quality, increasing trust and transparency and for reducing bias. Because complex machine learning models are essentially black boxes and too complicated to understand, we need to use approximations to get a better sense of how they work. One such approach is LIME, which stands for Local Interpretable Model-agnostic Explanations and is a tool that helps understand and explain the decisions made by complex machine learning models.
Continue reading at https://blog.codecentric.de/en/2018/01/look-beyond-accuracy-improve-trust-machine-learning/…
Links to the entire example code and more info are given at the end of the blog post.
The blog post is also available in German.
Before beginning work on a new data science project I like to do the following: 1. Get my work area ready by creating an R Project for use with the RStudio IDE. 2. Organize my work area by creating a series of directories to store my project inputs and outputs. I create ‘data’ (raw data), … Continue reading R Work Areas. Standardize and Automate.
Imagine you were to buy a car, would you just go to a store and buy the first one that you see? No, right? You usually consult few people around you, take their opinion, add your research to it and then go for the final decision. Let’s take a simpler scenario: whenever you go for … Continue reading How to implement Random Forests in R
Do you ever have to recode many values at once? It’s a frequent chore when preparing data. For example, suppose we had to replace state abbreviations with the full names:
abbs
One is an accident. Two is a coincidence. Three is a pattern. Perhaps it’s no coincidence that there are three new interfaces that use Stan’s C++ implementation of adaptive Hamiltonian Monte Carlo (currently an updated version of the no-U-turn sampler). ScalaStan embeds a Stan-like language in Scala. It’s a Scala package largely (if not entirely […]
The post Three new domain-specific (embedded) languages with a Stan backend appeared first on Statistical Modeling, Causal Inference, and Social Science.
Win-Vector LLC recently announced the rquery R package, an operator based query generator. In this note I want to share some exciting and favorable initial rquery benchmark timings. Let’s take a look at rquery’s new “ad hoc” mode (made convenient through wrapr‘s new “wrapr_applicable” feature). This is where rquery works on in-memory data.frame data by … Continue reading rquery: Fast Data Manipulation in R
In case you missed them, here are some articles from December of particular interest to R users. Hadley Wickham's Shiny app for making eggnog. Using R to analyze the vocal range of pop singers. A video tour of the data.table package from its creator, Matt Dowle. The European R Users Meeting (eRum) will be held in Budapest, May 14-18. Winners of the ASA Police Data Challenge student visualization contest. An introduction to seplyr, a re-skinning of the dplyr package to a standard R evaluation interface. How to run R in the Windows Subsystem for Linux, along with the rest of...
In this post we will use Keras to classify duplicated questions from Quora. The dataset first appeared in the Kaggle competition Quora Question Pairs and consists of approximately 400,000 pairs of questions along with a column indicating if th...
If you can walk away from a landing, it's a good landing. If you use the airplane the next day, it's an outstanding landing. -Chuck Yeager As the first airplane invented by Wright brothers in 1903, aviation accident became an inevitable tragedy as well as an attractive topic. In this project, I used the data […]
It is quite likely to get address info when scraping data from the web, but not geo-coordinates which may be required for further analysis like clustering. Thus geocoding is often needed to get a location’s coordinates by its address. There are several options, including one of the most popular, google geocoding API. This option can […]
Related Post
Spark RDDs Vs DataFrames vs SparkSQL – Part 3 : Web Server Log Analysis
Painlessly Merge Data into Actuarial Loss Development Triangles with R
The Mcomp Package
Building a Hacker News scraper with 8 lines of R code using rvest library
Building a Telecom Dictionary scraping web using rvest in R
One of the most powerful things about R is the ecosystem that has emerged around it. In addition to the R language itself and the many packages that extend it, you have a network of users, developers, governance bodies, software vendors and service providers that provide resources in technical information and support, companion applications, training and implementation services, and more. I gave the talk above at the useR! conference last year, but never posted the slides before because they weren't particularly useful without the associated talk track. Mark Sellors from Mango Solutions has thankfully filled the gap with his new...
08.01.2018 I recently discovered on a high dpi display (Retina-Display) enabled 13’’ notebook that the latest release of Bio7 for MacOSX is not displayed in high dpi but in a lower resolution which makes text and icons blurry (scaled) and unnecessary strains the eyes when editing R scripts with the R editor. So I uploaded […]
Knowledge of the way how R evaluates expressions is crucial to avoid hours of staring at the screen or hitting unexpected and difficult bugs.
We’ll start with an example of an issue I came accross a few months ago when using the purrr::map function....
How to internationalize exams2nops() by adding support for new natural languages in written R/exams (that can be automatically scanned and evaluated).
Idea
In addition to completely customizable PDF output, R/exams provides a standardized format called
“NOPS” for written exams with multiple-choice
(and/or single-choice) exercises that can be
automatically generated, scanned, and evaluated.
In order to assure that the automatic scanning works the title page has a fixed layout
that cannot be modified by R/exams users. However, to internationalize the format there
is a possibility for customizing the natural language support. A number of languages
is already available but it is not difficult to add further languages or to tweak
existing languages if desired.
Example
To illustrate how the language support works, once it has been fully incorporated into the
exams package, we set up a short exam with three exercises:
deriv2, tstat2,
swisscapital. All of these are readily available
in the package (and are actually in English).
library(&quot;exams&quot;)
myexam
A new package is now on the ghrr drat. It was uploaded four days ago to CRAN but still lingers in the inspect/ state, along with a growing number of other packages. But as some new packages have come through, I am sure it will get processed eventuall...
This is the considerably belated second part of my blog series on fitting diffusion models (or better, the 4-parameter Wiener model) with brms. The first part discusses how to set up the data and model. This second part is concerned with perhaps the most important steps in each model based data analysis, model diagnostics and […]
I’m starting off 2018 with dropping my involvement with jug. The reason for this being that I simply cannot find the time or interest anymore to maintain the code.
All of the open-source efforts that I started always had to do with a concrete need t...
I’ve always thought that it was ironic that most countries with the word “Democratic” in their name are exceptionally un-democratic in reality. So I was very interested in the following post on Reddit this week showing exactly this (https://www.reddit.com/r/dataisbeautiful/comments/7nkyek/countries_with_the_word_democratic_in_their/). However, there are only 8 countries that have “democratic” in their name: People’s Democratic Republic of […]
I’ve now down a few collages from R using magick: the faces of #rstats Twitter, We R-Ladies with Lucy D’Agostino McGowan, and a holiday card for R-Ladies. The faces of #rstats Twitter and holiday card collages were arranged at random, while the We ...
Data Visualization for Social Science will be published later this year by Princeton University Press. You can read a near-complete draft of the book at socviz.co. If you would like to receive one (1) email when the book is available for pre-order, ple...
A minor maintenance release of the RcppCNPy package arrived on CRAN this week.
RcppCNPy provides R with read and write access to NumPy files thanks to the cnpy library by Carl Rogers.
There is no code change here. But to process the vignette we rely ...
The R package wrapr now has a neat new feature: “wrapr_applicable”. This feature allows objects to declare a surrogate function to stand in for the object in wrapr pipelines. It is a powerful technique and allowed us to quickly implement a convenient new ad hoc query mode for rquery. A small effort in making a … Continue reading New wrapr R pipeline feature: wrapr_applicable
A maintenance release of the tint package arrived on CRAN earlier this week. Its name expands from tint is not tufte as the package offers a fresher take on the Tufte-style for html and pdf presentations.
A screenshot of the pdf variant is below.
...
Classification algorithm defines set of rules to identify a category or group for an observation. There is various classification algorithm available like Logistic Regression, LDA, QDA, Random Forest, SVM etc. Here I am going to discuss Logistic regression, LDA, and QDA. The classification model is evaluated by confusion matrix. This matrix is represented by a […]
Related Post
How to apply Linear Regression in R
Linear Regression in Python; Predict The Bay Area’s Home Prices
Building A Logistic Regression in Python, Step by Step
Multicollinearity in R
Scikit-Learn for Text Analysis of Amazon Fine Food Reviews
I'm pleased to announce that the RQuantLib Windows binaries are now up to 0.4.4!  The RQuantLib pre-built Windows binaries have been frozen on CRAN since 0.4.2, but now you can get version 0.4.4 binaries on Dirk's ghrr drat repo.
Installation is as simple as:
drat::addRepo(&quot;ghrr&quot;) # maybe use 'install.packages(&quot;drat&quot;)' first 
install.packages(&quot;RQuantLib&quot;, type=&quot;binary&quot;)
I will be able to create Windows binaries for future RQuantLib versions too, now that I have a Windows QuantLib build (version 1.11) to link against.
Dirk and I plan to talk with CRAN about getting the new binaries hosted there.  Regardless, they will always be available via the drat repo.
It looks pretty cool! Wednesday, Jan 10 Invited Talk: Predictive information criteria in hierarchical Bayesian models for clustered data. Sophia Rabe-Hesketh and Daniel Furr (U California, Berkely) 10:40-11:30am Does the New York City Police Department rely on quotas? Jonathan Auerbach (Columbia U) 11:30-11:50am Bayesian estimation of mechanical elastic constants. Ben Bales, Brent Goodlet, Tresa Pollock, […]
The post StanCon is next week, Jan 10-12, 2018 appeared first on Statistical Modeling, Causal Inference, and Social Science.
The new year begins with me being on the hook to crank out a book on advanced web-scraping in R by July (more on that in a future blog post). The bookdown🔗 package seemed to be the best way to go about doing this but I had only played with the toy/default examples of it... Continue reading →
In this post, taken from the book R Data Mining by Andrea Cirillo, we’ll be looking at how to scrape PDF files using R. It’s a relatively straightforward way to look at text mining – but it can be challenging if you don’t know exactly what you’re doing. Until January 15th, every single eBook and … Continue reading How to extract data from a PDF file with R
In diesem Artikel möchten wir Ihnen das Konzept der Markov Kette vorstellen, dessen Grundlagen veranschaulichen und Ihnen mehrere mögliche Anwendungsbereiche aufzeigen, in denen Sie mit einer gezielten statistischen Programmierung von Markov Ketten profitieren können. Eine Markov Kette ist ein stochastischer Prozess mit den vielfältigsten Anwendungsbereichen aus der Natur, Technik und Wirtschaft. Klassische Anwendungsmöglichkeiten aus der […]
Der Beitrag Die Markov Kette: Wie berechne ich die Gleichgewichtsverteilung? erschien zuerst auf Statistik Service.
The objective of this post is to create a method which easily combines loss runs, or listings of insurance claims, into triangles. Using only Excel, the common method is to create links between the excel files which must be updated manually for each new evaluation. This is prone to human error and is time-consuming. Using […]
Related Post
The Mcomp Package
Building a Hacker News scraper with 8 lines of R code using rvest library
Building a Telecom Dictionary scraping web using rvest in R
Scraping Javascript-rendered web content using R
Analysing Cryptocurrency Market in R
I have some big news about our R package cdata. We have greatly improved the calling interface and Nina Zumel has just written the definitive introduction to cdata. cdata is our general coordinatized data tool. It is what powers the deep learning performance graph (here demonstrated with R and Keras) that I announced a while … Continue reading Big cdata News
by [Błażej Moska](https://www.linkedin.com/in/b%C5%82a%C5%BCej-moska-9a316113a/), computer science student and data science intern Got stuck with too large a dataset? R speed drives you mad? Divide, parallelize and go with Rcpp! One of the frustrating moments while working with data is when you need results urgently, but your dataset is large enough to make it impossible. This happens often when we need to use algorithm with high computational complexity. I will demonstrate it on the example I've been working with. Suppose we have large dataset consisting of association rules. For some reasons we want to slim it down. Whenever two rules consequents are...
Do you know the Black Metal Cats Twitter account? As explained in this great introduction, it “combines kitties with heavy metal lyrics”. I know the account because I follow Scott Chamberlain who retweets them a lot, which I enjoy as far as one can...
The fifth update of the 3.6.x release of simmer, the Discrete-Event Simulator for R, is on CRAN. This release extends the attributes API by allowing users to set/get multiple attributes at once (a pretty straightforward change as well as useful; I don’t know why it didn’t occurred to me before…). Vectors as attributes and other data… Continuar leyendo simmer 3.6.5 →
“You don’t perceive objects as they are. You perceive them as you are.” “Your interpretation of physical objects has everything to do with the historical trajectory of your brain – and little to do with the objects themselves.” “The brain generates its own reality, even before it receives information coming in from the eyes and … Continue reading Deep Learning from first principles in Python, R and Octave – Part 1
We’re happy to announce the third release of the tibbletime package. This is a huge update, mainly due to a complete rewrite of the package. It contains a ton of new functionality and a number of breaking changes that existing users need to be aware ...
An arithmetic Le Monde mathematical puzzle to conclude 2017: Find (a¹,…,a¹³), a permutation of (1,…,13) such that a¹/a²+a³=a²+a³/a³+a⁴+a⁵=b¹
There are lots of tricks you can use to make R code run faster: use more efficient data structures; vectorize your R code; offload complex data management tasks to databases. Emily Robinson shares many of these R performance tips in a case study on A/B testing for Etsy. The tips are just as valuable as the process Emily shares for evaluating them — and also the process of asking the R community for help. Check out her post, linked below. Hooked on Data: Making R Code Faster : A Case Study
Here are some of the books that I found interesting and useful in 2017. Scrum: The Art of Doing Twice the Work in Half the Time by Jeff Sutherland Jeff Sutherland, one of the creators of the scrum methodology of project management lays down the rational for adopting scrum over more traditional project management frameworks. … Continue reading Helpful Data Science Reads
An interesting (if vaguely formulated) question on X validated: given two Gaussian variates that are maximally coupled, what is the correlation between these variates? The answer depends on the parameters of both Gaussian, with a correlation of one when both Gaussians are identical. Answering the question by simulation (as I could not figure out the […]
Another year, and lots of exciting reading!
Davidson, R. &amp; V. Zinde-Walsh, 2017. Advances in specification testing. Canadian Journal of Economics, online.
Dias, G. F. &amp; G. Kapetanios, 2018. Estimation and forecasting in vector autoregressive moving average models for rich datasets. Journal of Econometrics, 202, 75-91.  
González-Estrada, E. &amp; J. A. Villaseñor, 2017. An R package for testing goodness of fit: goft. Journal of Statistical Computation and Simulation, online.
Hajria, R. B., S. Khardani, &amp; H. Raïssi, 2017. Testing the lag length of vector autoregressive models:  A power comparison between portmanteau and Lagrange multiplier tests. Working Paper 2017-03, Escuela de Negocios y EconomÍa. Pontificia Universidad Católica de ValaparaÍso.
McNown, R., C. Y. Sam, &amp; S. K. Goh, 2018. Bootstrapping the autoregressive distributed lag test for cointegration. Applied Economics, 50, 1509-1521.
Pesaran, M. H. &amp; R. P. Smith, 2017. Posterior means and precisions of the coefficients in linear models with highly collinear regressors. Working Paper BCAM 1707, Birkbeck, University of London.
Yavuz, F. V. &amp; M. D. Ward, 2017. Fostering undergraduate data science. American Statistician, online. 
© 2018, David E. Giles
Makridakis Competitions (also known as the M Competitions or M-Competitions) are a series of competitions organized by teams led by forecasting researcher Spyros Makridakis and intended to evaluate and compare the accuracy of different forecasting methods. So far three competitions have taken place, named as M1 (1982), M2 (1993) and M3 (2000). The fourth competition […]
Related Post
Building a Hacker News scraper with 8 lines of R code using rvest library
Building a Telecom Dictionary scraping web using rvest in R
Scraping Javascript-rendered web content using R
Analysing Cryptocurrency Market in R
Time Series Analysis in R Part 3: Getting Data from Quandl
RStudio's Jenny Bryan (whose recent interviews here and here you should definitely check out) has some excellent advice for improving your workflow for R: Use. Projects. — @JennyBryan at #EARLConf2017 #rstats pic.twitter.com/r4a08JhWHT — David Smith (@revodavid) September 13, 2017 If you're you're routinely using setwd to manually change R's working directory, or using rm(list = ls()) to 'reset' your R session, that's a good sign that you're doing all of your R work in the default R workspace. It's a much better idea to set up a separate project (at its simplest, a separate directory) for each distinct piece of...
02.01.2018 A new version of Bio7 is available. The application Bio7 is an integrated development environment for ecological modeling scientific image analysis and statistical analysis. Over the years Bio7 evolved to a platform useful for a broad scientific context. This release includes a plethora of new features and improvements for the R editor, RShell, RMarkdown […]
Over the year 2017 the [crayon-5a4afb5f2a18f739005075-i/] package has grown from v1.6.0 to v2.3.1. Now it is much more mature and has more downloads. It even now has its own hex (thanks to Fotios Petropoulos): A lot of changes happened in 2017, and it is hard to mention all of them, but the major ones are: […]
To post your R job on the next post Just visit  this link and post a new R job  to the R community. You can post a job for  free  (and there are also “featured job” options available for extra exposure). Current R jobs Job seekers:  please follow the links below to learn more and apply for your R job of interest: Featured Jobs Full-Time Core Data Science @Facebook – PhD Intern (London 2018) Tal Galili London England, United Kingdom 21 Dec 2017 Full-Time R Shiny Dashboard Engineer in Health Tech Castor EDC – Posted by Castor EDC Amsterdam-Zuidoost Noord-Holland, Netherlands 12 Dec 2017 Part-Time Data Scientist Alchemy – Posted by Andreas Voniatis Anywhere 8 Dec 2017 Full-Time Data Analyst: Travel Behavior RSG – Posted by patricia.holland@rsginc.com San Diego California, United States 22 Nov 2017 Full-Time Customer Success Representative RStudio, Inc. – Posted by jclemens1 Anywhere 17 Nov 2017 More New R Jobs Full-Time Core Data Science @Facebook – PhD Intern (London 2018) Tal Galili London England, United Kingdom 21 Dec 2017 Full-Time Post-doc on Sensitivity Analysis for Bayesian Inference via… @ Melbourne, Australia Melbourne University – Posted by dicook Melbourne Victoria, Australia 21 Dec 2017 Full-Time R Shiny Dashboard Engineer in Health Tech Castor EDC – Posted by Castor EDC Amsterdam-Zuidoost Noord-Holland, Netherlands 12 […]
Blogging and social media for introverts - How to spot an introvert
You may have seen David Robinson’s recent post encouraging R users to start blogging. Some folk will willingly act on this advice, and others won’t.
For those that won...
- As we come close to the end of 2017, its time to look back. This has
been a great year for me in many ways. This blog started as a way to
write short pieces about using R for finance and promote my
book in an organic way.
Toda...
Often I use R to handle large datasets, analyze the data and filter out the data I don’t need. When all this is done, I usually use write.csv() to print my data off and reopen it in Google Sheets. My workflow would look something like this: full_data
(You can find all R⁶ posts here) A Twitter discussion: I'm going to keep my eyes out for this one! Would love to have an easy way to embed tweets in Rmd talks!— Jeff Hollister (@jhollist) December 30, 2017 that spawned from Maëlle’s recent look-back post turned into a quick function for capturing an image... Continue reading →
You’ve probably seen people posting their #Best9of2017, primarily on Instagram I’d say. I’m not an Instagram user, although I do have an account to spy on my younger sister and cousins, so I don’t even have 9 Instagram posts in total but I do l...
2017 is nearly at an end. We humans seem to need these cycles to help us on our path forward and have, throughout history, used these annual demarcation points as a time of reflection of what was, what is an what shall come next. To that end, I decided it was about time to help... Continue reading →
Not everyone has the luxury of taking some downtime at the end the year, but if you do have some free time, you may enjoy something on my short list of downtime reading. The books and articles here are not exactly “light reading”, nor are they literature for cuddling by the fire. Nevertheless, you may find something that catches your eye.
The Syncfusion series of free eBooks contains more than a few gems on a variety of programming subjects, including James McCaffrey’s R Programming Succinctly and Barton Poulson’s R Succinctly.
For a more ambitious read, mine the rich vein of SUNY Open Textbooks. My pick is Hiroki Sayama’s Introduction to the Modeling and Analysis of Complex Systems.
If you just can’t get enough of data science, then a few articles that caught my attention are:
Christopher Olah’s brief but mind-stretching post on Neural Networks, Manifolds, and Topology, which is good preparation for the Fujitsu Laboratories paper on Time Series Classification via Topological Data Analysis
The paper by Nguyen and Holmes on their Bayesian Unidimensional Scaling (BUDS) method for detecting patterns in high-dimensional data
Bou-Hamad et. al’s A review of survival trees, a valuable introduction to the literature on the subject
Rob Hyndman’s recent post on Some new time series packages
Mike Bostock’s beautiful and mind-altering post on Visualizing Algorithms.
Finally, if you really have some time on your hands, try searching through the 318M+ papers on PDFDRIVE.
Happy reading, and have a Happy and Prosperous New Year from all of us at RStudio!!
Your first step to becoming a top-performing data scientist is ....
The post The first step to becoming a top performing data scientist appeared first on SHARP SIGHT LABS.
Using tabulizer we’re able to extract information from PDFs so it comes in really handy when people publish data as a PDF! This post takes you through using tabulizer and tidyverse packages to scrape and clean up some budget data from PASS, an association for the Microsoft Data Platform community. The goal is to mainly show some of the tricks of the data wrangling trade that you may need to utilise when you scrape data from PDFs.
Sometimes you just want to add bananas from the commandline. Previously
I created a small script that takes an image and adds a dancing banana to the bottom left of the image. I wanted to make an API too, but that will have to wait till next year....
We are excited to announce the rquery R package. rquery is Win-Vector LLC‘s currently in development big data query tool for R. rquery supplies set of operators inspired by Edgar F. Codd‘s relational algebra (updated to reflect lessons learned from working with R, SQL, and dplyr at big data scale in production). As an example: … Continue reading Announcing rquery
In the riddle of this week on fivethirtyeight, the question is to find the average number of rounds when playing the following game: P=6 players sitting in a circle each have B=3 coins and with probability 3⁻¹ they give one coin to their right or left side neighbour, or dump the coin to the centre. […]
dplyr is one of the most popular r-packages and also part of tidyverse that’s been developed by Hadley Wickham. The mere fact that dplyr package is very famous means, it’s one of the most frequently used. Being a data scientist is not always about creating sophisticated models but Data Analysis (Manipulation) and Data Visualization play […]
Related Post
How to apply Monte Carlo simulation to forecast Stock prices using Python
Analysing iOS App Store iTunes Reviews in R
Handling ‘Happy’ vs ‘Not Happy’: Better sentiment analysis with sentimentr in R
Creating Reporting Template with Glue in R
Predict Employee Turnover With Python
Just too often on X validated, one sees questions displaying a complete ignorance of the basics that makes one purposelessly wonder what is the point of trying to implement advanced methods when missing the necessary background. And just as often, I reacted to the question by wondering out loud about this… In the current case, […]
Everybody is freaking out about the rise of the Bitcoin and the potential of the Blockchain technologies. The advent of cryptocurrencies, game changing use cases, disruption of established business models by disintermediation, etc.. By...
In this third and last post about the Sub-Gaussian property for the Beta distribution [1] (post 1 and post 2), I would like to show the interplay with the Bernoulli distribution as well as some connexions with optimal transport (OT is a hot topic in general, and also on this blog with Pierre’s posts on Wasserstein […]
The Revolutions team is celebrating Christmas today, and we're taking a break with family and enjoying good food. And given the number of Eggnogs that are being prepared — thanks to Hadley Wickham's eggnogr Shiny app — it might be a good idea to take the rest of the week off as well. (You can find the R source behind the eggnogr app here.) We'll be back again in the New Year, on January 2. Enjoy the holiday season, and a happy New Year to all.
Overview: Reinforcement learning uses “reward” signals to determine how to navigate through a system in the most valuable way. (I’m particularly interested in the variant of reinforcement learning called “Q-Learning” because the […]
This example is drawn from “A Painless Q-Learning Tutorial” at http://mnemstudio.org/path-finding-q-learning-tutorial.htm which explains how to manually calculate iterations using the updating equation for Q-Learning, based on the Bellman Equation (image from https://www.is.uni-freiburg.de/ressourcen/business-analytics/13_reinforcementlearning.pdf): The […]
I am excited to share a new deep learning model performance trajectory graph. Here is an example produced based on Keras in R using ggplot2: The ideas include: We plot model performance as a function of training epoch, data set (training and validation), and metric. For legibility we facet on metric, and facets are adjusted … Continue reading Plotting Deep Learning Model Performance Trajectories
Exactly a year ago I posted Kindle clippings.txt with R. Since then things have changed… I’m a Pythonista now! Consequently, I thought it would be fun to update that post and parse highlights with 3.6+ and pandas. Janky, but it works:
import panda...
Now that Twitter allows 280 characters, the code of some drawings I have made can fit in a tweet. In this post I have compiled a few of them. The first one is a cardioid inspired in string art (more info here): This other is based on Fermat’s spiral (more info here): A recurrence plot … Continue reading Tiny Art in Less Than 280 Characters →
Say what you will about 2017 - it sure made an impact. In the broader view I’m pretty sure all sensible people can agree that that impact was predominantly negative, but even so, small pockets of good can happen. So, here’s a short post about all (...
Last week I blogged about the dutchmasters color palettes package, which was inspired by the wonderful ochRe package. As mentioned I shamelessly copied the package. I replaced the list with character vectors containing hex colors and did a find and replace to make it dutchmasters instead of ochRe. This was pretty ugly. I realized that when we would refactor the ochRe functions, thus creating functions that create the functions, there would no longer be a need to copy-paste and find-and-replace. So that is what I did. I refactored and expanded ochRe’s chore into paletti. (Name chosen because I liked the ring of it). You grab it from Github, with devtools::install_github(&quot;edwinth/paletti&quot;).
Sure, this is a promo for a movie, but I'd love to have a full-length single of this: Relatedly, if you want to settle an argument about which pop diva has the greatest vocal range, Giora Simchoni used R to perform frequency analysis of their hits: That's all from us here at the blog for this week, and in fact for a little while: we're taking a break for the holidays. We'll be back on January 2, and in the meantime, enjoy!
It’s been a long time coming, but swatches🔗 is now on CRAN. What is “swatches”? First off, swatches has nothing to do with those faux-luxury brand Swiss-made timepieces. swatches is all about color. R/CRAN has plenty of color picking packages. The colourlovers🔗 📦 by @thosjleeper is one of my favs. But, color palettes have been... Continue reading →
There was a time when Web scraping was quite a difficult task requiring knowledge of XML Tree parsing and HTTP Requests. But with the likes of libraries like beautifulsoup (for Python) and rvest (for R), Web scraping has become a toy for any beginner to play with. This post aims to explain how insanely simple […]
Related Post
Building a Telecom Dictionary scraping web using rvest in R
Scraping Javascript-rendered web content using R
Analysing Cryptocurrency Market in R
Time Series Analysis in R Part 3: Getting Data from Quandl
Pulling Data Out of Census Spreadsheets Using R
Year over year, in the last classes before Christmas I ask my students to create a Christmas tree in R. Classes are about Techniques of data visualisation and usually, at this point, we are discussing interactive graphics and tools like rbokeh, ggiraph, vegalite, googleVis, D3, rCharts or plotly. I like this exercise because with most … Czytaj dalej chRistmas tRees
The official guidelines for the M4 competition have now been published, and there have been several developments since my last post on this.
There is now a prize for prediction interval accuracy using a scaled version of the Mean Interval Score. If th...
A guide through the available drivers and tools when using Amazon Redshift from R and/or RStudio. We examine RPostgreSQL, RPostgres and RJDBC.
This post will be about investigating the efficacy of contango as a volatility trading signal. For those that trade volatility … Continue reading →
The data.table package provides a high-performance interface for querying and manipulating data tables (a close cousin of data frames). In the video below (recorded at the H2OWorld conference), creator Matt Dowle gives a tour of the package and provides several examples of its use. If you'd like to see the details on the example Matt presents at the beginning of the talk, you can find the post how the City of Chicago uses data.table for food safety, here.
On my pretty and up-to-date CV, one of the first things one sees is my Github username, linking to my Github profile. What does a potential employer look at there? Hopefully not my non informative commit messages… My imitating a red Ampelmann, my bei...
A try to automatize the data preparation discretization stage. Converting numerical variables into categorical to be used in real scenarios.
Machine Learning and Regression Machine Learning (ML) is a field of study that provides the capability to a Machine to understand data and to learn from the data. ML is not only about analytics modeling but it is end-to-end modeling that broadly involves following steps: – Defining problem statement – Data collection. – Exploring, Cleaning […]
Related Post
Linear Regression in Python; Predict The Bay Area’s Home Prices
Building A Logistic Regression in Python, Step by Step
Multicollinearity in R
Scikit-Learn for Text Analysis of Amazon Fine Food Reviews
Survival Analysis – Part I
Animating dot plots to demonstrate patient flow - Dots..
This is not the blog post I’d originally intended to write. But I’m glad - because this one is so much better.
Some background. I’m one of the few Scotland based members of Ap...
For some time we have been teaching R users &quot;when working with wide tables on Spark or on databases: narrow to the columns you really want to work with early in your analysis.&quot; The idea behind the advice is: working with fewer columns makes for quicker queries. photo: Jacques Henri Lartigue 1912 The issue arises … Continue reading How to Greatly Speed Up Your Spark Queries
The main international R user conference, useR!, is typically hosted in Europe every other year. Back in 2016 when useR! was held in the USA, an alternative conference was held in Poland. Next year (when the useR! conference will be in Australia) the tradition continues, and the 2018 eRum (European R Users Meeting) will be held in Budapest, Hungary on May 14-16. Around 400-500 attendees are expected, and the program committee is sure to put together an excellent collection of workshops and talks. If you'd like to contribute, workshop proposals are due on January 14, and abstracts for talks and...
Mapping geospatial data using the leaflet package in R, and integrating the resultant interactive map into a WordPress blog post.
The post Surprising stories hide in seemingly mundane data appeared first on thinkr.
Last week, we created a map of world oil production, by country. This week, I’ll show you how to make a slight modification. I’ll show you how to highlight specific countries according to a variable in your data frame. The post How to highlight countries on a map appeared first on SHARP SIGHT LABS.
If you decide you want to use R in your existing Hugo blog, it’s really easy to convert over. There’s a single command you need to know from blogdown and the rest is working out your deployment process.
To create content, use the blogdown Rstudio a...
(For first-timers, R⁶ tagged posts are short &amp; sweet with minimal expository; R⁶ feed) At work-work I mostly deal with medium-to-large-ish data. I often want to poke at new or existing data sets w/o working across billions of rows. I also use Apache Drill for much of my exploratory work. Here’s how to uniformly sample... Continue reading →
Recently, I announced my workshop on Deep Learning with Keras and TensorFlow.
The next dates for it are January 18th and 19th in Solingen, Germany.
You can register now by following this link: https://www.codecentric.de/schulung/deep-learning-mit-keras...
  As a follow-up on my previous post on the sub-Gaussian property for the Beta distribution [1], I’ll give here a visual illustration of the proof. A random variable with finite mean is sub-Gaussian if there is a positive number such that: We focus on X being a Beta random variable. Its moment generating function is known as […]
The winners of the American Statistical Association Police Data Challenge have been announced. The ASA teamed up with the Police Data Initiative, which provides open data from local law enforcement agencies in the US, to create a competition for high school and college students to analyze crime data from Baltimore, Seattle and Cincinnati. In this post, we take a look at the winners of the visualization category. The winners of the Best Visualization for college students were Julia Nguyen, Katherine Qian, Youbeen Shim, Catherine Sun from University of Virginia. Their entry included several visualizations of crime data in Baltimore, including...
Whether „Driving Home for Christmas“, „Winter Wonderland“, „Let it snow!“ or „Last Christmas“ – every year christmas songs are taking over the charts again. While average Joe is joyfully putting on the next christmas song, the data scientist starts his journey of discovery through the snowy music history.   The data set comes from 55000+ Song … „Data Wonderland: Christmas songs from the viewpoint of a data scientist“ weiterlesen
For past few years , I was using featheras my favorite data writing and reading option in R (one reason was its cross platform compatible across Julia, Python and R), however, recently, observed it’s read and write time lines were not at all effective with large files of size __ 5 GB. And found fst format to be good for both read and write of large files in R, only disadvantage is it is cross platform compatible as feather. Especially, fst compression with 100% is good for storage of large file and it is more efficient in reading the same into R environment. Also, I feel no point in re-testing as they are already available at fst site.                                                                     ------ Happy R programming let me know your experiences.
This post aims to bring out some not-so-obvious subtle insights from analyzing TED Videos posted on TED.com. For those who do not know what is TED, Here’s the summary from Wikipedia: TED (Technology, Entertainment, Design) is a media organization which posts talks online for free distribution, under the slogan “ideas worth spreading”. This analysis uses […]
Related Post
Building a simple Sales Revenue Dashboard with R Shiny &amp; ShinyDashboard
Gender Diversity Analysis of Data Science Industry using Kaggle Survey Dataset in R
How Happy is Your Country? — Happy Planet Index Visualized
Exploring, Clustering, and Mapping Toronto’s Crimes
Spring Budget 2017: Circle Visualisation
We are thrilled to announce the dates for next year’s EARL London: 11-13 September 2018 (make sure you pop these dates in your calendars!).
Call for abstracts
Abstract...
Following five R in Insurance conferences, we are organising the first Insurance Data Science conference at Cass Business School London, 16 July 2018.
In 2013, we started with the aim to bring practitioners of industry and academia together to discuss and exchange ideas and needs from both sides.
R was and is a perfect glue between the two groups, a tool which both side embrace and which has fostered the knowledge transfer between the two. However, R is just one example and other languages serve this purpose equally well. Python is another popular language, but also Julia and Stan have gained momentum.
For that reason we have rebranded our conference series to “Insurance Data Science”. We believe by removing the explicit link to “R” we have more freedom to stay relevant and embrace whatever technology may evolve in the future.
We expect contributions to topics related to risk assessment, customer analytics, pricing, reserving, capital management, catastrophe and econometric modelling.
The keynote speakers are:
Prof. Dr. Gareth Peters (Heriot Watt University)
Eric Novik (CEO, Generable)
The conference will be followed by a full-day Stan in insurance workshop with Eric Novik and Paul-Christian Bürkner.
Key dates
1 February 2018: Registration opens
9 April 2018: Abstract submission deadline
31 May 2018: Early-bird registrations deadline
16 July 2018: Insurance Data Science Conference
17 July 2018: Stan in Insurance workshop
For more details visit https://insurancedatascience.org
What is Hierarchical Clustering? Clustering is a technique to club similar data points into one group and separate out dissimilar observations into different groups or clusters. In Hierarchical Clustering, clusters are created such that they have a predetermined ordering i.e. a hierarchy. For example, consider the concept hierarchy of a library. A library has many … Continue reading How to Perform Hierarchical Clustering using R
A simple Le Monde mathematical puzzle after two geometric ones I did not consider: Bob gets a 2×3 card with three integer entries on the first row and two integer entries on the second row such that (i) entry (1,1) is 1, (ii) summing up subsets of adjacent entries produces all integers from 1 to […]
Getting started with notebooks is as easy as clicking a button on Datazar. Notebooks are great tools that make — creating and sharing reproducible analysis — super easy. To create R or Python notebooks, follow these directions:
I.
Navigate to your project, or create a new one if you don’t have one yet.
Click on the “Create File” buton and a box containing several options will popup. Let’s go with “R Notebook” for this one. You can choose “Python Notebook” and the steps will be exactly the same. And just like that, you’ll be redirected to your freshly printed Notebook! First thing you’ll see is the first cell (a code text box). If you created an R Notebook type in message(&quot;hello world!&quot;) and then hit Shift+Enter.
If you’ve created a Python Notebook, type in print &quot;hello world!&quot; and then Shift+Enter. You’ll see the results for the cell displayed right below it.
You’ll see a new cell is created right below the result. On and on; you get the gist.
II.
If you want to load data that’s in your project and access it from your Notebook for your analysis, all you have to do is click on the “Load Files” button and click on the button next to the file you want to upload.
If you’re the using R Notebook, you can then import the dataset with data
Data Manipulation in R is now generally available on Amazon.
All book links will attempt geo-targeting so you end up at the right Amazon. Prices are in USD as most readers are American and the price will be the equivalent in local currency.
Data Mani...
Hello R-users, On November 20th we had the pleasure to host the 9th MilanoR meeting in Mikamai! Lots of you participated with passion and enthusiasm and we’re so glad of the outcome. Thank you all, from the bottom of our hearts! This time the meeting focused on a thrilling and very useful subject: Data Visualization. This […]
The post 9th MilanoR Meeting - Dataviz with R | Photos and presentations! appeared first on MilanoR.
Hello R users! We just launched another course: Working with Dates and Times in R by Charlotte Wickham! Dates and times are abundant in data and essential for answering questions that start with when, how long, or how often. However, they can be tricky, as they come in a variety of formats and can behave … Continue reading New Course: Working with Dates &amp; Times in R
I know some folks had a bit of fun with the previous post since it exposed the fact that I left out unique MQTT client id generation from the initial 0.1.0 release of the in-development package (client ids need to be unique). There have been some serious improvements since said post and I thought a... Continue reading →
In last weeks post we tantalised you with upcoming R &amp; data science conferences, but from a cost point view, not all R conferences are the same. Using the R conference site, it’s fairly easy to compare the cost of previous R conferences.
I select...
Tidy data — it’s one of those terms that tend to confuse people, and certainly confused me. It’s Codd’s third normal form, but you can’t go around telling that to people and expect to be understood. One form is ”long”, the other is ”wide”. One form is ”melted”, another ”cast”. One form is ”gathered”, the […]
The 3rd  paperback edition of both my books on cricket is now available on Amazon for $12.99 a) Cricket analytics with cricketr, Third Edition ($12.99). This book is based on my R package ‘cricketr‘, available on CRAN and uses ESPN Cricinfo Statsguru b) Beaten by sheer pace! Cricket analytics with yorkr, 3rd edition ($12.99). This … Continue reading The 3rd paperback editions of my books on Cricket, now on Amazon
Week 15 Gold Mining and Fantasy Football Projection Roundup now available. Go get that free agent gold! The post Gold-Mining Week 15 (2017) appeared first on Fantasy Football Analytics.
A new version of drat just arrived on CRAN as another no-human-can-delay-this automatic upgrade directly from the CRAN prechecks (though I did need a manual reminder from Uwe to remove a now stale drat repo URL -- bad @hrbrmstr -- from the README in...
Was enjoying Gabriel’s article Pipes in R Tutorial For Beginners and wanted call attention to a few more pipes in R (not all for beginners). data.table has essentially used the square bracket sequence “][” in a manner equivalent to piping in R since about 2006. Here is an example. The Bizarro Pipe “-__.;” has always … Continue reading More Pipes in R
A small maintenance release, version 0.6.13, of the digest package arrived on CRAN and in Debian yesterday.
digest creates hash digests of arbitrary R objects (using the 'md5', 'sha-1', 'sha-256', 'crc32', 'xxhash' and 'murmurhash' algorithms) permit...
I guess I wrote my R “hello world!” function 7 or 8 years ago while approaching R for the first time. And it is too little to illustrate the basic syntax of a programming language for a working program to a wannabe R programmer. Thus, here follows a collection of basic functions that may help … Continue reading “Print hello”​ is not enough. A collection of Hello world functions.
Some (opinionated) themes and highlights from this year’s NIPS conference:
Most of us see the internet through the lens of browsers and apps on our laptops, desktops, watches, TVs and mobile devices. These displays are showing us — for the most part — content designed for human consumption. Sure, apps handle API interactions, but even most of that communication happens over ports 80 or 443.... Continue reading →
Blog Post “Rtus @ Munich Re Datathon”
On the weekend of November 17. - 19. five brave data-knights from team “Rtus and the knights of the data.table” took on the challenge to compete in a datathon organized by Munich Re in its Munich-based innovation lab. Team Rtus was formed in April this year by a bunch of statistics students from LMU with the purpose to prove their data-skills in competitions with other teams from various backgrounds. The datathon was centered around the topic “the effects of climate change and hurricane events on modern reinsurance business” and after two days of very intensive battles with databases, web-crawlers and advanced machine learning modelling (using the famous R-library mlr), they managed to prevail against strong professional competitors and won the best overall price. After victories at the Datafest at Mannheim University and the Telefonica data challenge earlier this year, this was the last step to a hattrick for the core members of team Rtus.
In a previous post I looked at an approach for computing the differences between smooths estimated as part of a factor-smooth interaction using s()’s by argument. When a common-or-garden factor variable is passed to by, gam() estimates a separate smooth for each level of the by factor. Using the (Xp) matrix approach, we previously saw that we can post-process the model to generate estimates for pairwise differences of smooths. However, the by variable approach of estimating a separate smooth for each level of the factor my be quite inefficient in terms of degrees of freedom used by the model. This is especially so in situations where the estimated curves are quite similar but wiggly; why estimate many separate wiggly smooths when one, plus some simple difference smooths, will do the job just as well? In this post I look at an alternative to estimating separate smooths using an ordered factor for the by variable.
A big “thank you!!!” to Microsoft for hosting our new introduction to seplyr. If you are working R and big data I think the seplyr package can be a valuable tool. For how and why, please check out our new introductory article.
by John Mount, Win-Vector LLC [`seplyr`](https://winvector.github.io/seplyr/) is an [`R`](https://www.r-project.org) package that supplies improved standard evaluation interfaces for many common data wrangling tasks. The core of `seplyr` is a re-skinning of [`dplyr`](https://CRAN.R-project.org/package=dplyr)'s functionality to `seplyr` conventions (similar to how [`stringr`](https://CRAN.R-project.org/package=stringr) re-skins the implementing package [`stringi`](https://CRAN.R-project.org/package=stringi)). ## Standard Evaluation and Non-Standard Evaluation &quot;Standard evaluation&quot; is the name we are using for the value oriented calling convention found in many programming languages. The idea is: functions are only allowed to look at the values of their arguments and not how those values arise (i.e., they can not look at source code or variable...
You might have already seen or used the pipe operator when you're working with packages such as dplyr, magrittr,… But do you know where pipes and the famous %__% operator come from, what they exactly are, or how, when and why you should use them? Can you also come up with some alternatives? This tutorial … Continue reading Pipes in R Tutorial For Beginners
A few days ago Journal of Statistical Software has published our article (in collaboration with Marcin Kosiński) archivist: An R Package for Managing, Recording and Restoring Data Analysis Results. Why should you care? Let’s see. Starter Would you want to retrieve a ggplot2 object with the plot on the right? Just call the following line … Czytaj dalej archivist: Boost the reproducibility of your research
This week, here's another map: a map of world oil production that you can create using R.
As I've mentioned, projects like this – a few hundred lines of code, with a variety of functions from the tidyverse – are excellent practice for intermediate data scientists.
The post Mapping oil production by country in R appeared first on SHARP SIGHT LABS.
A boring little RVowpalWabbit package update to version 0.0.10 came in response to another CRAN request: We were switching directories to run tests (or examples) which is now discouraged, so we no longer do this as it turns that we can of course refe...
R has been available for Windows since the very beginning, but if you have a Windows machine and want to use R within a Linux ecosystem, that's easy to do with the new Fall Creator's Update (version 1709). If you need access to the gcc toolchain for building R packages, or simply prefer the bash environment, it's easy to get things up and running. Once you have things set up, you can launch a bash shell and run R at the terminal like you would in any Linux system. And that's because this is a Linux system: the Windows Subsystem...
One of the fast growing economies in the era of globalization is the Ethiopian economy. Among the lower income group countries, it has emerged as one of the rare countries to achieve a double digit growth rate in Grows Domestic Product (GDP). However, there is a great deal of debate regarding the double digit growth […]
Related Post
Outliers Detection and Intervention Analysis
Spark DataFrames: Exploring Chicago Crimes
Image Processing and Manipulation with magick in R
Analyzing the Bible and the Quran using Spark
Predict Customer Churn – Logistic Regression, Decision Tree and Random Forest
Among tulip fields, canals and sampling cheese, the museums of the Netherlands are one of its biggest tourist attractions. And for very good reasons! During the seventeenth century, known as the Dutch Golden Age, there was an abundance of talented painters. If you ever have the chance to visit the Rijksmuseum you will be in awe by the landscapes, households and portraits, painted with incredible detail and beautiful colors.
- In this post I’ll share my experience in setting up my own virtual
server for hosting shiny applications in Digital
Ocean. First, context. I’m working in a
academic project where we build a package for accessing financial ...
Microsoft had a cool API called “Emotion API”. With it you could submit a URL of a video, and the API would return a json file with the faces and emotions expressed in the video (per frame). However, that API never matured from preview mode, and in fact was deprecated on October 30th 2017 (it … Continue reading Using Microsoft’s Azure Face API to analyze videos (in R)
Intro Pipeline concept is definitely not new for software world, Unix pipe operator (|) links two tasks putting the output of one as the input of the other. In machine learning solutions it is pretty much usual to apply several transformation and manipulation to datasets, or to different portions or sample of the same dataset … Continue reading Leveraging pipeline in Spark trough scala and Sparklyr
You can also check this post, written in #blogdown, here: minimal-project-tree-r. Introduction The last two days arrived at my twitter feed some discussions on how bad are the following sentences at the beginning of your R script/notebook, sparked by @JennyBryan’s … Sigue leyendo →
A movie is said to satisfy the Bechdel Test if it satisfies the following three criteria: The movie has at least two named female characters ... who have a conversation with each other ... about something other than a man The website BechdelTest.com scores movies accordingly, granting one point for each of the criteria above for a maximum of three points. The recent Wonder Woman movie scores the full three points (Diana and her mother discuss war, for example), while Dunkirk, which features no named female characters, gets zero. (It was still a great film, however.) The website also offers...
Hi! Big announcement today as we just launched Introduction to the Tidyverse R course by David Robinson! This is an introduction to the programming language R, focused on a powerful set of tools known as the “tidyverse”. In the course you’ll learn the intertwined processes of data manipulation and visualization through the tools dplyr and … Continue reading New R Course: Introduction to the Tidyverse!
If you’re a Big Baller, you know when to stayeth in your lane but also when to switcheth lanes. The Big Baller Brand brothers just inked a deal to play professional basketball in Europe. They switched into a different lane to achieve their original goal. The knee-jerk reaction from the non-globally minded is that this […]
Slides from Münster Data Science Meetup
These are my slides from the Münster Data Science Meetup on December 12th, 2017.
My sketchnotes were collected from these two podcasts:
https://twimlai.com/twiml-talk-7-carlos-guestrin-explaining-predictions-machine-learning-models/
https://dataskeptic.com/blog/episodes/2016/trusting-machine-learning-models-with-lime
Sketchnotes: TWiML Talk #7 with Carlos Guestrin – Explaining the Predictions of Machine Learning Models &amp; Data Skeptic Podcast - Trusting Machine Learning Models with Lime
Example Code
the following libraries were loaded:
library(tidyverse) # for tidy data analysis
library(farff) # for reading arff file
library(missForest) # for imputing missing values
library(dummies) # for creating dummy variables
library(caret) # for modeling
library(lime) # for explaining predictions
Data
The Chronic Kidney Disease dataset was downloaded from UC Irvine’s Machine Learning repository: http://archive.ics.uci.edu/ml/datasets/Chronic_Kidney_Disease
data_file
A fishR user recently asked me
In the book that you published, I frequently use the stock-recruit curve code. The interface that shows both the Ricker/Beverton-Holt figure with the recruit per spawner to spawner figure (i.e., the dynamic ...
Here is a quick example for how to get started with some of the more sophisticated point pattern analysis tools that have been developed for ecologists – principally the adehabitathr package – but that are very useful for human data. Ecologists deploy point pattern analysis to establish the “home range” of a particular animal based […]
The present tutorial analyses the Tennis Grand Slam tournaments main results from the statistical point of view. Specifically, I try to answer the following questions: – How to fit the distribution of the Grand Slam tournaments number of victories across players? – How to compute the probability of having player’s victories greater than a specific […]
Related Post
Oneway ANOVA Explanation and Example in R; Part 2
Oneway ANOVA Explanation and Example in R; Part 1
One-way ANOVA in R
Cubic and Smoothing Splines in R
Chi-Squared Test – The Purpose, The Math, When and How to Implement?
BAGEL SHOP IDEA I was sitting in a bagel shop on Saturday with my 9 year old daughter. We had brought along hexagonal graph paper and a six sided die. We decided that we would choose a hexagon in the middle of the page and then roll the die to determine a direction: 1 up […]
The post Random walking appeared first on Decision Science News.
A quick personal note: today is my first day as a member of the Cloud Developer Advocates team at Microsoft! I'll still be blogging and events related to R, and supporting the R community, but now I'll be doing it as a member of a team dedicated to community outreach. As a bit of background, when I joined Microsoft back in 2015 via the acquisition of Revolution Analytics, I was thrilled to be able to continue my role supporting the R community. Since then, Microsoft as a whole has continue to ramp up its support of open source projects and...
By Gabriel Vasconcelos So I decided to have a quick look at the tuber package to extract YouTube data in R. My cousin is a singer (a hell of a good one) and he has a YouTube channel (dan vasc), … Continue reading →
Statistical models generally assume that All observations are independent from each other The distribution of the residuals follows , irrespective of the values taken by the dependent variable y When any of the two is not observed, more sophisticated modelling approaches are necessary. Let’s consider two hypothetical problems that violate the two respective assumptions, where y … Continue reading Linear mixed-effect models in R →
It’s that time of year when we need to start thinking about what R Conferences we would like to (and can!) attend. To help plan your (ahem) work trips, we thought it would be useful to list the upcoming main attractions.
We maintain a list of upcomin...
I've sent a slew of packages to CRAN recently (thanks to Swetlana and Uwe). There are updates to: caret was primarily updated to deal ...
I am pleased to announce that the R package styler, which I have worked on through Google Summer of Code 2017 with Kirill Müller and Yihui Xie, has reached a mature stage.
Shiny Server 1.5.6.875 and Shiny Server Pro 1.5.6.902 are now available.
This release of Shiny Server Pro includes floating license support and Shiny Server contains a small enhancement to the way errors are displayed. We recommend upgrading at your earliest convenience.
Shiny Server 1.5.6.875
Use HTTPS for Google Fonts on error page, which resolves insecure content errors on some browser when run behind SSL.
(PR #322)
Shiny Server Pro 1.5.6.902
This release adds floating license support through the license_type configuration directive.
Full documentation can be found at http://docs.rstudio.com/shiny-server/#floating-licenses.
Floating licensing allows you to run fully licensed copies of Shiny Server Pro easily in ephemeral instances, such as Docker containers, virtual machines, and EC2 instances. Instances don’t have to be individually licensed, and you don’t have to manually activate and deactivate a license in each instance. Instead, a lightweight license server distributes temporary licenses (“leases”) to each instance, and the instance uses the license only while it’s running.
This model is perfect for environments in which Shiny Server Pro instances are frequently created and destroyed on demand, and only requires that you purchase a license for the maximum number of concurrent instances you want to run.
I was invited to an exciting ecology &amp; R hackathon in my capacity as a co-editor for rOpenSci onboarding system of packages. It also worked well geographically since this hackathon was to take place in Ghent (Belgium) which is not too far away from...
    Hi all, This post deals with a strange phenomenon in R that I have noticed while working on unbiased MCMC. Reducing the problem to a simple form, consider the following code, which iteratively samples a vector ‘x’ and stores it in a row of a large matrix called ‘chain’ (I’ve kept the MCMC […]
Introduction On the 8th, 9th and 10th of December I participated at the IKEA hackaton. In one word it was just FANTASTIC! Well organized, good food, and participants from literally all over the world, even the heavy snow fall on … Continue reading →
Week 14 Gold Mining and Fantasy Football Projection Roundup now available. Go get that free agent gold! The post Gold-Mining Week 14 (2017) appeared first on Fantasy Football Analytics.
BNOSAC is working on building an application on top of open data from questions and answers given at the parliament in Belgium. It will basically show what our civil servants in parliament are busy with. If you are interested in co-developing, feel fre...
More sites are turning to services like Cloudflare due to just how stupid-easy it is to DDoS a site. Sometimes the DDoS is intentional (malicious). Sometimes it’s because your bot didn’t play nice (stop that, btw). Sadly, at some point, most of us with “vital” sites are going to have to pay protection money to... Continue reading →
R is an incredible tool for reproducible research. In the present series of blog posts I want to show how one can easily acquire data within an R session, documenting every step in a fully reproducible way. There are numerous data acquisition options for R users. Of course, I do not attempt to show all the data possibilities and tend to focus mostly on demographic data. If your prime interest lies outside human population statistics, it’s worth checking the amazing Open Data Task View.
It has been a while since my last post due to the major transition of moving back to Canada. This post will be a bit shorter than my previous ones but hopefully it will give some insight on practically investigating and analyzing open data that are becoming more popular these days. All the code for... Continue Reading →
Hi there!
Last Monday we celebrated a “Scientific Marathon” at Royal Botanic Garden in Madrid, a kind of mini-conference to talk about our research. I was talking about the relation between fungal spore size and environmental variables such as temperature and precipitation. To make my presentation more friendly, I created a GIF to explain the Brownian Motion model. In evolutionary biology, we can use this model to simulate the random variation of a continuous trait through time. Under this model, we can notice how closer species tend to maintain closer trait values due to shared evolutionary history. You have a lot of information about Brownian Motion models in evolutionary biology everywhere!
Here I will show you how I built a GIF to explain Brownian Motion in my talk using R and ImageMagick.
# First, we simulate continuous trait evolution by adding in each iteration # a random number from a normal distribution with mean equal to 0 and standard # deviation equal to 1. We simulate a total of 4 processes, to obtain at first # two species and a specieation event at the middle of the simulation, obtaining # a total of 3 species at the end. df1
Recently I wanted to download all the transcripts of a podcast (600+ episodes). The transcripts are simple txt files so in a way
I am not even ‘web’-scraping but just reading in 600 or so text files which is not really a big deal. I thought...
I’ve been writing on this blog less frequently in the past few months. Mostly this is because I’ve been working on a very intensive and engaging professional project that is much more hands-on (ie coding) than I’ve been at work for a while. So ...
It's awesome when you are asked to build a product demo and you end up building something you actually use yourself.
That is what happened to me with the Live Earthquake Shiny App.  A few...
R 3.4.3 (codename “Kite-Eating Tree”) was released last week. You can get the latest binaries version from here. (or the .tar.gz source code from here). As mentioned by David Smith, R 3.4.3 is primarily a bug-fix release: It fixes an issue with incorrect time zones on MacOS High Sierra, and some issues with handling Unicode characters. (Incidentally, representing international and … Continue reading &quot;R 3.4.3 is released (a bug-fix release)&quot;
Hi there! We just launched Working with Web Data in R by Oliver Keyes and Charlotte Wickham, our latest R course! Most of the useful data in the world, from economic data to news content to geographic information, lives somewhere on the internet – and this course will teach you how to access it. You’ll … Continue reading New DataCamp Course: Working with Web Data in R
Yesterday, at the last meeting of the Hamburg R User Group in this year, I had the pleasure to give a talk about Bayesian modelling and choosing (informative) priors in the rstanarm-package. You can download the slides of my talk here. Thanks to the Stan team and Tristan for proof reading my slides prior (
Why this post?
Currently, my team and I are building a Shiny app that serves as an interface for a forecasting model. The app allows business users to interact with predictions. However, we keep getting feature requests, such as, “Can we please have...
In case you missed them, here are some articles from November of particular interest to R users. R 3.4.3 &quot;Kite Eating Tree&quot; has been released. Several approaches for generating a &quot;Secret Santa&quot; list with R. The &quot;RevoScaleR&quot; package from Microsoft R Server has now been ported to Python. The call for papers for the R/Finance 2018 conference in Chicago is now open. Give thanks to the volunteers behind R. Advice for R user groups from the organizer of R-Ladies Chicago. Use containers to build R clusters for parallel workloads in Azure with the doAzureParallel package. A collection of R scripts...
tl;dr many analyses in the social sciences require many R packages jmv makes these common analyses available from one R package jmv can be used from jamovi, a graphical statistical spreadsheet, making it super-accessible
One of the beautiful gifts that R has got (that Python misses) is the package – Shiny. Shiny is an R package that makes it easy to build interactive web apps straight from R. Making Dashboard is an imminent wherever Data is available since Dashboards are good in helping Business make insights out of the […]
Related Post
Gender Diversity Analysis of Data Science Industry using Kaggle Survey Dataset in R
How Happy is Your Country? — Happy Planet Index Visualized
Exploring, Clustering, and Mapping Toronto’s Crimes
Spring Budget 2017: Circle Visualisation
Qualitative Research in R
The British Ecological Society has published a new volume in their Guides to Better Science series: A Guide to Reproducible Code in Ecology and Evolution (pdf). The introduction, by , describes its scope: A Guide to Reproducible Code covers all the basic tools and information you will need to start making your code more reproducible. We focus on R and Python, but many of the tips apply to any programming language. Anna Krystalli introduces some ways to organise files on your computer and to document your workflows. Laura Graham writes about how to make your code more reproducible and readable....
Recently, I was thinking about energy, solar energy, and battery technology, and I got curious about sources of lithium (which is used in batteries and related tech).
Using a few tools from R, I created a map of lithium production within about 15 minutes. Here, I'll show you how I did it, step by step.
The post Mapping lithium production using R appeared first on SHARP SIGHT LABS.
Downloading Annual Financial Reports and Corporate Events from B3 (formerly Bovespa) - Financial statements of companies traded at B3 (formerly Bovespa), the
Brazilian stock exchange, are available in its
website. Accessing the da...
In our last article we pointed out a dangerous silent result corruption we have seen when using the R dplyr package with databases. To systematically avoid this result corruption we suggest breaking up your dplyr::mutate() statements to be dependency-free (not assigning the same value twice, and not using any value in the same mutate it … Continue reading How to Avoid the dplyr Dependency Driven Result Corruption
To post your R job on the next post Just visit  this link and post a new R job  to the R community. You can post a job for  free  (and there are also “featured job” options available for extra exposure). Current R jobs Job seekers:  please follow the links below to learn more and apply for your R job of interest: Featured Jobs Full-Time Quantitative Market Research Analyst Gradient Metrics – Posted by kyblock Anywhere 6 Dec 2017 Full-Time Data Innovation Specialist /Technology Analyst- 20509 jessicaxtan Washington District of Columbia, United States 30 Nov 2017 Full-Time Data Scientist: Mobility Services Parkbob – Posted by ivan.kasanicky Wien Wien, Austria 29 Nov 2017 Full-Time Data Analyst: Travel Behavior RSG – Posted by patricia.holland@rsginc.com San Diego California, United States 22 Nov 2017 Full-Time Customer Success Representative RStudio, Inc. – Posted by jclemens1 Anywhere 17 Nov 2017 Full-Time Data Science Engineer Bonify – Posted by arianna@meritocracy Berlin Berlin, Germany 17 Nov 2017 More New R Jobs Full-Time Quantitative Market Research Analyst Gradient Metrics – Posted by kyblock Anywhere 6 Dec 2017 Full-Time Data Scientist in the Institute for Economics and Peace Institute for Economics and Peace – Posted by Institute for Economics and Peace Sydney New South Wales, Australia 5 Dec 2017 Full-Time Data Innovation Specialist /Technology Analyst- 20509 jessicaxtan Washington […]
I’m in the market for Christmas presents for my true love, @mrshrbrmstr, and thought I’d look to an age-old shopping list for inspiration. Just what would it set me back if I decided to mimic the 12 Days of Christmas in this modern day and age? Let’s try to do the whole thing in R... Continue reading →
We show how to use roxygen2 tags and templates for deprecating existing documented functions.
Introduction
Package roxygen2 provides a powerful way of documenting packages and objects inside them. In particular, great flexibility is available for o...
One of the biggest problems in Business to carry out any analysis is the availability of Data. That is where in many cases, Web Scraping comes very handy in creating that data that’s required. Consider the following case: To perform text analysis on Textual Data collected in a Telecom Company as part of Customer Feedback […]
Related Post
Scraping Javascript-rendered web content using R
Analysing Cryptocurrency Market in R
Time Series Analysis in R Part 3: Getting Data from Quandl
Pulling Data Out of Census Spreadsheets Using R
Extracting Tables from PDFs in R using the Tabulizer Package
Whether we're developing statistical models, training machine learning recognizers, or developing AI systems, we start with data. And while the suitability of that data set is, lamentably, sometimes measured by its size, it's always important to reflect on where those data come from. Data are not neutral: the data we choose to use has profound impacts on the resulting systems we develop. A recent article in Microsoft's AI Blog discusses the inherent biases found in many data sets: “The people who are collecting the datasets decide that, ‘Oh this represents what men and women do, or this represents all human...
Hi all!
A researcher asked me last week about how to compute wind average for an area using rWind. I wrote a simple function to do this using dplyr library (following the advice of my friend Javier Fajardo). The function will be added to rWind package as soon as possible. Meanwhile, you can test the results... enjoy!
# First, charge the new function library(dplyr) wind.region
We’ve released the newest version of NIMBLE on CRAN and on our website a week ago. Version 0.6-8 has a few new features, and more are on the way in the next few months. New features include: the proper Gaussian CAR (conditional autoregressive) model can now be used in BUGS code as dcar_proper, which behaves […]
My book ‘Practical Machine Learning with R and Python – Machine Learning in stereo’ is now available in both paperback ($9.99) and kindle ($6.97/Rs449) versions. In this book I implement some of the most common, but important Machine Learning algorithms in R and equivalent Python code. This is almost like listening to parallel channels of … Continue reading My book ‘Practical Machine Learning with R and Python’ on Amazon
Reassortment is an important strategy for influenza A viruses to
introduce a HA subtype that is new to human populations, which creates
the possibilities of pandemic.
A diagram showed above (Figure 2 of doi:10.1038/srep25549) is widely
used to illustrate the reassortment events. While such diagrams are
mostly manually draw and edit without software tool to automatically
generate. Here, I implemented the hybrid_plot function for producing
publication quality figure of reassortment events.
library(tibble)
library(ggplot2)
n
If you'd like to learn how you use R to develop AI applications, the Microsoft AI School now features a learning path focused on Microsoft R and SQL Server ML Services. This learning path includes eight modules, each comprising detailed tutorials and examples: Doing Data Analysis using R Analyzing Big Data with Microsoft R Server Introduction to SQL Server R Services Microsoft R for SQL Architects Microsoft R for T-SQL Professionals Microsoft R for Data Science Microsoft R with Spark on HDInsight Microsoft R Server for Data Science (with Spark) All of the Microsoft AI School learning paths are free...
This week magick 1.6 appeared on CRAN. This release is a big all-round maintenance update with lots of tweaks and improvements across the package.
The NEWS file gives an overview of changes in this version. In this post we highlight some changes.
library(magick)
stopifnot(packageVersion('magick') __= 1.6)
If you are new to magick, check out the vignette for a quick introduction.
Perfect Graphics Rendering
I have fixed a few small rendering imperfections in the graphics device. The native magick graphics device image_graph() now renders identical or better quality images as the R-base bitmap devices png, jpeg, etc.
One issue was that sometimes magick graphics would show a 1px black border around the image. It turned out this is caused by rounding of clipping coordinates.
When R calculates clipping area it often ends up at non-whole values. It is then up to the graphics device to decide what to do with the pixel that is partially clipped. Let’s show clipping in action:
testplot
Nic Crane, Data Scientist
At Mango, we’re seeing more and more clients making the decision to modernise their analytics process; moving away from SAS and on to R, Pyth...
Plumber is a package which allows you to create web APIs from your R code. If you're new to Plumber, you can find out more at www.rplumber.io.
We're excited to announce the v0.4.4 release of Plumber! This release adds a handful of oft-requested featur...
In my previous tutorial Arima Models and Intervention Analysis we took advantage of the strucchange package to identify and date time series level shifts structural changes. Based on that, we were able to define ARIMA models with improved AIC metrics. Furthermore, the attentive analysis of the ACF/PACF plots highlighted the presence of seasonal patterns. In […]
Related Post
Spark DataFrames: Exploring Chicago Crimes
Image Processing and Manipulation with magick in R
Analyzing the Bible and the Quran using Spark
Predict Customer Churn – Logistic Regression, Decision Tree and Random Forest
Find Your Best Customers with Customer Segmentation in Python
One very important benefit of stronger environmental protection is to reduce the damaging effects of pollution on human health.
In his very interesting article “The Consequences of Industrialization: Evidence from Water Pollution and Digestive Cance...
Towards the end of Part 1 of this short series on confounding, IPW, and (hopefully) marginal structural models, I talked a little bit about the fact that inverse probability weighting (IPW) can provide unbiased estimates of marginal causal effects in the context of confounding just as more traditional regression models like OLS can. I used an example based on a normally distributed outcome. Now, that example wasn’t super interesting, because in the case of a linear model with homogeneous treatment effects (i.e. no interaction), the marginal causal effect is the same as the conditional effect (that is, conditional on the confounders.) There was no real reason to use IPW in that example - I just wanted to illustrate that the estimates looked reasonable.
But in many cases, the conditional effect is different from the marginal effect. (And in other cases, there may not even be an obvious way to estimate the conditional effect - that will be the topic for the last post in this series). When the outcome is binary, the notion that conditional effects are equal to marginal effects is no longer the case. (I’ve touched on this before.) What this means, is that we can recover the true conditional effects using logistic regression, but we cannot estimate the marginal effect. This is directly related to the fact that logistic regression is linear on the logit (or log-odds) scale, not on the probability scale. The issue here is collapsibility, or rather, non-collapsibility.
A simulation
Because binary outcomes are less amenable to visual illustration, I am going to stick with model estimation to see how this plays out:
library(simstudy)
# define the data
defB
On 23 November Glenn Meyers gave a fascinating talk about The Bayesian Revolution in Stochastic Loss Reserving at the 10th Bayesian Mixer Meetup in London.
Glenn Meyers speaking at the Bayesian Mixer
Glenn worked for many years as a research actuary at Verisk/ ISO, he helped to set up the CAS Loss Reserve Database and published a monograph on Stochastic loss reserving using Bayesian MCMC models.
In this blog post I will go through the Correlated Log-normal Chain-Ladder Model from his presentation. It is discussed in more detailed in his monograph.
Glenn kindly shared his code as well, which I have used as a basis for this post.
Getting the data
The CAS Loss Reserve Database is an excellent data source to test reserving models. It is hosted on the CAS website and contains historical regulatory filings of US insurance companies.
The following code allows me to download the data and extract the information for one company, here company 353, which was the example company Glenn used as well.
library(data.table)
CASdata
Predictive modeling is fun. With random forest, xgboost, lightgbm and other elastic models… Problems start when someone is asking how predictions are calculated. Well, some black boxes are hard to explain. And this is why we need good explainers. In the June Aleksandra Paluszynska defended her master thesis Structure mining and knowledge extraction from random … Czytaj dalej Explain! Explain! Explain!
Week 13 Gold Mining and Fantasy Football Projection Roundup now available. Go get that free agent gold!
The post Gold-Mining – Week 13 (2017) appeared first on Fantasy Football Analytics.
A note to dplyr with database users: you may benefit from inspecting/re-factoring your code to eliminate value re-use inside dplyr::mutate() statements. If you are using the R dplyr package with a database or with Apache Spark: I respectfully advise you inspect your code to ensure you are not using any values created inside a dplyr::mutate() … Continue reading Please inspect your dplyr+database code
We’re pleased to announce version 1.5.10 of RStudio Connect and the general availability of RStudio Connect Execution Servers. Execution Servers enable horizontal scaling and high availability for all the content you develop in R. The 1.5.10...
Over the past few weeks, I had been noticing that some posts in the R-bloggers feed were getting truncated in Feedly. I don’t remember when I noticed that since I usually click through immediately from the headline entry to the R-bloggers page vs read in Feedly since ultimately I want to get to the author’s... Continue reading →
Last month the Australian people signaled their approval of legalizing same-sex marriage by a 62%:38% margin in a national survey. (On a personal note, I was elated and relieved by the result: my husband and I have discussed eventually retiring to Australia, and with this decision our marriage would be recognized there.) While fears of a surprise Brexit-like electoral backlash proved unfounded, researchers including R user Miles McBain explored the results for correlations to demographic variables. This process wasn't as simple as it might have been though: the Australian Bureau of Statistics released the results as a pair of Excel...
Can’t be bothered reading, tell me now
Getting started
Setting up R
Opening ports ready for RStudio
Installing RStudio
Nicer URLs
Adding SSL
Can’t be bothered reading, tell me now
Host RStudio server on an azure instance. Configure the instan...
Yesterday saw the public reboot of Binder / MyBinder (which I first wrote about a couple of years ago here), as reported in The Jupyter project blog post Binder 2.0, a Tech Guide and this practical guide: Introducing Binder 2.0 — share your interactive research environment. For anyone not familiar with Binder / MyBinder, it’s […]
Unlike Google PlayStore Developer Console for Android App, iOS App Store’s iTunes Connect does not help developers with the bulk download of App Store iTunes Reviews. So if you are an iOS App Developer or a Mobile App Product Manager, You are left with no option but to get paid subscription like App Annie to […]
Related Post
Handling ‘Happy’ vs ‘Not Happy’: Better sentiment analysis with sentimentr in R
Creating Reporting Template with Glue in R
Predict Employee Turnover With Python
Making a Shiny dashboard using ‘highcharter’ – Analyzing Inflation Rates
Time Series Analysis in R Part 2: Time Series Transformations
If you have any machine with an SSH server open to the world and you take a look at your logs, you may be alarmed to see so many login attempts from so many unknown IP addresses. DenyHosts is a pretty neat service for Unix-based systems which works in the background reviewing such logs and appending the… Continuar leyendo Visualising SSH attacks with R →
Recently I was faced with a file compressed in NASDAQ’s ITCH-protocol, as I wasn’t able to find an R-package that parses and loads the file to R for me, I spent (probably) way to much time to write one, so here it is. But you might wonder, what exactly is ITCH and why should I […]
To become a great data scientist, you need to master data visualization. One intermediate topic that you’ll need to learn is how to work with color. Specifically, you’ll need to learn how to manipulate the “fill” color of things like density plots ...
The post A quick introduction to using color in density plots appeared first on SHARP SIGHT LABS.
R 3.4.3 has been released, as announced by the R Core team today. As of this writing, only the source distribution (for those that build R themselves) is available, but binaries for Windows, Mac and Linux should appear on your local CRAN mirror within the next day or so. This is primarily a bug-fix release. It fixes an issue with incorrect time zones on MacOS High Sierra, and some issues with handling Unicode characters. (Incidentally, representing international and special characters is something that R takes great care in handling properly. It's not an easy task: a 2003 essay by Joel...
Lots of different visualizations have been proposed for understanding distributions. In this post, I am going show how to create my current favorite, which is...
By Gabriel Vasconcelos Today, November 29, 2017 was a crazy day in the Bitcoin world and the craziness is still going on as I write this post. The price range was of thousands of Dollars in a few hours. Bitcoins … Continue reading →
A few weeks ago I announced my latest project: Data Science instruction at the Census Bureau. In addition to announcing the project, I also snuck...
The post R Programmers: What is your biggest problem when working with Census data? appeared first on AriLamstein.com.
Several recent blog posts have explored the Secret Santa problem and provided solutions in R. This post provides a roundup of various solutions and how they are implemented in R. If you wanted to set up a &quot;Secret Santa&quot; gift exchange at the office, you could put everyone's name into a hat and have each participant draw a name at random. The problem is that someone might draw their own name, but if that happens you can just reshuffle all the names back into the hat and start the process over. That's essentially what the R code below, from a...
Our family has been reading, listening to and watching “A Christmas Carol” for just abt 30 years now. I got it into my crazy noggin to perform a sentiment analysis on it the other day and tweeted out the results, but a large chunk of the R community is not on Twitter and it would... Continue reading →
Kaggle recently released the dataset of an industry-wide survey that it conducted with 16K respondents. This article aims to understand how the argument of Gender Diversity plays out in Data Science Practice. Disclaimer: Yes, I understand this dataset is not the output of a Randomized Experiment hence cannot be a representative of the entire Data […]
Related Post
How Happy is Your Country? — Happy Planet Index Visualized
Exploring, Clustering, and Mapping Toronto’s Crimes
Spring Budget 2017: Circle Visualisation
Qualitative Research in R
Multi-Dimensional Reduction and Visualisation with t-SNE
Win-Vector LLC is proud to introduce two important new tool families (with documentation) in the 0.5.0 version of seplyr (also now available on CRAN): partition_mutate_se() / partition_mutate_qt(): these are query planners/optimizers that work over dplyr::mutate() assignments. When using big-data systems through R (such as PostgreSQL or Apache Spark) these planners can make your code faster … Continue reading Win-Vector LLC announces new “big data in R” tools
This week I have finished preliminary versions of two new R packages for time series analysis. The first (tscompdata) contains several large collections of time series that have been used in forecasting competitions; the second (tsfeatures) is designed...
Earlier today Rcpp passed 1250 reverse-dependencies on CRAN as another big milestone. The graph is on the left depicts the growth of Rcpp usage (as measured by Depends, Imports and LinkingTo, but excluding Suggests) over time.
Rcpp cleared 300 packa...
This seems easy as it requires just to change x parameter to y in the plot specification. Well, there are some edge cases where R users might get a in trouble!
library(plotly)
packageVersion('plotly')
[1] '4.7.1'
Before you go, let me just expla...
The prize
How do I enter?
FAQ
Today we are happy to announce our Stan contest. Something we feel very strongly at Jumping Rivers is giving back to the community. We have benefited immensely from hard work by numerous people, so when possible, we ...
For Cyber Week only, DataCamp offers the readers of R-bloggers over $150 off for unlimited access to its data science library. That’s over 90 courses and 5,200 exercises of which a large chunk are R-focused, plus access to the mobile app, Practice Challenges, and hands-on Projects. All by expert instructors such as Hadley Wickham (RStudio), Matt … Continue reading Cyber Week Only: Save 50% on DataCamp!
by Siddarth Ramesh, Data Scientist, Microsoft I’m an R programmer. To me, R has been great for data exploration, transformation, statistical modeling, and visualizations. However, there is a huge community of Data Scientists and Analysts who turn to Python for these tasks. Moreover, both R and Python experts exist in most analytics organizations, and it is important for both languages to coexist. Many times, this means that R coders will develop a workflow in R but then must redesign and recode it in Python for their production systems. If the coder is lucky, this is easy, and the R model...
Last night, the MünsteR R user-group had another great meetup:
Karin Groothuis-Oudshoorn, Assistant Professor at the University of Twente, presented her R package mice about Multivariate Imputation by Chained Equations.
It was a very interesting talk and here are my sketchnotes that I took during it:
MICE talk sketchnotes
Here is the link to the paper referenced in my notes: https://www.jstatsoft.org/article/view/v045i03
“The mice package implements a method to deal with missing data. The package creates multiple imputations (replacement values) for multivariate missing data. The method is based on Fully Conditional Specification, where each incomplete variable is imputed by a separate model. The MICE algorithm can impute mixes of continuous, binary, unordered categorical and ordered categorical data. In addition, MICE can impute continuous two-level data, and maintain consistency between imputations by means of passive imputation. Many diagnostic plots are implemented to inspect the quality of the imputations.”&quot; (https://cran.r-project.org/web/packages/mice/README.html)
For more information on the package go to http://stefvanbuuren.github.io/mice/.
Got some data, relating to how students move from one module to another. Rows are student ID, module code, presentation date. The flow is not well-behaved. Students may take multiple modules in one presentation, and modules may be taken in any order (which means there are loops…). My first take on the data was just […]
Win-Vector LLC has been working on porting some significant large scale production systems from SAS to R. From this experience we want to share how to simulate, in R with Apache Spark (via Sparklyr), a nifty SAS feature: the vectorized “block if(){}else{}” structure. When porting code from one language to another you hope the expressive … Continue reading Vectorized Block ifelse in R
Today I will show you a quick hack (OK it took me like 4 hours during my travels today yesterday and today),
on how to add a dancing banana to any picture.
Now, you might be thinking… Really, why would you add a dancing banana to
a picture, but...
A paper by Soheil Kolouri and co-authors was arXived last week about using Wasserstein distance for inference on multivariate Gaussian mixtures. The basic concept is that the parameter is estimated by minimising the p-Wasserstein distance to the empirical distribution, smoothed by a Normal kernel. As the general Wasserstein distance is quite costly to compute, the […]
If you are a data scientist who primarily works with R, chances are you had no formal training in software development. I certainly did not pick up many skills in that direction during my statistics masters. For years my workflow was basically load a dataset and hack away on it. In the best case my R-script came to some kind of conclusion or final data set, but usually it abruptly ended. Complex projects could result in a great number of scripts and data exports. Needless to say, reproducability was typically low and stress could run high when some delivery went wrong.
‘ImageMagick’ is one of the famous open source libraries available for editing and manipulating Images of different types (Raster &amp; Vector Images). magick is an R-package binding to ‘ImageMagick’ for Advanced Image-Processing in R, authored by Jeroen Ooms. magick supports many common image formats like png, jpeg, tiff and manipulations like rotate, scale, crop, trim, […]
Related Post
Analyzing the Bible and the Quran using Spark
Predict Customer Churn – Logistic Regression, Decision Tree and Random Forest
Find Your Best Customers with Customer Segmentation in Python
Principal Component Analysis – Unsupervised Learning
ARIMA models and Intervention Analysis
You can find this post in Blogdown format by clicking hereWhen it comes to analyzing survey data, you have to take into account the stochastic structure of the sample that was selected to obtain the data. Plots and graphics should not be an exception. ...
Earlier this year, I made a package that riffed off of ProPublica’s really neat voting cartograms (maps) for the U.S. House and Senate. You can see one for disaster relief spending in the House and one for the ACA “Skinny Repeal” in the Senate. We can replicate both here with the voteogram package (minus the... Continue reading →
The tenth annual R/Finance conference will be held in Chicago, June 1-2 2018. This is a fantastic conference for anyone working with R in the finance industry, or doing research around R in finance in the academic sector. This community-led, single-track conference always features a program of interesting talks in a convival atmosphere. If you'd like to contribute a 20-minute presentation or a 5-minute lightning talk, the call for papers is open though February 2, 2018. If you'd like to attend, registrations will open in March 2018. Microsoft is proud to be a continuing sponsor of R/Finance 2018. You can...
If you have ever spent any time investigating where the cool kids play in the data visualization world you will know it is all about...
The first edition of Polish R Users Conferences called Why R? took place on 27-29 September
at Warsaw University of Technology - Faculty of Mathematics and Information Science. The Polish R community is very strong. Dig in into the post to find out wh...
FOAAS upstream is still at release 1.1.0, but added a few new accessors a couple of months ago. So this new version of rfoaas updates to these: asshole(), cup(), fyyff(), immensity(), programmer(), rtfm(), thinking(). We also added test coverage and...
The pile of papers, books and data-sets, relating to previously released draft chapters of my Empirical software engineering book, has been growing, and cluttering up my mind. I decided to have a clear-out. A couple of things stood out. There are around 25 data-sets that have been promised but not yet arrived. If you encounter […]
Stan is probably the most interesting development in computational statistics in the last few years, at least for me. The version of Hamiltonian Monte-Carlo (HMC) implemented in Stan (NUTS, ) is extremely efficient and the range of probability distributions implemented in the Stan language allows to fit an extremely wide range of models. Stan has […]
Week 12 Gold Mining and Fantasy Football Projection Roundup now available. Go get that free agent gold!
The post Gold-Mining – Week 12 (2017) appeared first on Fantasy Football Analytics.
I love emoji ❤️ and I love xkcd, so this recent comic from Randall Munroe was quite a delight for me.
I sat there, enjoying the thought of these new sports like horse hole and multiplayer avocado and I thought, “I can make more of these in just the barest handful of lines of code”. This is largely thanks to the emo package by Hadley Wickham, which if you haven’t installed and started using yet, WHY NOT??? 😱
emoji_sports
The fourteenth release in the 0.12.* series of Rcpp landed on CRAN yesterday after a somewhat longer-than-usual gestation period (and word is it may have been due to some unrelated disturbances from lots of changes within the main r-devel build).
Thi...
Sentiment Analysis is one of the most obvious things Data Analysts with unlabelled Text data (with no score or no rating) end up doing in an attempt to extract some insights out of it and the same Sentiment analysis is also one of the potential research areas for any NLP (Natural Language Processing) enthusiasts. For […]
Related Post
Creating Reporting Template with Glue in R
Predict Employee Turnover With Python
Making a Shiny dashboard using ‘highcharter’ – Analyzing Inflation Rates
Time Series Analysis in R Part 2: Time Series Transformations
Time Series Analysis in R Part 1: The Time Series Object
If you wish to move forward from being a R user to become a R developer, it is time to take your programming skills to the next level. This course will give you an inner perspective of R working mechanisms, as well as tools for addressing your code's issues and to make it more efficient. Once these concepts are established, as whether you work alone or in team it is very important that your code is reproducibile, you will learn how to create R packages and use them as the fundamental unit of reproducible R code.
The post R live class | Professional R Programming | Nov 29-30 Milan appeared first on Quantide - R training &amp; consulting.
Now we’ve recovered from over indulging in Boston’s culinary delights, we’re ready to share our highlights from this year’s EARL Boston Conference. Day 1 highli...
Introduction Market Basket Analysis or association rules mining can be a very useful technique to gain insights in transactional data sets, and it can be useful for product recommendation. The classical example is data in a supermarket. For each customer … Continue reading →
Today is Thanksgiving Day here in the US, so we're taking the rest of the week off to enjoy the time with family. Even if you don't celebrate Thanksgiving, today is still an excellent day to give thanks to the volunteers who have contributed to the R project and its ecosystem. In particular, give thanks to the R Core Group, whose tireless dedication — in several cases over a period of more than 20 years — was and remains critical to the success and societal contributions of the language we use and love: R. You can contribute financially by becoming...
by Angela Li, founder and organizer of R-Ladies Chicago. This article also appears on Angela's personal blog. It’s been a few months since I launched R-Ladies Chicago, so I thought I’d sit down and write up some things that I’ve learned in the course of organizing this wonderful community. Looking back, there are a few things I wish someone told me at the beginning of the process, which I’ll share here over the course of a few weeks. The hope is that you can use these learnings to organize tech communities in your own area. Chicago #RLadies ❤️ R! Group...
We have been writing a lot on higher-order data transforms lately: Coordinatized Data: A Fluid Data Specification Data Wrangling at Scale Fluid Data Big Data Transforms. What I want to do now is &quot;write a bit more, so I finally feel I have been concise.&quot; The cdata R package supplies general data transform operators. The … Continue reading Arbitrary Data Transforms Using cdata
Here at Sharp Sight, we make a lot of maps. There are a few reasons for this. First, good maps are typically ‘information dense.’ You can get a lot of information at a glance from a good map. They are good visualization tools for finding and communicating insights. Second ...
The post Mapping “world cities” in R appeared first on SHARP SIGHT LABS.
A convoluted counting Le Monde mathematical puzzle: A film theatre has a waiting room and several projection rooms. With four films on display. A first set of 600 spectators enters the waiting room and vote for their favourite film. The most popular film is projected to the spectators who voted for it and the remaining […]
This package lets you analyze the variables of a dataset, to evaluate how is the shape of your data. Consider this the first step for modeling.
Recently, I've been doing a lot of work on the beta version of BCEA (I was after all born in Agrigento $-$ in the picture to the left $-$, which is a Greek city, so a beta version sounds about right...). The new version is only available as a beta-release from our GitHub repository - usual ways to install it are through the devtools package.There aren't very many changes from the current CRAN version, although the one thing I did change is kind of big. In fact, I've embedded the web-app functionalities within the package. So, it is now possible to launch the web-app from the current R session using the new function BCEAweb. This takes as arguments three inputs: a matrix e containing $S$ simulations for the measures of effectiveness computed for the $T$ interventions; a matrix c containing the simulations for the measures of costs; and a data frame or matrix containing simulations for the model parameters. In fact, none of the inputs is required and the user can actually launch an empty web-app, in which the inputs can be uploaded, say, from a spreadsheet (there are in fact other formats available).I think the web-app facility is not necessary when you've gone through the trouble of actually installing the R package and you're obviously using it from R. But it's helpful, nonetheless, for example in terms of producing some standard output (perhaps even more than the actual package $-$ which I think is more flexible) and of reporting, with the cool facility based on pandoc.This means there are a few more packages &quot;suggested&quot; on installation and potentially a longer compilation time for the package $-$ but nothing major. The new version is under testing but I may be able to release it on CRAN soon-ish... And there are other cool things we're playing around (the links here give all the details!).
Kirill Muller and I are excited to announce a 4-day R workshop in Toronto on January 23-26, 2018!
To register or for more information click here
Readers of this blog can receive a 10% discount by using promo code TIDYSHINYTEN (limited-time ...
by JS Tan (Program Manager, Microsoft) The R language is by and far the most popular statistical language, and has seen massive adoption in both academia and industry. In our new data-centric economy, the models and algorithms that data scientists build in R are not just being used for research and experimentation. They are now also being deployed into production environments, and directly into products themselves. However, taking your workload in R and deploying it at production capacity, and at scale, is no trivial matter. Because of R's rich and robust package ecosystem, and the many versions of R, reproducing...
Blogdown entry here.For those guys like me who are not such R geeks, this trick could be of interest. The package dplyr can be very useful when it comes to data manipulation and you can extract valuable information from a data frame. For example, when ...
Introduction This is the final and concluding part of my series on ‘Practical Machine Learning with R and Python’. In this series I included the implementations of the most common Machine Learning algorithms in R and Python. The algorithms implemented were 1. Practical Machine Learning with R and Python – Part 1 In this initial post, … Continue reading Practical Machine Learning with R and Python – Part 6
So you have a knack for all things data, know a dozen different Machine Learning algorithms, are a skillful user of Big Data tools, and have started learning Deep Learning — not much else there is to Data Science, right?
Well, I would submit that kno...
Motivation
Have you ever had the feeling that the creation of your data analysis report(s) resulted in looking up, copy-pasting and reuse of code from previous analyses?
This approach is time consuming and prone to errors. If you frequently analyze ...
In this post, I show how to build an interactive geographic dashboard using Displayr, Plotly and R. It is particularly fascinating in that it tracks...
Introduction Kaggle.com is a website designed for data scientists and data enthusiasts to connect and compete with each other. It is an open community that hosts […]
Introduction Recently, Dataiku 4.1.0 was released, it now offers much more support for R users. But wait a minute, Data-what? I guess some of you do not know Dataiku, so what is Dataiku in the first place? It is a … Continue reading →
You know the drill by now. Grab the tweets. Generate the report using RMarkdown. Push to Github. Publish to RPubs. This time it’s the Australian Bioinformatics &amp; Computational Biology Society Conference 2017, including the COMBINE symposium. Looks like a good time was had by all in Adelaide. A couple of quirks this time around. First, … Continue reading Twitter coverage of the Australian Bioinformatics &amp; Computational Biology Society Conference 2017
Twitter recently doubled the maximum length of a tweet to 280 characters, and while all users now have access to longer tweets, few have taken advantage of the opportunity. Bob Rudis used the rtweet package to analyze tweets sent with the #rstats hashtag since 280-char tweets were introduced, and most still kept below the old 280-character limit. The actual percentage differed by the Twitter client being used; I've reproduced the charts for the top 10 clients below. (Click the chart for even more clients, and details of the analysis including the R code that generated it.) That being said, some...
A bit about estimates of parameters Hi everyone! Today I want to tell you about parameters estimation of smooth functions. But before going into details, there are several things that I want to note. In this post we will discuss bias, efficiency and consistency of estimates of parameters, so I will use phrases like “efficient […]
Following my post on the M4 competition yesterday, Spyros Makridakis sent me these comments for posting here.
I would like to thank Rob, my friend and co-author, for his insightful remarks concerning the upcoming M4 competition. As Rob says, the two of...
After seeing a plot of best places to live in Spain and the USA based on the weather, I had to chime in and do the same thing for the Netherlands. The idea is simple, determine where you want to live based on your temperature preferences.
First ...
A maintenance release RcppClassic 0.9.9 is now at CRAN. This package provides a maintained version of the otherwise deprecated first Rcpp API; no new projects should use it.
Per a request from CRAN, we changed the call to Rcpp::Rcpp.plugin.maker() to...
Customer churn occurs when customers or subscribers stop doing business with a company or service, also known as customer attrition. It is also referred as loss of clients or customers. One industry in which churn rates are particularly useful is the telecommunications industry, because most customers have multiple options from which to choose within a […]
Related Post
Find Your Best Customers with Customer Segmentation in Python
Principal Component Analysis – Unsupervised Learning
ARIMA models and Intervention Analysis
A Gentle Introduction on Market Basket Analysis — Association Rules
Building A Book Recommender System – The Basics, kNN and Matrix Factorization
Introduction Residential real estate prices are fascinating... and frustrating. The homebuyer, the home-seller, the real estate agent, the economist, and the banker are all interested in […]
To post your R job on the next post Just visit  this link and post a new R job  to the R community. You can post a job for  free  (and there are also “featured job” options available for extra exposure). Current R jobs Job seekers:  please follow the links below to learn more and apply for your R job of interest: Featured Jobs Full-Time Customer Success Representative RStudio, Inc. – Posted by jclemens1 Anywhere 17 Nov 2017 Full-Time Data Science Engineer Bonify – Posted by arianna@meritocracy Berlin Berlin, Germany 17 Nov 2017 Freelance Dashboard/Visualization developer in R/Shiny International Finance Corporation – Posted by Oleksiy Anywhere 14 Nov 2017 Full-Time PROGRAMMER/SOFTWARE DEVELOPMENT ENGINEER/COMPUTATIONAL AND MACHINE LEARNING SPECIALIST fchiava Cambridge Massachusetts, United States 10 Oct 2017   More New R Jobs     Full-Time R&amp;D Database Developer @ Toronto, Canada Crescendo Technology Ltd – Posted by Crescendo Toronto Ontario, Canada 17 Nov 2017 Full-Time Customer Success Representative RStudio, Inc. – Posted by jclemens1 Anywhere 17 Nov 2017 Full-Time Data Science Engineer Bonify – Posted by arianna@meritocracy Berlin Berlin, Germany 17 Nov 2017 Freelance Dashboard/Visualization developer in R/Shiny International Finance Corporation – Posted by Oleksiy Anywhere 14 Nov 2017 Part-Time Development of User-Defined Calculations and Graphical Outputs for WHO’s Influenza Data World Health Organization – Posted by aspenh Anywhere 6 Nov 2017 Full-Time […]
A maintenance release 0.3.3.3.1 of RcppEigen is now on CRAN (and will get to Debian soon). It brings Eigen 3.3.* to R.
The impetus was a request from CRAN to change the call to Rcpp::Rcpp.plugin.maker() to only use :: as the function has in fact been...
I am writing a Rmarkdown document with plenty of tables, and I want them in a decent format, e.g. kable. However I don't want to format them one by one. For example, I have created the following data frame in dplyrdata2 %__% group_by(uf) %__% sum...
As most bloggers, I do check my analytics stats on a regular basis. What I do not really look at is social shares. For most blog posts traffic follows a clear pattern; 2 days of increased traffic, followed by a steady decrease to the base traffic volum...
The “M” competitions organized by Spyros Makridakis have had an enormous influence on the field of forecasting. They focused attention on what models produced good forecasts, rather than on the mathematical properties of those models. For that, Spyros deserves congratulations for changing the landscape of forecasting research through this series of competitions.
Makridakis &amp; Hibon, (JRSSA 1979) was the first serious attempt at a large empirical evaluation of forecast methods.
Seasonality of plague deaths
In the course of my ramblings through history, I recently came across Mark R. Welford and Brian H. Bossak Validation of Inverse Seasonal Peak Mortality in Medieval Plagues, Including the Black Death, in Comparison to Moder...
A long time ago, in a github repo far, far away there lived a tiny package that made it possible to create equal area, square U.S. state cartograms in R dubbed statebins🔗. Three years have come and gone and — truth be told — I’ve never been happy with that package. It never felt “right”... Continue reading →
I have just released some simple RStudio add-ins that are great for creating keyboard shortcuts when working with pipes in R.
You can install the add-ins from here (which also includes both installation instructions and use instructions/examples).
Connect();, the annual Microsoft developer conference, is wrapping up now in New York. The conference was the venue for a number of major announcements and talks. Here are some highlights related to data science, machine learning, and artificial intelligence: There have been several updates to Azure Machine Learning, including the ability to deploy models to the Azure IoT Edge using Docker containers for intelligent Internet-of-Things applications. Microsoft has joined the MariaDB Foundation, and now offers fully-managed MariaDB databases in Azure. (MySQL and MariaDB creator Monty Widenius made the announcement from Finland.) You can access MariaDB databases on Azure as you...
Network analysis presentation from the 7th Scottish QGIS user group. Featuring Ordnance Survey open data, QGIS, GRASS and R.
Week 11 Gold Mining and Fantasy Football Projection Roundup now available. Go get that free agent gold!
The post Gold-Mining – Week 11 (2017) appeared first on Fantasy Football Analytics.
I am happy to share that the latest version of padr just hit CRAN. This new version comprises bug fixes, performance improvements and new functions for formatting datetime variables. But above all, it introduces the custom paradigm that enables you to do asymmetric analysis.
Attendees at the May 2017 rOpenSci unconference. Photo credit: Nistara Randhawa
In May 2017, I helped run a wildly successful “unconference” that had a huge positive impact on the community I serve. rOpenSci is a non-profit in...
Among the many interesting talks I saw a the Domino Data Science Pop-Up in Chicago earlier this week was the presentation by Gene Lynes and Nick Lucius from the City of Chicago. The City of Chicago Tech Plan encourages smart communities and open government, and as part of that initiative the city has undertaken dozens of open-source, open-data projects in areas such as food safety inspections, preventing the spread of West Nile virus, and keeping sidewalks clear of snow. This talk was on the Clear Water initiative, a project to monitor the water quality of Chicago's many public beaches on...
I was fascinated by this xkcd comic about where to live based on your temperature preferences. I also thought it’d be fun to try to make a similar one from my R session! Since I’m no meteorologist and was a bit unsure of how to define winter and su...
IBM has a new set of corporate typefaces — dubbed “Plex” — and has released them with a generous open license. IBM Plex Sans is not too shabby: (that image was grifted from a Font Squirrel preview page) The digit glyphs are especially nice for charts and the font iself is just different enough from... Continue reading →
Udemy is offering readers of R-bloggers access to its global online learning marketplace for only $10 per course! This deal (offering over 50%-90% discount) is for hundreds of their courses – including many R-Programming, data science, machine learning etc. Click here to browse ALL (R and non-R) courses Advanced R courses:  Regression modelling Comprehensive Linear Modeling with R (15 Hours of video) Linear regression in R for Data Scientists (7 hours) Linear Mixed-Effects Models with R (11 Hours of video) Linear Regression, GLMs and GAMs with R (8 hours) Forecasting Models with R (5 hours) Time Series Analysis and Forecasting in R (3 Hours of video) Regression, Data Mining, Text Mining, Forecasting using R (32 hours) Statistical/machine learning Bayesian Computational Analyses with R (12 Hours of video) Data Science and Machine Learning Bootcamp with R (17 hours) Decision Tree – Theory, Application and Modeling using R (5 hours) Data Mining with R: Go from Beginner to Advanced! (12 hours) Machine Learning A-Z™: Hands-On Python &amp; R In Data Science (40 hours) Applied Multivariate Analysis with R (13 Hours of video)  Visualization Graphs in R (ggplot2, plotrix, base R) – Data Visualization with R Programming Language (5 Hours of video) Multivariate Data Visualization with R (7 Hours of video) Other topics Create Interactive […]
Last weekend, I discovered the pixel art. The goal is to reproduce a pixelated drawing. Anyone can do this without any drawing skills because you just have to reproduce the pixels one by one (on a squared paper). Kids and big kids can quickly become addicted to this. Example For this pixelated ironman, you need only 3 colors (black, yellow and red). At the beginning I thought this would be really easy and quick. It took me approximately 15 minutes to reproduce this. Children could take more than 1 hour to reproduce this, so it’s nice if you want to keep them busy. Make your own pixel art models On the internet, there are lots of models. There are also tutos on how to make models with Photoshop. Yet, I wanted to make an R package for making pixel art models, based on any pictures. The pipeline I came up with is the following: read an image with package magick downsize this image for processing use K-means to project colors in a small set of colors downsize the image and project colors plot the pixels and add lines to separate them I think there may be a lot to improve but from what I currently know about images, it’s the best I could come up with as a first shot. I made a package called pixelart, with an associated Shiny App. # Installation devtools::install_github(&quot;privefl/pixelart&quot;)
From Stephen Wolfram, a nice phrase to describe the sorts of thing you can create using tools like Jupyter notebooks, Rmd and Mathematica notebooks: computational essays that complements the “computational narrative” phrase that is also used to describe such documents. Wolfram’s recent blog post What Is a Computational Essay?, part essay, part computational essay,  is primarily […]
Just wrote a new R article: “Data Wrangling at Scale” (using Dirk Eddelbuettel’s tint template). Please check it out.
Can’t be bothered reading, tell me now A simple one-line tweak can significantly speed up package installation and updates. See my post at the Jumping Rivers blog (no point duplicating content in two places)  
The ggmap package allows you to download and plot basic maps from Google maps ...
The post How to plot basic maps with ggmap appeared first on SHARP SIGHT LABS.
The R language provides many different tools for creating maps and adding data to them. I’ve been using the leaflet package at work recently, so I thought I’d provide a short example here. Whilst searching for some data that might make a nice map, I came across this article at ABC News. It includes a … Continue reading Mapping data using R and leaflet
As a useful note to self, I paste here an easy example on the use of the pathview package by Weijun Luo to plot the log fold change of gene expression across a given KEGG pathway. This example is on Vitis vinifera (as the prefix vvi can suggest), but the approach is general. Basically, you […]
MRAN, the Microsoft R Application Network has been migrated to a new high-performance, high-availability server, and we've taken the opportunity to make a few upgrades along the way. You shouldn't notice any breaking changes (of course if you do, please let us know), but you should notice faster performance for the MRAN site and for the checkpoint package. (MRAN is also the home of daily archives of CRAN, which checkpoint relies on to deliver specific package versions for its reproducibility functions.) Among the improvements you will find: Faster search of R packages, including links to package vignettes and a dependencies...
This post will be about comparing strategies from the paper “Easy Volatility Investing”, along with a demonstration of R’s table.Drawdowns … Continue reading →
I have funding for a new post-doctoral research fellow, on a 2-year contract, to work with me and Professor Kate Smith-Miles on analysing large collections of time series data. We are particularly seeking someone with a PhD in computational statistics ...
The Programme for International Student Assessment (PISA) is a worldwide study of 15-year-old school pupils’ scholastic performance in mathematics, science, and reading. Every three years more than 500 000 pupils from 60+ countries are surveyed along with their parents and school representatives. The study yields in more than 1000 variables concerning performance, attitude and context … Czytaj dalej intsvy: PISA for research and PISA for teaching
There are many R packages that connect to the internet, whether it's to import data (readr), install packages from Github (devtools), connect with cloud services (AzureML), or many other web-connected tasks. There's one R package in particular that provides the underlying connection between R and the Web: curl, by Jeroen Ooms, who is also the new maintainer for R for Windows. (The name comes from curl, a command-line utility and interface library for connecting to web-based services). The curl package provides replacements for the standard url and download.file functions in R with support for encryption, and the package was recently...
Today I am delighted to announce an exciting new collaboration. I will be working with the US Census Bureau as a Data Science Instructor! Over...
The post New Project: Data Science Instruction at the US Census Bureau! appeared first on AriLamstein.com.
This is part two in a series I'm writing on network analysis. The first part is here. In this section I'm going to cover allocating resources, again using the St James' development in Edinburgh as an example. Most excitingly (for me) the end of this post covers the impact of changes in resource allocation.
I wanted to check a simple query: How many times has a particular topic been presented and from how many different presenters. Sounds interesting, tackling the problem should not be a problem, just that the end numbers may vary, since there will be some text analysis included. First of all, some web scraping and getting … Continue reading SQL Saturday statistics – Web Scraping with R and SQL Server
A definitely puzzled participant on X validated, confusing the Normal variate or variable used in the random walk Metropolis-Hastings step with its Normal density… It took some cumulated efforts to point out the distinction. Especially as the originator of the question had a rather strong a priori about his or her background: “I take issue […]
By now, virtually every major media outlet has covered the “280 Apocalypse”™. For those still not “in the know”, Twitter recently moved the tweet character cap to 280 after a “successful” beta test (some of us have different ideas of what “success” looks like). I had been on a hiatus from the platform for a... Continue reading →
Lately I’ve been thinking a lot about the connection between prediction models and the decisions that they influence. There is a lot of theory around this, but communicating how the various pieces all fit together with the folks who will use and be impacted by these decisions can be challenging. One of the important conceptual pieces […]
Make memorable plots with memery. memery is an R package that generates internet memes including superimposed inset graphs and other atypical features, combining the visual impact of an attention-grabbing meme with graphic results of data analysis. Version 0.3.0 of memery is now on CRAN. The latest development version and a package vignette are available on […]
We have just released a major update of the cdata R package to CRAN. If you work with R and data, now is the time to check out the cdata package. Among the changes in the 0.5.* version of cdata package: All coordinatized data or fluid data operations are now in the cdata package (no … Continue reading Update on coordinatized or fluid data
Motivation: More Precise Timestamps
R has excellent facilities for dealing with both dates and datetime objects.
For datetime objects, the POSIXt time type can be mapped to POSIXct and
its representation of fractional seconds since the January 1, 1970 “epoch” as
well as to the broken-out list representation in POSIXlt. Many add-on
packages use these facilities.
POSIXct uses a double to provide 53 bits of resolution. That is generally
good enough for timestamps down to just above a microsecond, and has served
the R community rather well.
But increasingly, time increments are measure in nanoseconds. Other languages uses a (signed)
64-bit integer to represent (integer) nanoseconds since the epoch. A bit over a year I realized
that we have this in R too—by combining the integer64 type in the
bit64 package by Jens Oehlschlaegel with the
CCTZ-based parser and formatter in my
RcppCCTZ package. And thus the nanotime package was created.
Leonardo Silvestri then significantly enhanced
nanotime by redoing it as an S4 class.
A simple example:
library(nanotime)
n
Working remotely has many benefits, but if you work remotely in an area like, say, rural Maine, one of those benefits is not massively speedy internet connections. Being able to go fast and furious on the internet is one of the many things I miss about our time in Seattle and it is unlikely that... Continue reading →
We’re in the heart of the academic season and there’s a lot going on. James Ramsey reported a critical performance regression bug in Stan 2.17 (this affects the latest CmdStan and PyStan, not the latest RStan). Sean Talts and Daniel Lee diagnosed the underlying problem as being with the change from char* to std::string arguments—you […]
The post Stan Roundup, 10 November 2017 appeared first on Statistical Modeling, Causal Inference, and Social Science.
R with Database and Big Data, our fifth autumnal class, presents the latest techniques to work with big data within the R environment. The large amount of data available nowadays is a tangled and hidden source of knowledge: being able to quickly and effectively unravel high value information from the vastness of data is the most powerful driver for success in this modern competitive market. Join us in Milan, 21-22 November.
The post R live class | R with Database and Big Data | Nov 21-22 Milan appeared first on Quantide - R training &amp; consulting.
By Emmanuel Awa, Francesca Lazzeri and Jaya Mathew, data scientists at Microsoft A few of us got to attend EARL conference in Boston last week which brought together a group of talented users of R from academia and industry. The conference highlighted various Enterprise Applications of R. Despite being a small conference, the quality of the talks were great and showcased various innovative ways in using some of the newer packages available for use in the R language. Some of the attendees were veteran R users while some were new comers to the R language, so there was a mix...
Week 10 Gold Mining and Fantasy Football Projection Roundup now available. Go get that free agent gold!
The post Gold-Mining – Week 10 (2017) appeared first on Fantasy Football Analytics.
The Happy Planet Index (HPI) is an index of human well-being and environmental impact that was introduced by NEF, a UK-based economic think tank promoting social, economic and environmental justice. It ranks 140 countries according to “what matters most — sustainable wellbeing for all”. This is how HPI is calculated: It’s tells us “how well nations are […]
Related Post
Exploring, Clustering, and Mapping Toronto’s Crimes
Spring Budget 2017: Circle Visualisation
Qualitative Research in R
Multi-Dimensional Reduction and Visualisation with t-SNE
Comparing Trump and Clinton’s Facebook pages during the US presidential election, 2016
Quentin F. Gronau, Henrik Singmann and Eric-Jan Wagenmakers have arXived a detailed documentation about their bridgesampling R package. (No wonder that researchers from Amsterdam favour bridge sampling!) [The package relates to a [52 pages] tutorial on bridge sampling by Gronau et al. that I will hopefully comment soon.] The bridge sampling methodology for marginal likelihood […]
Modern slot machines (fruit machine, pokies, or whatever those electronic gambling devices are called in your part of the world) are designed to be addictive. They're also usually quite complicated, with a bunch of features that affect the payout of a spin: multiple symbols with different pay scales, wildcards, scatter symbols, free spins, jackpots ... the list goes on. Many machines also let you play multiple combinations at the same time (20 lines, or 80, or even more with just one spin). All of this complexity is designed to make it hard for you, the player, to judge the real...
By Gabriel Vasconcelos Overview When working with time-series forecasting we often have to choose between a few potential models and the best way is to test each model in pseudo-out-of-sample estimations. In other words, we simulate a forecasting situation where … Continue reading →
Report Generation is a very important part in any Organization’s Business Intelligence and Analytics Division. The ability to create automated reports out of the given data is one of the most desirable things, that any innovative team would thrive for. And that is one area where SAS is considered to be more matured than R […]
Related Post
Predict Employee Turnover With Python
Making a Shiny dashboard using ‘highcharter’ – Analyzing Inflation Rates
Time Series Analysis in R Part 2: Time Series Transformations
Time Series Analysis in R Part 1: The Time Series Object
Parsing Text for Emotion Terms: Analysis &amp; Visualization Using R
A shiny new (mostly-but-not-completely maintenance) release of RQuantLib, now at version 0.4.4, arrived on CRAN overnight, and will get to Debian shortly. This is the first release in over a year, and it it contains (mostly) a small number of fixes t...
In case you missed them, here are some articles from October of particular interest to R users. A recent survey of competitors on the Kaggle platform reveals that Python (76%) and R (59%) are the preferred tools for building predictive models. Microsoft's &quot;Team Data Science Process&quot; has been updated with new guidelines on use of the IDEAR framework for R and Python. Microsoft R Open 3.4.2 is now available for Windows, Mac and Linux. Using the foreach package to estimate bias of rpart trees via bootstrapping. Replays of webinars on the Azure Data Science VM, and on document collection analysis...
MilanoR is a free event, open to all R users and enthusiasts or those who wish to learn more about R. The meeting consists of two talks (this time one about big data and and one about statistical learning) + a free buffet and networking time
The post 9th MilanoR Meeting: November 20th appeared first on MilanoR.
I am working on a super-secret project for which I am harvesting a highly confidential source of data: twitter 🙂 The idea is to gather a small amount of twitter data, but for a long time… maybe a year. I tried to use the package TwitteR, but it can only  grab up to a week of tweets… it’s not really… Read more →
R is an incredible tool for reproducible research. In the present series of blog posts I want to show how one can easily acquire data within an R session, documenting every step in a fully reproducible way. There are numerous data acquisition options for R users. Of course, I do not attempt to show all the data possibilities and tend to focus mostly on demographic data. If your prime interest lies outside human population statistics, it’s worth checking the amazing Open Data Task View.
First off, here are the previous posts in my Bayesian sampling series: Bayesian Simple Linear Regression with Gibbs Sampling in R Blocked Gibbs Sampling in R for Bayesian Multiple Linear Regression In the first post, I illustrated Gibbs Sampling - an algorithm for getting draws from a posterior when conditional posteriors are known. In the … Continue reading Metropolis-in-Gibbs Sampling and Runtime Analysis with Profviz →
The fourth update of the 3.6.x release of simmer, the Discrete-Event Simulator for R, is on CRAN. This release patches several bugs regarding resource priority and preemption management when seized amounts greater than one were involved. Check the examples available in the corresponding issues on GitHub (#114, #115, #116) to know if you are affected. It can be… Continuar leyendo simmer 3.6.4 →
A first maintenance release of RcppQuantuccia got to CRAN earlier today.
RcppQuantuccia brings the Quantuccia header-only subset / variant of QuantLib to R. At present it mostly offers calendaring, but Quantuccia just got a decent amount of new funct...
This is the 5th and probably penultimate part of my series on ‘Practical Machine Learning with R and Python’. The earlier parts of this series included 1. Practical Machine Learning with R and Python – Part 1 In this initial post, I touch upon univariate, multivariate, polynomial regression and KNN regression in R and Python 2.Practical … Continue reading Practical Machine Learning with R and Python – Part 5
For me, mathematics cultivates a perpetual state of wonder about the nature of mind, the limits of thoughts, and our place in this vast cosmos (Clifford A. Pickover – The Math Book: From Pythagoras to the 57th Dimension, 250 Milestones in the History of Mathematics) I am a big fan of Clifford Pickover and I … Continue reading Drawing 10 Million Points With ggplot: Clifford Attractors →
Last spring, I posted about my version of a modified age-bias plot. One reader commented on that post via Twitter – “Now that you solved the age-bias plot, how about the ‘best’ display of back-calculated length-at-age data, with Von...
Unlike @noamross, I am not an epidemiologist (NOTE: Noam battles pandemics before breakfast, so be super nice to him) but I do like to find kindred methodologies in other disciplines to help foster the growth of cybersecurity into something beyond it’s current Barnum &amp; Bailey state. I also love finding and exposing hidden APIs and... Continue reading →
rstudio::conf 2018, the conference on all things R and RStudio, is only a few months away. Now is the time to claim your spot or grab one of the few remaining seats at Training Days!
REGISTER NOW
Whether you’re already registered or still ...
While spreadsheets are fine tools for collecting and sharing data, the temptation is often there to also use them for in-depth analysis better suited to reproducible systems like R. Historian Jesse Sadler recently published the useful guide Excel vs R: A Brief Introduction to R, which provides useful advice to data analysts currently using spreadsheets on how to transition to R: Quantitative research often begins with the humble process of counting. Historical documents are never as plentiful as a historian would wish, but counting words, material objects, court cases, etc. can lead to a better understanding of the sources and...
The EARL Boston 2017 conference was held November 1 - 3 in Boston, Mass. There were some excellent presentations illustrating how R is being embraced in enterprises, especially in the financial and pharmaceutical industries. Matt Dancho, founder of Bus...
To post your R job on the next post Just visit  this link and post a new R job  to the R community. You can post a job for  free  (and there are also “featured job” options available for extra exposure). Current R jobs Job seekers:  please follow the links below to learn more and apply for your R job of interest: Featured Jobs Full-Time R programmer Model Answers – Posted by MichaelCheng Brisbane City Queensland, Australia 30 Oct 2017 Freelance Data Visualization Expert Nicole Van Herwaarden/IdeaConnection – Posted by NKVanHerwaarden Anywhere 23 Oct 2017 Freelance The R Foundation is looking for a volunteer to organize R’s contributed documentation R: The R Foundation – Posted by Tal Galili Anywhere 13 Oct 2017 Full-Time PROGRAMMER/SOFTWARE DEVELOPMENT ENGINEER/COMPUTATIONAL AND MACHINE LEARNING SPECIALIST fchiava Cambridge Massachusetts, United States 10 Oct 2017 Full-Time Data Scientist Edelweiss Business Services – Posted by Aakash Gupta Mumbai Maharashtra, India 20 Sep 2017 More New R Jobs Full-Time Data Scientist for H Labs @ Chicago, Illinois, United States Heidrick &amp; Struggles – Posted by Heidrick1 Chicago Illinois, United States 2 Nov 2017 Full-Time Business Data Analytics Faculty Maryville University – Posted by tlorden St. Louis Missouri, United States 2 Nov 2017 Full-Time R programmer Model Answers – Posted by MichaelCheng Brisbane City Queensland, Australia 30 Oct 2017 […]
A maintenance release of our pinp package for snazzier one or two column vignettes is now on CRAN as of yesterday.
In version 0.0.3, we disabled the default \pnasbreak command we inherit from the PNAS LaTeX style. That change turns out to have been t...
My husband and I recently started watching the wonderful series “Parks and recreation” which was recommended to me by my fellow R-Lady Jennifer Thompson in this very convincing thread. The serie was even endorsed by other R-Ladies. Jennifer told me...
A maintenance release of the tint package arrived on CRAN earlier today. Its name expands from tint is not tufte as the package offers a fresher take on the Tufte-style for html and pdf presentations.
A screenshot of the pdf variant is below.
Thi...
Today we’re pleased to announce a new category of community.rstudio.com dedicated to R administrators: https://community.rstudio.com/c/r-admin.
There are already multiple places where you can get help with R, Shiny, the RStudio IDE, and the tidyverse. There are, however, far fewer resources for R admins: people who work with R in production, in large organizations, and in complex environments. We hope this new category will serve as a useful and friendly place to connect with fellow R admins to discuss the issues they deal with. We expect this category to include:
Discussions about best practices and ideas
General questions to fellow admins about RStudio Pro products, designed to ease friction in R administrator workflows
An exchange of ideas on domain-specific use cases and configurations
If you’re an existing RStudio customer, this forum is a complement to RStudio’s direct support:
Folks from RStudio will participate, but only lightly moderate topics and discussions.
RStudio commercial license holders should still feel free to report Pro
product problems to support@rstudio.com.
If you think a topic needs RStudio support’s attention, please suggest that
the poster contact RStudio support directly. You can also tag @support in a reply.
You can also check this post, written in #blogdown, here: taming-exam-results-with-pdf. Introduction There are several ways to mine tables and other content from a pdf, using R. After a lot of trial &amp; error, here’s how I managed to extract … Sigue leyendo →
The RStudio team has created another very useful cheat sheet for R: Working with Strings. This cheat sheet provides an example-laden menu of operations you can perform on strings (character verctors) in R using the stringr package. While base R provides a solid set of string manipulation functions, the stringr package functions are simpler, more consistent (making them easy to use with the pipe operator), and more like the Ruby or Python way of handling string operations. The back page of the cheat sheet also provides a handy guide to regular expressions a useful (if often frustrating) tool for extracting...
Our article &quot;Let’s Have Some Sympathy For The Part-time R User&quot; includes two points: Sometimes you have to write parameterized or re-usable code. The methods for doing this should be easy and legible. The first point feels abstract, until you find yourself wanting to re-use code on new projects. As for the second point: I … Continue reading Let X=X in R
Despite my week-long Twitter consumption sabbatical (helped — in part — by the nigh week-long internet and power outage here in Maine), I still catch useful snippets from folks. My cow-orker @dabdine shunted a tweet by @terrencehart into a Slack channel this morning, and said tweet contained a link to this little gem. Said gem... Continue reading →
Week 9 Gold Mining and Fantasy Football Projection Roundup now available. Go get that free agent gold!
The post Gold-Mining – Week 9 (2017) appeared first on Fantasy Football Analytics.
This tutorial has been taken from Machine Learning with R Second Edition   by Brett Lantz. Use the code  MLR250RB at the checkout to save 50% on the RRP. Or pick up this title with 4 others for just $50 – get the R-Bloggers bundle. The global financial crisis of 2007-2008 highlighted the importance of transparency … Continue reading How to identify risky bank loans using C.50 decision trees
Motivation I have had a lot of fun exploring The US cities’ Crime data via their Open Data portals. Because Toronto’s crime data was simply not available. Not until the summer of this year, Toronto police launch a public safety data portal to increase transparency between the public and officers. I recently have had the […]
Related Post
Spring Budget 2017: Circle Visualisation
Qualitative Research in R
Multi-Dimensional Reduction and Visualisation with t-SNE
Comparing Trump and Clinton’s Facebook pages during the US presidential election, 2016
Analyzing Obesity across USA
In this post I detail problems that emerge when trying to estimate the parameters of GARCH models.
After the release of meme package, I received several feedbacks from users.
The most usefule one is the comment on my blog post:
Sercan Kahveci
Greetings Mr. Yu,
I am very happy that this package exists. Thank you for making it! I would like to request a feature, to ensure the package is able to compete with professional meme-creation tools like memegenerator and paint.net. Since memes often use the font Impact, in white and with black outline, I believe the package would be more powerful if it also did that automatically.
Regards,
Sercan Kahveci, MSc
Content creator at Questionable Research Memes on Facebook
https://www.facebook.com/QResearchMemes/
The words, ‘compete with professional meme-creation tools’, stimulated me to develop text plotting with background outline effect.
Now this feature is available in meme v__=0.0.7, which can be downloaded from CRAN.
Here is an example:
library(meme)
u
by Jocelyn Barker, Data Scientist at Microsoft I have a confession to make. I am not just a statistics nerd; I am also a role-playing games geek. I have been playing Dungeons and Dragons (DnD) and its variants since high school. While playing with my friends the other day it occurred to me, DnD may have some lessons to share in my job as a data scientist. Hidden in its dice rolling mechanics is a perfect little experiment for demonstrating at least one reason why practitioners may resist using statistical methods even when we can demonstrate a better average performance...
This past weekend, violent windstorms raged through New England. We — along with over 500,000 other Mainers — went “dark” in the wee hours of Monday morning and (this post was published on Thursday AM) we still have no utility-provided power nor high-speed internet access. The children have turned iFeral, and being a remote worker... Continue reading →
One of my first baby steps into the open source world, was when I answered this SO question over four years ago. Recently I revisited the post and saw that Z.Lin did a very nice and more modern implementation, using dplyr and facetting in ggplot2. I decided to merge here ideas with mine to create a general function that makes MM plots. I also added two features: counts, proportions, or percentages to the cells as text and highlighting cells by a condition.
I have written the following post about Predictive Maintenance and flexdashboard at my company codecentric’s blog:
Predictive Maintenance is an increasingly popular strategy associated with Industry 4.0; it uses advanced analytics and machine lea...
According to a recent analysis of Stack Overflow &quot;Developer Stories&quot;, where programmer candidates list the technologies the would and would not like to work with, R is the least disliked programming language: This is probably related to the fact that there's high demand in the job market for fast-growing technologies, which is a disincentive for listing them on the &quot;would not work with&quot; section of an online resume. As author David Robinson notes: If you’ve read some of our other posts about the growing and shrinking programming languages, you might notice that the least disliked tags tend to be fast-growing...
Release 1.4 of the magick package introduces
a new feature called image convolution that
was requested by Thomas L. Pedersen. In this post we explain what this is all about.
Kernel Matrix
The new image_convolve() function applies a kernel over the image. Kernel convolution means that each pixel value is recalculated using the weighted neighborhood sum defined in the kernel matrix. For example lets look at this simple kernel:
library(magick)
kern % image_negate()
As with the blurring, the original image can be blended in with the transformed one, effectively sharpening the image along edges.
img %__% image_convolve('DoG:0,0,2', scaling = '100, 100%')
The ImageMagick documentation has more examples of convolve with various avaiable kernels.
The last and final Le Monde puzzle is a bit of a disappointment, to wit: A 4×4 table is filled with positive and different integers. A 3×3 table is then deduced by adding four adjacent [i.e. sharing a common corner] entries of the original table. Similarly with a 2×2 table, summing up to a unique […]
It is widely recognized that aggressive securitization of housing loans played a decisive role in the creating of a house price bubble in the US and the subsequent financial crisis. An open question is in how far financial experts on securitization wer...
The long awaited promises will be released soon!
Being as impatient as I am when it comes to new technology, I decided to play with currently available implementation of promises that Joe Cheng shared and presented recently in London at EARL confere...
Later this month I’m speaking at the 2017 Beijing Workshop on Forecasting, to be held on Saturday 18 November at the Central University of Finance and Economics.
I’m giving four talks as part of the workshop. Other speakers are Junni Zhang, Lei Song, Hui Bu, Feng Li and Yanfei Kang.
Full program details are available online.
Just last week we organised the 2nd rOpenSci ozunconference, the sibling rOpenSci unconference, held in Australia. Last year it was held in Brisbane, this time around, the ozunconf was hosted in Melbourne, from October 27-27, 2017.
At the ozunconf, we ...
Competitive predictive modeling site Kaggle conducted a survey of participants in prediction competitions, and the 16,000 responses provide some insights about that user community. (Whether those trends generalize to the wider community of all data scientists is unclear, however.) One question of interest asked what tools Kagglers use at work. Python is the most commonly-used tool within this community, and R is second. (Respondents could select more than one tool.) Interestingly, the rankings varied according to the job title of the respondent. R and Python received top-ranking for every job-title subgroup except one (database administrators, who preferred SQL), according to...
Ezra Haber Glenn, the author of the acs package in R, recently posted about the Free Software Foundation’s “Social Benefit” Award on the acs mailing...
The post Free Software Foundation “Social Benefit” Award Nominations appeared first on AriLamstein.com.
Following up on the initial 0.0.1 release of linl, Aaron and I are happy to announce release 0.0.2 which reached the CRAN network on Sunday in a smooth 'CRAN-pretest-publish' auto-admittance. linl provides a simple-yet-powerful Markdown---and RMarkdo...
This is a book by James P. Howard, II, I received from CRC Press for review in CHANCE. (As usual, the customary warning applies: most of this blog post will appear later in my book review column in CHANCE.) It consists in a traditional introduction to numerical analysis with backup from R codes and packages. […]
Our pinp package for snazzier one or two column vignette received it second update. Now at version 0.0.3, it arrived on CRAN on Saturday with minimal fuzz as an 'CRAN-pretest-publish' transition.
We added more frontmatter options, documented more, an...
It's been over a year since we first introduced introduced the Team Data Science Process (TDSP). The data, technology and practices behind Data Science continue to evolve, and the TDSP has evolved in parallel. Over the past year, several new facets have been added, including: The IDEAR (Interactive Data Exploration, Analysis and Reporting) framework, an open source extension to R and Python designed to standardize the process of data exploration and reporting; Guidance for use of Spark 2.0, including an end-to-end Spark v2.0 walkthrough; Guidance for use of in-database Python with SQL Server, including an end-to-end in-database Python tutorial; Instantiation...
As many folks know, I live in semi-rural Maine and we were hit pretty hard with a wind+rain storm Sunday to Monday. The hrbrmstr compound had no power (besides a generator) and no stable/high-bandwidth internet (Verizon LTE was heavily congested) since 0500 Monday and still does not as I write this post. I’ve played with... Continue reading →
Discover how to tell the story that lives in your data: using the most modern R tools for data visualization, learn how to build beautiful, effective and flexible plots. Then, step by step, discover how to embed visualizations and tables in a powerful Shinyapp, to make your data easily navigable and let their insights emerge . Join the fourth course of our term: Data Visualization and Dashboard with R, in November 7-8
The post R live class | Data Visualization and Dashboard with R | Nov 7-8 Milan appeared first on Quantide - R training &amp; consulting.
Shiny is an amazing R package which lets the R developers and users build amazing web apps using R itself. It lets the R users analyze, visualize and deploy their machine learning models directly in the form of the web app. This package lets you host standalone apps on a webpage or embed them in […]
Related Post
Time Series Analysis in R Part 2: Time Series Transformations
Time Series Analysis in R Part 1: The Time Series Object
Parsing Text for Emotion Terms: Analysis &amp; Visualization Using R
Using MongoDB with R
Finding Optimal Number of Clusters
This post on the heatmaply package is based on my recent paper from the journal bioinformatics (a link to a stable DOI). The paper was published just last week, and since it is released as CC-BY, I am permitted (and delighted) to republish it here in full. My co-authors for this paper are Jonathan Sidi, Alan O’Callaghan, and Carson Sievert. Summary: heatmaply is an R … Continue reading &quot;heatmaply: an R package for creating interactive cluster heatmaps for online publishing&quot;
Motivation The sheer size of the airline industry provides a reason to care about it: it affects not only millions of people directly (flyers, pilots, engineers, […]
The post Increasing Airline Customer Satisfaction appeared first on NYC Data Science Academy Blog.
New big release (200-pages) in the Data Science Live Book -open source-
Download available in PDF, eBook and Kindle versions!
Almost 2 years ago, I wrote a short post on the German data science market by analysing open position on the key job platforms; Stepstone, Monster and Indeed. Recently I received requests to update the post with fresh data.
So here it comes. To add a ...
This report describes changes in version 1.6-0 of the ‘gridSVG’ package for R. The most important result of these changes is that ‘gridSVG’ is now much faster at generating SVG files (in situations where it used to be painfully slow). Paul Murrell Download
This is the 4th installment of my ‘Practical Machine Learning with R and Python’ series. In this part I discuss classification with Support Vector Machines (SVMs), using both a Linear and a Radial basis kernel, and Decision Trees. Further, a closer look is taken at some of the metrics associated with binary classification, namely accuracy … Continue reading Practical Machine Learning with R and Python – Part 4
In my recent research I have worked on understanding the key correlates of Brexit. This paper is joint work with my co-authors Sascha Becker and Dennis Novy and has now been published in Economic Policy. After having gone through the peer-review process, I am very happy to share the data and the underlying code. On … Continue reading Capturing the Brexit vote in data →
As part of our consulting practice Win-Vector LLC has been helping a few clients stand-up advanced analytics and machine learning stacks using R and substantial data stores (such as relational database variants such as PostgreSQL or big data systems such as Spark). Often we come to a point where we or a partner realize: &quot;the … Continue reading Big Data Transforms
A tidytext analysis of Faith No More lyrics - Is this a midlife crisis?
I wanted to ease myself back into text mining,specifically using the tidytext package as I haven’t had to do any at work for well over a year.
I’ve been thinking ab...
In our next meetup, we’ll have Hadley (I think no introduction needed). Packages made easy...
It’s time to branch out into a new area of data visualisation: proportion area plots. These plots use area to show proportion between different related values. A common type of proportional area plots are tree maps. We are going to be using the same principle but with circles. A common subject for area visualisation is […]
Related Post
Qualitative Research in R
Multi-Dimensional Reduction and Visualisation with t-SNE
Comparing Trump and Clinton’s Facebook pages during the US presidential election, 2016
Analyzing Obesity across USA
Can we predict flu deaths with Machine Learning and R?
I missed two weeks and haven’t had time to create a dedicated blog for Stan yet, so we’re still here. This is only the update for this week. From now on, I’m going to try to concentrate on things that are done, not just in progress so you can get a better feel for the […]
The post Stan Roundup, 27 October 2017 appeared first on Statistical Modeling, Causal Inference, and Social Science.
Microsoft R Open (MRO), Microsoft's enhanced distribution of open source R, has been upgraded to version 3.4.2 and is now available for download for Windows, Mac, and Linux. This update upgrades the R language engine to the latest R 3.4.2 and updates the bundled packages. MRO is 100% compatible with all R packages. MRO 3.4.2 points to a fixed CRAN snapshot taken on October 15 2017, and you can see some highlights of new packages released since the prior version of MRO on the Spotlights page. As always you can use the built-in checkpoint package to access packages from an...
The Projects chapter of my Empirical software engineering book has been added to the draft pdf (download here). This material turned out to be harder to bring together than I had expected. Building software projects is a bit like making sausages in that you don’t want to know the details, or in this case those […]
I've been using the ggplot2 package a lot recently. When creating a legend or tick marks on the axes, ggplot2 uses the levels of a character or factor vector. Most of the time, I am working with coded variables that use some abbreviation of the &quot;true&quot; meaning (e.g. &quot;f&quot; for female and &quot;m&quot; for male or single characters for some single character for a location: &quot;S&quot; for Stuttgart and &quot;M&quot; for Mannheim).In my plots, I don't want these codes but the full name of the level. Since I am not aware of any super-fast and easy to use function in base R (let me know in the comments if there is one), I came up with a very simple function and put this in my .Rprofile (that means that it is available whenever I start R). I called it &quot;replace.levels&quot;. The dot before the name means that it is invisible and does not show up in my Global Environment overview in RStudio. You have to call it with .replace.levels(), of course.This is how it works:.replace.levels takes two arguments: vec and replace.listvec is the vector where substitutions shall be made.replace.list is a list with named elements. The names of the elements are substituted for the contents of the elements.The function also checks if all elements (both old and new ones) appear in vec. If not, it throws an error. It is a very simple function, but it saves me a lot of typing.Example:a
Cryptocurrency market has been growing rapidly that being an Analyst, It intrigued me what does it comprise of. In this post, I’ll explain how can we analyse the Cryptocurrency Market in R with the help of the package coinmarketcapr. Coinmarketcapr package is an R wrapper around coinmarketcap API. To get started, Let us load the […]
Related Post
Time Series Analysis in R Part 3: Getting Data from Quandl
Pulling Data Out of Census Spreadsheets Using R
Extracting Tables from PDFs in R using the Tabulizer Package
Extract Twitter Data Automatically using Scheduler R package
An Introduction to Time Series with JSON Data
Week 8 Gold Mining and Fantasy Football Projection Roundup now available. Go get that free agent gold!
The post Gold-Mining – Week 8 (2017) appeared first on Fantasy Football Analytics.
by Błażej Moska, computer science student and data science intern One of the most important thing in predictive modelling is how our algorithm will cope with various datasets, both training and testing (previously unseen). This is strictly connected with the concept of bias-variance tradeoff. Roughly speaking, variance of an estimator describes, how do estimator value ranges from dataset to dataset. It's defined as follows: \[ \textrm{Var}[ \widehat{f} (x) ]=E[(\widehat{f} (x)-E[\widehat{f} (x)])^{2} ] \] \[ \textrm{Var}[ \widehat{f} (x)]=E[(\widehat{f} (x)^2]-E[\widehat{f} (x)]^2 \] Bias is defined as follows: \[ \textrm{Bias}[ \widehat{f} (x)]=E[\widehat{f}(x)-f(x)]=E[\widehat{f}(x)]-f(x) \] One could think of a Bias as an ability to...
Leveraging tidyverse packages httr, stringr &amp; purrr - Introduction
McDonald’s is a nostalgic component of America 🇺🇸 and a pioneer of fast food operations and real estate ventures, as depicted in the 2016 film, The Founder, about Ray Kroc. As a kid I traveled to different McDonald’s across the east coast and noticed a difference in the classic hamburger preparation for adding mustard (i.e. in Maryland and not in Upstate New York). After some Google research, I noticed others had documented the regional differences in the use of mustard and but no aggregated data set existed detailing which McDonald’s added mustard to their hamburgers.
I hypothesized that these deviations in food prep could be identified from yelp.com reviews. The process below explains the approaches I took to gather data from the web with the yelp API and the development of a shiny web application which detects string patterns in reviews for the keyword ‘mustard’ for a specific McDonald’s.
API Process
This script highly references Jenny Bryan’s yelpr example!
library(yelpr) # devtools::install_github(&quot;jennybc/ryelp&quot;)
library(httr)
library(stringr)
library(purrr)
# 1. Create an application on the [Yelp developers site](https://www.yelp.com/developers/v3/manage_app) and agree to the Terms and aggreements
## Set your credentials as environment variables. Sys.setenv(YELP_CLIENT_ID='**************')
Sys.setenv(YELP_SECRET='*****************************')
# 2. search for businesses by creating an app
yelp_app
Two new Microsoft webinars are taking place over the next week that may be of interest: AI Development in Azure using Data Science Virtual Machines The Azure Data Science Virtual Machine (DSVM) provides a comprehensive development and production environment to Data Scientists and AI-savvy developers. DSVMs are specialized virtual machine images that have been curated, configured, tested and heavily used by Microsoft engineers and data scientists. DSVM is an integral part of the Microsoft AI Platform and is available for customers to use through the Microsoft Azure cloud. In this session, we will first introduce DSVM, familiarize attendees with the...
This blog post was born out of pure curiosity about the robustness of the IRIS Dataset. Biological datasets do not need to be that big in comparison to datasets of customers, consumption, stock and anything that might be volatile. When still at the university, on one occasion I can remember, we were measuring the length … Continue reading Robust IRIS Dataset?
I developed a tiny toy package, meme, which is now on CRAN. As it’s name indicated, it was designed to create memes, which are captioned photos that are intended to be funny, riduculous.
meme()
The package is quite simple. You can use meme() function to add meme captions, and this is all the package supposed to do:
library(meme)
u
We’re into the third day of Business Science Demo Week. Hopefully by now you’re getting a taste of some interesting and useful packages. For those that may have missed it, every day this week we are demo-ing an R package: tidyquant (Monday), timetk (Tuesday), sweep (Wednesday), tibbletime (Thursday) and h2o (Friday)! That’s five packages in five days! We’ll give you intel on what you need to know about these packages to go from zero to hero. Today is sweep, which has broom-style tidiers for forecasting. Let’s get going!
Previous Demo Week Demos
class(Monday)
I developed a tiny toy package, meme, which is now on CRAN. As it’s name indicated, it was designed to create memes, which are captioned photos that are intended to be funny, riduculous.
meme()
The package is quite simple. You can use meme() function to add meme captions, and this is all the package supposed to do:
library(meme)
u
The slackr package, by Bob Rudis, contains functions that make it possible to interact with the Slack messaging platform. The great part of Slack is that it has mobile devices application interface. We take advantage of this by extending slackr’s interaction with with the server:
Write to a channel (current) - slackr
Compile TeX snippets directly to a channel - tex_slackr
tex_slackr(xtable::xtable(mtcars))
This function uses the texPreview package to compile and generate the TeX output.
Read channel history (new) - history_slackr
Edit messages on a channel (new) - edit_slackr
Delete channel messages (new) - delete_slackr
This lets us interact with R and Slack in new ways, by getting active updates on long simulations directly to your (and your team’s) mobile device and multitask away from your computer.
devtools::install_github('hrbrmstr/slackr')
Progress Bars
Create text progress bar that is sent directly to a Slack channel.
progress_slackr
The recent Association of Public Data Users (APDU) Conference gave me the opportunity to meet some people who have made tremendous contributions to the world...
The post Meeting Titans of Open Data appeared first on AriLamstein.com.
Some Announcements: Dr. Nina Zumel will be presenting “Myths of Data Science: Things you Should and Should Not Believe”, Sunday, October 29, 2017 10:00 AM to 12:30 PM at the She Talks Data Meetup (Bay Area). ODSC West 2017 is soon. It is our favorite conference and we will be giving both a workshop and … Continue reading Some Announcements
R has a rich and infinitely flexible graphics system, and you can easily embed R graphics into Microsoft Office documents like PowerPoint or Word. The one thing I dread hearing after delivering such a document, though, is &quot;how can I tweak that graphic?&quot;. I could change the colors or fonts or dimensions in R, of course, but sometimes people just want to watch the world burn tweak graphics to their hearts' content. If you're in that situation, you have a couple of options for using R to create Office documents with graphics, and make those graphics editable. Both options work...
CRAN contains up to date (October 2017) more than 11500 R packages. If you want to scroll through all of these, you probably need to spend a few days, assuming you need 5 seconds per package and there are 8 hours in a day.
Since R version 3.4, we can also get a dataset will all packages, their dependencies, the package title, the description and even the installation errors which the packages have. Which makes the CRAN database with all packages an excellent dataset for doing text mining. If you want to get that dataset, just do as follows in R:
library(tools)crandb
Kmeans is one of the most popular and simple algorithm to discover underlying structures in your data. The goal of kmeans is simple, split your data in k different groups represented by their mean. The mean of each group is assumed to be a good summary of each observation of this cluster. Kmeans Algorithm We assume […]
The post Machine Learning Explained: Kmeans appeared first on Enhance Data Science.
jsTree is a R package that is a standalone htmlwidget for the jsTree JavaScript library. It can be run from the R console directly into the Rstudio Viewer, be used in a Shiny application or be part of an Rmarkdown html output.
Installation
#CRAN
install.packages('jsTree')
#Github
devtools::install_github('metrumresearchgroup/jsTree')
The motivation for the package came from the shinyTree by Jeff Allen, which has an early version of the JavaScript library hardcoded into a Shiny application.
The input for the htmlwidget is a character vector and the heirarchy is set by the ‘/’, as to mimic the delimiter of a file path.
The main purpose of this package is vizualize folder hierarchy, much like in the Files tab in RStudio. But, instead of looking in local directories it will be used for remote repositories, such as github, bitbucket and svn repositories. This is implemented in the vcs package and a future post will show utility that package gives.
To show the basic functionality of jsTree we will use a dataset that contains all superfund sites from the EPA. You can access the tibble here. The data
library(htmlTable)
library(jsTree)
library(widgetframe)
# Read in the data
superfundsite
There was something about them that made me uneasy, some longing and at the same time some deadly fear – Dracula (Stoker, Bram) Twitter is a very good source of inspiration. Some days ago I came across with this: Mi presentación sobre análisis de textos con R para el primer meetup de @RLadiesSantiago: https://t.co/KPU89vyeND #rladies … Continue reading A Shiny App to Create Sentimental Tweets Based on Project Gutenberg Books →
MicrosoftML is an R package for machine learning that works in tandem with the RevoScaleR package. (In order to use the MicrosoftML and RevoScaleR libraries, you need an installation of Microsoft Machine Learning Server or Microsoft R Client.) A great way to see what MicrosoftML can do is to take a look at the on-line book Machine Learning with the MicrosoftML Package Package by Ali Zaidi. The book includes worked examples on several topics: Exploratory data analysis and feature engineering Regression models Classification models for computer vision Convolutional neural networks for computer vision Natural language processing Transfer learning with pre-trained...
When I was an R newbie, I was taught to load packages by using the command library(package). In my Linear Models class, the instructor likes to use require(package). This made me wonder, are the commands interchangeable? What’s the difference, and […]
This post will be about pulling free data from AlphaVantage, and gauging interest for a volatility trading subscription service. So … Continue reading →
I’m pleased to announce the latest update from my sjPlot-package on CRAN. Beside some bug fixes and minor new features, the major update is a new function, plot_model(), which is both an enhancement and replacement of sjp.lm(), sjp.glm(), sjp.lmer(), sjp.glmer() and sjp.int(). The latter functions will become deprecated in the next updates and removed somewhen […]
To post your R job on the next post Just visit  this link and post a new R job  to the R community. You can post a job for  free  (and there are also “featured job” options available for extra exposure). Current R jobs Job seekers:  please follow the links below to learn more and apply for your R job of interest: Featured Jobs Full-Time Data Scientist Cincinnati Reds – Posted by MichaelSchatz Anywhere 13 Oct 2017 Freelance The R Foundation is looking for a volunteer to organize R’s contributed documentation R: The R Foundation – Posted by Tal Galili Anywhere 13 Oct 2017 Full-Time Rays Research &amp; Development Analyst Tampa Bay Rays – Posted by kferris10 Saint Petersburg Florida, United States 11 Oct 2017 Full-Time PROGRAMMER/SOFTWARE DEVELOPMENT ENGINEER/COMPUTATIONAL AND MACHINE LEARNING SPECIALIST fchiava Cambridge Massachusetts, United States 10 Oct 2017 Freelance R Shiny Developer – Work From Home Data Science Talent – Posted by datasciencetalent Frankfurt am Main Hessen, Germany 6 Oct 2017 Full-Time Data Scientist Edelweiss Business Services – Posted by Aakash Gupta Mumbai Maharashtra, India 20 Sep 2017 More New R Jobs Freelance Statistician / Econometrician / R Programmer for Academic Statistical Research Academic Research – Posted by empiricus Anywhere 21 Oct 2017 Full-Time Postdoc – Norwegian institute of public health – […]
MilanoR Staff is happy to announce the 9th MilanoR Meeting! The meeting will take place on November 20th, from 7pm to about 9:30 pm, in Mikamai (close to the Pasteur metro station) [save the date, more info soon]. This time we want to focus on a specific topic: data visualization with R. Do you feel you have something to input? Send us your contribution, even in term of a short video if you cannot attend the meeting.
The post 9th MilanoR meeting on November 20th: call for presentations! appeared first on MilanoR.
Nowadays, a lot of interesting time series data is freely available that allows us to compare important economic, social and environmental trends across countries.
I feel that one can learn a lot by surfing through the data sections on the websites of...
Aaron Wolen and I are pleased to announce the availability of the initial 0.0.1 release of our new linl package on the CRAN network. It provides a simple-yet-powerful Markdown---and RMarkdown---wrapper the venerable LaTeX letter class. Aaron had done...
What’s an effective way of helping a student run a desktop application when their own computer won’t run the application, for whatever reason, locally? Virtualised software, running remotely, provides one solution. So here’s an example of a project that looks at doing just that:  DIT4C (“Data Intensive Tools for the Cloud”), ‘a platform for hosting data analysis […]
Way back in July of 2009, the first version of the twitteR package was published by Geoff Jentry in CRAN. Since then it has seen 28 updates, finally breaking the 0.x.y barrier into 1.x.y territory in March of 2013 and receiving it’s last update in July of 2015. For a very long time, the twitteR... Continue reading →