When I wrote the original post, I wasn’t planning on writing a follow-up. Certainly not the week after. But what a difference a week can make in a dynamic system like the US stock market. While re-running the computations testing the latest version of RStudio, I noticed something surprising – President Trump’s rally has advanced […]
The post Markets Performance after Election: Day 239 appeared first on Quintuitive.
Here's a refresher on the history of the R project: 1992: R development begins as a research project in Auckland, NZ by Robert Gentleman and Ross Ihaka 1993: First binary versions of R published at Statlib [see update, below] 1995: R first distributed as open-source software, under GPL2 license 1997: R core group formed 1997: CRAN founded (by Kurt Hornik and Fritz Leisch) 1999: The R website, r-project.org, founded 2000: R 1.0.0 released (February 29) 2001: R News founded (later to become the R Journal) 2003: R Foundation founded 2004: First UseR! conference (in Vienna) 2004: R 2.0.0 released 2009:...
Motivation
On a previous post I explained how to install R on Ubuntu 16.04 without further complications.
Now here are the equivalent steps on Ubuntu 17.10. The presented script also installs Java and common packages as I wrote this for a fresh install...
In the last two posts, I’ve focused purely on statistical topics – one-way ANOVA and dealing with multicollinearity in R. In this post, I’ll deviate from the pure statistical topics and will try to highlight some aspects of qualitative research. More specifically, I’ll show you the procedure of analyzing text mining and visualizing the text […]
Related Post
Multi-Dimensional Reduction and Visualisation with t-SNE
Comparing Trump and Clinton’s Facebook pages during the US presidential election, 2016
Analyzing Obesity across USA
Can we predict flu deaths with Machine Learning and R?
Graphical Presentation of Missing Data; VIM Package
How much pollution reduction do we get from a MWh electricity produced by wind or solar power? In principle, this corresponds to the avoided pollution of a fossil fuel plant that reduces its output due to the higher production of renewable electricity....
20.10.2017 Beside a massive of amount of R packages and ImageJ plugins Bio7 can be extended with Eclipse plugins useful for data science and programming. Some of them could also be very useful for R related developments (e.g., to develop R packages or distribute Shiny apps). Installation of Eclipse Plugins One  way to install Eclipse […]
In this post ‘Practical Machine Learning with R and Python – Part 3’,  I discuss ‘Feature Selection’ methods. This post is a continuation of my 2 earlier posts Practical Machine Learning with R and Python – Part 1 Practical Machine Learning with R and Python – Part 2 While applying Machine Learning techniques, the data … Continue reading Practical Machine Learning with R and Python – Part 3
Week76 Gold Mining and Fantasy Football Projection Roundup now available. Go get that free agent gold!
The post Gold-Mining – Week 7 (2017) appeared first on Fantasy Football Analytics.
One of the specialist smoother types in the mgcv package is the Markov Random Field (MFR) smooth. This smoother essentially allows you to model spatial data with an intrinsic Gaussian Markov random field (GMRF). GRMFs are often used for spatial data measured over discrete spatial regions. MRFs are quite flexible as you can think about them as representing an undirected graph whose nodes are your samples and the connections between the nodes are specified via a neighbourhood structure. I’ve become interested in using these MRF smooths to include information about relationships between species. However, these smooths are not widely documented in the smoothing literature so working out how best to use them to do what we want has been a little tricky once you move beyond the typical spatial examples. As a result I’ve been fiddling with these smooths, fitting them to some spatial data I came across in a tutorial Regional Smoothing in R from The Pudding. In this post I take a quick look at how to use the MRF smooth in mgcv to model a discrete spatial data set from the US Census Bureau.
While there are hundreds of excellent books and websites devoted to R, the canonical source of truth regarding the R system remains the R manuals. You can find the manuals at your local CRAN mirror and on your laptop as part of the R distribution (try Help __ Manuals in RGui, or Help __ R Help in RStudio to find them). Unlike books, the R manuals are updated by the R Core Team with every new release, so if you're not sure how the base R system is supposed to work this is the place to check. Note that the...
At Plotly, we are commonly asked about thematic maps — especially county-level choropleth maps. This style of map provides a visual illustration of variation across a geographic area. Some pertinent uses are population density, economic measurements, crime statistics, and election results. With Plotly, there are multiple ways to bring county-level choropleths to life. For example, using ggplotly […]
Ensemble learning methods are widely used nowadays for its predictive performance improvement. Ensemble learning combines multiple predictions (forecasts) from one or multiple methods to overcome accuracy of simple prediction and to avoid possible over...
19.10.2017 I released a new version of Bio7 for MacOSX. (Screenshots: Bio7 with Dark theme enabled. Scrollbars are set to “When scrolling” in the preferences.) For an overview of the new features please consult the release notes of Windows and Linux here: Installation: Download and extract the installation file from http://bio7.org. If you start Bio7 […]
Taxis are plentiful and convenient in New York City, but the city is also served by a wide network of commuter bicycles (Citi Bikes). If you need to get from, say, the West Village to the Garment District, are you better off time-wise hailing a cab, or heading over to the nearest Citi Bike station? Data scientist Todd W. Schnieder crunched the number on travel times for both taxis and Citi Bikes to figure out which was better. Neither is universally the best, but for some trips taxis are most often the fastest, and for others bikes are faster. An...
By Gabriel Vasconcelos The Julia programming language is growing fast and its efficiency and speed is now well-known. Even-though I think R is the best language for Data Science, sometimes we just need more. Modelling is an important part of … Continue reading →
Plotly serves a large bioinformatics and biostats research community. These users leverage the uniquely interactive features of plotly charts for dendrograms, heatmaps, volcano plots, and other visualizations common in this field. Here are 7 resources in Python and R created by plotly bioinformatics and biostats researchers. 1. Jupyter notebook: Visualizing bioinformatics data with plotly and python   […]
A recent riddle [from The Riddle] that I pondered about during a [long!] drive to Luxembourg last weekend was about splitting a square field into three lots of identical surface for a minimal length of separating wire… While this led me to conclude that the best solution was a T like separation, I ran a […]
17.10.2017 A new version of Bio7 for Windows and Linux is available. The MacOSX version will be released soon, too. The application Bio7 is an integrated development environment for ecological modelling, scientific image analysis and statistical analysis. Beside many different tools it contains a complete Graphical User Interface for R. The application itself is based […]
Note: cross-posted with the Stack Overflow blog.
If you hang out on Meta Stack Overflow, you may have noticed news from time to time about A/B tests of various features here at Stack Overflow. We use A/B testing to compare a new version to a baseline f...
One of our senior data scientists, Olga Mierzwa-Sulima spoke at the userR! conference in Brussels to a packed house. The seats were full and there were audience members spilling out the doors.
Source: https://twitter.com/matlabulous/status/882530484...
Coming back to markets and trading (after a while), the feeling has been that the markets, and the economy as a whole, are doing good. How good? Since I haven’t been following things closely, I had to do some forensics. Friday was the 234th trading day after the election. There were claims at different points […]
The post Markets Performance after Election appeared first on Quintuitive.
What a wonderful week at the Astrostat [Indian] summer school in Autrans! The setting was superb, on the high Vercors plateau overlooking both Grenoble [north] and Valence [west], with the colours of the Fall at their brightest on the foliage of the forests rising on both sides of the valley and a perfect green on […]
R is an incredible tool for reproducible research. In the present series of blog posts I want to show how one can easily acquire data within an R session, documenting every step in a fully reproducible way. There are numerous data acquisition options for R users. Of course, I do not attempt to show all the data possibilities and tend to focus mostly on demographic data. If your prime interest lies outside human population statistics, it’s worth checking the amazing Open Data Task View.
The ROpenSci team has started publishing a new series of interviews with the goal of “demystifying the creative and development processes of R community members”. I had the great pleasure of being interviewed by Kelly O'Briant earlier this year, and the interview was published on Friday. Thanks for being a great interviewer, Kelly! I'm looking forward to hearing from other R community members as the the rest of the series is published. ROpenSci blog: .rprofile: David Smith
Today we are excited to announce the availability of RStudio Professional Drivers. There are, of course, many ways to connect to Databases using R. RStudio Professional Drivers are specifically intended for use with our professional products...
For those who aren’t familiar with the colourpicker package, it provides a colour picker for R that can be used in Shiny, as well as other related tools. Today it’s leaving behind its 0.x days and moving on to version 1.0!
colourpicker h...
This weekend, I decided it was time: I was going to update my Python environment and get Keras and Tensorflow installed so I could start doing tutorials (particularly for deep learning) using […]
Introduction It is hard to wander around New York City without seeing rows of dozens of bright blue Citibikes planted in the middle of busiest nooks […]
The post Citibike Business Opportunity: Advertising appeared first on NYC Data Science Academy Blog.
Traffic fatalities in the news
Fatalities from traffic crashes have been in the news in New Zealand recently, for tragic reasons. For once, data beyond the short term have come into the debate. Sam Warburton has written two good articles on the topi...
In this post we will explore how to make SVG tables in R using plotly. The tables are visually appealing and can be modified on the fly using simple drag and drop. Make sure you install the latest version of Plotly i.e. v 4.7.1.9 from Github using devtools::install_github(&quot;ropensci/plotly) The easiest way to create a SVG […]
By Yuri Fonseca In this post we are going to make an Uber assignment simulation and calculate some metrics of waiting time through simulation. Setting Suppose we live in a 100×100 block city where each block takes 1 minute to … Continue reading →
Take a look at this list of 13 hot startups, from a list compiled by Alex Bresler. Perhaps one of them is the next Juicero? FAR ATHERA: A CLINICAL AI PLATFORM THAT CAN BE ACCESSED ON DEMAND. ZAPSY: TRY-AT-HOME SERVICE FOR CONSUMER ELECTRONICS. MADESS: ON-DEMAND ACCESS TO CLEAN WATER. DEERG: AI RADIOLOGIST IN A HOME SPER: THE FASTEST, EASIEST WAY TO BUY A HOME WITHOUT THE USER HAVING TO WEAR ANYTHING. PLILUO: VENMO FOR B2B SAAS. LANTR: WE HELP DOCTORS COLLECT 2X MORE ON CANDLES AND KEROSENE. ABS: WE PROVIDE FULL-SERVICE SUPPORT FOR SUBLIME, VIM, AND EMACS. INSTABLE DUGIT: GITHUB...
R is a powerful language used widely for business analytics. More companies are seeking business analysts with knowledge of R. In this R for Business Analysts […]
The post R for Business Analysts, NYC Data Science Academy in-person training | November 6 @NYC appeared first on NYC Data Science Academy Blog.
In this 2nd part of the series “Practical Machine Learning with R and Python – Part 2”, I continue where I left off in my first post Practical Machine Learning with R and Python – Part 2. In this post I cover the some classification algorithmns and cross validation. Specifically I touch -Logistic Regression -K Nearest … Continue reading Practical Machine Learning with R and Python – Part 2
Three years ago I referenced the Seinfeld Streak used in an earlier post of regular updates to to the Rcpp Gallery:
This is sometimes called Jerry Seinfeld's secret to productivity: Just keep at it. Don't break the streak.
and showed the first chart of GitHub streaking
And two year ago a first follow-up appeared in this post:
And a year ago we had a followup last year
And as it October 12 again, here is the new one:
Again, special thanks go to Alessandro Pezzè for the Chrome add-on GithubOriginalStreak.
This post by Dirk Eddelbuettel originated on his Thinking inside the box blog. Please report excessive re-aggregation in third-party for-profit settings.
Introduction I often see advertisement for The Bold and The Beautiful, I have never watched a single episode of the series. Still, even as a data scientist you might be wondering how these beautiful ladies and gentlemen from the show … Continue reading →
Last week's R-themed crossword from R-Ladies DC was popular, so here's another R-related crossword, this time by Barry Rowlingson and published on page 39 of the June 2003 issue of R-news (now known as the R Journal). Unlike the last crossword, this one follows the conventions of a British cryptic crossword: the grid is symmetrical, and eschews 4x4 blocks of white or black squares. Most importantly, the clues are in the cryptic style: rather than being a direct definition, cryptic clues pair wordplay (homonyms, anagrams, etc) with a hidden definition. (Wikipedia has a good introduction to the types of clues...
If you want to find the structure hidden behind your data, this is right class for you: you will learn how to group similar observations using Clustering; how to “naturally” aggregate your variables using Dimensionality Reduction; how to predict outcomes using Regression and Classification (LMs, GLMs, Trees, Random forests, Neural networks). In other words, you will get a full-immersion in the Data Mining and the Machine Learning world, using R. Our third live class in Milan, on October 25-26
The post R live class | Machine Learning with R | Oct 25-26 appeared first on Quantide - R training &amp; consulting.
Sometime in 2009, I began listening to a science podcast titled This Week in Virology, or TWiV for short. I thought it was pretty good and listened regularly up until sometime in 2016, when it seemed that most episodes were approaching two hours in duration. I listen to several podcasts when commuting to/from work, which … Continue reading XML parsing made easy: is that podcast getting longer?
Clara Schartner, Data Scientist
It will come as no surprise that cats and ggplot are among our favourite things here at Mango, luckily there is an easy way to combine both.
Using the function annotation_custom in the popular ggplot2 package it is possible to display images on a plot i.e. points of a scatterplot. This way data can be displayed in a more fun, creative way. In keeping with the cat theme I have chosen a data set about cats and a cat icon based on Mango the cat. The MASS package provides a data set called cats which contains the body weight, heart weight and sex of adult cats.
library(MASS)
data(cats)
head(cats)
set.seed(1234)
cats
To post your R job on the next post Just visit  this link and post a new R job  to the R community. You can post a job for  free  (and there are also “featured job” options available for extra exposure). Current R jobs Job seekers:  please follow the links below to learn more and apply for your R job of interest: Featured Jobs Full-Time Rays Research &amp; Development Analyst Tampa Bay Rays – Posted by kferris10 Saint Petersburg Florida, United States 11 Oct 2017 Full-Time PROGRAMMER/SOFTWARE DEVELOPMENT ENGINEER/COMPUTATIONAL AND MACHINE LEARNING SPECIALIST fchiava Cambridge Massachusetts, United States 10 Oct 2017 Freelance R Shiny Developer – Work From Home Data Science Talent – Posted by datasciencetalent Frankfurt am Main Hessen, Germany 6 Oct 2017 Full-Time Data Scientist Edelweiss Business Services – Posted by Aakash Gupta Mumbai Maharashtra, India 20 Sep 2017 Full-Time Solutions Engineer for RStudio @ U.S.A. (or anywhere) RStudio – Posted by nwstephens Anywhere 8 Sep 2017 More New R Jobs Full-Time Rays Research &amp; Development Analyst Tampa Bay Rays – Posted by kferris10 Saint Petersburg Florida, United States 11 Oct 2017 Full-Time PROGRAMMER/SOFTWARE DEVELOPMENT ENGINEER/COMPUTATIONAL AND MACHINE LEARNING SPECIALIST fchiava Cambridge Massachusetts, United States 10 Oct 2017 Freelance R Shiny Developer – Work From Home Data Science Talent – Posted by datasciencetalent […]
Week 6 Gold Mining and Fantasy Football Projection Roundup now available. Go get that free agent gold!
The post Gold-Mining – Week 6 (2017) appeared first on Fantasy Football Analytics.
Halfiax, here we come. I (Bob, not Andrew) am going to be giving a talk on Stan and then Mitzi and I will be teaching a course on Stan after that. The public is invited, though space is limited for the course. Here are details if you happen to be in the Maritime provinces. TALK: […]
The post Halifax, NS, Stan talk and course Thu 19 Oct appeared first on Statistical Modeling, Causal Inference, and Social Science.
Another quick preview of my R-packages, especially sjPlot, which now also support brmsfit-objects from the great brms-package. To demonstrate the new features, I load all my „core“-packages at once, using the strengejacke-package, which is only available from GitHub. This package simply loads four packages (sjlabelled, sjmisc, sjstats and sjPlot). First, I fit two sample models, … More support for Bayesian analysis in the sj!-packages #rstats #rstan #brms weiterlesen
The Azure Data Lake store is an Apache Hadoop file system compatible with HDFS, hosted and managed in the Azure Cloud. You can store and access the data within directly via the API, by connecting the filesystem directly to Azure HDInsight services, or via HDFS-compatible open-source applications. And for data science applications, you can also access the data directly from R, as this tutorial explains. To interface with Azure Data Lake, you'll use U-SQL, a SQL-like language extensible using C#. The R Extensions for U-SQL allow you to reference an R script from a U-SQL statement, and pass data from...
How does Linear Discriminant Analysis work and how do you use it in R? This post answers these questions and provides an introduction to Linear...
A reminder that when idle queries pop into your head, the answer can often be found using R + online data. And a brief excursion into accessing the Weather Underground. One interesting aspect of Australian life, even in coastal urban areas like Sydney, is that sometimes it just stops raining. For weeks or months at … Continue reading Feels like a dry winter – but what does the data say?
One of the really appealing features of the mgcv package for fitting GAMs is the functionality it exposes for fitting quite complex models, models that lie well beyond what many of us may have learned about what GAMs can do. One of those features that I use a lot is the ability to model the smooth effects of some covariate (x) in the different levels of a factor. Having estimated a separate smoother for each level of the factor, the obvious question is, which smooths are different? In this post I’ll take a look at one way to do this using by-variable smooths.
Python has been getting some attention recently for its impressive growth in usage. Since both R and Python are used for data science, I sometimes get asked if R is falling by the wayside, or if R developers should switch course and learn Python. My answer to both questions is no. First, while Python is an excellent general-purpose data science tool, for applications where comparative inference and robust predictions are the main goal, R will continue to be the prime repository of validated statistical functions and cutting-edge research for a long time to come. Secondly, R and Python are both...
We're only a few weeks away!
Don't miss keynotes from Mara Averick, David Robinson and Tareef Kawaf!
They will be joined by speakers from DataCamp, Crunch, Pfizer, GfK, ...
We use text mining a lot in day-to-day data mining operations. In order to share our knowledge on this, to show that R is an extremely mature platform to do business-oriented text analytics and to give you practical experience with text mining, our course on Text Mining with R is scheduled for the 3rd consecutive year at LStat, the Leuven Statistics Research Center (Belgium) as well as at the Data Science Academy in Brussels. Courses are scheduled 2 times in November 2017 and also in March 2018.
This course is a hands-on course covering the use of text mining tools for the purpose of data analysis. It covers basic text handling, natural language engineering and statistical modelling on top of textual data. The following items are covered.
Text encodings
Cleaning of text data, regular expressions
String distances
Graphical displays of text data
Natural language processing: stemming, parts-of-speech tagging, tokenization, lemmatisation
Sentiment analysis
Statistical topic detection modelling and visualization (latent diriclet allocation)
Visualisation of correlations &amp; topics
Word embeddings
Document similarities &amp; Text alignment
Feel free to register at the following dates:
18-19/10/2017: Statistical machine learning with R. Leuven (Belgium). Subscribe here
08+10/11/2017: Text mining with R. Leuven (Belgium). Subscribe here
27-28/11/2017: Text mining with R. Brussels (Belgium). http://di-academy.com/bootcamp + send mail to training@di-academy.com
19-20/12/2017: Applied spatial modelling with R. Leuven (Belgium). Subscribe here
20-21/02/2018: Advanced R programming. Leuven (Belgium). Subscribe here
08-09/03/2018: Computer Vision with R and Python. Leuven (Belgium). Subscribe here
22-23/03/2018: Text Mining with R. Leuven (Belgium). Subscribe here
I have a character string snipped from a lyric by Daniel Son the Necklace Don. Let’s get all Weird Al Yankovic with it. I’ll revise some words with the datzen::yankovise() function. You can swap out words using a dictionary of name-value pairs. Above, we have a bar from 2 Chainz aka the Hair Weave Killer. […]
I’m delighted to announce the release of version 1.0.0 of the dplyrXdf package. dplyrXdf began as a simple (relatively speaking) backend to dplyr for Microsoft Machine Learning Server/Microsoft R Server’s Xdf file format, but has now become a broader suite of tools to ease working with Xdf files. This update to dplyrXdf brings the following new features: Support for the new tidyeval framework that powers the current release of dplyr Support for Spark and Hadoop clusters, including integration with the sparklyr package to process Hive tables in Spark Integration with dplyr to process SQL Server tables in-database Simplified handling of...
I’ve blogged a bit about robots.txt — the rules file that documents a sites “robots exclusion” standard that instructs web crawlers what they can and cannot do (and how frequently they should do things when they are allowed to). This is a well-known and well-defined standard, but it’s not mandatory and often ignored by crawlers... Continue reading →
This time I elaborate on a much more specific subject that will mostly concern biologists and geneticists. I will try my best to outline the approach as to ensure non-experts will still have a basic understanding. This tutorial illustrates the power of genome-wide association (GWA) studies by mapping the genetic determinants of cholesterol levels using … Continue reading Genome-wide association studies in R →
I just arrived in Autrans, on the Plateau du Vercors overlooking Grenoble and the view is fabulistic! Trees have started to turn red and yellow, the weather is very mild, and my duties are restricted to teaching ABC to a group of enthusiastic astronomers and cosmologists..! Second advanced course on ABC in the mountains this […]
We’re excited to announce the general availability of RStudio 1.1. Highlights include:
A connections tab which makes it easy to connect to, explore, and view data in a variety of databases.
A terminal tab which provides fluid shell integration with the IDE, xterm emulation, and even support for full-screen terminal applications.
An object explorer which can navigate deeply nested R data structures and objects.
A new, modern dark theme and Retina-quality icons throughout.
Dozens of other small improvements and bugfixes.
RStudio Server Pro 1.1 is also now available. Some of the new Pro features include:
Support for floating licenses, which make it easy to run RStudio Server Pro in Docker containers, virtual machines, and cloud computing instances.
Improved session management, which allows analysts to label, multi-select, force quit and otherwise self-manage their R sessions.
Tools for administrators, including the ability to send users notifications and automatically clean up unused sessions, freeing disk space and resources.
You can download RStudio v1.1 today, and let us know what you think on the RStudio IDE community forum.
In April I spoke for the Royal Statistical Society (Glasgow branch) at their event celebrating 20 years of CRAN. The other speakers were Charis Chanialidis and Colin Gillespie. Like most people I find… Continue reading →
We will show how to perform multivariate random sampling using one of the Markov Chain Monte Carlo (MCMC) algorithms, called the Gibbs sampler.
Random sampling with rabbit on the bed plane
via GIPHY
To start, what are MCMC algorithms and what are they based on?
Suppose we are interested in generating a random variable with a distribution of , over .
If we are not able to do this directly, we will be satisfied with generating a sequence of random variables , which in a sense tending to a distribution of . The MCMC approach explains how to do so:
Build a Markov chain , for , whose stationary distribution is .
If the structure is correct, we should expect random variables to converge to .
In addition, we can expect that
for function , occurs:
with probability equals to 1.
In essence, we want the above equality to occur for any arbitrary random variable .
One of the MCMC algorithms that guarantee the above properties is the so-called Gibbs sampler.
Gibbs Sampler - description of the algorithm.
Assumptions:
is defined on the product space We are able to draw from the conditional distributions
,
where
Algorithm steps:
Select the initial values For repeat:
For sample from distribution Repeat step 2 until the distribution of vector stabilizes.
The subsequent iterations of point 2 will provide a randomization of .
How do we understand point 3?
We are most satisfied when average coordinates stabilize to some accuracy.
Intuition in two-dimensional case:Source: [3]
Gibbs sampling for randomization with a two-dimensional normal distribution.
We will sample from the distribution of , where
and .
Knowing joint density of , it’s easy to show, that:
R implementation:
gibbs_normal_sampling
Have you been to an event recently involving free software or a related topic? How did you find it? Are you organizing an event and don't want to fall into the trap of using Facebook or Meetup or other services that compete for a share of your commun...
An entry about our Bayesian Essentials book on X validated alerted me to a typo in the derivation of the Gaussian posterior..! When deriving the posterior (which was left as an exercise in the Bayesian Core), I just forgot the term expressing the divergence between the prior mean and the sample mean. Mea culpa!!!Filed under: […]
Today we are introducing tibbletime v0.0.2, and we’ve got a ton of new features in store for you. We have functions for converting to flexible time periods with the ~period formula~ and making/calculating custom rolling functions with rollify() (plus...
Last year I spent a bit of time learning about routing and network analysis. I created a map of distance from each GB postcode to the nearest railway station. At the time my… Continue reading →
If you have previously attempted to analyze NFL data, it is likely that you have tried to scrape ESPN or football-reference, which provides a wealth on statistics surrounding game data. However, if you ever wanted to obtain truly in-depth data, then it...
Let’s say you have data containing a categorical variable with 50 levels. When you divide the data into train and test sets, chances are you don’t have all 50 levels featuring in your training set. This often happens when you divide the data set into train and test sets according to the distribution of the … Continue reading Linear / Logistic Regression in R: Dealing With Unknown Factor Levels in Test Data
A little over a year ago, we decided to propose a data visualization course at the first-year level. We had been thinking about this for awhile, but never had the time to teach it given the scheduling constraints we had. … Continue reading →
Election results are in
The New Zealand Electoral Commission have released the final count of votes and seats from the 2017 election. Moving in the direction most sensible observers expected, the National Party lost two seats compared to the “on-th...
It was recently brought to my attention by Hannah Frick that there are now sooo many R-Ladies chapters around the world! R-Ladies is a world-wide organization to promote gender diversity in the R community, and I’m very grateful to be part of this co...
Thanks to Paul Teetor, getSymbols() can now import data from Alpha Vantage!  This feature is part of the quantmod 0.4-11 release, and provides another another data source to avoid any Yahoo Finance API changes*.Alpha Vantage is a free web service that provides real-time and historical equity data.  They provide daily, weekly, and monthly history for both domestic and international markets, with up to 20 years of history. Dividend and split adjusted close prices are available for daily data. They also provide near real-time price bars at a resolution of 1 minute or more, for up to 10 recent days.
All you need to get started is a one-time registration for an API key.  Alpha Vantage has clean, documented, public API that returns either JSON-encoded data or a CSV file.  The arguments to getSymbols.av() closely follow the native API, so be sure to use their documentation!
To get started, install the latest quantmod from CRAN.  Then you call:
  getSymbols(&quot;MSFT&quot;, src = &quot;av&quot;, api.key = &quot;[your key]&quot;) 
Where you replace &quot;[your key&quot;] with the API key you receive after registration.  You can use setDefaults() to set your API key one time, and use it for all getSymbols.av() calls.
  setDefaults(&quot;getSymbols.av&quot;, api.key = &quot;[your key]&quot;)
*Speaking of API changes, this release also includes a fix for a Yahoo Finance change (#174).
I missed last week and almost forgot to add this week’s. Jonah Gabry returned from teaching a one-week course for a special EU research institute in Spain. Mitzi Morris has been knocking out bug fixes for the parser and some pull requests to refactor the underlying type inference to clear the way for tuples, sparse […]
The post Stan Biweekly Roundup, 6 October 2017 appeared first on Statistical Modeling, Causal Inference, and Social Science.
Introduction This is the 1st part of a series of posts I intend to write on some common Machine Learning Algorithms in R and Python. In this first part I cover the following Machine Learning Algorithms Univariate Regression Multivariate Regression Polynomial Regression K Nearest Neighbors Regression The code includes the implementation in both R and … Continue reading Practical Machine Learning with R and Python – Part 1
The members of the R Ladies DC user group put together an R-themed crossword for a recent networking event. It's a fun way to test out your R knowledge. (Click to enlarge, or download a printable version here.) If you get stuck, you can find the answers here or at the link below. And if you'd like to join your local R-Ladies chapter, you can find a list of meetups here. Github (rladies): meetup-presentations_dc / NetworkingCrosswordPuzzle-2017
In my previous tutorial Structural Changes in Global Warming I introduced the strucchange package and some basic examples to date structural breaks in time series. In the present tutorial, I am going to show how dating structural changes (if any) and then Intervention Analysis can help in finding better ARIMA models. Dating structural changes consists […]
Related Post
A Gentle Introduction on Market Basket Analysis — Association Rules
Building A Book Recommender System – The Basics, kNN and Matrix Factorization
Explaining complex machine learning models with LIME
Text Message Classification
Analyzing Google Trends Data in R
Checking residual distributions for non-normal GLMs
Quantile-quantile plots
If you are fitting a linear regression with Gaussian (normally
distributed) errors, then one of the standard checks is to make sure the
residuals are approximately normally d...
In case you missed them, here are some articles from September of particular interest to R users. The mathpix package converts images of hand-drawn equations to their LaTeX equivalent. R 3.4.2 is released. Applying image featurization to the problem of classifying wood knots in lumber. Microsoft ML Server 9.2, which provides operationalization for R, is released. A roundup of news stories related to R from the Ignite conference. A tutorial on launching a Spark cluster with R on HDInsight. A map of pirate attacks, and an ethics lesson on scraping data. A preview of ALTREP, the new vector implementation that...
I just updated simstudy to version 0.1.5 (available on CRAN) so that it now includes several new distributions - exponential, discrete uniform, and negative binomial.
As part of the release, I thought I’d explore the negative binomial just a bit, particularly as it relates to the Poisson distribution. The Poisson distribution is a discrete (integer) distribution of outcomes of non-negative values that is often used to describe count outcomes. It is characterized by a mean (or rate) and its variance equals its mean.
Added variation
In many situations, when count data are modeled, it turns out that the variance of the data exceeds the mean (a situation called over-dispersion). In this case an alternative model is used that allows for the greater variance, which is based on the negative binomial distribution. It turns out that if the negative binomial distribution has mean \(\mu\), it has a variance of \(\mu + \theta \mu^2\), where \(\theta\) is called a dispersion parameter. If \(\theta = 0\), we have the Poisson distribution, but otherwise the variance of a negative binomial random variable will exceed the variance of a Poisson random variable as long as they share the same mean, because \(\mu __ 0\) and \(\theta \ge 0\).
We can see this by generating data from each distribution with mean 15, and a dispersion parameter of 0.2 for the negative binomial. We expect a variance around 15 for the Poisson distribution, and 60 for the negative binomial distribution.
library(simstudy)
library(ggplot2)
# for a less cluttered look
theme_no_minor
This is a somewhat belated introduction of a package that we published on CRAN at the beginning of the year already, but I hadn’t found the time to blog about this earlier. In the R environment and beyond, a large number of packages exist that estimate posterior distributions via MCMC sampling, either for specific statistical models (e.g.…
If you work on natural language processing in a day-to-day setting which involves statistical engineering, at a certain timepoint you need to process your text with a number of text mining procedures of which the following ones are steps you must do before you can get usefull information about your text
Tokenisation (splitting your full text in words/terms)
Parts of Speech (POS) tagging (assigning each word a syntactical tag like is the word a verb/noun/adverb/number/...)
Lemmatisation (a lemma means that the term we &quot;are&quot; is replaced by the verb to &quot;be&quot;, more information: https://en.wikipedia.org/wiki/Lemmatisation)
Dependency Parsing (finding relationships between, namely between &quot;head&quot; words and words which modify those heads, allowing you to look to words which are maybe far away from each other in the raw text but influence each other)
If you do this in R, there aren't much available tools to do this. In fact there are none which
do this for multiple language
do not depend on external software dependencies (java/python)
which also allow you to train your own parsing &amp; tagging models.
Except R package udpipe (https://github.com/bnosac/udpipe, https://CRAN.R-project.org/package=udpipe) which satisfies these 3 criteria.
If you are interested in doing the annotation, pre-trained models are available for 50 languages (see ?udpipe_download_model) for details. Let's show how this works on some Dutch text and what you get of of this.
library(udpipe)dl
Days are getting shorter and the leaves begin to change: autumn is here and this means only one more month until the [R] Kenntnis-Tagen 2017 on November 8 and 9. A diverse agenda, cross-industry networking and a clear business context – many aspects make the [R] Kenntnis-Tage a special event and are reasons for you to … „The [R] Kenntnis-Tage 2017: A special event“ weiterlesen
I’ve been pretty quiet on the blog front recently. That’s because I overhauled my site, migrating it to Hugo (the foundation of blogdown). Just doing one extra thing on top of my usual workload, I also did another thing. I wrote a book too!
I’m a big book fan, and I’m especially a Kindle Unlimited fan (all the books you can read for £8 per month, heck yeah!) so I wanted to make books that I could publish and see on Kindle Unlimited.
One of my favourite activities in R is using Markdown to create business reports. Most of my work I export to MS Word to communicate analytical results with my colleagues. For my academic work and eBooks, I prefer LaTeX to produce … Continue reading →
The post Writing academic articles using R Markdown and LaTeX appeared first on The Devil is in the Data.
A new member has just joined the family of Data Science Virtual Machines on Azure: The Deep Learning Virtual Machine. Like other DSVMs in the family, the Deep Learning VM is a pre-configured environment with all the tools you need for data science and AI development pre-installed. The Deep Learning VM is designed specifically for GPU-enabled instances, and comes with a complete suite of deep learning frameworks including Tensorflow, PyTorch, MXNet, Caffe2 and CNTK. It also comes witth example scripts and data sets to get you started on deep learning and AI problems, including: Jupyter notebooks to compare performance and...
More nzelect New Zealand election data on CRAN
Version 0.4.0 of my nzelect R package is now on CRAN. The key changes from version 0.3.0 are:
election results by voting place are now available back to 2002 (was just 2014)
polling place locations...
This is part 3 of a multi-part guide on working with time series data in R. You can find the previous parts here: Part 1, Part 2. Generated data like that used in Parts 1 and 2 is great for sake of example, but not very interesting to work with. So let’s get some real-world […]
Related Post
Pulling Data Out of Census Spreadsheets Using R
Extracting Tables from PDFs in R using the Tabulizer Package
Extract Twitter Data Automatically using Scheduler R package
An Introduction to Time Series with JSON Data
Get Your Data into R: Import Data from SPSS, Stata, SAS, CSV or TXT
RProtoBuf provides R bindings for the Google Protocol Buffers (&quot;ProtoBuf&quot;) data encoding and serialization library used and released by Google, and deployed fairly widely in numerous projects as a language and operating-system agnostic protocol.
A new releases RProtoBuf 0.4.11 appeared on CRAN earlier today. Not unlike the other recent releases, it is mostly a maintenance release which switches two of the vignettes over to using the pinp package and its template for vignettes.
Changes in RProtoBuf version 0.4.11 (2017-10-03)
The RProtoBuf-intro and RProtoBif-quickref vignettes were converted to Rmarkdown using the templates and style file from the pinp package.
A few minor internal upgrades
CRANberries also provides a diff to the previous release. The RProtoBuf page has copies of the (older) package vignette, the 'quick' overview vignette, a unit test summary vignette, and the pre-print for the JSS paper. Questions, comments etc should go to the GitHub issue tracker off the GitHub repo.
This post by Dirk Eddelbuettel originated on his Thinking inside the box blog. Please report excessive re-aggregation in third-party for-profit settings.
Develop a wide variety of statistical models with R, from the simplest Linear Regression to the most sophisticated GLM models: a two-days live course to master statistics for Data Science, with R
The post R live class – Statistics for Data Science appeared first on Quantide - R training &amp; consulting.
I have a new post on the Stack Overflow blog today about the complex, interrelated ecosystems of software development. On the data team at Stack Overflow, we spend a lot of time and energy thinking about tech ecosystems and how technologies are related to each other. One way to get at this idea of relationships between technologies is tag correlations, how often technology tags at Stack Overflow appear together relative to how often they appear separately. One place we see developers using tags at Stack Overflow is on their Developer Stories. If we are interested in how technologies are connected and how they are used together, developers’ own descriptions of their work and careers is a great place to get that.
I released the data for this network structure as a dataset on Kaggle so you can explore it for yourself! For example, the post for Stack Overflow includes an interactive visualization created using the networkD3 package but we can create other kinds of visualizations using the ggraph package. Either way, trusty igraph comes into play.
library(readr)
library(igraph)
library(ggraph)
stack_network
For many of us data scientists, whatever the tools we use to conduct research or perform an analysis, our superiors are going to want the results as a Microsoft Office document. Most likely it's a Word document or a PowerPoint presentation, it probably has to follow the corporate branding guidelines to boot. The OfficeR package, by David Gohel, addresses this problem by allowing you to take a Word or PowerPoint template and programatically insert text, tables and charts generated by R into the template to create a complete document. (The OfficeR package also represents a leap forward from the similar...
What type of online dashboard will work best for your data? This post reviews eight types of online dashboards to assist you in choosing the...
Recently I’ve been working with a client to help their analysts improve their proficiency with R. A major challenge in engagements like this is figuring...
The post A “Pre-Training” R Survey appeared first on AriLamstein.com.
In an effort to provide context to the frequent mass shootings in the United States, Kieran Healy (Associate Professor of Sociology at Duke University) created this updated chart comparing assault death rates in the US to that of 23 other advanced democracies. The chart shows the rate (per 100,000 citizens) of death caused by assaults (stabbings, gunshots, etc. by a third party). Assaults are used rather than gun deaths specifically, as that's the only statistic for which readily comparable data is available. The data come from the OECD Health Status Database through 2015, the most recent complete year available. The...
Processing R markdown (Rmd) documents with Eclipse/StatET external tools requires a different setup than processing ‘regular’ knitr documents (Rnw). I was having problems getting the whole rmarkdown -__ pandoc workflow working on Eclipse, but the following fix seems to have resolved it, and I can generate Word or HTML documents from a single .Rmd file […]
@drsimonj here to share a ggplot2-based function for plotting path analysis/structural equation models (SEM) fitted with Yves Rosseel’s lavaan package.
 Background
SEM and its related methods (path analysis, confirmatory factor analysis, etc.) can be visualized as Directed Acyclic Graphs with nodes representing variables (observed or latent), and edges representing the specified relationships between them. For this reason, we will use Thomas Lin Pedersen’s tidygraph and ggraph packages. These packages work together to work with relational structures in a tidy format and plot them using ggplot2.
 The function
Below is a function ggsem(), which takes a fitted lavaan object and returns a ggplot2 object representing the nodes, edges, and parameter values. It handles regression paths, correlations, latent factors, and factor loadings.
library(tidyverse)
library(tidygraph)
library(ggraph)
library(lavaan)
# Plot a fitted lavaan object
ggsem % distinct(to) %__% transmute(metric = to, latent = TRUE)
# Node properties
param_nodes % filter(lhs == rhs) %__% transmute(metric = lhs, e = est.std) %__% left_join(latent_nodes) %__% mutate(latent = if_else(is.na(latent), FALSE, latent))
# Complete Graph Object
param_graph % mutate_all(funs((. - mean(.)) / sd(.)))
 Path Analysis
Let’s define a simple path model where diamond price is predicted by its carats, in turn, predicted by its x-axis length.
model
Have you ever wondered what was in your MacDonalds menu? Or in your DoubleCheese Burger (well it’s my favorite one)? A wonderful dataset was released a few months ago, it contains all the nutrition facts from McDonald’s items. You can find the dataset here. In addition to this, I released a new version of D3partitionR a few […]
The post Explore your McDonalds Meal with Shiny and D3partitionR appeared first on Enhance Data Science.
One of rOpenSci’s aims is to build capacity of software users and developers and foster a sense of pride in their work. What better way to do that than to encourage you to participate in Hacktoberfest, a month-long celebration of open source software...
I’ve recently been binge-reading The Guardian Experience columns. I’m a big fan of The Guardian life and style section regulars: the blind dates to which I dedicated a blog post, Oliver Burkeman’s This column will change your life, etc. Experienc...
Market Basket Analysis is one of the key techniques used by large retailers to uncover associations between items. It works by looking for combinations of items that occur together frequently in transactions. To put it another way, it allows retailers to identify relationships between the items that people buy. Association Rules are widely used to […]
Related Post
Building A Book Recommender System – The Basics, kNN and Matrix Factorization
Explaining complex machine learning models with LIME
Text Message Classification
Analyzing Google Trends Data in R
Gradient boosting in R
Although I first came across Jupyter – then IPython – notebooks in October 2012 (I think…), it took me another six months or so before I started playing them and pitched them for the then nascent TM351 course. We decided to explore the notebooks when the course/module team first met around about October 2013. Four […]
Ahh, leaves falling, parents crying, collegicians biking uphill with a bag of in-n-out in between their teeth. Must be the new academic school year! I figured it’s a good time to introduce my work-in-progress datzen package of miscellaneous #rstats functions.  You can bee-line straight to the github readme with more examples. Or stick around and I’ll highlight […]
I am a runner but also a Body Pump enthusiast. Body Pump is a group fitness class of the Les Mills company, in which you train different muscle groups using a weighted bar – whose total weight you modulate with plates in order to adapt it to your fit...
The Internet Archive recently announced a new service they’ve dubbed ‘Third Eye’. This service scrapes the chyrons that annoyingly scroll across the bottom-third of TV news broadcasts. IA has a vast historical archive of TV news that they’ll eventually process, but — for now — the more recent broadcasts from four channels are readily available.... Continue reading →
Zenodo
Zenodo is a repository which allows everybody to deposit free of charge any type of research output, in all disciplines of science.
EFSA is piloting it’s use for creating a knowledge base on all types of food safety related evidence(data, doc...
Modern websites are complex beasts. They house photo galleries, interactive visualizations, web fonts, analytics code and other diverse types of content. Despite the potential for diversity, many web sites share similar “tech stacks” — the components that come together to make them what they are. These stacks consist of web servers (often with special capabilities),... Continue reading →
For this final post, I will cover some advanced topics and discuss how to use data tables within user generated functions. Once again, let’s use the Chicago crime data. Let’s start by subseting the data. The following code takes the first 50000 rows within the dat dataset, selects four columns, creates three new columns pertaining […]
I am pleased to announce that tidytext 0.1.4 is now on CRAN!
This release of our package for text mining using tidy data principles has an excellent collection of delightfulness in it. First off, all the important functions in tidytext now support supp...
This post will be about implementing and investigating the running Kelly Criterion — that is, a constantly adjusted Kelly Criterion … Continue reading →
One of the assumptions of Classical Linear Regression Model is that there is no exact collinearity between the explanatory variables. If the explanatory variables are perfectly correlated, you will face with these problems: Parameters of the model become indeterminate Standard errors of the estimates become infinitely large However, the case of perfect collinearity is very […]
Related Post
Scikit-Learn for Text Analysis of Amazon Fine Food Reviews
Survival Analysis – Part I
Linear Regression from Scratch in Python
Generalized Additive Models
Second step with non-linear regression: adding predictors
If you want to rapidly master data science, you need to ...
The post How to rapidly master data science appeared first on SHARP SIGHT LABS.
Downloading Quarterly Financial Reports from Bovespa - Financial statements of companies traded at B3 (formerly Bovespa), the
Brazilian stock exchange, are available in its
website. Accessing the data for a
single company is strai...
Motivation Before trying a new restaurant, we frequently consult with review platforms, such as Yelp, Zomato, or Google, where we can read comments from previous diners. […]
The post Real time Yelp reviews analysis and response solutions for restaurant owners appeared first on NYC Data Science Academy Blog.
R 3.4.2 (codename “Short Summer”) was released yesterday. You can get the latest binaries version from here. (or the .tar.gz source code from here). As mentioned by David Smith, R 3.4.2 includes a performance improvement for names: c() and unlist() are now more efficient in constructing the names(.) of their return value, thanks to a proposal by Suharto Anggono. (PR#17284) The full list of bug … Continue reading &quot;R 3.4.2 is released (with several bug fixes and a few performance improvements)&quot;
Statistics involves a lot of mathematics, so one of the nice things about report-generation systems for R like Rmarkdown is that it makes it easy to include nicely-formatted equations by using the LaTeX syntax. So, if we want to include the density function of the Guassian Normal distribution: $$ \frac{1}{{\sigma \sqrt {2\pi } }} e^ { - \frac{ - \left( {x - \mu } \right)^2 }{2\sigma ^2} } $$ we can just add the following markup to the document: \[ \frac{1}{{\sigma \sqrt {2\pi} }} e^{-\frac{-(x-\mu)^2}{2\sigma ^2}} \] Creating that markup can be a little tricky, but it generally follows naturally...
Organizations are nowadays storing huge amounts of data related to various business processes. Process mining provides different methods and techniques to analyze and improve these processes. This allows companies to gain a competitive advantage. Process mining initiated with the discovery … Continue reading →
In my recent post, My travels through the realms of Data Science, Machine Learning, Deep Learning and (AI), I had recounted my journey in the domains of of Data Science, Machine Learning (ML), and more recently Deep Learning (DL) all of which are useful while analyzing data. Of late, I have come to the realization that … Continue reading Using Linear Programming (LP) for optimizing bowling change or batting lineup in T20 cricket
The thirteenth release in the 0.12.* series of Rcpp landed on CRAN this morning, following a little delay because Uwe Ligges was traveling and whatnot. We had announced its availability to the mailing list late last week. As usual, a rather substanti...
Banaue rice terraces. Photo: Jon Rawlinson In a previous article, we showed the use of partial pooling, or hierarchical/multilevel models, for level coding high-cardinality categorical variables in vtreat. In this article, we will discuss a little more about the how and why of partial pooling in R. We will use the lme4 package to fit … Continue reading Partial Pooling for Lower Variance Variable Encoding
The R Core team today announced the release of R 3.4.2. This release fixes a number of minor bugs and also includes a performance improvement to the commonly-used function c when applied to vectors with a names attribute. Like all minor releases, this release is backwards compatible with prior releases in the R 3.4.x series. Binary builds of R 3.4.2 will begin to appear on your local CRAN mirror within the next day or so. This release also marks the final Windows build of R by Duncan Murdoch, who has been volunteering his time to the R Core group for...
Week 4 Gold Mining and Fantasy Football Projection Roundup now available. Go get that free agent gold!
The post Gold-Mining – Week 4 (2017) appeared first on Fantasy Football Analytics.
The goodness of fit for data reduction techniques such as MDS and t-SNE can be easily assessed with Shepard diagrams. A Shepard diagram compares how far apart your data points are before and after you transform...
Yesterday and today I attended the data2day, a conference about Big Data, Machine Learning and Data Science in Heidelberg, Germany. Topics and workshops covered a range of topics surrounding (big) data analysis and Machine Learning, like Deep Learnin...
I occasionally hang out on StackOverflow and often use an answer as an opportunity to fill a package void for a particular need. docxtractr and qrencoder are two (of many) packages that were birthed from SO answers. I usually try to answer with inline code first then expand the functionality into a package (if warranted).... Continue reading →
This is the last post in the series of articles about using Multi-Channel Attribution in marketing. In previous two articles (part 1 and part 2), we’ve reviewed a simple and powerful approach based on Markov chains that allows you to effectively attribute marketing channels. In this article, we will review another fascinating approach that marries heuristic […]
The post Marketing Multi-Channel Attribution model based on Sales Funnel with R appeared first on AnalyzeCore - data is beautiful, data is a story.
by Bob Horton and Vanja Paunic, Microsoft AI and Research Data Group Training deep learning models from scratch requires large data sets and significant computational reources. Using pre-trained deep neural network models to extract relevant features from images allows us to build classifiers using standard machine learning approaches that work well for relatively small data sets. In this context, a deep learning solution can be thought of as incorporating layers that compute features, followed by layers that map these features to outcomes; here we’ll just map the features to outcomes ourselves. We explore an example of using a pre-trained deep...
A maintenance release of RcppZiggurat is now on the CRAN network for R. It switched the vignette to the our new pinp package and its two-column pdf default.
The RcppZiggurat package updates the code for the Ziggurat generator which provides very fas...
This tutorial was inspired by a this post published at DataScience+ by Bidyut Ghosh. Special thanks also to Dani Navarro, The University of New South Wales (Sydney) for the book Learning Statistics with R (hereafter simply LSR) and the lsr packages available through CRAN. I highly recommend it. Let’s load the required R packages library(ggplot2) […]
Related Post
One-way ANOVA in R
Cubic and Smoothing Splines in R
Chi-Squared Test – The Purpose, The Math, When and How to Implement?
Missing Value Treatment
R for Publication by Page Piccinini
For R users interested in Machine Learning, you can attend our upcoming course on Machine Learning with R which is scheduled on 16-17 October 2017 in Leuven, Belgium. This is now the 4th year this course is given at the university of Leuven so we made quite some updates since the first time this was given 4 years ago.
During the course you'll learn the following techniques from a methodological as well as a practical perspective: naive bayes, trees, feed-forward neural networks, penalised regression, bagging, random forests, boosting and if time permits graphical lasso, penalised generalised additive models, support vector machines.
Subscribe here: https://lstat.kuleuven.be/training/coursedescriptions/statistical-machine-learning-with-r
For a full list of training courses provided by BNOSAC - either in-house or in-public: go to http://www.bnosac.be/training
For R users interested in text mining with R, applied spatial modelling with R, advanced R programming or computer vision, you can also subscribe for the following courses, scheduled at the University of Leuven.
16-17/10/2017: Statistical machine learning with R. Leuven (Belgium). Subscribe here
08+10/11/2017: Text mining with R. Leuven (Belgium). Subscribe here
19-20/12/2017: Applied spatial modelling with R. Leuven (Belgium). Subscribe here
20-21/02/2018: Advanced R programming. Leuven (Belgium). Subscribe here
08-09/03/2018: Computer Vision with R and Python. Leuven (Belgium). Subscribe here
22-23/03/2018: Text Mining with R. Leuven (Belgium). Subscribe here
 
 
Hi there! We proud to launch our latest R &amp; machine learning course, Supervised Learning in R: Classification! By Brett Lantz.
This beginner-level introduction to machine learning covers four of the most common classification algorithms. You will ...
Sometimes we don’t even realize how common machine learning (ML) is in our daily lives. Various “intelligent” algorithms help us for instance with finding the most important facts (Google), they suggest what movie to watch (Netflix), or influence our shopping decisions (Amazon). The biggest international companies quickly recognized the potential of machine learning and transferred it to business solutions.
Nowadays not only big companies are able to use ML. Imagine — not so abstract — situation when a company tries to predict customer behavior based on some personal data. Just a few years ago, the best strategy to solve this problem would be to hire a good data science team. Nowadays, thanks to growing ML popularity, it is available even for small start-ups.
Today, I would like to present you a demo of how to solve difficult business problems with ML. We will take advantage of mljar.com service and its R API. With just a few lines of code we will be able to achieve very good results.
Microsoft R Server has received a new name and a major update: Microsoft ML Server 9.2 is now available. ML Server provides a scalable production platform for R — and now Python — programs. The basic idea is that a local client can push R or Python code and have it operationalized on the remote server. ML Server is also included with the Data Science Virtual Machine and HDInsight Spark clusters on Azure. This video gives a high-level overview of the process, or you can also see details of deploying an R model or a Python model as a web...
A few short weeks after the more substantial 0.0.9 release of RcppAnnoy, we have a quick bug-fix update.
RcppAnnoy is our Rcpp-based R integration of the nifty Annoy library by Erik. Annoy is a small and lightweight C++ template header library for ve...
In part one, I provided an initial walk through of some nice features that are available within the data.table package. In particular, we saw how to filter data and get a count of rows by the date. Let us now add a few columns to our dataset on reported crimes in the city of Chicago. […]
Analysis of the effects of the Graham-Cassidy Bill on the ACA population - I have been trying to decipher for myself, what is in the current (well, yesterday’s) Graham-Cassidy health care bill. I saw this image on many news outlets a few da...
Microsoft Cognitive Services Vision API in R A little while ago I did a brief tutorial of the Google Vision API using RoogleVision created by Mark Edmonson. I couldn’t find anything similar to that in R for the Microsoft Cognitive Services API so I thought I would give it a shot. I whipped this example […]
Atmospheric air pollution is one of the most important environmental concerns in many countries around the world, and it is strongly affected by meteorological conditions. Accordingly, in this set of exercises we use openair package to work and analyze air quality and meteorological data. This packages provides tools to directly import data from air quality […]
Related exercise sets:
Working with air quality and meteorological data Exercises (Part-3) Working with air quality and meteorological data Exercises (Part-2) Working with air quality and meteorological data Exercises (Part-1) Explore all our (__1000) R exercises
Find an R course using our R Course Finder directory
regexSelect is a small package that uses Shiny modules to solve a problem in Shiny selectize objects - regular expression (regex) searching. You can quickly filter the values in the selectize object, while being able to add that new regex que...
R for Data Science is our first course of the autumn term. It takes place in October 4-5 in a location close to Milano Lima.
If you want to deepen your data analysis knowledge using the most modern R tools, or you want to figure out if R is the right solution for you, this is the your class. This course is for people that are willing to learn about R and would like to get an overview of its capabilities for data science.
The post R live class | R for Data Science | Oct 4-5 Milan appeared first on Quantide - R training &amp; consulting.
To post your R job on the next post Just visit  this link and  post  a new R job  to the R community. You can post a job for  free  (and there are also “featured job” options available for extra exposure). Current R jobs Job seekers:  please follow the links below to learn more and apply for your R job of interest: Featured Jobs Full-Time Data Scientist Edelweiss Business Services  – Posted by  Aakash Gupta Mumbai Maharashtra, India 20 Sep  2017 Full-Time Solutions Engineer for RStudio @ U.S.A. (or anywhere) RStudio  – Posted by  nwstephens Anywhere 8 Sep  2017 More New Jobs Freelance R Programmer &amp; Statistician Empiricus – Posted by  empiricus Anywhere 25 Sep 2017 Full-Time Data Scientist Edelweiss Business Services  – Posted by  Aakash Gupta Mumbai Maharashtra, India 20 Sep 2017 Full-Time Postdoctoral Data Scientist: GIS and mHealth Harvard T.H. Chan School of Public Health  – Posted by  Christine Choirat Boston Massachusetts, United States 14 Sep 2017 Full-Time Data Scientist @ London Hiscox  – Posted by  alan.south London England, United Kingdom 13 Sep 2017 Freelance Crafty developer: R->C++ SherandoResearch  – Posted by  feersum_monkey Anywhere 8 Sep 2017 Full-Time Solutions Engineer for RStudio @ U.S.A. (or anywhere) RStudio  – Posted by  nwstephens […]
For many years, I actively avoided the data.table package and preferred to utilize the tools available in either base R or dplyr for data aggregation and exploration. However, over the past year, I have come to realize that this was a mistake. Data tables are incredible and provide R users with a syntatically concise and […]
An interactive infographic can be used to communicate a lot of information in an engaging way. With the right tools, they are also relatively straightforward to create. In this post, I show step-by-step how to...
It's been a big day for the team here at Microsoft, with a flurry of announcements from the Ignite conference in Orlando. We'll provide more in-depth details in the coming days and weeks, but for now here's a brief roundup of the news related to data science: Microsoft ML Server 9.2 is now available. This is the new name for what used to be called Microsoft R Server, and now also includes support for operationalizing the Python language as well as R. This 2-minute video explains how to deploy models with ML Server (and with this release, real-time scoring is...
One of the services that the R package vtreat provides is level coding (what we sometimes call impact coding): converting the levels of a categorical variable to a meaningful and concise single numeric variable, rather than coding them as indicator variables (AKA &quot;one-hot encoding&quot;). Level coding can be computationally and statistically preferable to one-hot encoding … Continue reading Custom Level Coding in vtreat
In Part 1 of this series, we got started by looking at the ts object in R and how it represents time series data. In Part 2, I’ll discuss some of the many time series transformation functions that are available in R. This is by no means an exhaustive catalog. If you feel I left […]
Related Post
Time Series Analysis in R Part 1: The Time Series Object
Parsing Text for Emotion Terms: Analysis &amp; Visualization Using R
Using MongoDB with R
Finding Optimal Number of Clusters
Analyzing the first Presidential Debate
spiderbar, spiderbar Reads robots rules from afar. Crawls the web, any size; Fetches with respect, never lies. Look Out! Here comes the spiderbar. Is it fast? Listen bud, It's got C++ under the hood. Can you scrape, from a site? Test with can_fetch(), TRUE == alright Hey, there There goes the spiderbar. (Check the end... Continue reading →
We at Appsilon are excited about RStudio introducing promises in R quite soon which is going to be a huge step forward in programming in R (we have already used futures and similar libraries to run code asynchronously, however this is going to be a sta...
eXtreme Gradient Boosting is a machine learning model which became really popular few years ago after winning several Kaggle competitions. It is very powerful algorithm that use an ensemble of weak learners to obtain a strong learner. Its R implementation is available in xgboost package and it is really worth including into anyone’s machine learning […]
Related exercise sets:
Model Evaluation 2 How to prepare and apply machine learning to your dataset Hacking statistics or: How I Learned to Stop Worrying About Calculus and Love Stats Exercises (Part-5) Explore all our (__1000) R exercises
Find an R course using our R Course Finder directory
A maintenance update RcppGSL 0.3.3 is now on CRAN. It switched the vignette to the our new pinp package and its two-column pdf default.
The RcppGSL package provides an interface from R to the GNU GSL using the Rcpp package.
No user-facing new code or...
A new version of the RcppCNPy package arrived on CRAN yesterday.
RcppCNPy provides R with read and write access to NumPy files thanks to the cnpy library by Carl Rogers.
This version updates internals for function registration, but otherwise mostly s...
A bug-fix release RcppClassic 0.9.8 for the very recent 0.9.7 release which fixes a build issue on macOS introduced in 0.9.7. No other changes.
Courtesy of CRANberries, there are changes relative to the previous release.
Questions, comments etc shoul...
Statistics are often taught in school by and for people who like Mathematics. As a consequence, in those class emphasis is put on leaning equations, solving calculus problems and creating mathematics models instead of building an intuition for probabilistic problems. But, if you read this, you know a bit of R programming and have access […]
Related exercise sets:
Hacking statistics or: How I Learned to Stop Worrying About Calculus and Love Stats Exercises (Part-8) Hacking statistics or: How I Learned to Stop Worrying About Calculus and Love Stats Exercises (Part-6) Hacking statistics or: How I Learned to Stop Worrying About Calculus and Love Stats Exercises (Part-4) Explore all our (__1000) R exercises
Find an R course using our R Course Finder directory
I am pleased to announce that vtreat version 0.6.0 is now available to R users on CRAN. vtreat is an excellent way to prepare data for machine learning, statistical inference, and predictive analytic projects. If you are an R user we strongly suggest you incorporate vtreat into your projects. vtreat handles, in a statistically sound … Continue reading Upcoming data preparation and modeling article series
By Gabriel Vasconcelos Regression Trees In this post I am going to discuss some features of Regression Trees an Random Forests. Regression Trees are know to be very unstable, in other words, a small change in your data may drastically … Continue reading →
Welcome everybody, we are proud to introduce the brand new web page and
blog http://www.R-exams.org/. This provides a central access point for
the open-source software “exams” implemented in the
R system for statistical computing.
R/exams is a one-...
We have dabbled with RevoScaleR before , In this exercise we will work with H2O , another high performance R library which can handle big data very effectively .It will be a series of exercises with increasing degree of difficulty . So Please do this in sequence . H2O requires you to have Java installed […]
Related exercise sets:
Big Data analytics with RevoScaleR Exercises-2 Data wrangling : I/O (Part-1) Density-Based Clustering Exercises Explore all our (__1000) R exercises
Find an R course using our R Course Finder directory
If you'd like to get started using R with Spark, you'll need to set up a Spark cluster and install R and all the other necessary software on the nodes. A really easy way to achieve that is to launch an HDInsight cluster on Azure, which is just a managed Spark cluster with some useful extra components. You'll just need to configure the components you'll need, in our case R and Microsoft R Server, and RStudio Server. This tutorial explains how to launch an HDInsight cluster for use with R. It explains how to size the cluster and launch the...
t-SNE is a very powerful technique that can be used for visualising (looking for patterns) in multi-dimensional data. Great things have been said about this technique. In this blog post I did a few experiments with t-SNE in R to learn about this technique and its uses. Its power to visualise complex multi-dimensional data is […]
Related Post
Comparing Trump and Clinton’s Facebook pages during the US presidential election, 2016
Analyzing Obesity across USA
Can we predict flu deaths with Machine Learning and R?
Graphical Presentation of Missing Data; VIM Package
Creating Graphs with Python and GooPyCharts
[edit: Juho Kokkala corrected my homework. Thanks! I updated the post. Also see some further elaboration in my reply to Andrew’s comment. As Andrew likes to say …] So far, Giancarlo Stanton has hit 56 home runs in 555 at bats over 149 games. Miami has 10 games left to play. What’s the chance he’ll […]
The post Will Stanton hit 61 home runs this season? appeared first on Statistical Modeling, Causal Inference, and Social Science.
There are substantial differences between ad-hoc analyses (be they: machine learning research, data science contests, or other demonstrations) and production worthy systems. Roughly: ad-hoc analyses have to be correct only at the moment they are run (and often once they are correct, that is the last time they are run; obviously the idea of reproducible … Continue reading My advice on dplyr::mutate()
This past Tuesday was Talk Like A Pirate Date, the unofficial holiday of R (aRRR!) users worldwide. In recognition of the day, Bob Rudis used R to create this map of worldwide piracy incidents from 2013 to 2017. The post provides a useful and practical example of extracting data from a website without an API, otherwise known as &quot;scraping&quot; data. In this case the website was the International Chamber of Commerce, and the provided R code demonstrates several useful R packages for scraping: rvest, for extracting data from a web pages' HTML source purrr, and specifically the safely function, for...
Exploratory Data Analysis of Tropical Storms in R The disastrous impact of recent hurricanes, Harvey and Irma, generated a large influx of data within the online community. I was curious about the history of hurricanes and tropical storms so I found a data set on data.world and started some basic Exploratory data analysis (EDA). EDA […]
Week 3 Gold Mining and Fantasy Football Projection Roundup now available. Go get that free agent gold!
The post Gold-Mining – Week 3 (2017) appeared first on Fantasy Football Analytics.
R has a lot of packages for users to analyse posts on social media. As an experiment in this field, I decided to start with the biggest one: Facebook. I decided to look at the Facebook activity of Donald Trump and Hillary Clinton during the 2016 presidential election in the United States. The winner may […]
Related Post
Analyzing Obesity across USA
Can we predict flu deaths with Machine Learning and R?
Graphical Presentation of Missing Data; VIM Package
Creating Graphs with Python and GooPyCharts
Visualizing Tennis Grand Slam Winners Performances
Well I walk upon the river like it’s easier than land (Love is All, The Tallest Man on Earth) The Metropolitan Museum of Art provides here a dataset with information on more than 450.000 artworks in its collection. You can do anything you want with these data: there are no restrictions of use. Each record … Continue reading Visualizing the Spanish Contribution to The Metropolitan Museum of Art →
Editors Note: It has been heartbreaking watching the images from México City. Teresa Ortiz, co-organizer of R-Ladies CDMX reports on efforts of data scientists to help. Our thoughts are with them, and with the people of México.
It has been a hard couple of days around here.
In less than 2 weeks, México has gone through two devastating earthquakes and the damages keep adding. Nevertheless, the response from the citizens has been outstanding and Mexican data-driven initiatives have not stayed behind.
An example is codeandoMexico, an open innovation platform which brings together top talent through public challenges, codeandoMexico developed a Citizen’s quick response center with a directory of emergency phone numbers, providing the location of places that need help, maps, and more. Most of the information in the site is updated from first-hand experience of volunteers that are helping throughout the affected areas. Technical collaboration with codeandoMexico is open to everyone by attending to issues on their website these include extending the services offered by the site or attending to bugs.
Social networks are being extensively used to spread information about where and how to help, however often times tweets and facebook posts are fake or obsolete. There are several efforts taking place to overcome this difficulty; R-Ladies CDMX’s cofounder Silvia Gutierrez is working in one such project to map information on shelters, affected buildings and more (http://bit.ly/2xfIwZm).
The full scope of the damage is yet to be seen, but the support, both from México and other countries, is encouraging.
Euler Problem 32 returns to pandigital numbers, which are numbers that contain one of each digit. Like so many of the Euler Problems, these numbers serve no practical purpose whatsoever, other than some entertainment value. You can find all pandigital … Continue reading →
The post Pandigital Products: Euler Problem 32 appeared first on The Devil is in the Data.
Changes are coming to the internals of the R engine which promise to improve performance and reduce memory use, with dramatic impacts in some circumstances. The changes were first proposed by Gabe Becker at the DSC Conference in 2016 (and updated in 2017), and the implementation by Luke Tierney and Gabe Becker is now making its way into the development branches of the R sources. ALTREP is an alternative way of representing R objects internally. (The details are described in this draft vignette, and there are a few more examples in this presentation.) It's easiest to illustrate by example. Today,...
This is the second exercise set on answering probability questions with simulation. Finishing the first exercise set is not a prerequisite. The difficulty level is about the same – thus if you are looking for a challenge aim at writing up faster more elegant algorithms. As always, it pays off to read the instructions carefully […]
Related exercise sets:
Answer probability questions with simulation Probability functions intermediate Lets Begin with something sample Explore all our (__1000) R exercises
Find an R course using our R Course Finder directory
D3partitionR is an R package to visualize interactively nested and hierarchical data using D3.js and HTML widget. These last few weeks I’ve been working on a major D3partitionR update which is now available on GitHub. As soon as enough feedbacks are collected, the package will be on uploaded on the CRAN. Until then, you can […]
The post Major update of D3partitionR: Interactive viz’ of nested data with R and D3.js appeared first on Enhance Data Science.
Today, I have given a webinar for the Applied Epidemiology Didactic of the University of Wisconsin - Madison titled “From Biology to Industry. A Blogger’s Journey to Data Science.”
I talked about how blogging about R and Data Science helped me b...
A first update 0.0.2 of the pinp package arrived on CRAN just a few days after the initial release.
We added a new vignette for the package (see below), extended a few nice features, and smoothed a few corners.
The NEWS entry for this release f...
I was just going to make a quick announcement to let folks know that I’ve updated the simstudy package to version 0.1.4 (now available on CRAN) to include functions that allow conversion of columns to factors, creation of dummy variables, and most importantly, specification of outcomes that are more flexibly conditional on previously defined variables. But, as I was coming up with an example that might illustrate the added conditional functionality, I found myself playing with package flexmix, which uses an Expectation-Maximization (EM) algorithm to estimate latent classes and fit regression models. So, in the end, this turned into a bit more than a brief service announcement.
Defining data conditionally
Of course, simstudy has always enabled conditional distributions based on sequentially defined variables. That is really the whole point of simstudy. But, what if I wanted to specify completely different families of distributions or very different regression curves based on different individual characteristics? With the previous version of simstudy, it was not really easy to do. Now, with the addition of two key functions, defCondition and addCondition the process is much improved. defCondition is analogous to the function defData, in that this new function provides an easy way to specify conditional definitions (as does defReadCond, which is analogous to defRead). addCondition is used to actually add the data column, just as addColumns adds columns.
It is probably easiest to see in action:
library(simstudy)
# Define baseline data set
def
Beth Ashlee, Data Scientist
After a successful first-ever EARL San Francisco in June, it was time to head back to the birth place EARL - London. With more abstracts subm...
The above title was the title of my talk this evening at our Bay Area R Users Group. I had been asked to talk about my new book, and I presented four of the myths that are dispelled in the book. Hadley also gave an interesting talk, “An introduction to tidy evaluation,” involving some library … Continue reading Regression Analysis — What You Should’ve Been Taught But Weren’t, and Were Taught But Shouldn’t Have Been →
Infographics, dashboards, and reports often need to highlight or visualize a single number. But how do you highlight a single number so that it has an impact and looks good? It can be a big...
Hi! We have added R API for mljar - so you can run sklearn, xgboost, lightGBM, Keras, RGF from one R line :) Please check it on https://github.com/mljar/mljar-api-R
International Code Talk Like A Pirate Day almost slipped by without me noticing (September has been a crazy busy month), but it popped up in the calendar notifications today and I was glad that I had prepped the meat of a post a few weeks back. There will be no ‘rrrrrr’ abuse in this post,... Continue reading →
Mango Solutions are delighted to announce the full EARL Boston agenda for this coming November!
As always, we received more abstracts than we could accept and all were o...
The USGS has followed up their visualization of Hurricane Harvey rainfalls with an updated version of the animation, this time showing the rain and flooding from Hurricane Irma in Florida: Another #rstats #dataviz! Precip and #flooding from #HurricaneIrma 💧 #opensource code: https://t.co/rpocPQe7zR #openscience pic.twitter.com/rGX1SNiYEM — USGS R Community (@USGS_R) September 15, 2017 This iteration improves the Harvey version by displaying rainfall in a fine grid over the state rather than on a county-by-county basis. Once again you can find the R code and data on Github, and an interactive version of the chart is available at the link below. USGS...
Many of the methods used in time series analysis and forecasting have been around for quite some time but have taken a back seat to machine learning techniques in recent years. Nevertheless, time series analysis and forecasting are useful tools in any data scientist’s toolkit. Some recent time series-based competitions have recently appeared on kaggle, […]
Related Post
Parsing Text for Emotion Terms: Analysis &amp; Visualization Using R
Using MongoDB with R
Finding Optimal Number of Clusters
Analyzing the first Presidential Debate
GoodReads: Machine Learning (Part 3)
I have written a blog post about why I love R and prefer it to other languages. The post is on my new site, but since it isn’t on R-bloggers yet I am also posting the link here:
Working in Data Science, I often feel like I have to justify using R...
Making a line chart in ggplot2 is pretty straightforward, if you know how it works ...
The post How to make a line chart with ggplot2 appeared first on SHARP SIGHT LABS.
International Talk Like A Pirate Day
Q: What has 8 legs, 8 arms and 8 eyes?
A: 8 pirates.
Avast, ye scurvy scum! Today be September 19, the International Talk Like A Pirate Day. While it’s a silly “holiday”, it’s a great chance to ...
Take the mystery out of CRAN level package maintenance - Thinking of creating a new package? Dread the task of function documentation? Afraid to run devtools::check(build_args = '--as-cran') and get bombarded by Errors, Warnings, and Notes (o...
The fourth EARL London conference took place last week, and once again it was an enjoyable and informative showcase of practical applications of R. Kudos to the team from Mango for hosting a great event featuring interesting talks and a friendly crowd. As always, there were more talks on offer than I was able to attend (most of the event was in three parallel tracks), but here are a few highlights that I was able to catch: Jenny Bryan from RStudio gave an outstanding keynote: &quot;Workflows: You Should Have One&quot;. The talk was a really useful collection of good practices...
I was socially engineered by @yoniceedee into creating today’s post due to being prodded with this tweet: Where to see the best fall foliage, based on your location: https://t.co/12pQU29ksB pic.twitter.com/JiywYVpmno— Vox (@voxdotcom) September 18, 2017 Since there aren’t nearly enough sf and geom_sf examples out on the wild, wild #rstats web, here’s a short one... Continue reading →
Employee turnvover (attrition) is a major cost to an organization, and predicting turnover is at the forefront of needs of Human Resources (HR) in many organizations. Until now the mainstream approach has been to use logistic regression or survival cur...
Invitation to the 3rd Big Data Analytics World Championships 2017 We invite members of the R-Bloggers/R-Users community to participate in the 2017 TEXATA Big Data Analytics World Championships (www.texata.com).  You’re Free Discount Code is: 2017RUSERS (normally $30 fully paid entry).   Here are important dates for the TEXATA educational business competition: Round 1 on 30th September 2017 (Online) Round … Continue reading Big Data Analytics World Championship (30th September 2017) – Free Entry Code and details
As mentioned in the previous post (https://statcompute.wordpress.com/2017/06/29/model-operational-loss-directly-with-tweedie-glm/), we often need to model non-negative numeric outcomes with zeros in the operational loss model development. Tweedie GLM provides a convenient interface to model non-negative losses directly by assuming that aggregated losses are the Poisson sum of Gamma outcomes, which however might not be well supported empirically from […]
This is common case when working with data that your source is a remote database. Usual ways to cope this when using R is either to load all the data into R or to perform the heaviest joins and aggregations with SQL before loading the data. Both of them have cons: the former one is […]
Related exercise sets:
R with remote databases Exercises (Part-1) Descriptive Analytics-Part 5: Data Visualisation (Categorical variables) Descriptive Analytics-Part 4 : Data Manipulation Explore all our (__1000) R exercises
Find an R course using our R Course Finder directory
A rather boring and otherwise uneventful release 0.9.7 of RcppClassic is now at CRAN. This package provides a maintained version of the otherwise deprecated first Rcpp API; no new projects should use it.
Once again no changes in user-facing code. But...
A puzzling Le Monde mathematical puzzle for which I could find no answer in the allotted time!: A most democratic electoral system allows every voter to have at least one representative by having each of the N voters picking exactly m candidates among the M running candidates and setting the size n of the representative […]
A brandnew and very exciting (to us, at least) package called pinp just arrived on CRAN, following a somewhat unnecessarily long passage out of incoming. It is based on the PNAS LaTeX Style offered by the Proceeding of the National Academy of Science...
A new version of drat arrived earlier today on CRAN as another no-human-can-delay-this automatic upgrade directly from the CRAN prechecks. It is mostly a maintenance release ensuring PACKAGES.rds is also updated, plus some minor other edits.
drat sta...
On this set of exercises, we are going to use the lm and glm functions to perform several generalized linear models on one dataset. Since this is a basic set of exercises we will take a closer look at the arguments of these functions and how to take advantage of the output of each function […]
Related exercise sets:
Forecasting: Linear Trend and ARIMA Models Exercises (Part-2) Parallel Computing Exercises: Snow and Rmpi (Part-3) Density-Based Clustering Exercises Explore all our (__1000) R exercises
Find an R course using our R Course Finder directory
Creating a meaningful visualization from data with long lists can be challenging. While word clouds are often the popular choice, they are not always the best option. This post illustrates seven alternatives to word clouds...
House effects in state space models of voting intention
“House effects” is the term commonly used in models of voting intention to refer to the different statistical biases of different pollsters. In my Model B state space model of the New Zealan...
I’ve blathered about trust before 1 2, but said blatherings were in a “what if” context. Unfortunately, the if has turned into a when, which begged for further blathering on a recent FOSS ecosystem cybersecurity incident. The gg_spiffy @thomasp85 linked to a post by the SK-CSIRT detailing the discovery and take-down of a series of... Continue reading →
I’m working on the next update of my sjPlot-package, which will get a generic plot_model() method, which plots any kind of regression model, with different plot types being supported (forest plots for estimates, marginal effects and predictions, including displaying interaction terms, …). The package also supports rstan resp. rstanarm models. Since these are typically presented […]
INTRODUCTION Dear reader, If you are a newbie in the world of machine learning, then this tutorial is exactly what you need in order to introduce yourself to this exciting new part of the data science world. This post includes a full machine learning project that will guide you step by step to create a […]
Related exercise sets:
How to prepare and apply machine learning to your dataset Summarizing dataset to apply machine learning – exercises Visualizing dataset to apply machine learning-exercises Explore all our (__1000) R exercises
Find an R course using our R Course Finder directory
Microsoft R Open (MRO), Microsoft's enhanced distribution of open source R, has been upgraded to version 3.4.1 and is now available for download for Windows, Mac, and Linux. This update upgrades the R language engine to R 3.4.1 and updates the bundled packages. MRO is 100% compatible with all R packages. MRO 3.4.1 points to a fixed CRAN snapshot from September 1 2017, and you can see some highlights of new packages released since MRO 3.4.0 on the Spotlights page. As always you can use the built-in checkpoint package to access packages from an earlier date (for compatibility) or a...
R functions can be used to create chart templates, which keep the look and feel of reports consistent. This post gives step by step guide on how to create chart templates using R functions. R...
A collection of liars in this Le Monde mathematical puzzle: A circle of 16 liars and truth-tellers is such that everyone states that their immediate neighbours are both liars. How many liars can there be? A circle of 12 liars and truth-tellers is such that everyone state that their immediate neighbours are one liar plus […]
Most R users are quite familiar with data frames: the data.frame is the fundamental object type for working with columnar data in R. But for SQL Server users, the data frame is an important concept to understand, since it will be the main object type in R used to store data from SQL tables. This guide to working with data frames with SQL Server R Services provides the basic concepts of R data frames and how to generate them and manipulate them within a SQL Server procedure. If you find that article useful, you might also find these prior articles...
The latest version of GetTDData offers function get.yield.curve to
download the current Brazilian yield curve directly from
Anbima. The yield curve is
a financial tool that shows how, based on current prices of fixed income
instru...
Week 2 Gold Mining and Fantasy Football Projection Roundup now available. Go get that free agent gold!
The post Gold-Mining – Week 2 (2017) appeared first on Fantasy Football Analytics.
In this set of exercises, we are going to explore some applications of probability functions and how to plot some density functions. The package MASS will be used in this set. Note: We are going to use random numbers functions and random processes functions in R such as runif. A problem with these functions is […]
Related exercise sets:
Probability functions intermediate Hacking statistics or: How I Learned to Stop Worrying About Calculus and Love Stats Exercises (Part-4) Lets Begin with something sample Explore all our (__1000) R exercises
Find an R course using our R Course Finder directory
The ggpubr R package facilitates the creation of beautiful ggplot2-based graphs for researcher with non-advanced programming backgrounds.
The current material presents a collection of articles for simply creating and customizing publication-ready...
A new minor release of RcppRedis arrived on CRAN last week, following the release 0.2.0 of RcppMsgPack which brought the MsgPack headers forward to release 2.1.5. This required a minor and rather trivial change in the code. When the optional RcppMsgP...
It’s been a long time coming but I finally moved my blog from Jekyll/Bootstrap on Github pages to blogdown, Hugo and Netlify! Moreover, I also now have my own domain name www.shirin-glander.de. :-)
I followed the blogdown ebook to set up my blog. I ...
In this article I am going to talk about the non-parametric techniques used for survival analysis. To comprehend this article effectively, you’ll need basic understanding of probability, statistics and R. If you have any questions regarding the concept or the code, feel free to comment, I’ll be more than happy to get back to you. […]
Related Post
Linear Regression from Scratch in Python
Generalized Additive Models
Second step with non-linear regression: adding predictors
Weather forecast with regression models – part 4
Weather forecast with regression models – part 3
A new and much enhanced version of RcppMsgPack arrived on CRAN a couple of days ago. It came together following this email to the r-package-devel list which made it apparent that Travers Ching had been working on MessagePack converters for R which re...
I’ve blogged about my in-development R package hgr a before and it’s slowly getting to a CRAN release. There are two new features to it that are more useful in an interactive session than in a programmatic context. Since they build on each other, we’ll take them in order. New S3 print() Method Objects created... Continue reading →
In our previous portfolio volatility work, we covered how to import stock prices, convert to returns and set weights, calculate portfolio volatility, and calculate rolling portfolio volatility.
Now we want to break that total portfolio volatility into its constituent parts and investigate how each asset contributes to the volatility. Why might we want to do that?
For our own risk management purposes, we might want to ensure that our risk isn’t too concentrated in one asset. Not only might this lead to a less-diversified portfolio than we thought we had, but it also might indicate that our initial assumptions about a particular asset were wrong - or, at least, they have become less right as the asset has changed over time.
Similarly, if this portfolio is governed by a mandate from, say, an institutional client, that client might have a preference or even a rule that no asset or sector can rise above a certain threshold of risk contribution. That institutional client might require a report like this from each of their outsourced managers, so they can sum the constituents.
With that motivation in mind, let’s get prices, returns, and set weights for five ETFs.
library(timetk)
library(tidyverse)
library(tidyquant)
library(highcharter)
symbols % reduce(merge) %__%
`colnames
Nine R user communities already exist in France and there is a much large number of R communities around the world. It was time for Grenoble to start its own! The goal of the R user group is to facilitate the identification of local useRs, to initiate contacts, and to organise experience and knowledge sharing sessions. […]
Data wrangling is the process of importing, cleaning, and transforming raw data into actionable information for analysis. It is a time-consuming process that is estimated to take about 60-80% of analysts’ time. In this series, we will go through this process. It will be a brief series with the goal of crafting the reader’s skills […]
Related exercise sets:
Data wrangling : Cleansing – Regular expressions (2/3) Data wrangling : Cleansing – Regular expressions (1/3) Regular Expressions Exercises – Part 1 Explore all our (__1000) R exercises
Find an R course using our R Course Finder directory
Starting from October, our new live R courses are available to everyone! We have updated them to the latest R functionalities and organized them to ideally form a complete data science path with R: either you seek R introduction or R developer tools, statistical models or Big Data, Shiny dashboards or Machine learning techniques, we bet that we have a course for it. The post New R Live Courses in Milan! Autumn term calendar appeared first on Quantide - R training &amp; consulting.
It can be found on the RStudio cheatsheet page. Suggestions and pull requests are always welcome.
We’re entering a golden age of the “analytics developer.” Cutting edge companies like AirBnB, Apple, and S&amp;P are no longer buying off-the-shelf BI software. Instead, they build their own targeted BI platforms that succeed where traditional Big BI falls flat: Agility, collaboration, browser experience, and web-based data visualization. If your company strives to change the world […]
Fashion-MNIST
About three weeks ago the Fashion-MNIST dataset of Zalando’s article images, which is a great replacement of classical MNIST dataset, was released. In the following article we will try to build a strong classifier using H2O and R.
Eac...
Atmospheric air pollution is one of the most important environmental concerns in many countries around the world, and it is strongly affected by meteorological conditions. Accordingly, in this set of exercises we use openair package to work and analyze air quality and meteorological data. This packages provides tools to directly import data from air quality […]
Related exercise sets:
Working with air quality and meteorological data Exercises (Part-2) Working with air quality and meteorological data Exercises (Part-1) Descriptive Analytics-Part 5: Data Visualisation (Categorical variables) Explore all our (__1000) R exercises
Find an R course using our R Course Finder directory
R Course in Rio de Janeiro We are preparing an R for Data-Science course (direct link here) in partnership with the IBPAD (Brazilian Institute of Research and Data Analysis). It is a great course for those who want to have … Continue reading →
Shiny Server 1.5.4.869 and Shiny Server Pro 1.5.4.872 are now available.
Both the new Shiny Server and Shiny Server Pro releases include bug fixes and enhancements, and we recommend upgrading at your earliest convenience.
Shiny Server 1.5.4.869
Clickjacking protection can now be enabled using the frame_options directive. This directive was already available in Shiny Server Pro, but is now available in Shiny Server as well.
A bug that caused &quot;Error: Can't set headers after they are sent.&quot; to appear in error logs was fixed.
Several bugs in license-manager were fixed.
Shiny Server Pro 1.5.4.872
The “utilization scheduler” component, which manages the way requests are routed to R processes, has been significantly improved. It has been made more robust, and applications now respond more efficiently to increased load.
auth_pam: The performance of multiple simultaneous logins was improved.
auth_ldap: A bug was fixed that caused LDAP to return no groups when a username contained a backslash.
Want to know what's capturing the attention of the producers at the 24-hour cable news stations? There's no equivalent of Twitter's trending topics for the likes of CNN or BBC News, but the newsflash package for R by Bob Rudis can extract the latest trending topics from the TV news stations. It seems TV news topics were much less homogenous yesterday pic.twitter.com/hbKx5cqaf2 — boB Rudis (@hrbrmstr) September 12, 2017 The newsflash package is an interface to the GDELT Project's Television Explorer, which provides access to the closed-captioning transcripts from seven major cable-news stations, with archives available for the past 6...
A few months ago, I published a post where I was trying to map the dependencies relationships between R packages. Today I want to do something similar with package contributors. My idea is to reconstruct a social graph where each node would be a person (presumably a developer), and two persons would be connected by an edge if they have collaborated on the same package. Thus I would be able to explore the CRAN social network! This post is also...Lire la suite Lire la suite
Earlier this week, Amazon announced that it it has started to search for a new city in which to build a second headquarters.
... we're going to scrape some data and visualize it to analyze possible locations for their new HQ.
The post Visualizing possible cities for Amazon’s new headquarters (using R) appeared first on SHARP SIGHT LABS.
We pride ourselves on the fact that we work on some of the most difficult business applications of Shiny. These range from sales, manufacturing, management tools, to even satellite data analysis tools.
Many of our clients include multinational enterpr...
A new online textbook, Data Visualization for Social Science, will teach you everything you need to know about creating beautiful and elegant data visualizations using the ggplot2 package and the R language. The author Kieran Healy is a professor of Sociology at Duke University who has designed data visualizations featured in outlets like The Washington Post, Vox, The Rachel Maddow Show, Slate, and the Boston Globe. While the data sets featured in the book are generally from the social sciences, the book nonetheless provides an excellent introduction to visualization with R for any domain. This book is not a general...
I’m excited to announce that plotly 4.7.1 is now on CRAN! Along with some important bug fixes and numerous improvements to the underlying plotly.js library, this release includes an exciting new R-specific feature — the ability to modify (i.e., update without a full redraw) an existing plotly graph inside a shiny app via the new […]
Edward tufte keyNoTe Edward Tufte will be kicking off PLOTCON NYC this year with his keynote talk. Edward is a Professor Emeritus of Political Science, Statistics, and Computer Science at Yale University. He wrote, designed, and self-published 4 classic books on data visualization including, “The Visual Display of Quantitative Information.” The New York Times described ET […]
In the last set of exercises , you have seen the basic functionalities of RevoScaleR .In this exercise set we will explore RevoScaleR further. get the Credit card fraud data set from revolutionanalytics and lets get started Answers to the exercises are available here.Please check the documentation before starting these exercise set If you obtained […]
Related exercise sets:
Big Data analytics with RevoScaleR Exercises Big Data Manipulation in R Exercises Density-Based Clustering Exercises Explore all our (__1000) R exercises
Find an R course using our R Course Finder directory
Then felt I like some watcher of the skies  When a new planet swims into his ken;  Or like stout Cortez when with eagle eyes  He star’d at the Pacific—and all his men  Look’d at each other with a wild surmise—  Silent, upon a peak in Darien.    On First Looking into Chapman’s Homer by … Continue reading My travels through the realms of Data Science, Machine Learning, Deep Learning and (AI)
This post explains how to use R to automatically write and send emails based on automatically computed analyses (yep, everything automated). This means that when analysis changes or is updated, the email body text changes...
This is common case when working with data that your source is a remote database. Usual ways to cope this when using R is either to load all the data into R or to perform the heaviest joins and aggregations with SQL before loading the data. Both of them have cons: the former one is […]
Related exercise sets:
Descriptive Analytics-Part 5: Data Visualisation (Spatial data) Descriptive Analytics-Part 5: Data Visualisation (Categorical variables) Descriptive Analytics-Part 2 : Data Imputation Explore all our (__1000) R exercises
Find an R course using our R Course Finder directory
As with many aspects of the tidyverse, its non-standard evaluation (NSE) implementation is not something entirely new, but built on top of base R. What makes this one so challenging to get your mind around, is that the Honorable Doctor Sir Lord General and friends brought concepts to the realm of the mortals that many of us had no, or only a vague, understanding of. Earlier, I gave an overview of the most common actions in tidy eval. Although appreciated by many, it left me unsatisfied, because it made clear to me I did not really understand NSE. Neither in base R, nor in tidy eval. Therefore, I bit the bullet and really studied it for a few evenings. Starting with base R NSE, and later learning what tidy eval actually adds to it. I decided to share the things I learned in this, rather lengthy, blog. I think it captures the essentials in NSE, although it surely is incomplete and might be even erronous at places. Still, I hope you find it worthwhile and it will help you understand NSE better and apply it with more confidence.
Update to GetHFData (Version 1.4) - GetHFData newest version (1.4) allows the recreation of the LOB (limit order book) based on order data. A limit order book is the standard format that trading occurs in most exchanges. Order are...
Statistics are often taught in school by and for people who like Mathematics. As a consequence, in those class emphasis is put on leaning equations, solving calculus problems and creating mathematics models instead of building an intuition for probabilistic problems. But, if you read this, you know a bit of R programming and have access […]
Related exercise sets:
Hacking statistics or: How I Learned to Stop Worrying About Calculus and Love Stats Exercises (Part-4) Hacking statistics or: How I Learned to Stop Worrying About Calculus and Love Stats Exercises (Part-6) Hacking statistics or: How I Learned to Stop Worrying About Calculus and Love Stats Exercises (Part-2) Explore all our (__1000) R exercises
Find an R course using our R Course Finder directory
Earlier this year, the GDELT Project released their Television Explorer that enabled API access to closed-caption tedt from television news broadcasts. They’ve done an incredible job expanding and stabilizing the API and just recently released “top trending tables” which summarise what the “top” topics and phrases are across news stations every fifteen minutes. You should... Continue reading →
The long view
There’s been noise in the public debate on the forthcoming New Zealand election about the value of voting intention polls. There’s often not enough attention to the long view on these things, so here is a graphic of 15 years of poll...
R Markdown in corporate settings
I’ve been busy recently writing a paper at work using R Markdown, the wonderful tool provided by the folks at RStudio “to weave together narrative text and code to produce elegantly formatted output”. I don’t ...
INTRODUCTION Dear reader, If you are a newbie in the world of machine learning, then this tutorial is exactly what you need in order to introduce yourself to this exciting new part of the data science world. This post includes a full machine learning project that will guide you step by step to create a […]
Related exercise sets:
How to prepare and apply machine learning to your dataset Summarizing dataset to apply machine learning – exercises Building Shiny App exercises part 6 Explore all our (__1000) R exercises
Find an R course using our R Course Finder directory
On August 26 Hurricane Harvey became the largest hurricane to make landfall in the United States in over 20 years. (That record may yet be broken by Irma, now bearing down on the Florida peninsula.) Harvey's rains brought major flooding to Houston and other coastal areas in the Gulf of Mexico. You can see the rainfall generated by Harvey across Texas and Louisiana in this animation from the US Geological Survey of county-by-county precipitation as the storm makes its way across land. Watch #Harvey move thru SE #Texas spiking rainfall rates in each county (blue colors) Interactive version: https://t.co/qFrnyq3Sbm pic.twitter.com/md0hiUs9Bb...
Infographics. I’ve seen good examples. I’ve seen more bad examples. In general, I prefer a good chart to an infographic. That said, there’s a “genre” of infographic that I do think is useful, which I’ll call “if X were 100 Y”. A good example: if the world were 100 people. That method of showing proportions … Continue reading Infographic-style charts using the R waffle package
We have started working on a new rOpenSci package called writexl. This package wraps the very powerful libxlsxwriter library which allows for exporting data to Microsoft Excel format.
The major benefit of writexl over other packages is that it is completely written in C and has absolutely zero dependencies. No Java, Perl or Rtools are required.
Getting Started
The write_xlsx function writes a data frame to an xlsx file. You can test that data roundtrips properly by reading it back using the readxl package. Columns containing dates and factors get automatically coerced to character strings.
library(writexl)
library(readxl)
write_xlsx(iris, &quot;iris.xlsx&quot;)
# read it back
out
Classification is a supervised machine learning technique in which the dataset which we are analyzing has some inputs \(X_i\) and a response variable \(Y\) which is a discrete valued variable.Discrete valued means the variable has a finite set of values.In more specific terms in classification the response variable has some categorical values.In R we call […]
Related Post
Analyzing Google Trends Data in R
Gradient boosting in R
Radial kernel Support Vector Classifier
Random Forests in R
Network analysis of Game of Thrones
Edit: A version posted at 7:30pm and removed at 8:30pm (eastern time zone) had last years figures. This one now points to the correct database. Football is Back! But I'm[...]
The post Gold-Mining – Week 1 (2017) appeared first on Fantasy Football Analytics.
We are very excited to announce the initial release of our newest R package,
tibbletime. As evident from the name, tibbletime is built on top of the
tibble package (and more generally on top of the tidyverse) with the main
purpose of being able to crea...
In case you missed them, here are some articles from August of particular interest to R users. Using the featurizeText function in the MicrosoftML package to extract ngrams from unstructured text. A joyplot visualizes the probabilities associated with phrases like &quot;highly likely&quot; and &quot;little chance&quot; by a sample of 46 Redditors. Two examples of creating 3-D animations in R: a stereo cube, and the Charleston in motion capture data. A tutorial on creating thematic maps in R, from ComputerWorld. Some tips on using R to query data in Power BI. Using the Rcpp package to calculate a membership matrix for...
Some thoughts about Bayesian Kriging in INLA
I have been playing around with spatial modelling in the R INLA package. This blog just records a few thoughts I have had about using INLA for kriging (spatial interpolation). I am keen to discuss these ide...
Post from Pablo Bernabeu’s blog. Principal Component Analysis (PCA) is a technique used to find the core components that underlie different variables. It comes in very useful whenever doubts arise about the true origin of three or more variables. There are two main methods for performing a PCA: naive or less naive. In the naive method, … Continue reading Naive Principal Component Analysis (using R)
Part of 3 in the series Random Number GenerationCombined linear congruential generators, as the name implies, are a type of PRNG (pseudorandom number generator) that combine two or more LCGs (linear congruential generators). The combination of two or more LCGs into one random number generator can result in a marked...
The post Combined Linear Congruential Generators with R appeared first on Aaron Schlegel.
We're down to single digits for the 2017 EARL London!
Next Tuesday's workshops only have limited places left, so if you had plans to attend, make sure you book now. For ...
I’m currently looking for a new research assistant to help (primarily) with some modelling and R coding as part of a project on forecasting mobile phone sales. The position is likely to last for about 6–9 months, and will be casual.
Requirements Based in Melbourne. I’d rather not communicate remotely. Able to work at least 20 hours per week. Some of that can be from home if necessary, but you do need to be at Monash University (Clayton campus) at least some of the time.
What and why If there could only be one file format to save your charts in, it should be SVG. … More
The netCDF format is popular in sciences that analyse sequential spatial data. It is a self-describing, machine-independent data format for creating, accessing and sharing array-oriented information. The netCDF format provides spatial time-series such as meteorological or environmental data. This article shows how to visualise … Continue reading →
The post Analysing soil moisture data in NetCDF format with the ncdf4 library appeared first on The Devil is in the Data.
For a second year running, there will be another rOpenSci OzUnconference in Australia. This one will be held in Melbourne, on 26-27 October 2017.
Unlike regular conferences, there are no talks and there is no pre-determined agenda. It brings together s...
Version 3.4 of the Knime Analytics Platform, the open-source data science workflow toolbox, was released back in July. With that release came new integrations with Azure and Microsoft R Server, which were highlighted in a recent blog post. With Knime 3.4, you can use Microsoft R Server packages in an R node, and connect to data services in Azure. Knime 3.4 includes a new example: predicting departure delays from the Airlines data set. To run this example, you'll need to configure Knime to make Microsoft R Client or Microsoft R Server the default R engine used by Knime. With that...
Webinar Series: Data Science Undergraduate Education Join the National Academies of Sciences, Engineering, and Medicine for a webinar series on undergraduate data science education. Webinars will take place on Tuesdays from 3-4pm ET starting onSeptember 12 and ending on November 14. See below for the … Continue reading →
In this first volume of symplyR, we are excited to share our Practical Guides to Partioning Clustering.
The course materials contain 3 chapters organized as follow:
K-Means Clustering Essentials
Contents:
K-means basic ideas
K-means algorithm
...
Hi there! We just launched a new course: Multiple and Logistic Regression by Ben Baumer!
In this course, you'll take your skills with simple linear regression to the next level. By learning multiple and logistic regression techniques you will gain the...
&quot;After you've mastered a small number of R functions, visualizations like this become easy (and, they're great practice).&quot;
The post Mapping the largest cities in Asia using R appeared first on SHARP SIGHT LABS.
On Thursday, September 14th I will be giving a lightning talk at the Association of Public Data Users (APDU) Conference in Alexandria, Virginia. The talk...
The post Upcoming talk at the Association of Public Data Users (APDU) Conference appeared first on AriLamstein.com.
I have written the following post about Data Science for Fraud Detection at my company codecentric’s blog:
Fraud can be defined as “the crime of getting money by deceiving people” (Cambridge Dictionary); it is as old as humanity: whenever two...
This exercise set provides practice using the fast and concise data.table package. If you are new to the syntax it is recommended that you start by solving the set on the basics of data.table before attempting this one. We will use data on used cars (Toyota Corollas) on sale during 2004 in the Netherlands. There […]
Related exercise sets:
Basics of data.table: Smooth data exploration Stock prices analysis part 1 exercises Data Manipulation with Data Table -Part 1 Explore all our (__1000) R exercises
Find an R course using our R Course Finder directory
In this post, I will talk about parallelism in R. This post will likely be biased towards the solutions I use. For example, I never use mcapply nor clusterApply. I prefer to always use foreach. In this post, we will focus on how to parallelize R code on your computer. I will use mainly silly examples, just to show one point at a time. Basics of foreach You can install the foreach package with install.packages(&quot;foreach&quot;). library(foreach)
The next event in the Effective Applications of the R Language (EARL) conference series takes place next week, with EARL London 2017. The EARL conference series got its start in London, and the London event remains the biggest and brightest of the venues. This year's program is no exception, with an impressive raft of tutorials and keynote speakers, and three full tracks of contributed talks focusing on real-world applications of R. Here are a few highlights you might want to check out. From the Monday tutorial sessions, Jenny Bryan's session on Writing R Functions for Fun and Profit promises to...
In a previous post, I derived and coded a Gibbs sampler in R for estimating a simple linear regression. In this post, I will do the same for multivariate linear regression. I will derive the conditional posterior distributions necessary for the blocked Gibbs sampler. I will then code the sampler and test it using simulated … Continue reading Blocked Gibbs Sampling in R for Bayesian Multiple Linear Regression →
This post will investigate the idea of reducing leverage when drawdowns are small, and increasing leverage as losses accumulate. It’s … Continue reading →
Background Let me preface this by saying I am not making a recommendation to use the Hoffman method. Neither am I advocating for reference interval mining from routine data. There are many challenges associated with this kind of effort. That's for another post I think. However, I am going to how one does the calculations … Continue reading Mining Your Routine Data for Reference Intervals: Hoffman, Bhattacharya and Maximum Likelihood →
My students, Vincent Yang and Harrison Nguyen, and I have developed a new data visualization package, cdparcoord, available now on CRAN. It can be viewed as an extension of the freqparcoord package written by a former grad student, Yingkang Xie and myself, which I have written about before in this blog. The idea behind both … Continue reading cdparcoord: Parallel Coordinates Plots for Categorical Data →
Downloading and reading bibliometric data from Lattes - Lattes is the largest and unique platform for
academic curriculumns. There you can find information about the academic
work of ALL Brazilian scholars. It includes institution...
I recently posted about using a Python module to convert HTML to usable text. Since then, a new package has hit CRAN dubbed htm2txt that is 100% R and uses regular expressions to strip tags from text. I gave it a spin so folks could compare some basic output, but you should definitely give htm2txt... Continue reading →
Google Trends shows the changes in the popularity of search terms over a given time (i.e., number of hits over time). It can be used to find search terms with growing or decreasing popularity or to review periodic variations from the past such as seasonality. Google Trends search data can be added to other analyses, […]
Related Post
Gradient boosting in R
Radial kernel Support Vector Classifier
Random Forests in R
Network analysis of Game of Thrones
Structural Changes in Global Warming
GitHub vs. GitLab
Git is a distributed implementation of version control. Many people have written very eloquently about why it is a good idea to use version control, not only if you collaborate in a team but also if you work on your own; one example ...
A common experience for those in the social sciences migrating to R from SPSS or STATA is that some procedures that happened at the click of a button will now require more work or are too obscured by the unfamiliar language to see how to accomplish. One such procedure that I’ve experienced is when calculating […]
Related exercise sets:
Introduction to copulas Exercises (Part-1) Multiple Regression (Part 3) Diagnostics Introduction to copulas Exercises (Part-2) Explore all our (__1000) R exercises
Find an R course using our R Course Finder directory
This was just going to be a few Tweets but it ended up being a bit of a rollercoaster of learning for me, and I haven’t blogged in far too long, so I’m writing it up quickly as a ‘hey...Continue Reading →
Question: how hard is it to count rows using the R package dplyr? Answer: surprisingly difficult. When trying to count rows using dplyr or dplyr controlled data-structures (remote tbls such as Sparklyr or dbplyr structures) one is sailing between Scylla and Charybdis. The task being to avoid dplyr corner-cases and irregularities (a few of which … Continue reading It is Needlessly Difficult to Count Rows Using dplyr
LASSO has been a popular algorithm for the variable selection and extremely effective with high-dimension data. However, it often tends to “over-regularize” a model that might be overly compact and therefore under-predictive. The Elastic Net addresses the aforementioned “over-regularization” by balancing between LASSO and ridge penalties. In particular, a hyper-parameter, namely Alpha, would be used […]
With over 11,000 packages in the main CRAN (Consolidated R Archive Network) collection, users of R face a challenge when looking for tools to carry out their statistical and computational tasks, especially in subject areas outside their main domain of expertise. Moreover, a lot of packages have similar or overlapping features. Some are, to be…
By Gabriel Vasconcelos In many cases, especially if you are dealing with forecasting models, it is natural to use a large set of models to forecast the same variable and then select the best model using some error measure. For … Continue reading →
We’re excited to announce the alphavantager package, a lightweight R interface to the Alpha Vantage API! Alpha Vantage is a FREE API for retreiving real-time and historical financial data. It’s very easy to use, and, with the recent glitch with the...
While working on a large client project using Sparklyr and multinomial regression we recently ran into a problem: Apache Spark chooses the order of multinomial regression outcome targets, whereas R users are used to choosing the order of the targets (please see here for some details). So to make things more like R users expect, … Continue reading Permutation Theory In Action
INTRODUCTION Dear reader, If you are a newbie in the world of machine learning, then this tutorial is exactly what you need in order to introduce yourself to this exciting new part of the data science world. This post includes a full machine learning project that will guide you step by step to create a […]
Related exercise sets:
How to prepare and apply machine learning to your dataset Building Shiny App exercises part 4 Hacking statistics or: How I Learned to Stop Worrying About Calculus and Love Stats Exercises (Part-5) Explore all our (__1000) R exercises
Find an R course using our R Course Finder directory
PeerJ Preprints has recently published a collection of articles that focus on the practical side of statistical analysis: Practical Data Science for Stats. While the articles are not peer-reviewed, they have been selected and edited by Jennifer Bryan and Hadley Wickham, both well-respected members of the R community. And while the articles provide great advice for any data scientist, the content does heavily feature the use of R, so it's particularly useful to R users. There are 16 articles in the collection (with possibly more to come?). Here are just a few examples that caught my eye: Bryan: &quot;Excuse me,...
What is OpenML?
The field of Machine Learning has grown tremendously over the last years, and is a key component of data-driven science. Data analysis algorithms are being invented and used every day, but their results and experiments are published almost exclusively in journals or separated repositories. However, data by itself has no value. It’s the ever-changing ecosystem surrounding data that gives it meaning.
OpenML is a networked science platform that aims to connect and organize all this knowledge online, linking data, algorithms, results and people into a coherent whole so that scientists and practitioners can easy build on prior work and collaborate in real time online.
This is cross-posted from the NumFOCUS blog. NumFOCUS is a nonprofit that supports and promotes world-class, innovative, open source scientific computing, including rOpenSci.
Scott use this for NumFOCUS SEO? At rOpenSci, we create and curate sof...
An new version 0.0.9 of RcppAnnoy, our Rcpp-based R integration of the nifty Annoy library by Erik, is now on CRAN. Annoy is a small and lightweight C++ template header library for very fast approximate nearest neighbours.
This release corrects an is...
tmap More maps of the Highlands?
Yep, same as last time, but no need to install dev versions of anything, we can get awesome maps courtesy of the tmap package.
Get the shapefile from the last post
library(tmap)
library(tmaptools)
library(viridis)
scot
Part 2 of 2 in the series Random Number GenerationMultiplicative congruential generators, also known as Lehmer random number generators, is a type of linear congruential generator for generating pseudorandom numbers in . The multiplicative congruential generator, often abbreviated as MLCG or MCG, is defined as a recurrence relation similar to...
The post Multiplicative Congruential Generators in R appeared first on Aaron Schlegel.
In this post, I show a method for extracting small amounts of data from somewhat large Census Bureau Excel spreadsheets, using R.  The objects of interest are expenditures of state and local governments on hospital capital in Iowa for the years 2004 to 2014. The data can be found at http://www2.census.gov/govs/local/. The files at the […]
Related Post
Extracting Tables from PDFs in R using the Tabulizer Package
Extract Twitter Data Automatically using Scheduler R package
An Introduction to Time Series with JSON Data
Get Your Data into R: Import Data from SPSS, Stata, SAS, CSV or TXT
Exporting Data from R to TXT, CSV, SPSS or Stata
Last week I wrote about how you can use the MicrosoftML package in Microsoft R to featurize images: reduce an image to a vector of 4096 numbers that quantify the essential characteristics of the image, according to an AI vision model. You can perform a similar featurization process with text as well, but in this case you have a lot more control of the features used to represent the text. Tsuyoshi Matsuzaki demonstrates the process in a post at the MSDN Blog. The post explores the Multi-Domain Sentiment Dataset, a collection of product reviews from Amazon.com. The dataset includes reviews...
Are you solving problems and innovating with R?
Are you working with R in a commercial setting?
Do you enjoy sharing your knowledge?
If you said yes to any of the above,...
In this set of exercises, we are going to explore some of the probability functions in R by using practical applications. Basic probability knowledge is required. In case you are not familiarized with the function apply, check the R documentation. Note: We are going to use random numbers functions and random processes functions in R […]
Related exercise sets:
Lets Begin with something sample Probability functions beginner Hacking statistics or: How I Learned to Stop Worrying About Calculus and Love Stats Exercises (Part-6) Explore all our (__1000) R exercises
Find an R course using our R Course Finder directory
Recently I noticed that the R package sparklyr had the following odd behavior: suppressPackageStartupMessages(library(&quot;dplyr&quot;)) library(&quot;sparklyr&quot;) packageVersion(&quot;dplyr&quot;) #__ [1] '0.7.2.9000' packageVersion(&quot;sparklyr&quot;) #__ [1] '0.6.2' packageVersion(&quot;dbplyr&quot;) #__ [1] '1.1.0.9000' sc * Using Spark: 2.1.0 d [1] NA ncol(d) #__ [1] NA nrow(d) #__ [1] NA … Continue reading Why to use the replyr R package
In my sordid past, I was a data science consultant. One thing about data science that they don’t teach you at school is that senior managers in most large companies require reports to be in PowerPoint....
Use R as your travel guide and plan your next Pacific island hopping holiday with the igraph package. This code analyses flight routes and finds routes. Continue reading →
The post Pacific Island Hopping using R and iGraph appeared first on The Devil is in the Data.
What kind of probability are people talking about when they say something is &quot;highly likely&quot; or has &quot;almost no chance&quot;? The chart below, created by Reddit user zonination, visualizes the responses of 46 other Reddit users to &quot;What probability would you assign to the phase: &quot; for various statements of probability. Each set of responses has been converted to a kernel destiny estimate and presented as a joyplot using R. Somewhat surprisingly, the results from the Redditors hew quite closely to a similar study of 23 NATO intelligence officers in 2007. In that study, the officers — who were accustomed...
I’ve been using R or its predecessors for about 30 years, which means I know a lot about R, and also that I don’t necessarily know how to use modern R tools. Lately, I’ve been trying to unlearn some old approaches, and to re-learn them using the ...
Data wrangling, is the process of importing, cleaning and transforming raw data into actionable information for analysis. It is a time-consuming process which is estimated to take about 60-80% of analyst’s time. In this series we will go through this process. It will be a brief series with goal to craft the reader’s skills on […]
Related exercise sets:
Regular Expressions Exercises – Part 1 Data wrangling : Cleansing – Regular expressions (1/3) Data wrangling : Transforming (2/3) Explore all our (__1000) R exercises
Find an R course using our R Course Finder directory
Are you a person who likes to try new products? Are you curious about which hair products are popular and trendy? If you're excited about getting […]
The post Web Scraping Influenster: Find a Popular Hair Care Product for You appeared first on NYC Data Science Academy Blog.
Register now for ODSC West in San Francisco, November 2-4 and save 60% with code RB60 until September 1st. R continues to hold its own in the data science landscape thanks in no small part to its flexibility.  That flexibility allows R to integrate with some of the most popular data science tools available. Given … Continue reading R in the Data Science Stack at ODSC
Today, we’re continuing our blog series on new features in RStudio 1.1. If you’d like to try these features out for yourself, you can download a preview release of RStudio 1.1.
I Only Work in Black
For those of us that like to work in black or very very dark grey, the dark theme can be enabled from the ‘Global Options’ menu, selecting the ‘Appearance’ tab and choosing an ‘Editor theme’ that is dark.
Icons are now high-DPI, a ‘Modern’ and ‘Sky’ theme were also added, read more about them under Using RStudio Themes.
All panels support themes: Code editor, Console, Terminal, Environment, History, Files, Connections, Packages, Help, Build and VCS. Other features like Notebooks, Debugging, Profiling , Menus and the Object Explorer support this theme as well.
However, the Plots and Viewer panes render with the default colors of your content and therefore, require additional packages to switch to dark themes. For instance, shinythemes provides the darkly theme for Shiny and ggthemes support for light = FALSE under ggplot. If you are a package author, consider using rstudioapi::getThemeInfo() when generating output to these panes.
Enjoy!
Introduction Two things that recently came to my attention were AutoML (Automatic Machine Learning) by h2o.ai and the fashion MNIST by Zalando Research. So as a test, I ran AutoML on the fashion mnist data set. H2o AutoML As you all … Continue reading →
A second fix-up release is needed following on the recent bi-monthly RcppArmadillo release as well as the initial follow-up as it turns out that OS X / macOS is so darn special that it needs an entire separate treatment for OpenMP. Namely to turn it...
If you have tried to communicate research results and data visualizations using R, there is a good chance you will have come across one of its great limitations. R is painful when you need to...
I’m pleased to announce that splashr is now on CRAN. (That image was generated with splashr::render_png(url = &quot;https://cran.r-project.org/web/packages/splashr/&quot;)). The package is an R interface to the Splash javascript rendering service. It works in a similar fashion to Selenium but is fear more geared to web scraping and has quite a bit of power under the... Continue reading →
The Internet Movie Database (Imdb) is a great source to get information about movies. Keras provides access to some part of the cleaned dataset (e.g. for sentiment classification). While sentiment classification is an interesting topic, I wanted to see...
Far away, this ship has taken me far away (Starlight, Muse) Madrid City has an Open Data platform where can be found around 300 data sets about a number of topics. One of these sets is the one I used for this experiment. It contains information about cycling accidents  happened in the city from January … Continue reading The Cycling Accident Map of Madrid City →
R is often used to visualize and animate 2-dimensional data. (Here are just a few examples.) But did you know you can create 3-dimensional animations as well? As Thomas Lins Pedersen explains in a recent blog post, the trick is in using the persp function to translate points in 3-D space into a 2-D projection. This function is normally used to render a 3-D scatterplot or wireframe plot, but if you instead capture its output value, it returns a transformation matrix. You can then use the trans3d function to with this matrix to transform points in 3-D space. Thomas demonstrates...
Atmospheric air pollution is one of the most important environmental concerns in many countries around the world, and it is strongly affected by meteorological conditions. Accordingly, in this set of exercises we use openair package to work and analyze air quality and meteorological data. This packages provides tools to directly import data from air quality […]
Related exercise sets:
Working with air quality and meteorological data Exercises (Part-1) Forecasting: Time Series Exploration Exercises (Part-1) Data table exercises: keys and subsetting Explore all our (__1000) R exercises
Find an R course using our R Course Finder directory
It’s not even Labor Day, so it seems to be a bit early to start planning for next year’s R conferences. But, early-bird pricing for RStudio::Conf 2018 ends this Thursday.
The conference which will be held in San Diego between January 31st and February 3rd promises to match and even surpass this year’s event. In addition to keynotes from Di Cook (Monash University and Iowa State University), J.J. Allaire (RStudio Founder, CEO &amp; Principal Developer), Shiny creator Joe Cheng, and Chief Scientist Hadley Wickham, a number of knowledgeable (and entertaining) speakers have already committed including quant, long-time R user and twitter humorist JD Long (@CMastication), Stack Overflow’s David Robinson (@drob) and ProPublica editor Olga Pierce (@olgapierce).
Making the deadline for early-bird pricing will get you a significant savings. The “full seat” price for the conference is $495 before midnight EST August 31, 2017 and $695 thereafter. You can register here.
For a good idea of the kinds of things you can expect to learn at RStudio::Conf 2018 have a look at the videos from this year’s event.
I was about to embark on setting up a background task to sift through R package PDFs for traces of functions that “omit NA values” as a surprise present for Colin Fay and Sir Tierney: [Please RT]#RStats folks, @nj_tierney &amp; I need your help for {naniar}!When does R silently drop/omit NA? https://t.co/V5elyGcG8Z pic.twitter.com/VScLXFCl2n— Colin Fay... Continue reading →
&quot;... once you have mastered the essential functions as isolated units, you need to learn how to put them together.&quot;
The post Tidyverse practice: mapping large European cities appeared first on SHARP SIGHT LABS.
When it comes to clumsy column headers namely., wide ones with spaces and special characters, I see many get panic and change the headers in the source file, which is an awkward option given variety of alternatives that exist in R for handling them.
One easy handling of such scenarios is using library(janitor), as name suggested can be employed for cleaning and maintaining. Janitor has function by name clean_names() which can be useful while directly importing the data itself as show in the below example:&quot; library(janitor); newdataobject % clean_names() &quot; Author undertook several projects, courses and programs in data sciences for more than a decade, views expressed here are from his industry experience. He can be reached at mavuluri.pradeep@gmail or besteconometrician@gmail.com for more details.Find more about author at http://in.linkedin.com/in/pradeepmavuluri
I am excited to announce the redesign and reorganization of shiny.rstudio.com, also known as the Shiny Dev Center. The Shiny Dev Center is the place to go to learn about all things Shiny and to keep up to date with it as it evolves.
The goa...
An arithmetic Le Monde mathematical puzzle (that first did not seem to involve R programming because of the large number of digits in the quantity involved): An integer x with less than 100 digits is such that adding the digit 1 on both sides of x produces the integer 99x.  What are the last nine […]
A new version 0.2.0 of the RcppSMC package arrived on CRAN earlier today (as a very quick pretest-publish within minutes of submission).
RcppSMC provides Rcpp-based bindings to R for the Sequential Monte Carlo Template Classes (SMCTC) by Adam Johanse...
The R package seplyr has a neat new feature: the function seplyr::expand_expr() which implements what we call “the string algebra” or string expression interpolation. The function takes an expression of mixed terms, including: variables referring to names, quoted strings, and general expression terms. It then “de-quotes” all of the variables referring to quoted strings and … Continue reading Neat New seplyr Feature: String Interpolation
Suppose as a business manager you have the responsibility for testing and comparing the lifetimes of four brands (Apollo, Bridgestone, CEAT and Falken) of automobile tyres. The lifetime of these sample observations are measured in mileage run in ’000 miles. For each brand of automobile tyre, sample of 15 observations have been collected. On the […]
Related Post
Cubic and Smoothing Splines in R
Chi-Squared Test – The Purpose, The Math, When and How to Implement?
Missing Value Treatment
R for Publication by Page Piccinini
Assessing significance of slopes in regression models with interaction
To post your R job on the next post Just visit this link and post a new R job to the R community. You can post a job for free (and there are also “featured job” options available for extra exposure). Current R jobs Job seekers: please follow the links below to learn more and apply for your R job of interest: Featured Jobs Full-Time Technical Project Manager Virginia Tech Applied Research Corporation – Posted by miller703 ArlingtonVirginia, United States 15 Jul2017 Full-Time Software Developer Virginia Tech Applied Research Corporation – Posted by miller703 ArlingtonVirginia, United States 15 Jul2017 Full-Time Solutions Engineer RStudio – Posted by nwstephens Anywhere 14 Jul2017 Full-Time Customer Success Rep RStudio – Posted by jclemens1 Anywhere 12 Jul2017   More New Jobs Freelance Optimization Expert IdeaConnection – Posted by Sherri Ann O’Gorman Anywhere 22 Aug2017 Full-Time Data journalist for The Economist @London The Economist – Posted by cooberp London England, United Kingdom 18 Aug2017 Full-Time Technical Business Analyst Investec Asset Management – Posted by IAM London England, United Kingdom 14 Aug2017 Full-Time Senior Data Scientist @ Dallas, Texas, U.S. Colaberry Data Analytics – Posted by Colaberry_DataAnalytics Dallas Texas, United States 8 Aug2017 Full-Time Financial Analyst/Modeler @ Mesa, Arizona, U.S. MD Helicopters – Posted by swhalen Mesa Arizona, United States 31 Jul2017 Full-Time Research volunteer in Cardiac Surgery @ Philadelphia, Pennsylvania, U.S. Thomas Jefferson University – Posted by CVSurgery Philadelphia Pennsylvania, United States 31 Jul2017   In R-users.com you can see all […]
Computerworld's Sharon Machlis has published a very useful tutorial on creating geographic data maps with R. (The tutorial was actually published back in March, but I only came across it recently.) While it's been possible to create maps in R for a long time, some recent packages and data APIs have made the process much simpler. The tutorial is based on the following R packages: sf, a package that provides simple data structures for objects representing spatial geometries including map features like borders and rivers tmap and tmaptools, packages for creating static and interactive &quot;thematic maps&quot; like choropleths and spatial...
Background At the AACC meeting recently, there was an enthusiastic discussion of standardization of reporting for serum protein electrophoresis (SPEP) presented by a working group headed up by Dr. Chris McCudden and Dr. Ron Booth, both of the University of Ottawa. One of the discussions pertained to how monoclonal bands, especially small ones, should be … Continue reading Non-Linear Regression: Application to Monoclonal Peak Integration in Serum Protein Electrophoresis →
I recently had the honor of speaking at the 2017 American Community Survey (ACS) Data Users Conference. My talk focused on choroplethr, the suite of...
The post Recording of my Talk at the ACS Data Users Conference appeared first on AriLamstein.com.
By Yuri Fonseca Basic idea In this post we will discuss briefly about pricing optimization. The main idea behind this problem is the following question: As manager of a company/store, how much should I charge in order to maximize my … Continue reading →
First thing’s first: maybe you shouldn’t care about tidyeval. Maybe you don’t need it. If you exclusively work interactively, I don’t think that learning about tidyeval is important. I can only speak for me, and explain to you why I personally ...
Here’s a small preview of forthcoming features in the ggeffects-package, which are already available in the GitHub-version: For marginal effects from models fitted with glmmTMB() or glmer() resp. glmer.nb(), confidence intervals are now also computed. If you want to test these features, simply install the package from GitHub: library(devtools) devtools::install_github(&quot;strengejacke/ggeffects&quot;) Here are three examples: library(glmmTMB) […]
Introduction Like all other sharing systems,  Airbnb the housing sharing system, Uber the car sharing system, Citi Bike is the network of bicycle rental stations intended […]
The post NYC Citi Bike Visualization - A Hint of Future Transportation appeared first on NYC Data Science Academy Blog.
Statistics are often taught in school by and for people who like Mathematics. As a consequence, in those class emphasis is put on leaning equations, solving calculus problems and creating mathematics models instead of building an intuition for probabilistic problems. But, if you read this, you know a bit of R programming and have access […]
Related exercise sets:
Hacking statistics or: How I Learned to Stop Worrying About Calculus and Love Stats Exercises (Part-6) Hacking statistics or: How I Learned to Stop Worrying About Calculus and Love Stats Exercises (Part-2) Hacking statistics or: How I Learned to Stop Worrying About Calculus and Love Stats Exercises (Part-5) Explore all our (__1000) R exercises
Find an R course using our R Course Finder directory
PUTTING 35 INCHES OF RAIN IN PERSPECTIVE We are always interested in putting numbers into perspective, so we were interested in this article in which they put the Hurricane Harvey’s rain into perspective. They’re predicting 30-40 inches of rain in a few days in Texas. They asked an expert to put that into perspective and […]
The post How much will that Texas rain be appeared first on Decision Science News.
Immersion is among the most effective ways to learn any language. Immersing where new and advanced users come together to improve their use of the R language is a rare opportunity. rstudio::conf 2018 is that time and place!
REGISTER TODAY
...
There was a discussion on Twitter about the need to read in “.msg” files using R. The “MSG” file format is one of the many binary abominations created by Microsoft to lock folks and users into their platform and tools. Thankfully, they (eventually) provided documentation for the MSG file format which helped me throw together... Continue reading →
wrapr is an R package that supplies powerful tools for writing and debugging R code. Primary wrapr services include: let() %.__% (dot arrow pipe) := (named map builder) λ() (anonymous function builder) DebugFnW() let() let() allows execution of arbitrary code with substituted variable names (note this is subtly different than binding values for names as … Continue reading wrapr: R Code Sweeteners
This week, the entire Columbia portion of the Stan team is out of the office and we didn’t have an in-person/online meeting this Thursday. Mitzi and I are on vacation, and everyone else is either teaching, TA-ing, or attending the Stan course. Luckily for this report, there’s been some great activity out of the meeting […]
The post Stan Weekly Roundup, 25 August 2017 appeared first on Statistical Modeling, Causal Inference, and Social Science.
In Power BI, the dashboarding and reporting tool, you can use R to filter, transform, or restructure data via the Query Editor. For example, you could use the mice package to impute missing values, or use the tidytext package to assign sentiment scores to text inputs. As Imke Feldmann explains, there are lots of useful tricks you can accomplish using R in the query editor, including: Passing multiple Power BI tables into the R script Parameterizing R scripts Modifying file names and output data types for use in Power BI Generating multiple table outputs from an R script For the...
Tidy evaluation is a bit challenging to get your head around. Even after reading programming with dplyr several times, I still struggle when creating functions from time to time. I made a small summary of the most common actions I perform, so I don’t have to dig in the vignettes and on stackoverflow over and over. Each is accompanied with a minimal example on how to implement it. I thought others might find this useful too, so here it is in a blog post. This list is meant to be a living thing so additions and improvements are most welcome. Please do a PR on this file or send an email.
INTRODUCTION Dear reader, If you are a newbie in the world of machine learning, then this tutorial is exactly what you need in order to introduce yourself to this exciting new part of the data science world. This post includes a full machine learning project that will guide you step by step to create a […]
Related exercise sets:
Vector exercises Evaluate your model with R Exercises Neural networks Exercises (Part-2) Explore all our (__1000) R exercises
Find an R course using our R Course Finder directory
The BH package on CRAN was updated today to version 1.65.0. BH provides a sizeable portion of the Boost C++ libraries as a set of template headers for use by R, possibly with Rcpp as well as other packages.
This release upgrades the version of Boost ...
by Błażej Moska, computer science student and data science intern Suppose that we have performed clustering K-means clustering in R and are satisfied with our results, but later we realize that it would also be useful to have a membership matrix. Of course it would be easier to repeat clustering using one of the fuzzy kmeans functions available in R (like fanny, for example), but since it is slightly different implementation the results could also be different and for some reasons we don’t want them to be changed. Knowing the equation we can construct this matrix on our own, after...
Boosting is another famous ensemble learning technique in which we are not concerned with reducing the variance of learners like in Bagging where our aim is to reduce the high variance of learners by averaging lots of models fitted on bootstrapped data samples generated with replacement from training data, so as to avoid overfitting. Another […]
Related Post
Radial kernel Support Vector Classifier
Random Forests in R
Network analysis of Game of Thrones
Structural Changes in Global Warming
Deep Learning with R
Over the years I've produced quite a lot of code for power calculations and simulations of different longitudinal linear mixed models. Over the summer I bundled together these calculations for the designs I most typically encounter into an R package. T...
In this set of exercise , you will explore how to handle bigdata with RevoscaleR package from Microsoft R (previously Revolution Analytics).It comes with Microsoft R client . You can get it from here . get the Credit card fraud data set from revolutionanalytics and lets get started Answers to the exercises are available here.Please […]
Related exercise sets:
Vector exercises Cross Tabulation with Xtabs exercises Data Exploration with Tables exercises Explore all our (__1000) R exercises
Find an R course using our R Course Finder directory
Part of 1 in the series Random Number GenerationA Linear congruential generator (LCG) is a class of pseudorandom number generator (PRNG) algorithms used for generating sequences of random-like numbers. The generation of random numbers plays a large role in many applications ranging from cryptography to Monte Carlo methods. Linear congruential...
The post Linear Congruential Generator in R appeared first on Aaron Schlegel.
I needed to clean some web HTML content for a project and I usually use hgr::clean_text() for it and that generally works pretty well. The clean_text() function uses an XSLT stylesheet to try to remove all non-“main text content” from an HTML document and it usually does a good job but there are some pages... Continue reading →
Introduction
Although there are several good books on principal component methods (PCMs) and related topics, we felt that many of them are either too theoretical or too advanced.
This book provides a solid practical guidance to summarize, visu...
Hello, R users! This week we're continuing to bridge the gap between computers and human language with the launch Sentiment Analysis in R: The Tidy Way by Julia Silge!
Text datasets are diverse and ubiquitous, and sentiment analysis provides an approa...
I’ve been developing a course at DataCamp over the past several months, and I am happy to announce that it is now launched!
The course is Sentiment Analysis in R: the Tidy Way and I am excited that it is now available for you to explore and learn...
Mango Solutions are thrilled to announce that Tareef Kawaf, President of RStudio, will be joining us at EARL Boston as our third Keynote Speaker.
Tareef is an experience...
I wanted to give everyone a heads up about a major rebrand and some probable downtime happening over the weekend. I’m going to be consolidating my Locke Data consulting company materials, the blog, the talks, and my package documentation into
The post Notice: Changes to the site appeared first on Locke Data. Locke Data are a data science consultancy aimed at helping organisations get ready and get started with data science.
Google Trends shows the changes in the popularity of search terms over a given time (i.e., number of hits over time). It can be used to find search terms with growing or decreasing popularity or...
Before recession a commercial tool was popular in the country, hence, uncertainty around tools and technology was not much; however, after recession, incertitude (i.e. uncertainty) around tools and technology have pre-occupied and occupying data sc...
Euler problem 30 is another number crunching problem that deals with numbers to the power of five. Two other Euler problems dealt with raising numbers to a power. The previous problem looked at permutations of powers and problem 16 asks for … Continue reading →
The post Digit fifth powers: Euler Problem 30 appeared first on The Devil is in the Data.
Minard's chart depicting Napoleon's 1812 march on Russia is a classic of data visualization that has inspired many homages using different time-and-place data. If you'd like to recreate the original chart, or create one of your own, Andrew Heiss has created a tutorial on using the ggplot2 package to re-envision the chart in R: The R script provided in the tutorial is driven by historical data on the location and size of Napoleon's armies during the 1812 campaign, but you could adapt the script to use new data as well. Andrew also shows how to combine the chart with a...
The data.table package provides perhaps the fastest way for data wrangling in R. The syntax is concise and is made to resemble SQL. After studying the basics of data.table and finishing this exercise set successfully you will be able to start easing into using data.table for all your data manipulation needs. We will use data […]
Related exercise sets:
Vector exercises Data frame exercises Vol. 2 Instrumental Variables in R exercises (Part-1) Explore all our (__1000) R exercises
Find an R course using our R Course Finder directory
Over the last few days, Rcpp passed another noteworthy hurdle. It is now used by over 10 percent of packages on CRAN (as measured by Depends, Imports and LinkingTo, but excluding Suggests). As of this morning 1130 packages use Rcpp out of a total of...
Some time ago I started working with Bayesian methods, using the great rstanarm-package. Beside the fantastic package-vignettes, and books like Statistical Rethinking or Doing Bayesion Data Analysis, I also found the ressources from Tristan Mahr helpful to both better understand Bayesian analysis and rstanarm. This motivated me to implement tools for Bayesian analysis into my […]
If you want to work on large data science projects (analyses and machine learning) you need to be able to perform dozens of small tasks ...
For example, you'll need to be able to fluently perform dozens of little bits of data wrangling, just like this ...
The post Simple practice: data wrangling the iris dataset appeared first on SHARP SIGHT LABS.
What do women do in films? If you analyze the stage directions in film scripts — as Julia Silge, Russell Goldenberg and Amber Thomas have done for this visual essay for ThePudding — it seems that women (but not men) are written to snuggle, giggle and squeal, while men (but not women) shoot, gallop and strap things to other things. This is all based on an analysis of almost 2,000 film scripts mostly from 1990 and after. The words come from pairs of words beginning with &quot;he&quot; and &quot;she&quot; in the stage directions (but not the dialogue) in the screenplays...
I’ve blathered about my crawl_delay project before and am just waiting for a rainy weekend to be able to crank out a follow-up post on it. Working on that project involved sifting through thousands of Web Archive (WARC) files. While I have a nascent package on github to work with WARC files it’s a tad... Continue reading →
routr is now available on CRAN, and I couldn’t be happier. It’s release marks
the completion of an idea that stretches back longer than my attempts to bring
network visualization and ggplot2 together (see this post for ref).
While my PhD was stil...
I have a new visual essay up at The Pudding today, using text mining to explore how women are portrayed in film.
The R code behind this analysis in publicly available on GitHub.
I was so glad to work with the talented Russell Goldenberg and...
The recent release of the blscrapeR package brings the “tidyverse” into the fold. Inspired by my recent collaboration with Kyle Walker on his excellent tidycensus package, blscrapeR has been optimized for use within the tidyverse as of the current ...
The R package seplyr supplies a few neat new coding notations. An Abacus, which gives us the term “calculus.” The first notation is an operator called the “named map builder”. This is a cute notation that essentially does the job of stats::setNames(). It allows for code such as the following: library(&quot;seplyr&quot;) names
What is a choice simulator? A choice simulator is an online app or an Excel workbook that allows users to specify different scenarios and get predictions. Here is an example of a choice simulator. Choice simulators have...
A/B Testing is a familiar task for many working in business analytics. Essentially, A/B Testing is a simple form of hypothesis testing with one control group and one treatment group. Classical frequentist methodology instructs the analyst to estimate the expected effect of the treatment, calculate the required sample size, and perform a test to determine […]
Related exercise sets:
Hacking statistics or: How I Learned to Stop Worrying About Calculus and Love Stats Exercises (Part-4) Hacking statistics or: How I Learned to Stop Worrying About Calculus and Love Stats Exercises (Part-2) Cross Tabulation with Xtabs exercises Explore all our (__1000) R exercises
Find an R course using our R Course Finder directory
Do you want to get your own simmer hexagon sticker? Just fill in this form and get one send to you for free.
Check out r-simmer.org or CRAN for more information on simmer, a discrete-event simulation package for R.
Background Sometimes we might want to compare three or four tube types for a particular analyte on a group of patients or we might want to see if a particular analyte is stable over time in aliqioted samples. In these experiments are essentially doing the multivariable analogue of the paired t-test. In the tube-type experiment, … Continue reading Compare Tube Types with R – Repeated Measures ANOVA →
I will be at the AI Summit in San Francisco next month, which means I can't make it to Ignite in Orlando this year. Which is a bit of a shame, because there's a fantastic Data Science track at Ignite. There are 25 sessions on offer, with presentations from my Microsoft colleagues on Microsoft R, Cognitive Toolkit, Bot Framework, the Team Data Science Process, and much more. There will also be applications of these technologies and systems presented by the Portland Trail Blazers, Enbridge, and Jack Henry and Associates. Big data machine learning with Microsoft R Server Patterns, Architecture, &amp;...
Just before the summer holidays, BNOSAC presented a talk called Computer Vision and Image Recognition algorithms for R users at the UseR conference. In the talk 6 packages on Computer Vision with R were introduced in front of an audience of about 250 p...
In the previous post (https://statcompute.wordpress.com/2017/06/29/model-operational-loss-directly-with-tweedie-glm), it has been explained why we should consider modeling operational losses for non-material UoMs directly with Tweedie models. However, for material UoMs with significant losses, it is still beneficial to model the frequency and the severity separately. In the prevailing modeling practice for operational losses, it is often convenient to […]
I recently posted an update regarding our R package revisit, aimed at partially remedying the reproducibility crisis, both in the sense of (a) providing transparency to data analyses and (b) flagging possible statistical errors, including misuse of significance testing. One person commented to me that it may not be important for the package to include … Continue reading Wrong on an Astronomical Scale →
The last months, I have worked on brand logo detection in R with Keras. Starting with a model from scratch adding more data and using a pretrained model. The goal is to build a (deep) neural net that is able to identify brand logos in images.
Just ...
The dropout approach developed by Hinton has been widely employed in deep learnings to prevent the deep neural network from overfitting, as shown in https://statcompute.wordpress.com/2017/01/02/dropout-regularization-in-deep-neural-networks. In the paper http://proceedings.mlr.press/v38/korlakaivinayak15.pdf, the dropout can also be used to address the overfitting in boosting tree ensembles, e.g. MART, caused by the so-called “over-specialization”. In particular, while first few […]
On the heels of the very recent bi-monthly RcppArmadillo release comes a quick bug-fix release 0.7.960.1.1 which just got onto CRAN (and I will ship a build to Debian in a moment).
There were three distinct issues I addressed in three quick pull req...
Probability is at the heart of data science. Simulation is also commonly used in algorithms such as the bootstrap. After completing this exercise, you will have a slightly stronger intuition for probability and for writing your own simulation algorithms. Most of the problems in this set have an exact analytical solution, which is not the case […]
Related exercise sets:
Hacking statistics or: How I Learned to Stop Worrying About Calculus and Love Stats Exercises (Part-4) Probability functions beginner Combinations Exercises Explore all our (__1000) R exercises
Find an R course using our R Course Finder directory
I am very glad to find that someone figure out how to use ggjoy with ggtree.
I really love ggjoy and believe it can be a good tool to visualize gene set enrichment (GSEA) result. DOSE/clusterProfiler support several visualization methods.
running...
The dropout approach developed by Hinton has been widely employed in the context of deep learnings to prevent the deep neural network from over-fitting, as shown in https://statcompute.wordpress.com/2017/01/02/dropout-regularization-in-deep-neural-networks. In the paper http://proceedings.mlr.press/v38/korlakaivinayak15.pdf, the dropout is also proposed to address the over-fitting in tree boosting ensembles, e.g. MART, caused by the so-called “over-specialization”. In particular, while […]
A new cross tab tool
I recently put up a simple web app, built with R Shiny, to let users explore the relationship between party vote in the 2014 New Zealand general election and a range of demographic and attitudinal questions in the New Zealand Elec...
Prompted by both my own struggles with wgrib2 compilation and a plea on the rNOMADS email listserv, I’m going to describe how to compile and install wgrib2 on Mac OS. First of all, some background: wgrib2 is an excellent utility written by Wesley Ebisuzaki at NOAA.  It allows for a number of swift and stable […]
dplyr is one of the most popular R packages. It is powerful and important. But is it in fact easily comprehensible?dplyr makes sense to those of us who use it a lot. And we can teach part time R users a lot of the common good use patterns. But, is it an easy task to … Continue reading Is dplyr Easily Comprehensible?
simplyR is a web space where we’ll be posting practical and easy guides for solving real important problems using R programming language.
As we aren’t fans of unnecessary complications, we’ll keep the content of our tutorials / R codes as simpl...
Hello Readers, This is a notification post – Did you realize our website has moved? The blog is live at New JA Blog under the domain http://www.journeyofanalytics.com . You can read about the rent analysis post here. If you received this post AND an email from anu_analytics, then please disregard this post. If you received this … Continue reading Zillow Rent Analysis
Income inequality data
Ideally the Gini coefficient to estimate inequality is based on original household survey data with hundreds or thousands of data points. Often this data isn’t available due to access restrictions from privacy or other concer...
INTRODUCTION The ggvis package is used to make interactive data visualizations. The fact that it combines shiny’s reactive programming model and dplyr’s grammar of data transformation make it a useful tool for data scientists. This package may allows us to implement features like interactivity, but on the other hand every interactive ggvis plot must be […]
Related exercise sets:
How to create interactive data visualizations with ggvis ggvis Exercises (Part-1) How to create visualizations with iPlots package in R Explore all our (__1000) R exercises
Find an R course using our R Course Finder directory
Finally, -after 24h of failed attempts-, I could get my favourite Hugo theme up and running with R Studio and Blogdown. All the steps I followed are detailed in my new Blogdown entry, which is also a GitHub repo. After … Sigue leyendo →
Making your code run faster is often the primary goal when using parallel programming techniques in R, but sometimes the effort of converting your code to use a parallel framework leads only to disappointment, at least initially. Norman Matloff, author of Parallel Computing for Data Science: With Examples in R, C++ and CUDA, has shared chapter 2 of that book online, and it describes some of the issues that can lead to poor performance. They include: Communications overhead, particularly an issue with fine-grained parallelism consisting of a very large number of relatively small tasks; Load balance, where the computing resources...
Some time ago (last year actually 😳) I had a blast developing a feature for ggforce which had been on my mind for far to long than
its limited utility warranted. The idea was to showcase the new facetting extension powers I’d added to ggplot2 by...
RStudio is excited to announce the availability of RStudio Server Pro on the Google Cloud Platform.
RStudio Server Pro GCP is identical to RStudio Server Pro, but with additional convenience for data scientists, including pre-installation o...
Introduction My first car was a 13 year Mitsubishi Colt, I paid 3000 Dutch Guilders for it. I can still remember a friend that would not like me to park this car in front of his house because of possible … Continue reading →
Ava Yang
It’s Game of Thrones time again as the battle for Westeros is heating up. There are tons of ideas, ingredients and interesting analyses out there and I was craving for my own flavour. So step zero, where is the data?
Jenny Bryan’s purrr tutorial introduced the list got_chars, representing characters information from the first five books, which seems not much fun beyond exercising list manipulation muscle. However, it led me to an API of Ice and Fire, the world’s greatest source for quantified and structured data from the universe of Ice and Fire including the HBO series Game of Thrones. I decided to create my own API functions, or better, an R package (inspired by the famous rwar package).
The API resources cover 3 types of endpoint - Books, Characters and Houses. GoTr pulls data in JSON format and parses them to R list objects. httr’s Best practices for writing an API package by Hadley Wickham is another life saver.
The package contains: - One function got_api() - Two ways to specify parameters generally, i.e. endpoint type + id or url - Three endpoint types
## Install GoTr from github
#devtools::install_github(&quot;MangoTheCat/GoTr&quot;)
library(GoTr)
library(tidyverse)
library(listviewer)
# Retrieve books id 5
books_5
As you may have noticed, we have made a few changes to our apps for the 2017 season to bring you a smoother and quicker experience while also adding more[...]
The post 2017 App Update appeared first on Fantasy Football Analytics.
The first &quot;official&quot; version of R, version 1.0.0, was released on February 29, 200. But the R Project had already been underway for several years before then. Sharing this tweet, from yesterday, from R Core member Peter Dalgaard: It was twenty years ago today, Ross Ihaka got the band to play.... #rstats pic.twitter.com/msSpPz2kyA — Peter Dalgaard (@pdalgd) August 16, 2017 Twenty years ago, on August 16 1997, the R Core Group was formed. Before that date, the committers to R were the projects' founders Ross Ihaka and Robert Gentleman, along with Luke Tierney, Heiner Schwarte and Paul Murrell. The email...
The bi-monthly RcppArmadillo release is out with a new version 0.7.960.1.0 which is now on CRAN, and will get to Debian in due course.
And it is a big one. Lots of nice upstream changes from Armadillo, and lots of work on our end as the Google Summe...
On this set of exercises, we are going to explore some of the probability functions in R with practical applications. Basic probability knowledge is required. Note: We are going to use random number functions and random process functions in R such as runif, a problem with these functions is that every time you run them […]
Related exercise sets:
Hacking statistics or: How I Learned to Stop Worrying About Calculus and Love Stats Exercises (Part-4) Hacking statistics or: How I Learned to Stop Worrying About Calculus and Love Stats Exercises (Part-6) Lets Begin with something sample Explore all our (__1000) R exercises
Find an R course using our R Course Finder directory
Somebody nice reached out and gave us this wonderful feedback on our new Supervised Learning in R: Regression (paid) video course. Thanks for a wonderful course on DataCamp on XGBoost and Random forest. I was struggling with Xgboost earlier and Vtreat has made my life easy now :). Supervised Learning in R: Regression covers a … Continue reading Thank You For The Very Nice Comment
Microsoft Cognitive Services provides several APIs for image recognition, but if you want to build your own recognizer (or create one that works offline), you can use the new Image Featurizer capabilities of Microsoft R Server. The process of training an image recognition system requires LOTS of images — millions and millions of them. The process involves feeding those images into a deep neural network, and during that process the network generates &quot;features&quot; from the image. These features might be versions of the image including just the outlines, or maybe the image with only the green parts. You could further...
On May 31, I made a post here about our R package revisit, which is designed to help remedy the reproducibility crisis in science. The intended user audience includes reviewers of research manuscripts submitted for publication, scientists who wish to confirm the results in a published paper, and explore alternate analyses, and members of the … Continue reading Update on Our ‘revisit’ Package →
A geographic bubble chart is a straightforward method to visualise quantitative information with a geospatial relationship. Last week I was in Vietnam helping the Phú Thọ Water Supply Joint Stock Company with their data science. They asked me to create … Continue reading →
The post Visualising Water Consumption using a Geographic Bubble Chart appeared first on The Devil is in the Data.
If you’ve read the book “R for Data Science” or plan to, now you can dive deeper with co-author and RStudio Master Instructor Garrett Grolemund, winner of the Excellence in CE Award at JSM 2017!
This two day workshop provides a comprehensive overview of what is now called the Tidyverse, a core set of R packages that are essential to Data Science. You will visualize, transform, and model data in R and work with date-times, character strings, and untidy data formats. Along the way, you will learn to use the brightest stars in the tidyverse: ggplot2, dplyr, tidyr, readr, purrr and tibble, along with stringr, lubridate, hms, and forcats.
When? 9 a.m.to 5 p.m. Thursday October 5th and Friday the 6th
Where? 20 F Street NW Conference Center, Washington, D.C.
Who? Garrett Grolemund, Data Scientist and Master Instructor at RStudio.
Garrett excels at teaching, statistics, and teaching statistics. He wrote the popular lubridate package and is the author of Hands On Programming with R in addition to Data Science with R. He holds a PhD in Statistics and specializes in Data Visualization.
Build your skills and learn from the best at this rare in-person workshop - the only East coast workshop from RStudio in 2017.
Register here: https://www.rstudio.com/workshops/master-the-tidyverse/
As of today, there are just 30+ seats left. Discounts are available for academics (students or faculty) and for 5 or more attendees from any organization. Email training@rstudio.com if you have any questions about the workshop that you don’t find answered on the registration page.
I often create character variables (i.e. variables with strings of text as their values) in SAS, and they sometimes don’t render as expected.  Here is an example involving the built-in data set SASHELP.CLASS. Here is the code: data c1;      set sashelp.class;      * define a new character variable to classify someone as tall or […]
Data wrangling, is the process of importing, cleaning and transforming raw data into actionable information for analysis. It is a time-consuming process which is estimated to take about 60-80% of analyst’s time. In this series we will go through this process. It will be a brief series with goal to craft the reader’s skills on […]
Related exercise sets:
Regular Expressions Exercises – Part 1 Density-Based Clustering Exercises Data wrangling : I/O (Part-2) Explore all our (__1000) R exercises
Find an R course using our R Course Finder directory
  A new version of MODIStsp (1.3.3) is on CRAN as of today ! Below, you can find a short description of the main improvements.Processing speed improvements
  Processing of MODIS layers after download (i.e., scale and offset calibration, computation of Spectral Indexes and Quality Indicators) is now much faster.  As you can see in the figure, processing time was almost halved on my (not so fast) laptop. This was achieved by modifying all computation functions so to use `raster::calc()` and `raster::overlay()` (more on this in a later post).Although speed is also limited by download speed and compression options, this will allow to save quite some time when working on large areas and with many MODIS layers.
Test mode
  MODIStsp 1.3.3 also introduces a test mode. Although it was mainly developed to facilitate unit testing, it is also available to the user. Running:MODIStsp(test = X) # with X equal to a number between 0 and 6  will run MODIStsp expoiting different sets of processing parameters (available as JSON files in the \Test_files subfolder of MODIStsp installation). We hope this will help us in more easily pinpoint and solve eventual problems signalled by users.Bug fixes  Several bugs discovered after v1.3.2 release were fixed. We thank MODIStsp users for their feedback and help in improving the package ! You can find a list of the main fixes in our NEWS page.
I’m pleased to announce that fiery has been updated to version 1.0 and is now
available on CRAN. As the version bump suggests, this is a rather major update to the package, fixing and improving upon the framework based on my experience
with it, as...
In this post, I describe a recent case where I used rlang’s tidy evaluation
system to do some data-cleaning. This example is not particularly involved, but
it demonstrates is a basic but powerful idea: That we can capture the
expressions that a user...
Last year, Buzzfeed broke the story that US law enforcement agencies were using small aircraft to observe points of interest in US cities, thanks to analysis of public flight-records data. With the data journalism team no doubt realizing that the Flightradar24 data set hosted many more stories of public interest, the challenge lay in separating routine, day-to-day aircraft traffic from the more unusual, covert activities. So they trained an artificial intelligence model to identify unusual flight paths in the data. The model, implemented in the R programming language, applies a random forest algorithm to identify flight patterns similar to those...
rstudio::conf, the conference on all things R and RStudio, will take place February 2 and 3, 2018 in San Diego, California, preceded by Training Days on January 31 and February 1. We are pleased to announce that this year’s conference includes contributed talks and e-posters, and diversity scholarships. More information below!
Contributed talks and e-posters
rstudio::conf() is accepting proposals for contributed talks and e-posters for the first time! Contributed talks are 20 minutes long, and will be scheduled alongside talks by RStudio employees and invited speakers. E-posters will be shown during the opening reception on Thursday evening: we’ll provide a big screen, power, and internet; you’ll provide a laptop with an innovative display or demo.
We are particularly interested in talks and e-posters that:
Showcase the use of R and RStudio’s tools to solve real problems.
Expand the tidyverse to reach new domains and audiences.
Communicate using R, whether it’s building on top of RMarkdown, Shiny, ggplot2, or something else altogether.
Discuss how to teach R effectively.
To present you’ll also need to register for the conference.
Apply now!
Applications close Sept 15, and you’ll be notified of our decision by Oct 1.
Diversity scholarships
We’re also continuing our tradition of diversity scholarships, and this year we’re doubling the program to twenty recipients. We will support underrepresented minorities in the R community by covering their registration (including workshops), travel, and accommodation.
Apply now!
Applications close Sept 15, and you’ll be notified of our decision by Oct 1.
To master data science, you need to practice.
This sounds easy enough, but in reality, many people have no idea how to practice.
Ultimately, you need to ....
The post Simple practice: basic maps with the Tidyverse appeared first on SHARP SIGHT LABS.
Atmospheric air pollution is one of the most important environmental concerns in many countries around the world, and it is strongly affected by meteorological conditions. Accordingly, in this set of exercises we use openair package to work and analyze air quality and meteorological data. This packages provides tools to directly import data from air quality […]
Related exercise sets:
Data table exercises: keys and subsetting Density-Based Clustering Exercises Data wrangling : I/O (Part-1) Explore all our (__1000) R exercises
Find an R course using our R Course Finder directory
A new version of the acs package is now on CRAN. I recommend that all users of choroplethr update to this version. Here is how to...
The post acs v2.1.1 is now on CRAN appeared first on AriLamstein.com.
When monitoring a data source, model, or other automated process, it’s convienent to have method for easily delivering performance metrics and notifying you whenever something is amiss. One option is to use a dashboard; however, this requires active time and effort to grab numbers and catch errors. An alternative approach is to send an email […]
Related exercise sets:
Building Shiny App Exercises (part 5) Building Shiny App exercises part 1 Parallel Computing Exercises: Foreach and DoParallel (Part-2) Explore all our (__1000) R exercises
Find an R course using our R Course Finder directory
Timo Grossenbacher, data journalist with Swiss Radio and TV in Zurich, had a bit of a surprise when he attempted to recreate the results of one of the R Markdown scripts published by SRF Data to accompany their data journalism story about vested interests of Swiss members of parliament. Upon re-running the analysis in R last week, Timo was surprised when the results differed from those published in August 2015. There was no change to the R scripts or data in the intervening two-year period, so what caused the results to be different? Image credit: Timo Grossenbacher The version of...
By Gabriel Vasconcelos When I first started using R there was no such thing as the tidyverse. Although some of the tidyverse packages were available independently, I learned to treat my data mostly using brute force combining pieces of information … Continue reading →
Post by Adi Sarid (Sarid Institute for Research Services LTD.) Lately I’ve been working a lot with the shinydashboard library. Like shiny, it allows any R programmer to harness the power of R and create professional looking interactive apps. The thing about shinydashboards is that it makes wonderfully looking dashboards. What I’ve been doing with the dashboards … Continue reading Shinydashboards from right to left (localizing a shinydashboard to Hebrew)
We are very excited to announce a new (paid) Win-Vector LLC video training course: Supervised Learning in R: Regression now available on DataCamp The course is primarily authored by Dr. Nina Zumel (our chief of course design) with contributions from Dr. John Mount. This course will get you quickly up to speed covering: What is … Continue reading Supervised Learning in R: Regression
Background From time to time I have wanted to bring an online table into an R dataframe. While in principle, the data can be cut and paste into Excel, sometimes the table is very large and sometimes the columns get goofed up in the process. Fortunately, there are a number of R tools for accomplishing … Continue reading Parse an Online Table into an R Dataframe – Westgard’s Biological Variation Database →
RProtoBuf provides R bindings for the Google Protocol Buffers (&quot;ProtoBuf&quot;) data encoding and serialization library used and released by Google, and deployed fairly widely in numerous projects as a language and operating-system agnostic protocol.
A new releases RProtoBuf 0.4.10 just appeared on CRAN. It is a maintenance releases replacing one leftover errorenous use of package= in .Call with the correct PACKAGE= (as requsted by CRAN). It also integrates a small robustification in the deserializer when encountering invalide objects; this was both reported and fixed by Jeffrey Shen.
Changes in RProtoBuf version 0.4.10 (2017-08-13)
More careful operation in deserializer checking for a valid class attribute (Jeffrey Shen in #29 fixing #28)
At the request of CRAN, correct one .Call() argument to PACKAGE=; update routine registration accordingly
CRANberries also provides a diff to the previous release. The RProtoBuf page has an older package vignette, a 'quick' overview vignette, a unit test summary vignette, and the pre-print for the JSS paper. Questions, comments etc should go to the GitHub issue tracker off the GitHub repo.
This post by Dirk Eddelbuettel originated on his Thinking inside the box blog. Please report excessive re-aggregation in third-party for-profit settings.
I’m very happy to announce that reqres has been released on CRAN. reqres is
a new (in R context) approach to working with HTTP messages, that is, the requests you send to a server and the respons it returns. The uunderlying mechanics of a web ser...
(General reminder abt “R⁶” posts in that they are heavy on code-examples, minimal on expository. I try to design them with 2-3 “nuggets” embedded for those who take the time to walk through the code examples on their systems. I’ll always provide further expository if requested in a comment, so don’t hesitate to ask if... Continue reading →
Statistics are often taught in school by and for people who like Mathematics. As a consequence, in those class emphasis is put on leaning equations, solving calculus problems and creating mathematics models instead of building an intuition for probabilistic problems. But, if you read this, you know a bit of R programming and have access […]
Related exercise sets:
Hacking statistics or: How I Learned to Stop Worrying About Calculus and Love Stats Exercises (Part-2) Hacking statistics or: How I Learned to Stop Worrying About Calculus and Love Stats Exercises (Part-4) Hacking statistics or: How I Learned to Stop Worrying About Calculus and Love Stats Exercises (Part-5) Explore all our (__1000) R exercises
Find an R course using our R Course Finder directory
Emojis were again quite en vogue this week on Twitter with Romain François doing some awesome stuff for the emo package, in particular this teeny tiny animated clock. It reminded me of my own emoji animated clock that I had done a while ago for repres...
INTRODUCTION The ggvis package is used to make interactive data visualizations. The fact that it combines shiny’s reactive programming model and dplyr’s grammar of data transformation make it a useful tool for data scientists. This package may allows us to implement features like interactivity, but on the other hand every interactive ggvis plot must be […]
Related exercise sets:
How to create interactive data visualizations with ggvis Data science for Doctors: Cluster Analysis Exercises iPlots exercises Explore all our (__1000) R exercises
Find an R course using our R Course Finder directory
For a begineer in R or any language,regular expression might seem like a daunting task . Rebus package in R gives a lowers the barrier for common regular expression tasks and is useful for a begineer or even for advanced users for most of the common regex skills in a more intuitive yet verbose way […]
Related exercise sets:
Regular Expressions Exercises – Part 1 Hacking Strings with stringi Hacking strings with stringr Explore all our (__1000) R exercises
Find an R course using our R Course Finder directory
This week, more Stan! Charles Margossian is rock star of the week, finishing off the algebraic solver math library fixture and getting all plumbed through Stan and documented. Now you can solve nonlinear sets of equations and get derivatives with the implicit function theory all as part of defining your log density. There is a […]
The post Stan Weekly Roundup, 11 August 2017 appeared first on Statistical Modeling, Causal Inference, and Social Science.
In case you missed them, here are some articles from July of particular interest to R users. A tutorial on using the rsparkling package to apply H20's algorithms to data in HDInsight. Several exercises to learn parallel programming with the foreach package. A presentation on the R6 class system, by Winston Chang. Introducing &quot;joyplots&quot;, a ggplot2 extension for visualizing multiple time series or distributions (with a nod to Joy Division). SQL Server 2017, with many new R-related capabilities, is nearing release. Ali Zaidi on using neural embeddings with R and Spark to analyze Github comments. R ranks #6 in the...
Nathan Eastwood, Data Scientist
I was recently introduced to the Twitter Waterflow Problem and I decided it was interesting enough to try and complete the challenge in R. Consider the following picture:
This plot shows a series of walls and empty valleys. We can represent this picture by an array of integers, where the value at each index is the height of the wall. So in this case, the array of integers can be defined as:
wall
Welcome to the eight post in the ramblingly random R rants series, or R4 for short. We took a short break over the last few weeks due to some conferencing followed by some vacationing and general chill.
But we're back now, and this post gets us back to initial spirit of (hopefully) quick and useful posts. Perusing yesterday's batch of CRANberries posts, I noticed a peculiar new directory shown the in the diffstat output we use to compare two subsequent source tarballs. It was entitled .aspell/, in the top-level directory, and in two new packages by R Core member Kurt Hornik himself.
The context is, of course, the not infrequently-expressed desire to customize the spell checking done on CRAN incoming packages, see e.g. this r-package-devel thread.
And now we can as I verified with (the upcoming next release of) RcppArmadillo, along with a recent-enough (i.e. last few days) version of r-devel. Just copying what Kurt did, i.e. adding a file .aspell/defaults.R, and in it pointing to rds file (named as the package) containing a character vector with words added to the spell checker's universe is all it takes. For my package, see here for the peculiars.
Or see here:
edd@bud:~/git/rcpparmadillo/.aspell(master)$ cat defaults.R Rd_files
INTRODUCTION The ggvis package is used to make interactive data visualizations. The fact that it combines shiny’s reactive programming model and dplyr’s grammar of data transformation make it a useful tool for data scientists. This package may allows us to implement features like interactivity, but on the other hand every interactive ggvis plot must be […]
Related exercise sets:
How to create visualizations with iPlots package in R Data wrangling : Reshaping Building Shiny App exercises part 10 Explore all our (__1000) R exercises
Find an R course using our R Course Finder directory
I’m happy to announce that version 0.10.0 beta of the dplyrXdf package is now available. You can get it from Github: install_github(&quot;RevolutionAnalytics/dplyrXdf&quot;, build_vignettes=FALSE) This is a major update to dplyrXdf that adds the following features: Support for the tidyeval framework that powers the latest version of dplyr Works with Spark and Hadoop clusters and files in HDFS Several utility functions to ease working with files and datasets Many bugfixes and workarounds for issues with the underlying RevoScaleR functions This (pre-)release of dplyrXdf requires Microsoft R Server or Client version 8.0 or higher, and dplyr 0.7 or higher. If you’re using...
Hey R users! Here's another course launched this week: Spatial Statistics in R by Barry Rowlingson.
Everything happens somewhere, and increasingly the place where all these things happen is being recorded in a database. There is some truth behind the ...
Part of 8 in the series Numerical AnalysisThe Trapezoidal Rule is another of Closed Newton-Cotes formulas for approximating the definite integral of a function. The trapezoidal rule is so named due to the area approximated under the integral representing a trapezoid. Although there exist much more accurate quadrature methods, the...
The post The Trapezoidal Rule of Numerical Integration in R appeared first on Aaron Schlegel.
Have you embraced the tidyverse? Do you now want to expand it to meet your needs? Then this is a NEW two-day hands on workshop designed for you! The goal of this workshop is to take you from someone who uses tidyverse functions to someone who can extend the tidyverse by:
Writing expressive code using advanced functional programming techniques
Designs consistent APIs using analogies to existing tools
Uses the S3 object system to make user friendly values
Can bundle functions with documentation and tests into a package to share with others.
The class is taught by Hadley Wickham, Chief Scientist at RStudio, a member of the R Foundation, and Adjunct Professor at Stanford University and the University of Auckland. He builds tools (both computational and cognitive) to make data science easier, faster, and more fun. Much of the material for the course is drawn from two of his existing books, Advanced R and R Packages, but the course also includes a lot of new material that will eventually become a book called “Tidy tools”.
Register here: https://www.rstudio.com/workshops/extending-the-tidyverse/
As of today, there are just 30+ seats left. Discounts are still available for academics (students or faculty) and for 5 or more attendees from any organization. Email training@rstudio.com if you have any questions about the workshop that you don’t find answered on the registration page.
A few weeks ago, our CTO Marek Rogala gave an engaging talk at the User Conference in Brussels. Despite being one of the last talks, he drew a crowd that filled the room and had significant viewership online.
Marek’s talk was entitled shiny.collecti...
Hello R users, new course hot off the press today by Nina Zumel - Supervised Learning in R: Regression!
From a machine learning perspective, regression is the task of predicting numerical outcomes from various inputs. In this course, you'll learn abo...
Data Science for Water Utilities will be a new book that explains how to use R to undertake analytics for problems typical to water utilities. Continue reading →
The post Data Science for Water Utilities Using R appeared first on The Devil is in the Data.
The riddle this week on The Riddler was about finding the largest sequence of integers between 1 and 100 such that each integer is only used once and always followed by a multiple or a factor. A basic R code searching at random [and programmed during a massive downpour on Skye] led to a solution […]
by Le Zhang (Data Scientist, Microsoft) and Graham Williams (Director of Data Science, Microsoft) Microsoft's Cognitive Toolkit (better known as CNTK) is a commercial-grade and open-source framework for deep learning tasks. At present CNTK does not have a native R interface but can be accessed through Keras, a high-level API which wraps various deep learning backends including CNTK, TensorFlow, and Theano, for the convenience of modularizing deep neural network construction. The latest version of CNTK (2.1) supports Keras. The RStudio team has developed an R interface for Keras making it possible to run different deep learning backends, including CNTK, from...
Mango Solutions are excited to announce that Mara Averick, who many know as @dataandme on Twitter, will be joining us at Boston EARL on 1-3 November as one of our keynot...
Motivation “Hey, Yu-Han. I am going to Time Square to celebrate New Year this December. Do you have any recommendation about hotels or short-term rentals?” Once […]
The post NYC Airbnb Insights appeared first on NYC Data Science Academy Blog.
In SQL Server 2016 and later, you can publish an R function to the database as a stored procedure. This makes it possible to run your R function on the SQL Server itself, which makes the power of that server available for R computations, and also eliminates the time required to move data to and from the server. It also makes your R function available as a resource to DBAs for use in SQL queries, even if they don't know the R language. Neils Berglund recently posted a detailed tutorial on using the sqlrutils package to publish an R function...
While understanding and spending time improving specific techniques, and strengthening indvidual muscles is important, occasionally it is necessary to do some rounds of actual sparring to see your flow and spot weaknesses. This exercise sets forces you to use all that you have practiced: to scrape links, download data, regular expressions, merge data and then […]
Related exercise sets:
Data wrangling : I/O (Part-1) Data wrangling : I/O (Part-2) Web Scraping Exercises Explore all our (__1000) R exercises
Find an R course using our R Course Finder directory
Today I made a confession on Twitter: I told the world I had spent my whole career not naming chunks in RMarkdown documents. Even if I had said one should name them when teaching RMarkdown. But it was also a tweet for showing off since I was working on...
Correspondence analysis is a popular tool for visualizing the patterns in large tables. To many practitioners it is probably a black box. Table goes in, chart comes out. In this post I explain the mathematics...
Background At the AACC meeting last week, some of my friends were bugging me that I had not made a blog post in 10 months. Without getting into it too much, let's just say I can blame Cerner. Thanks also to a prod from a friend, here is an approach to a fairly common problem. … Continue reading Determine the CV of a Calculated Lab Reportable – Bioavailable Testosterone →
If something takes less time if done through parallel processing, why not do it and save time? Modern laptops and PCs today have multi core processors with sufficient amount of memory available and one can use it to generate outputs quickly. Parallelizing your codes has its own numerous advantages. Instead of waiting several minutes or … Continue reading Implementing Parallel Processing in R
Many introductions to Bayesian analysis use relatively simple didactic examples (e.g. making inference about the probability of success given bernoulli data). While this makes for a good introduction to Bayesian principles, the extension of these principles to regression is not straight-forward. This post will sketch out how these principles extend to simple linear regression. Along … Continue reading Bayesian Simple Linear Regression with Gibbs Sampling in R →
Trend of US Immigration Enforcement In the coming months I’ll be digging into the immigration enforcement data posted on data.world. I encourage anyone to take this data and either add to the project or to do something on their own. I will be bringing in external data sources to merge as well (which I did […]
Tomas Kalibera, the newest member of the R Core Team, has been working for the last several years with fellow Core Team member Luke Tierney implementing R's byte-code compiler and interpreter. Byte-compiling R code often improves its speed of execution, and usually happens without you having to take any explicit action. R's base and recommended packages are compiled when R is installed, and your scripts and functions are automatically compiled &quot;just-in-time&quot; as you execute them. At the useR!2017 conference last month, Tomas Kalibera gave a very interesting talk: Taking Advantage of the Byte Compiler. In particular, he shows that not...
Since I submitted the geojsonR package I was interested in running geospatial MongoDB queries using GeoJson data. I decided to use PyMongo (through the reticulate package) after opening two Github issues here and here. In my opinion, the PyMo...
K-Means Clustering is a well known technique based on unsupervised learning. As the name mentions, it forms ‘K’ clusters over the data using mean of the data. Unsupervised algorithms are a class of algorithms one should tread on carefully. Using the wrong algorithm will give completely botched up results and all the effort will go … Continue reading Exploring Assumptions of K-means Clustering using R
Support vector machines are a famous and a very strong classification technique which does not uses any sort of probabilistic model like any other classifier but simply generates hyperplanes or simply putting lines ,to separate and classify the data in some feature space into different regions. Support Vector Classifiers are majorly used for solving a […]
Related Post
Random Forests in R
Network analysis of Game of Thrones
Structural Changes in Global Warming
Deep Learning with R
Unsupervised Learning and Text Mining of Emotion Terms Using R
Here's an app that illustrates how fertility and death rates impact the US age structure diagram over time.  The app uses a simple population model to forecast the US age structure in 100 years, starting from 2010 census data and user-selected fertility and death rates.You can find and play with the app at https://johnyagecic.shinyapps.io/AgeStructureShiny/E-mail JYagecic@gmail.com for the R code.
This week, a very promising new R blog was launched, namely the blog of Eric Persson a.k.a as expersso on Twitter. I had really been looking forward to this because expersso’s code screenshots have always been quite cool, so seeing his no longer bein...
I have some more thoughts on the topic: “the part-time R-user.” I am thinking a bit more about the diversity R users. It occurs to me simply dividing R users into two groups, beginning and advanced, neglects a very important group: the part-time R user. This leaves us teachers and package developers with an unfortunate … Continue reading More on “The Part-Time R-User”
Timeline, Merging &amp; Flash charts This is part 10 of our series and we are going to explore the features of some interesting types of charts that googleVis provides like Timeline, Flash and learn how to merge two googleVis charts to one. Read the examples below to understand the logic of what we are going […]
Related exercise sets:
Data Visualization with googleVis exercises part 3 Data Visualization with googleVis exercises part 2 Data Visualization with googleVis exercises part 4 Explore all our (__1000) R exercises
Find an R course using our R Course Finder directory
A popular approach to deal with missing values is to impute the data to get a complete dataset on which any statistical method can be applied. Many imputation methods are available and provide a completed dataset in any cases, whatever the number of individuals and/or variables, the percentage of missing values, the pattern of missing […]
“The idea of imputation is both seductive and dangerous” (R.J.A Little &amp; D.B. Rubin). Indeed, a predicted value is considered as an observed one and the uncertainty of prediction is ignored, conducting to bad inferences with missing values. That is why Multiple Imputation is recommended. The missMDA package quickly generates several imputed datasets with quantitative variables and/or categorical […]
I keep seeing years represented by coloured bars. First it was that demographic tsunami chart. Then there are examples like the one on the right, which came up in a web search today. I even saw one (whispers) at work today. I get what they are trying to do – illustrate trends within categories over … Continue reading Years as coloured bars
Calculating Gini coefficients
Stats NZ release a series of working papers, and a recent one caught my eye because of my interest in inequality statistics (disclaimer - I am working at Stats NZ at the moment, but on completely different things). Workin...
The accidental aRt tumblr (mentioned here a few years ago) continues to provide a steady stream of images that wouldn't look out of place in a modern art gallery, but which in fact are data visualizations (mostly attempted in R), gone wrong. (Here's a typical recent entry.) But now, Giora Simchoni has taken this concept to the next level by writing an R package that deliberately creates works of art. The kandinsky package (available now on Github) takes any dataset as input, and creates an image reminiscent of the work of abstract painter Wassily Kandinsky. The data points are used...
You’d almost think we were Europeans based on how much we’ve slowed down over the summer. Imad Ali, Jonah Gabry, and Ben Goodrich finished the online pkgdown-style documentation for all the Stan Development Team supported R packages. They can be accessed via http://mc-stan.org/(package_name), e.g., rstan: http://mc-stan.org/rstan rstanarm: http://mc-stan.org/rstanarm shinystan: http://mc-stan.org/shinytan loo: http://mc-stan.org/loo bayesplot: http://mc-stan.org/bayesplot The […]
The post Stan Weekly Roundup, 3 August 2017 appeared first on Statistical Modeling, Causal Inference, and Social Science.
When I started writing about methods for better &quot;parametric programming&quot; interfaces for dplyr for R dplyr users in December of 2016 I encountered three divisions in the audience: dplyr users who had such a need, and wanted such extensions. dplyr users who did not have such a need (&quot;we always know the column names&quot;). dplyr … Continue reading Let’s Have Some Sympathy For The Part-time R User
INTRODUCTION R Markdown is one of the most popular data science tools and is used to save and execute code, create exceptional reports whice are easily shareable. The documents that R Markdown provides are fully reproducible and support a wide variety of static and dynamic output formats. Using markdown syntax, which provides an easy way […]
Related exercise sets:
How to create reports with R Markdown in RStudio R Markdown exercises part 1 Building Shiny App exercises part 1 Explore all our (__1000) R exercises
Find an R course using our R Course Finder directory
Chatbots have become a rage since some time now, with firms from various sectors investing in such bots that would reduce or remove the need of employing a call centre whilst maintaining similar levels of efficiency, if not greater. A notable example i...
Sitting on top of R’s external pointers, the RcppXPtr class provides
a powerful and generic framework for
Passing user-supplied C++ functions
to a C++ backend. This technique is exploited in the
RcppDE package, an
efficient C++ based implementation of the
DEoptim package that
accepts optimisation objectives as both R and compiled functions (see
demo(&quot;compiled&quot;, &quot;RcppDE&quot;) for further details). This solution has a
couple of issues though:
Some repetitive scaffolding is always needed in order to bring the XPtr to R space.
There is no way of checking whether a user-provided C++ function
complies with the internal signature supported by the C++ backend,
which may lead to weird runtime errors.
Better XPtr handling with RcppXPtrUtils
In a nutshell, RcppXPtrUtils provides functions for dealing with these
two issues: namely, cppXPtr and checkXPtr. As a package author,
you only need to 1) import and re-export cppXPtr to compile code and
transparently retrieve an XPtr, and 2) use checkXPtr to internally
check function signatures.
cppXPtr works in the same way as Rcpp::cppFunction, but instead of
returning a wrapper to directly call the compiled function from R, it
returns an XPtr to be passed to, unwrapped and called from C++. The
returned object is an R’s externalptr wrapped into a class called
XPtr along with additional information about the function signature.
library(RcppXPtrUtils)
ptr
By Gabriel Vasconcelos In a late post I discussed the Double Selection (DS), a procedure for inference after selecting controls. I showed an example of the consequences of ignoring the variable selection step discussed in an article by Belloni, Chernozhukov … Continue reading →
Impact of the conservation optimism hashtag
The hashtag #conservationoptimism became popular during the recent International Congress for Conservation Biology symposium. Michael Burgass asked me what its twitter impact was, so here is a quick analysi...
Here is a course with videos that present Hierarchical clustering and its complementary with principal component methods. Four videos present a course on clustering, how to determine the number of clusters, how to describe the clusters and how to perform the clustering when there are lots of individuals and/or lots of variables. Then  you will […]
Just getting back from the most fun meetup I have been to in quite some time: episode 23 (by their count) of Open Source Open Mic hosted by Matt Godbolt and Joe Walnes here in Chicago. Nothing but a sequence of lightning talks. Plus beer and pizza. S...
Part 1 of 7 in the series Numerical AnalysisNumerical differentiation is a method of approximating the derivative of a function at particular value . Often, particularly in physics and engineering, a function may be too complicated to merit the work necessary to find the exact derivative, or the function itself...
The post Numerical Differentiation with Finite Differences in R appeared first on Aaron Schlegel.
Given a short review of a product, like &quot;I couldn't put it down!&quot;, can you predict what the product is? In that case it's pretty easy — it's for a book — but this general problem of text categorization comes up in a lot of natural language analysis problems. In his talk at useR!2017 (shown below), Microsoft data scientist Angus Taylor demonstrates how to build a text categorization model in R. He applies a convolutional neural network (trained using the R interface to the MXNET deep learning platform) to Amazon review data, and creates a small Shiny app to categorize...
The foreach statement, which was introduced in the previous set of exercises of this series, can work with various parallel backends. This set allows to train in working with backends provided by the snow and Rmpi packages (on a single machine with multiple CPUs). The name of the former package stands for “Simple Network of […]
Related exercise sets:
Parallel Computing Exercises: Foreach and DoParallel (Part-2) Parallel Computing Exercises: Snowfall (Part-1) Creating Sample Datasets – Exercises Explore all our (__1000) R exercises
Find an R course using our R Course Finder directory
(This article was first published on Blog, and kindly contributed to R-bloggers) To leave a comment for the author, please follow the link and comment on their blog: Blog. R-bloggers.com offers daily e-mail updates about R news and tutorials on topics such as: Data science, Big Data, R jobs, visualization (ggplot2, Boxplots, maps, animation), programming (RStudio, Sweave, LaTeX, SQL, Eclipse, git, hadoop, Web Scraping) statistics (regression, PCA, time series, trading) and more...
Solution to Euler Problem 27 using the R language. Find the product of the coefficients for the quadratic expression that produces the most primes. Continue reading →
The post Generating Quadratic Primes: Euler Problem 27 appeared first on The Devil is in the Data.
The Solutions section of the Cortana Intelligence Gallery provides more than two dozen working examples of applying machine learning, data science and artificial intelligence to real-world problems. Each solution provides sample data, scripts for model training and evaluation, and reporting of predictions. You can deploy a complete stack in Azure to implement the solution with the click of a button, or follow instructions to deploy on your own hardware. The internals of each solution is fully documented and open source, so you can easily customize it to your needs. Here's a brief overview of some solutions that have recently been...
Click here to explore the data for yourself
First off — I’ll admit that was my poor attempt at a click-bait title. But if you’re still reading the next paragraph, that means it was successful!
Table of contents
Background
Am I...
Data wrangling is a task of great importance in data analysis. Data wrangling, is the process of importing, cleaning and transforming raw data into actionable information for analysis. It is a time-consuming process which is estimated to take about 60-80% of analyst’s time. In this series we will go through this process. It will be […]
Related exercise sets:
Data table exercises: keys and subsetting Data wrangling : Transforming (2/3) Merging Dataframes Exercises Explore all our (__1000) R exercises
Find an R course using our R Course Finder directory
On November 8 and 9, Kassel will once more become the meeting point for the German-speaking R community. From the usage of R in the automotive industry to risk analysis, from data mining with caret to R Markdown: The [R] Kenntnis-Tage 2017 are again standing for an exciting program – always with a focus on … „[R] Kenntnis-Tage 2017: Register now and benefit from the Summer Special“ weiterlesen
ISMB (Intelligent Systems for Molecular Biology – which sounds rather old-fashioned now, doesn’t it?) is the largest conference for bioinformatics and computational biology. It is held annually and, when in Europe, jointly with the European Conference on Computational Biology (ECCB). I’ve had the good fortune to attend twice: in Brisbane 2003 (very enjoyable early in … Continue reading Twitter Coverage of the ISMB/ECCB Conference 2017
If you have ever looked with any depth at statistical computing for multivariate analysis, there is a good chance you have come across the singular value decomposition (SVD). It is a workhorse for techniques that decompose data, such as correspondence analysis and principal...
K-MEANS CLUSTERING, A WORKHORSE OF DATA SCIENCE AND MACHINE LEARNING CLICK TO ENLARGE The USA is a large country. How different are people’s experiences of the weather depending on where they live? To look into this question, we downloaded high temperature data for over 1,300 airport weather stations in the contiguous USA for every day […]
The post Five kinds of weather you’ll meet in America appeared first on Decision Science News.
Adi Sarid (Tel Aviv university and Sarid Research Institute LTD.) July-2017 Background A while back I participated in an R workshop, in the annual convention of the Israeli Association for Statistics. I had the pleasure of talking with Tal Galili and Jonathan Rosenblatt which indicated that a lot of Israeli R users run into difficulties … Continue reading HebRew (using Hebrew in R)
At the useR! conference last month, Jim Hester gave a talk about two packages that provide a modern database interface for R. Those packages are the odbc package (developed by Jim and other members of the RStudio team), and the DBI package (developed by Kirill Müller with support from the R Consortium). To communicate with databases, a common protocol is ODBC. ODBC is an open, cross-platform standard for interfacing with databases, whatever the database software and variant of the SQL language it implements. R has long had the RODBC package created by Brian Ripley, but the new odbc package provides...
The reticulate package provides a very clean &amp; concise interface bridge between R and Python which makes it handy to work with modules that have yet to be ported to R (going native is always better when you can do it). This post shows how to use reticulate to create parquet files directly from R... Continue reading →
Statistics are often taught in school by and for people who like Mathematics. As a consequence, in those class emphasis is put on leaning equations, solving calculus problems and creating mathematics models instead of building an intuition for probabilistic problems. But, if you read this, you know a bit of R programming and have access […]
Related exercise sets:
Nonparametric Tests Exercises Hacking statistics or: How I Learned to Stop Worrying About Calculus and Love Stats Exercises (Part-2) Hacking statistics or: How I Learned to Stop Worrying About Calculus and Love Stats Exercises (Part-3) Explore all our (__1000) R exercises
Find an R course using our R Course Finder directory
In my last posts ([here](http://flovv.github.io/Logo_detection_deep_learning/ and here, I described how one can detect logos in images with R. The first results were promising and achieved a classification accuracy of ~50%. In this post i will detail h...
This post will be directed towards those newer in investing, with an explanation of drawdowns–in my opinion, a simple and … Continue reading →
Next Tuesday (August 8) I will be giving a talk at the Bay Area R Users Group (BARUG). The talk is titled Beyond Popularity: Monetizing R...
The post Upcoming Talk at the Bay Area R Users Group (BARUG) appeared first on AriLamstein.com.
By Gabriel Vasconcelos In a recent paper that can be downloaded here, Carvalho, Masini and Medeiros show that estimating counterfactuals in a non-stationary framework (when I say non-stationary it means integrated) is a tricky task. It is intuitive that the … Continue reading →
H2O.ai is an open-source AI platform that provides a number of machine-learning algorithms that run on the Spark distributed computing framework. Azure HDInsight is Microsoft's fully-managed Apache Hadoop platform in the cloud, which makes it easy to spin up and manage Azure clusters of any size. It's also easy to to run H2O on HDInsight: H2O AI Platform is available as an application on HDInsight, which pre-installs everything you need as the cluster is created. You can also drive H2O from R, but the R packages don't come auto-installed on HDInsight. To make this easy, the Azure HDInsight team has...
Dealing with a lot of dimensions can be painful for machine learning algorithms. High dimensionality will increase the computational complexity, increase the risk of overfitting (as your algorithm has more degrees of freedom) and the sparsity of the data will grow. Hence, dimensionality reduction will project the data in a space with less dimension to […]
The post Machine Learning Explained: Dimensionality Reduction appeared first on Enhance Data Science.
Using the Google Vision API in R Utilizing RoogleVision After doing my post last month on OpenCV and face detection, I started looking into other algorithms used for pattern detection in images. As it turns out, Google has done a phenomenal job with their Vision API. It’s absolutely incredible the amount of information it can […]
To post your R job on the next post Just visit this link and post a new R job to the R community. You can post a job for free (and there are also “featured job” options available for extra exposure). Current R jobs Job seekers: please follow the links below to learn more and apply for your R job of interest: Featured Jobs Freelance Data Scientists – PhD Paradise – Germany Data Science Talent – Posted by datasciencetalent Frankfurt am Main Hessen, Germany 22 Jul2017 Full-Time Technical Project Manager Virginia Tech Applied Research Corporation – Posted by miller703 Arlington Virginia, United States 15 Jul2017 Full-Time Software Developer Virginia Tech Applied Research Corporation – Posted by miller703 Arlington Virginia, United States 15 Jul2017 Full-Time Marketing Manager RStudio – Posted by bill@rstudio.com Anywhere 14 Jul2017 Full-Time Solutions Engineer RStudio – Posted by nwstephens Anywhere 14 Jul2017 Full-Time Lead Data Scientist Golden Rat Studios – Posted by goldenrat Los Angeles California, United States 13 Jul2017 Full-Time Customer Success Rep RStudio – Posted by jclemens1 Anywhere 12 Jul2017 Full-Time Data analyst – infectious disease mapping John Drake (University of Georgia) – Posted by John Drake Athens Georgia, United States 16 Jun2017   More New Jobs Full-Time Financial Analyst/Modeler @ Mesa, Arizona, U.S. MD Helicopters – Posted by swhalen MesaArizona, United States 31 Jul2017 Full-Time Research volunteer in Cardiac Surgery @ Philadelphia, Pennsylvania, U.S. Thomas Jefferson University – Posted by CVSurgery PhiladelphiaPennsylvania, United States 31 Jul2017 […]
To explain the “optimal transport” problem, we usually start with Gaspard Monge’s “Mémoire sur la théorie des déblais et des remblais“, where the the problem of transporting a given distribution of matter (a pile of sand for instance) into another (an excavation for instance). This problem is usually formulated using distributions, and we seek the “optimal” transport from one distribution to the other one. The formulation, in the context of distributions has been formulated in the 40’s by Leonid Kantorovich, e.g. from the distribution … Continue reading Matching, Optimal Transport and Statistical Tests →
Histogram &amp; Calendar chart This is part 9 of our series and we are going to explore the features of two interesting types of charts that googleVis provides like histogram and calendar charts. Read the examples below to understand the logic of what we are going to do and then test yous skills with the […]
Related exercise sets:
Data Visualization with googleVis exercises part 2 Data Visualization with googleVis exercises part 4 Data Visualization with googleVis exercises part 3 Explore all our (__1000) R exercises
Find an R course using our R Course Finder directory
Course materials (GitHub) This was a PhD course given in the spring of 2017 at Linköping University. The course was organised by the graduate school Forum scientium and was aimed at people who might be interested in using R for data analysis. The materials developed from a part of a previous PhD course from a […]
“Memorable dataviz with the R program” awarded people’s choice prize
For the past two years Dr Nick Hamilton has invited me to give a talk on creating data visuals with the R program at the wonderful UQ Winterschool in
Bioinformatics.
This year...
In the third part in a series on Tidy Time Series Analysis, we’ll use the runCor function from TTR to investigate rolling (dynamic) correlations. We’ll again use tidyquant to investigate CRAN downloads. This time we’ll also get some help from the...
In my talk at useR! earlier this month, I emphasized the fact that a major impediment to obtaining good speed from parallelizing an algorithm is systems overhead of various kinds, including: Contention for memory/network. Bandwidth limits — CPU/memory, CPU/network, CPU/GPU. Cache coherency problems. Contention for I/O ports. OS and/or R limits on number of sockets … Continue reading Understanding Overhead Issues in Parallel Computation →
On 26-28 September 2017, I will be running my 3-day workshop in Perth on “Forecasting: principles and practice” based on my book of the same name.
Topics to be covered include seasonality and trends, exponential smoothing, ARIMA modelling, dynamic regression and state space models, as well as forecast accuracy methods and forecast evaluation techniques such as cross-validation.
Workshop participants are expected to be familiar with basic statistical tools such as multiple regression, but no knowledge of time series or forecasting will be assumed.
The Win-Vector public R packages now all have new pkgdown documentation sites! (And, a thank-you to Hadley Wickham for developing the pkgdown tool.)
Please check them out (hint: vtreat is our favorite).
The package sites:
cdata
replyr
seplyr
sigr
vtre...
A good six years ago I blogged about plotOBOS() which charts a moving average (from one of several available variants) along with shaded standard deviation bands. That post has a bit more background on the why/how and motivation, but as a teaser here is the resulting chart of the SP500 index (with ticker ^GSCP):
 
The code uses a few standard finance packages for R (with most of them maintained by Joshua Ulrich given that Jeff Ryan, who co-wrote chunks of these, is effectively retired from public life). Among these, xts had a recent release reflecting changes which occurred during the four (!!) years since the previous release, and covering at least two GSoC projects. With that came subtle API changes: something we all generally try to avoid but which is at times the only way forward. In this case, the shading code I used (via polygon() from base R) no longer cooperated with the beefed-up functionality of plot.xts(). Luckily, Ross Bennett incorporated that same functionality into a new function addPolygon --- which even credits this same post of mine.
With that, the updated code becomes
## plotOBOS -- displaying overbough/oversold as eg in Bespoke's plots
##
## Copyright (C) 2010 - 2017 Dirk Eddelbuettel
##
## This is free software: you can redistribute it and/or modify it
## under the terms of the GNU General Public License as published by
## the Free Software Foundation, either version 2 of the License, or
## (at your option) any later version.
suppressMessages(library(quantmod)) # for getSymbols(), brings in xts too
suppressMessages(library(TTR)) # for various moving averages
plotOBOS
INTRODUCTION R Markdown is one of the most popular data science tools and is used to save and execute code, create exceptional reports whice are easily shareable. The documents that R Markdown provides are fully reproducible and support a wide variety of static and dynamic output formats. Using markdown syntax, which provides an easy way […]
Related exercise sets:
How to create reports with R Markdown in RStudio Building Shiny App exercises part 1 Shiny Applications Layouts Exercises (Part-6) Explore all our (__1000) R exercises
Find an R course using our R Course Finder directory
There has been a bit of a buzz recently about so-called “joyplots.” Wilke described joyplots as “partially overlapping line plots that create the impression of a mountain range.” I would describe them as partially overlapping densit...
The foreach package provides a simple looping construct for R: the foreach function, which you may be familiar with from other languages like Javascript or C#. It's basically a function-based version of a &quot;for&quot; loop. But what makes foreach useful isn't iteration: it's the way it makes it easy to run those iterations in parallel, and save time on multi-CPU and distributed systems. If you want to get familiar with the foreach function, Parallelizing Loops at Microsoft Docs will introduce you to foreach oops (and the companion concept, iterators), and the various &quot;backends&quot; you can use to make the loops...
One of my tweets that referenced an excellent post about the ethics of web scraping garnered some interest: Apologies for a Medium link but if you do ANY web scraping, you need to read this #rstats // Ethics in Web Scraping https://t.co/y5YxvzB8Fd— boB Rudis (@hrbrmstr) July 26, 2017 If you load that up that tweet... Continue reading →
In the last set of exercises, we worked on the basic concepts of string manipulation with stringr. In this one we will go further into hacking strings universe and learn how to use stringi package.Note that stringi acts as a backend of stringr but have many more useful string manipulation functions compared to stringr and […]
Related exercise sets:
Hacking strings with stringr Higher Order Functions Exercises Character Functions (Advanced) Explore all our (__1000) R exercises
Find an R course using our R Course Finder directory
There has been a resurgence of interest in data visualizations inspired by Joy Division’s Unknown Pleasures album cover. These so-called “Joy Plots” are easier to create thanks to the development of the “ggjoy” R package and also some nice code posted using D3. I produced a global population map (details here) using a similar technique in 2013 and since […]
Here’s the roundup for this past week. Michael Betancourt added case studies for methodology in both Python and R, based on the work he did getting the ML meetup together: RStan workflow PyStan workflow Michael Betancourt, along with Mitzi Morris, Sean Talts, and Jonah Gabry taught the women in ML workshop at Viacom in NYC […]
The post Stan Weekly Roundup, 28 July 2017 appeared first on Statistical Modeling, Causal Inference, and Social Science.
The third update of the 3.6.x release of simmer, the Discrete-Event Simulator for R, is on CRAN. First of all and once again, I must thank Duncan Garmonsway (@nacnudus) for writing a new vignette: “The Bank Tutorial: Part II”. Among various fixes and performance improvements, this release provides a way of knowing the progress of a simulation.… Continuar leyendo simmer 3.6.3 →
I have written the following post about Social Network Analysis and Topic Modeling of codecentric’s Twitter friends and followers for codecentric’s blog:
Recently, Matthias Radtke has written a very nice blog post on Topic Modeling of the c...
A geometric Le Monde mathematical puzzle: Given a pizza of diameter 20cm, what is the way to cut it by two perpendicular lines through a point distant 5cm from the centre towards maximising the surface of two opposite slices?  Using the same point as the tip of the four slices, what is the way to […]
I went to UseR-2017 in Brussels. It was my first time at an UseR (I have been to the first satRday), AND I LOVED IT!
There were many interesting talks, I am so going to use Fast Frugal Trees in the future for instance and I saw a lot of shiny app...
We have several announcements regarding Business Science R packages. First, as of this week the R package formerly known as timekit has changed to timetk for time series tool kit. There are a few “breaking” changes because of the name change, and t...
It’s no secret that I’m a fan of Apache Drill. One big strength of the platform is that it normalizes the access to diverse data sources down to ANSI SQL calls, which means that I can pull data from parquet, Hie, HBase, Kudu, CSV, JSON, MongoDB and MariaDB with the same SQL syntax. This also... Continue reading →
R is an object-oriented language with several object-orientation systems. There's the original (and still widely-used) S3 class system based on the &quot;class&quot; attribute. There's the somewhat stricter, signature-based S4 class system. There are reference classes (also called R5), which provide R objects with multiple references without duplicating data in memory. And now there's the R6 class system, implemented as an R package by Winston Chang, So why yet another class system for R? As Winston explains in his useR!2017 talk embedded below, the S3 and S4 class systems are functional object-oriented systems, where class methods are separate from objects, and...
Part of 6 in the series Numerical AnalysisThe divided differences method is a numerical procedure for interpolating a polynomial given a set of points. Unlike Neville’s method, which is used to approximate the value of an interpolating polynomial at a given point, the divided differences method constructs the interpolating polynomial...
The post Divided Differences Method of Polynomial Interpolation appeared first on Aaron Schlegel.
Hello R-users, On June 29th we had the great pleasure to host the RStudio in Milano at Impact Hub. It went absolutely well with great participation, thank you all! This post is just to give the materials to those of you who could not make it or just want to go through it again and […]
The post RStudio meets MilanoR - Presentations, photos and video! appeared first on MilanoR.
I was recently confronted to a situation that required going from a long dataset to a wide dataset, but with a small twist: there were two datasets, which I had to merge into one. You might wonder what kinda crappy twist that is, right? Well, let’s t...
TL;DR
C++ templates and function overloading are incompatible with R’s C API, so polymorphism must be achieved via run-time dispatch, handled explicitly by the programmer.
The traditional technique for operating on SEXP objects in a generic manner entails a great deal of boilerplate code, which can be unsightly, unmaintainable, and error-prone.
The desire to provide polymorphic functions which operate on vectors and matrices is common enough that Rcpp provides the utility macros RCPP_RETURN_VECTOR and RCPP_RETURN_MATRIX to simplify the process.
Subsequently, these macros were extended to handle an (essentially) arbitrary number of arguments, provided that a C++11 compiler is used.
Background
To motivate a discussion of polymorphic functions, imagine that we desire a function (ends) which, given an input vector x and an integer n, returns a vector containing the first and last n elements of x concatenated. Furthermore, we require ends to be a single interface which is capable of handling multiple types of input vectors (integers, floating point values, strings, etc.), rather than having a separate function for each case. How can this be achieved?
R Implementation
A naïve implementation in R might look something like this:
ends
I recently bought a new laptop and began installing essential software all over again, including R of course! And I wanted all the libraries that I had installed in my previous laptop. Instead of installing libraries one by one all over again, I did the following: Step 1: Save a list of packages installed in … Continue reading Quick Way of Installing all your old R libraries on a New Device
This article describes how to use Virtual tags to analyse SCADA data. Virtual tags provide context o SCADA or Historian data by combining information from various tags with meta data about these tags. Continue reading →
The post How Virtual Tags have transformed SCADA data analysis appeared first on The Devil is in the Data.
Earlier this month, I, along with John Nash, Spencer Graves, and Ludovic Vannoorenberghe, organized a session at useR!2017 focused on discovering, learning about, and evaluating R packages. You can check out the recording of the session.
There are m...
I am working on v.0.4.0 of the padr package this summer. Two new features that will be added are wrappers around seq.Date and seq.POSIXt. Since it is going to take a while before the new release is on CRAN, I go ahead and do an early presentation of these functions. Date and datetime parsing in base R are powerful and comprehensive, but also tedious. They can slow you down in your programming or analysis. Luckily, good wrappers and alternatives exist, at least the ymd{_h}{m}{s} suite from lubridate and Dirk Eddelbuettel’s anytime. These functions remove much of the overhead of date and datetime parsing, allowing for quick formatting of vectors in all kinds of formats. They also alleviate the pain of using seq.Date() and seq.POSIXt a little, because the from and the to arguments should be parsed dates or datetimes. Take the following example.
This is a joyplot: a series of histograms, density plots or time series for a number of data segments, all aligned to the same horizontal scale and presented with a slight overlap. Peak time for sports and leisure #dataviz. About time for a joyplot; might do a write-up on them. #rstats code at https://t.co/Q2AgW068Wa pic.twitter.com/SVT6pkB2hB — Henrik Lindberg (@hnrklndbrg) July 8, 2017 The name &quot;Joy Plot&quot; was apparently coined by Jenny Bryan in April 2017, in response to one of Lindberg's earlier visualizations using this style. (The community appears to have settled on 'joyplot' since then.) The name refers to...
by Aimée Gott, Senior Consultant
If you follow us on Twitter you might have noticed that Mango has been doing a bit of travelling this summer. He’s been to San Franci...
Recently, I got a question on one of my previous blog posts, if there is possibility to pass two queries in same run-time as an argument to external procedure sp_execute_external_script. Some of the  arguments of the procedure sp_execute_external_script are enumerated. This is valid for the inputting dataset and as the name of argument @input_data_1 suggests, … Continue reading Passing two SQL queries to sp_execute_external_script
Statistics are often taught in school by and for people who like Mathematics. As a consequence, in those class emphasis is put on leaning equations, solving calculus problems and creating mathematics models instead of building an intuition for probabilistic problems. But, if you read this, you know a bit of R programming and have access […]
Related exercise sets:
Hacking statistics or: How I Learned to Stop Worrying About Calculus and Love Stats Exercises (Part-2) Hacking statistics or: How I Learned to Stop Worrying About Calculus and Love Stats Exercises (Part-1) Lets Begin with something sample Explore all our (__1000) R exercises
Find an R course using our R Course Finder directory
My sjstats-package has been updated on CRAN. The past updates introduced new functions for various purposes, e.g. predictive accuracy of regression models or improved support for the marvelous glmmTMB-package. The current update, however, added some ANOVA tools to the package. In this post, I want to give a short overview of these new functions, which […]
Matt @stiles is a spiffy data journalist at the @latimes and he posted an interesting chart on U.S. Attorneys General longevity (given that the current US AG is on thin ice): Only Watergate and the Civil War have prompted shorter tenures as AG (if Sessions were to leave now). A daily viz: https://t.co/aJ4KDsC5kC pic.twitter.com/ZoiEV3MhGp— Matt... Continue reading →
SQL Server 2017, the next major release of the SQL Server database, has been available as a community preview for around 8 months, but now the first full-featured release candidate is available for public preview. For those looking to do data science with data in SQL Server, there are a number of new features compared to SQL Server 2017 which might be of interest: SQL Server 2017 will be supported on Linux, specifically on Red Hat, SUSE, and Ubuntu (as well as within a Docker container). You can read some of the backstory on how SQL Server came to Linux...
July 21-22 saw the 18th incarnation of the Bioinformatics Open Source Conference, which generally precedes the ISMB meeting. I had the great pleasure of attending BOSC way back in 2003 and delivering a short presentation on Bioperl. I knew almost nothing in those days, but everyone was very kind and appreciative. My trusty R code … Continue reading Twitter Coverage of the Bioinformatics Open Source Conference 2017
The MODIStsp website, which lay abandoned since several months on github pages, recently underwent a major overhaul thanks to pkgdown. The new site is now available at http://lbusett.github.io/MODIStsp/
We hope that the revised website will allow to navigate MODIStsp-related material much more easily than either github or the standard CRAN documentation, and will therefore help users in better and more easily exploiting MODIStsp functionality.The restyling was possible thanks to the very nice &quot;pkgdown&quot; R package (http://hadley.github.io/pkgdown), which allows to easily create a static documentation website.Though pkgdown does already quite a good job in creating a bare-bones website exploiting just the material available in a standard devtools-based R package file structure, I was surprised at how simply the standard website could be tweaked to obtain a very nice (IMO) final result without needing any particular background on html, css, etc. !(On that, I plan to soon post a short guide containing a few tips and tricks I learnt in the last days for setting up and configuring a pkgdown website, so stay tuned !)
What are people saying about International Congress for Conservation Biology 2017 on Twitter?
Here’s a brief analysis of tweets from the International Congress for Conservation Biology to date. The conference started on the starting 23rd July I was ...
At the useR!2017 conference earlier this month, my colleague Ali Zaidi gave a presentation on using Neural Embeddings to analyze GitHub pull request comments (processed using the tidy text framework). The data analysis was done using R and distributed on Spark, and the resulting neural network trained using the Microsoft Cognitive Toolkit. You can see the slides here, and you can watch the presentation below.
One of the many nice things about summer is the time and space it allows for blogging. And, after a very stimulating SRTL conference (Statistics Reasoning, Teaching and Learning) in Rotorua, New Zealand, there’s lots to blog about. Let’s begin … Continue reading →
Recently a blog
post made its
rounds on the internet describing how it is possible to speed up plot creation
in ggplot2 by first creating a blank canvas and then later adding the plot
elements on top of it. The main takeaway plot is reproduced below:
...
Highcharts has long been a favourite visualisation library of mine, and I’ve written before about Highcharter, my preferred way to use Highcharts in R. Highcharter has a nice simple function, hcboxplot(), to generate boxplots. I recently generated some for a project at work and was asked: can we see how many observations make up the … Continue reading Hacking Highcharter: observations per group in boxplots
Ensemble Learning is a type of Supervised Learning Technique in which the basic idea is to generate multiple Models on a training dataset and then simply combining(average) their Output Rules or their Hypothesis \( H_x \) to generate a Strong Model which performs very well and does not overfits and which balances the Bisa-Variance Tradeoff […]
Related Post
Network analysis of Game of Thrones
Structural Changes in Global Warming
Deep Learning with R
Unsupervised Learning and Text Mining of Emotion Terms Using R
Using MCA and variable clustering in R for insights in customer attrition
Stippling is the creation of a pattern simulating varying degrees of solidity or shading by using small dots (Wikipedia).StippleGen is a piece of software that renders images using stipple patterns, which I discovered on Xi’an’s blog a couple days ago. StippleGen uses an algorithm by Adrian Secord (described here) that turns out to be related […]
The content in this blog comes from a shiny application proof of concept using IMDB movie data.  To view the application: IMDB Movie Data App on […]
The post Runtime vs. Success (Using IMDB) appeared first on NYC Data Science Academy Blog.
The title may seem tautological, but since the arrival of dplyr 0.7.x, there have been some efforts at using dplyr without actually using it that I can’t quite understand. The tidyverse has raised passions, for and against it, for some time already. There are excellent alternatives out there, and I myself use them when I find it suitable. But when I choose to… Continuar leyendo Programming with dplyr by using dplyr →
This blog post will be fairly short, given the minor nature of the update.
Several users complained about OAUTH2 authentication not working, which I didn’t know because I usually use the legacy authentication method! Luckily, GitHub user leocwlau re...
Annotation &amp; Sankey Charts In the eighth part of our series we are going to learn about the features of some interesting types of charts. More specifically we will talk about Annotation and Sankey charts. Read the examples below to understand the logic of what we are going to do and then test yous skills […]
Related exercise sets:
Data Visualization with googleVis exercises part 4 Data Visualization with googleVis exercises part 2 Data Visualization with googleVis exercises part 3 Explore all our (__1000) R exercises
Find an R course using our R Course Finder directory
In the second part in a series on Tidy Time Series Analysis, we’ll again use tidyquant to investigate CRAN downloads this time focusing on Rolling Functions. If you haven’t checked out the previous post on period apply functions, you may want to re...
It has been a while since any kind of significant update has been released for the ggetern library, and the other day a minor update was submitted to CRAN, which is now available. There were a few minor bug fixes, and the inclusion of a few new geometries which some of you may find of […]
The post Version 2.2.1 Released appeared first on ggtern: ternary diagrams in R.
seplyr is an R package that makes it easy to program over dplyr 0.7.*. To illustrate this we will work an example. Suppose you had worked out a dplyr pipeline that performed an analysis you were interested in. For an example we could take something similar to one of the examples from the dplyr 0.7.0 … Continue reading Tutorial: Using seplyr to Program Over dplyr
At my day job I’m a data manager and statistician for an epidemiology project called CHAI lead by Cathryn Tonne. CHAI means “Cardio-vascular health effects of air pollution in Telangana, India” and you can find more about it in our recently publi...
I recently read the high quality book Global Inequality by Branko Milanovic. When reading this sort of thing, I often find I can increase my engagement with a topic by playing around with the data myself. In this day and age, this is much more likely...
So the goal here is to focus back on Greenville County and have even more granularity. I look at median house prices near Greenville and then overlay the park data downloaded earlier. This time, for the Census data, I use the tidycensus package that came out recently. Furthermore, instead of using ggplot2 to create a static map, I use the leaflet package to create an interactive map, and, furthermore integrate data from disparate sources in a convenient way.
IEEE Spectrum has published its fourth annual ranking of of top programming languages, and the R language is again featured in the Top 10. This year R ranks at #6, down a spot from its 2016 ranking (and with an IEEE score — derived from search, social media, and job listing trends — tied with the #5 place-getter, C#). Python has taken the #1 slot from C, jumping from its #3 ranking in 2016. For R (a domain specific language for data science) to rank in the top 10, and for Python (a general-purpose language with many data science applications)...
It was another productive week in Stan land. The big news is that Jonathan Auerbach reports that A team of Columbia students (mostly Andrew’s, including myself) recently won first place in a competition predicting elementary school enrollment. I heard 192 entered, and there were 5 finalists….Of course, we used Stan (RStan specifically). … Thought it […]
The post Stan Weekly Roundup, 21 July 2017 appeared first on Statistical Modeling, Causal Inference, and Social Science.
Introduction R Markdown is one of the most popular data science tools and is used to save and execute code to create exceptional reports whice are easily shareable. The documents that R Markdown provides are fully reproducible and support a wide variety of static and dynamic output formats. R Markdown uses markdown syntax, which provides […]
Related exercise sets:
Vector exercises Building Shiny App exercises part 1 Building Shiny App Exercises (part-9) Explore all our (__1000) R exercises
Find an R course using our R Course Finder directory
US Census Data The US Census collects a number of demographic measures and publishes aggregate data through its website. There are several ways to use Census data in R, from the Census API to the USCensus2010 package. If you are interested in geopolitical data in the US, I recommend exploring both these options - the Census API requires a key for each person who uses it, and the package requires downloading a very large dataset. The setups for both require some effort, but once that effort is done you don’t have to do it again. The acs package in R allows you to access the Census API easily. I highly recommend checking it out, and that’s the method we will use here. Note that I’ve already defined the variable api_key - if you are trying to run this code you will need to first run something like api_key
In this post, I will talk about my package bigstatsr, which I’ve just presented in a lightning talk of 5 minutes at useR!2017. You can listen to me in action there. I should have chosen a longer talk to explain more about this package, maybe next time. I will use this post to give you a more detailed version of the talk I gave in Brussels. Motivation behind bigstatsr I’m a PhD student in predictive human genetics. I’m basically trying to predict someone’s risk of disease based on their DNA mutations. These DNA mutations are in the form of large matrices so that I’m currently working with a matrix of 15K rows and 300K columns. This matrix would take approximately 32GB of RAM if stored as a standard R matrix. When I began studying this dataset, I had only 8GB of RAM on my computer. I now have 64GB of RAM but it would take only copying this matrix once to make my computer begin swapping and therefore slowing down. I found a convenient solution by using the object big.matrix provided by the R package bigmemory (Kane, Emerson, and Weston 2013). With this solution, you can access a matrix that is stored on disk almost as if it were a standard R matrix in memory. Yet, for these special matrices, some useful statistical functions were missing or not fast enough. So, I implemented these. It was a good experience about programming optimized and parallelized algorithms. I’m aware that there are other packages that come with bigmemory, such as biganalytics and bigalgebra, that already implement some of the features I will talk about in this post. I will discuss why I don’t use these packages. However, I use the work of other packages such as biglasso and RSpectra. Introduction to bigmemory # loading package bigstatsr (and bigmemory) library(bigstatsr) # initializing some matrix on disk: wrapper to bigmemory::big.matrix() mat
“‘Demographic tsunami’ will keep Sydney, Melbourne property prices high” screams the headline. While the census showed Australia overall is aging, there’s been a noticeable lift in the number of people aged between 25 to 32. As the accompanying graph shows… Whoa, that is one ugly chart. First thought: let’s not be too hard on Fairfax … Continue reading Chart golf: the “demographic tsunami”
International Congress for Conservation Biology: What analysis program do conservation scientists use?
With the International Congress for Conservation Biology starting 23rd July I was wondering, what analysis programs are most commonly used by conser...
I’m starting to incorporate more Rcpp in my R work, and so decided to spend some time exploring the Rcpp Gallery. One example by John Merrill caught my eye. He provides a C++ solution to transforming an list of lists into a data frame, and shows impressive speed savings compared to as.data.frame. This got me thinking about how […]
Virtually every dataset has them; id variables that link a record to a subject and/or time point. Often one column, or a combination of columns, forms the unique id of a record. For instance, the combination of patient_id and visit_id, or ip_adress and visit_time. The first step in most of my analyses is almost always checking the uniqueness of a variable, or a combination of variables. If it is not unique, may assumptions about the data may be wrong, or there are data quality issues. Since I do this so often, I decided to make a little wrapper around this procedure. The unique_id function will return TRUE if the evaluated variables indeed are the unique key to a record. If not, it will return all the records for which the id variable(s) are duplicated so we can pinpoint the problem right away. It uses dplyr v.0.7.1, so make sure that it is loaded.
Quirks about running Rcpp on Windows through RStudio This is a quick note about some tribulations I had running Rcpp (v. 0.12.12) code through RStudio (v. 1.0.143) on a Windows 7 box running R (v. 3.3.2). I also have RTools v. 3.4 installed. I fully admit that this may very well be specific to my […]
Rafael Irizarry from the Harvard T.H. Chan School of Public Health has presented a number of courses on R and Biostatistics on EdX, and he recently also provided an index of all of the course modules as YouTube videos with supplemental materials. The EdX courses are linked below, which you can take for free, or simply follow the series of YouTube videos and materials provided in the index. Data Analysis for the Life Sciences Series Statistics and R Introduction to Linear Models and Matrix Algebra Statistical Interference and Modeling for High-throughput Experiments High-Dimensional Data Analysis A companion book and associated...
Plumber is a package which allows you to create web APIs from your R code. If you're new to Plumber, you can take a look at www.rplumber.io to learn more about how to use the package to create your APIs.
Over the years, we've noticed a handful of thin...
Proposed solution to Euler Problem 26 in the R language. Find the value of d Continue reading →
The post The Viral Recurring Decimal: Euler Problem 26 appeared first on The Devil is in the Data.
Ligature fonts are fonts which sometimes map multiple characters to a single glyph, either for readability or just because it looks neat. Importantly, this only affects the rendering of the text with said font, while the distinct characters remain in … Continue reading →
Part 1 of 5 in the series Numerical AnalysisNeville’s method evaluates a polynomial that passes through a given set of and points for a particular value using the Newton polynomial form. Neville’s method is similar to a now-defunct procedure named Aitken’s algorithm and is based on the divided differences recursion...
The post Neville’s Method of Polynomial Interpolation appeared first on Aaron Schlegel.
Machine learning is often split between three main types of learning: supervised learning, unsupervised learning, and reinforcement learning. Knowing the differences between these three types of learning is necessary for any data scientist. The big picture The type of learning is defined by the problem you want to solve and is intrinsic to the goal of […]
The post Machine Learning Explained: supervised learning, unsupervised learning, and reinforcement learning appeared first on Enhance Data Science.
Data wrangling is a task of great importance in data analysis. Data wrangling, is the process of importing, cleaning and transforming raw data into actionable information for analysis. It is a time-consuming process which is estimated to take about 60-80% of analyst’s time. In this series we will go through this process. It will be […]
Related exercise sets:
Building Shiny App exercises part 6 Density-Based Clustering Exercises Hacking statistics or: How I Learned to Stop Worrying About Calculus and Love Stats Exercises (Part-2) Explore all our (__1000) R exercises
Find an R course using our R Course Finder directory
Jonah “ShinyStan” Gabry, Mike “Riemannian NUTS” Betancourt, and I will be giving a three-day short course next month in New York, following the model of our successful courses in 2015 and 2016. Before class everyone should install R, RStudio and RStan on their computers. (If you already have these, please update to the latest version […]
The post Short course on Bayesian data analysis and Stan 23-25 Aug in NYC! appeared first on Statistical Modeling, Causal Inference, and Social Science.
The ArCo package 0.2 is now available on CRAN. The functions are now more user friendly. The new features are: Default function for estimation if the user does not inform the functions fn and p.fn. The default model is Ordinary … Continue reading →
A new version of RcppAPT -- our interface from R to the C++ library behind the awesome apt, apt-get, apt-cache, ... commands and their cache powering Debian, Ubuntu and the like -- arrived on CRAN yesterday.
We added a few more functions in order to ...
At RStudio, I have the pleasure of interacting with data science teams around the world. Many of these teams are led by R users stepping into the role of analytic admins. These users are responsible for supporting and growing the R user base in their organization and often lead internal R user groups.
One of the most successful strategies to support a corporate R user group is the creation of an internal R package. This article outlines some common features and functions shared in internal packages. Creating an R package is easier than you might expect. A good place to start is this webinar on package creation.
Logos and Custom CSS
Interestingly, one powerful way to increase the adoption of data science outputs - plots, reports, and even slides - is to stick to consistent branding. Having a common look and feel makes it easier for management to recognize the work of the data science team, especially as the team grows. Consistent branding also saves the R user time that would normally be spent picking fonts and color schemes.
It is easy to include logos and custom CSS inside of an R package, and to write wrapper functions that copy the assets from the package to a user’s local working directory. For example, this wrapper function adds a logo from the RStudioInternal package to the working directory:
getLogo
Hello R users, we've got a brand new course today: Writing Efficient R Code by Colin Gillespie.
The beauty of R is that it is built for performing data analysis. The downside is that sometimes R can be slow, thereby obstructing our analysis. For thi...
The development version of my new R package seplyr is performing in practical applications with dplyr 0.7.* much better than even I (the seplyr package author) expected. I think I have hit a very good set of trade-offs, and I have now spent significant time creating documentation and examples. I wish there had been such … Continue reading seplyr update
The source tarball for JAGS 4.3.0 is now available from Sourceforge. Binary distributions will be available later. See the updated manual for details of the features in the new version. This version is fully compatible with the current version of … Continue reading →
It’s Sunday, and I [Bob] am just sitting on the couch peacefully ggplotting to illustrate basic sample spaces using spinners (a trick I’m borrowing from Jim Albert’s book Curve Ball). There’s an underlying continuous outcome (i.e., where the spinner lands) and a quantization into a number of regions to produce a discrete outcome (e.g., “success” […]
The post Animating a spinner using ggplot2 and ImageMagick appeared first on Statistical Modeling, Causal Inference, and Social Science.
If you use an API key to access a secure service, or need to use a password to access a protected database, you'll need to provide these &quot;secrets&quot; in your R code somewhere. That's easy to do if you just include those keys as strings in your code — but it's not very secure. This means your private keys and passwords are stored in plain-text on your hard drive, and if you email your script they're available to anyone who can intercept that email. It's also really easy to inadvertently include those keys in a public repo if you use...
How to take into account and how to compare information from different information sources? Multiple Factor Analysis is a principal Component Methods that deals with datasets that contain quantitative and/or categorical variables that are structured by groups. Here is a course with videos that present the method named Multiple Factor Analysis. Multiple Factor Analysis (MFA) […]
Blue dragonflies dart to and fro I tie my life to your balloon and let it go (Warm Foothills, Alt-J) In my last post I did some drawings based on L-Systems. These drawings are done sequentially. At any step, the state of the drawing can be described by the position (coordinates) and the orientation of … Continue reading Plants →
Statistics are often taught in school by and for people who like Mathematics. As a consequence, in those class emphasis is put on leaning equations, solving calculus problems and creating mathematics models instead of building an intuition for probabilistic problems. But, if you read this, you know a bit of R programming and have access […]
Related exercise sets:
Hacking statistics or: How I Learned to Stop Worrying About Calculus and Love Stats Exercises (Part-2) Hacking statistics or: How I Learned to Stop Worrying About Calculus and Love Stats Exercises (Part-1) Data Science for Doctors – Part 3 : Distributions Explore all our (__1000) R exercises
Find an R course using our R Course Finder directory
R Documentation at Stack Overflow One of the nice features of R is the ease of data acquisition. I am now working on the examples of data acquisition form different sources within an R session. Soon I am going publish a long-read with an overview of demographic data acquisition in R.
The new constants package is available on CRAN. This small package provides the CODATA 2014 internationally recommended values of the fundamental physical constants (universal, electromagnetic, physicochemical, atomic…), provided as symbols for direct use within the R language. Optionally, the values with errors and/or the values with units are also provided if the errors and/or the units packages are installed as well.… Continuar leyendo constants 0.0.1 →
Here is a course with videos that present Multiple Correspondence Analysis in a French way. The most well-known use of Multiple Correspondence Analysis is: surveys. Four videos present a course on MCA, highlighting the way to interpret the data. Then  you will find videos presenting the way to implement MCA in FactoMineR, to deal with […]
In an older post, I discussed a number of functions that are useful for programming in R. I wanted to expand on that topic by covering other functions, packages, and tools that are useful. Over the past year, I have been working as an R programmer and these are some of the new learnings that […]
In an earlier post I discussed how to avoid overfitting when using Support Vector Machines. This was achieved using cross validation. In cross validation, prediction accuracy is maximized by varying the cost parameter. Importantly, prediction accuracy is...
The Ecosystems chapter of my Empirical software engineering book has been added to the draft pdf (download here). I don’t seem to be able to get away from rewriting everything, despite working on the software engineering material for many years. Fortunately the sparsity of the data keeps me in check, but I keep finding new […]
I’m extremely pleased to announce that the sergeant package is now on CRAN or will be hitting your local CRAN mirror soon. sergeant provides JDBC, DBI and dplyr/dbplyr interfaces to Apache Drill. I’ve also wrapped a few goodies into the dplyr custom functions that work with Drill and if you have Drill UDFs that don’t... Continue reading →
It’s easy to get stuck in the day-to-day at the office and there’s never time to upskill or even think about career development. However, to really grow and develop ...
The annual useR!2017 conference took place July 4-7 in Brussels, and in every dimension it was the best yet. It was the largest (with over 1,100 R users from around the world in attendance), and yet still very smoothly run with many amazing talks and lots of fun for everyone. If you weren't able to make it to Brussels, take a look at these recaps from Nick Strayer &amp; Lucy D'Agostino McGowan, Once Upon Data and DataCamp to get a sense of what it was like, or simply take a look at this recap video: From my personal point of...
Last month the French elections viewed through Twitter: a semantic analysis post showed how the two contenders were perceived on Twitter during three key events of the campaign (Macron leaks, presidential debate and election day). The goal of the post is to show how to perform this twitter analysis using R. Collecting tweets in real time with […]
The post Twitter analysis using R (Semantic analysis of French elections) appeared first on Enhance Data Science.
In time series analysis, structural changes represent shocks impacting the evolution with time of the data generating process. That is relevant because one of the key assumptions of the Box-Jenkins methodology is that the structure of the data generating process does not change over time. How can structural changes be identified ? The strucchange package […]
Related Post
Deep Learning with R
Unsupervised Learning and Text Mining of Emotion Terms Using R
Using MCA and variable clustering in R for insights in customer attrition
Web Scraping and Applied Clustering Global Happiness and Social Progress Index
Key Phrase Extraction from Tweets
This is the fourth part of the series on volatility modelling. For other parts of the series follow the tag volatility. In this exercise set we will explore GARCH-M and E-GARCH models. We will also use these models to generate rolling window forecasts, bootstrap forecasts and perform simulations. Answers to the exercises are available here. […]
Related exercise sets:
Volatility modelling in R exercises (Part-3) Volatility modelling in R exercises (Part-2) Forecasting: ARIMAX Model Exercises (Part-5) Explore all our (__1000) R exercises
Find an R course using our R Course Finder directory
Ensuring your analytic IP is given the attention it deserves
It is now widely recognised that data is the key to making informed business decisions. As such, models and ...
One of the many great packages of rOpenSci has implemented the open source engine Tesseract.
Optical character recognition (OCR) is used to digitize written or typed documents, i.e. photos or scans of text documents are “translated” into a digital...
Table, Org Chart &amp; Tree Map In the seventh part of our series we are going to learn about the features of some interesting types of charts. More specifically we will talk about Table, Org Chart and Tree Map. Read the examples below to understand the logic of what we are going to do and […]
Related exercise sets:
Data Visualization with googleVis exercises part 2 Data Visualization with googleVis exercises part 3 Data Visualization with googleVis exercises part 4 Explore all our (__1000) R exercises
Find an R course using our R Course Finder directory
This week’s R bulletin covers some interesting plotting methods. Hope you like this R weekly bulletin. Enjoy reading! Shortcut Keys 1. Source a file – Ctrl+Shift+O 2. Source the current document – Ctrl+Shift+S 3. Find in Files – Ctrl+Shift+F Problem Solving Ideas Using Multiple Colors for Plotting a Variable Plotting a variable with multiple colors...
The post R Weekly Bulletin Vol – XIII appeared first on .
Cryptocurrencies are a fascinating phenomenon and excellent data source for analysis with R.  This post will introduce how to get a list of available currencies into R.  Wikipedia maintains a page with a list of cryptocurrencies.  These can be read from HTML into a data frame.library(httr)library(XML)results = GET(&quot;https://en.wikipedia.org/wiki/List_of_cryptocurrencies&quot;)doc = readHTMLTable(doc=content(results, &quot;text&quot;))View(doc[1])This list is not complete.  Also, an API would be preferable to reading a wiki page.  A publicly available API at cryptocompare.com provides similar information in JSON format.library(jsonlite)library(data.table)response = fromJSON('https://www.cryptocompare.com/api/data/coinlist')df = data.table::rbindlist(response$Data, fill=TRUE)View(df)With a bit of shiny code, this data can be served up in a web application and published to shinyapps.io... so you can skip the R coding and review the data here if you prefer:https://casimir.shinyapps.io/cryptocurr/
I have been writing a lot (too much) on the R topics dplyr/rlang/tidyeval lately. The reason is: major changes were recently announced. If you are going to use dplyr well and correctly going forward you may need to understand some of the new issues (if you don’t use dplyr you can safely skip all of … Continue reading dplyr 0.7 Made Simpler
Why should you do this?
On Install R and Python via Homebrew I described some potential problems when using R or Python libraries.
After doing a fresh OS X install I had some problems with \(\LaTeX\) and XQuartz. The root of the problem is directories location that is easily solved with Homebrew as it doesn't touch the system and provides symlinks for its packages.
What do you need to do this?
You'll need Homebrew and XCode installed. How to do that? check these links
Install Homebrew
Install XCode
Its possible to install just Command Line Tools for XCode from Apple Developers instead of full XCode. The script below contains a line to install that.
How do you do this?
This script will install coreutils, bash, git and more updated tools.
Don't forget to read How to install and codesign GDB on OS X El Capitan
to make it work.
Mango Solutions are delighted to announce that David Robinson, Data Scientist at Stack Overflow, will be keynoting at EARL Boston 2017. David analyzes data on the world...
This is first of the set of exercise on string manipulation with stringr Answers to the exercises are available here. If you obtained a different (correct) answer than those listed on the solutions page, please feel free to post your answer as a comment on that page. Exercise 1 use a stringr function to merge […]
Related exercise sets:
Regular Expressions Exercises – Part 1 Character Functions (Advanced) Building Shiny App exercises part 1 Explore all our (__1000) R exercises
Find an R course using our R Course Finder directory
Another week, another bunch of Stan updates. Kevin Van Horn and Elea McDonnell Feit put together a tutorial on Stan [GitHub link] that covers linear regression, multinomial logistic regression, and hierarchical multinomial logistic regression. Andrew has been working on writing up our “workflow,” by which he means not only the three-step process of Ben Goodrich […]
The post Stan Weekly Roundup, 14 July 2017 appeared first on Statistical Modeling, Causal Inference, and Social Science.
My lightning talk at the useR!2017 conference was R in Minecraft: a five-minute tour of the miner and craft packages and associated book designed to teach kids how to use the R language while manipulating the world of Minecraft. You can see my talk below (starting at the 10:40 mark), and the slides (mostly created in Minecraft) are available for download as well. Check the rest of that video for some other interesting lighting talks from that session: [00:00] Christian Thiele: The cutpointr package: Improved and tidy estimation of optimal cutpoints [05:20] Edwin Thoen: Preparing Datetime Data with Padr [10:40]...
By the by, I’ve posted a first attempt at an R package – ergastR to wrap the ergast developer API, which is where I get chunks of data from for my f1datajunkie tinkerings. You can find it on Github: psychemedia/ergastR. The function names are the ones used in the Wrangling F1 Data With R book. The […]
OpenCPU 2.0 provides the most robust system available today for building and deploying R based apps and services. The server exposes a simple HTTP API for calling with R functions, scripts and managing data, which provides a very solid basi...
Teaching R to students with little to no experience in programming or data analysis is a challenging task. Our talk at useR!2017 showed how different ingredients of our course Exploratory and Descriptive Data Analysis at UHasselt are used to facilitate … Continue reading →
The presentations and tutorials are starting to become available at the Channel 9 useR!2017 page. There's a wealth of amazing content to explore there already, and I wanted to call out one presentation in particular: Uwe Ligges' keynote presentation, 20 years of CRAN. There are many reasons for the success of R: the language itself, the mathematical and statistical libraries, the data visualization tools, the community, and many other factors. But in my opinion, the most important factor that has driven the growth of R over the last 20 years has been CRAN. This repository of R extensions has been...
Part 2 of 4 in the series Numerical AnalysisPolynomial interpolation is the method of determining a polynomial that fits a set of given points. There are several approaches to polynomial interpolation, of which one of the most well known is the Lagrangian method. This post will introduce the Lagrangian method...
The post Lagrangian Polynomial Interpolation with R appeared first on Aaron Schlegel.
The R Consortium is undertaking a multi-year effort to survey the whole R world. In a rather low-key blog post at the end of last month, the R Consortium’s technical committee, the Infrastructure Steering Committee (ISC), launched its prototype survey of R users. The idea is to use the information gleaned from the exploratory questions in this first survey to craft a more refined version that can be sent out annually. So far, we have received approximately twelve hundred responses. Our goal for this first attempt, is to collect ten thousand completed surveys.
The map below indicates that our coverage is pretty good so far, but we would like to see more responses from Asia, Africa and South America. Additional responses from other areas of the world are most welcome, as well.
Please take the survey yourself and help us spread the word on social media, by word of mouth, and any other way you can think of. The survey will be live until September 15th.
A month ago, I started playing with the deep learning framework Keras for R. As a use-case I picked logo detection in images. While the training of a net worked out fine, the results were mediocre.
(Check out the full post to for details on the model a...
Here is a course with videos that present Principal Component Analysis in a French way. Three videos present a course on PCA, highlighting the way to interpret the data. Then  you will find videos presenting the way to implement in FactoMineR, to deal with missing values in PCA thanks to the package missMDA and lastly […]
Introduction If I mention Power BI to fellow data scientists I often get strange looks. However, I quite like the tool, it is an easy and fast way to share results, KPI’s and graphs with others. With the latest release, Power BI now … Continue reading →
Summary: The application of Surface Renewal (SR) analysis as an alternative to energy and gaseous flux measurements using eddy covariance has gained prominence as a less costly and reliable method.  The R code provided in this post is the first of two posts. In this first post  R programming language is applied to calculate ramp … Continue reading Surface Renewal Analysis for Energy Flux Exchanges in an Ecosystem: 1: Calculating Ramp Characteristics using R.
I’m pleased to announce the release of haven 1.1.0. Haven is designed to faciliate the transfer of data between R and SAS, SPSS, and Stata. It makes it easy to read SAS, SPSS, and Stata file formats in to R data frames, and makes it easy to save your R data frames in to SPSS and Stata if you need to collaborate with others using closed source statistical software.
Install the latest version by running:
install.packages(&quot;haven&quot;)
haven 1.1.0 is a small release that fixes a bunch of smaller issues. See the release notes for full details. The most important bug was ensuring that read_sav() once again correctly returns system defined missings as NA (rather than NaN). Other highlights include:
Preliminary support for reading and writing SAS transport files with
read_xpt() and write_xpt().
An experimental cols_only argument in read_sas() that allows you to
read only selected columns.
An update to the bundled ReadStat code, fixing many smaller reading and
writing bugs.
Better checks in write_sav() and write_dta() making it more likely that
you’ll get a clear error in R instead of producing an invalid file.
A big thanks goes to community members Evan Cortens and Patrick Kennedy who helped make this release possible with their code contributions.
In April 2017, Sberbank, Russia’s oldest and largest bank, created a Kaggle competition with the goal of predicting realty prices in Moscow. Given Russia’s volatile economy, […]
The post From Khrushchev to Kaggle: The Russian Real Estate Market appeared first on NYC Data Science Academy Blog.
In general, foreach is a statement for iterating over items in a collection without using any explicit counter. In R, it is also a way to run code in parallel, which may be more convenient and readable that the sfLapply function (considered in the previous set of exercises of this series) or other apply-alike functions. […]
Related exercise sets:
Parallel Computing Exercises: Snowfall (Part-1) List exercises List Vol.2 Exercises Explore all our (__1000) R exercises
Find an R course using our R Course Finder directory
Motivation
This thread on Twitter prompted some questions for me about who actually turns up to vote in New Zealand’s elections. With limited time, I can’t answer many of the important questions raised in that thread and the article it refers to ...
Here is a course with videos that present Multiple Correspondence Analysis in a French way. The most well-known use of Multiple Correspondence Analysis is: surveys. Four videos present a course on MCA, highlighting the way to interpret the data. Then  you will find videos presenting the way to implement MCA in FactoMineR, to deal with […]
Instructions on how to use R and iGraph to analyse the Enron email corpus. This article describes how to research relationships between employees. Continue reading →
The post Using the iGraph package to Analyse the Enron Corpus appeared first on The Devil is in the Data.
Last summer, just prior to starting as an intern working for Hadley, I published
fiery, a flexible webserver framework
that I had worked on during the spring. The reason why I developed fiery amids
offerings such as shiny, opencpu, and plumbr was that ...
The main aim of this project is to study the states which had the most obese population among adults and children as well as teens in USA. Secondly, another objective of this project is to learn how to scrape data in R from an HTML page using rvest package and generate beautiful maps using ggplot […]
Related Post
Can we predict flu deaths with Machine Learning and R?
Graphical Presentation of Missing Data; VIM Package
Creating Graphs with Python and GooPyCharts
Visualizing Tennis Grand Slam Winners Performances
Plotting Data Online via Plotly and Python
Following the talk at MCM 2017 about the strange impact of the random generator on the outcome of an MCMC generator, I tried in Montréal airport the following code on the banana target of Haario et al. (1999), copied from Soetaert and Laine and using the MCMC function of the FME package: library(FME) Banana
For R dplyr users one of the promises of the new rlang/tidyeval system is an improved ability to program over dplyr itself. In particular to add new verbs that encapsulate previously compound steps into better self-documenting atomic steps. Let’s take a look at this capability. First let’s start dplyr. suppressPackageStartupMessages(library(&quot;dplyr&quot;)) packageVersion(&quot;dplyr&quot;) ## [1] '0.7.1.9000' A … Continue reading Better Grouped Summaries in dplyr
by Fang Zhou, Data Scientist; Graham Williams, Director of Data Science, all at Microsoft Credit Risk Scoring is a classic but increasingly important operation in banking as banks are becoming far more risk careful when lending for mortgages, credit card payments or other commercial purposes, in an industry known for fierce competition and the global financial crisis. With an accurate credit risk scoring model, a bank is able to predict the likelihood of default on a transaction. This will in turn help evaluate the potential risk posed by lending money to consumers and to mitigate losses due to bad debt,...
From the current Riddler, a problem that only requires a few lines of code and a few seconds of reasoning. Or not. N households each stole the earnings from one of the (N-1) other households, one at a time. What is the probability that a given household is not burglarised? And what are the expected […]
In this post I’ll tell you about a new package I’ve built and also take you under the hood to show you a really cool thing thats going on: asynchronous API calls. The package: postlightmercury I created a package called postlightmercury which is now on cran. The package is a wrapper for the Mercury web parser by Postlight.... Read more »
Since its foundation just a little over two years ago, the R Consortium has been dedicated to providing support to the R Project and the R community. Already, the R Consortium has channeled the contributions from its corporate members to fund more than 25 projects, working groups, and community initiatives. Recently funded initiatives include a code coverage tool for R, improved connectivity between R and databases, new methods for handling spatial data, the R-hub package builder, and the SatRdays and R-Ladies community programs. (Many of these projects were presented at the recent useR!2017 conference in Brussels, which was awesome to...
Geographical Charts In part 6 of this series we are going to see some amazing geographical charts that googleVis provides. Read the examples below to understand the logic of what we are going to do and then test yous skills with the exercise set we prepared for you. Lets begin! Answers to the exercises are […]
Related exercise sets:
Data Visualization with googleVis exercises part 2 Data Visualization with googleVis exercises part 3 Data Visualization with googleVis exercises part 4 Explore all our (__1000) R exercises
Find an R course using our R Course Finder directory
By Gabriel Vasconcelos S3 objects R has three object systems: S3, S4 and RC. S3 is by far the easiest to work with and it can make you codes much understandable and organized, especially if you are working on a … Continue reading →
This week, the useR!2017 conference took place in Brussels. useR! is the annual international R User conference and is the main meeting of the R user and developer community.
More than 1,000 people from all over the world came to Brussels to discover...
R 3.4.1 (codename “Single Candle”) was released several days ago. You can get the latest binaries version from here. (or the .tar.gz source code from here). As mentioned last week by David Smith, R 3.4.1 includes several Windows related bug fixed: including an issue sometimes encountered when attempting to install packages on Windows, and problems displaying functions including Unicode characters … Continue reading &quot;R 3.4.1 is released – with some Windows related bug-fixes&quot;
In this article, I provide an R script for easily downloading closing prices for stocks included in the S&amp;P 500 index.
The R package dplyr is an extremely useful resource for data cleaning, manipulation, visualisation and analysis. It contains a large number of very useful functions and is, without doubt, one of my top 3 R packages today (ggplot2 and reshape2 being the others). When I was learning how to use dplyr for the first time,… Continue reading Useful dplyr Functions (w/examples) →
In case you missed them, here are some articles from June of particular interest to R users. R 3.4.1 &quot;Single Candle&quot; has been released. The Scientific Computing Coordinator at the FDA explains how R is used at the FDA and by sponsors for clinical trial submissions. Several useful tips related to including images in Rmarkdown documents. A review of one of R's best features — its community. It's now possible to include interactive R visualizations in Power BI reports (like graphics created with plotly or htmlwidgets). The Azure Data Science Virtual Machine for Windows now supports GPU-based computations with Microsoft...
For many R users the magrittr pipe is a popular way to arrange computation and famously part of the tidyverse. The tidyverse itself is a rapidly evolving centrally controlled package collection. The tidyverse authors publicly appear to be interested in re-basing the tidyverse in terms of their new rlang/tidyeval package. So it is natural to … Continue reading What is magrittr’s future in the tidyverse?
Excellent summary here.
Shiny is an R Package to deploy web apps using an R backend. Let’s face it, Shiny is awesome! It brings all the power of R to a simple web app with interactivity, user inputs, and interactive visualizations. If you don’t know Shiny yet, you can access a selection of apps on Show me shiny. As […]
The post The R Shiny packages you need for your web apps! appeared first on Enhance Data Science.
This is the third part of the series on volatility modelling. For other parts of the series follow the tag volatility. In this exercise set we will use GARCH models to forecast volatility. Answers to the exercises are available here. Exercise 1 Load the rugarch and the FinTS packages. Next, load the m.ibmspln dataset from […]
Related exercise sets:
Forecasting: Linear Trend and ARIMA Models Exercises (Part-2) Volatility modelling in R exercises (Part-2) Volatility modelling in R exercises (Part-1) Explore all our (__1000) R exercises
Find an R course using our R Course Finder directory
Dataframes are used in R to hold tabular data. Think of the prototypical spreadsheet or database table: a grid of data arranged into rows and columns. That’s a dataframe. The tibble R package
provides a fresh take on dataframes to fix some longstan...
The new issue of The R Journal is now available! You may Download the complete issue, or choose your topic of interest from the following links:   Table of contents Editorial  Roger Bivand 4 Contributed Research Articles iotools: High-Performance I/O Tools for R  Taylor Arnold, Michael J. Kane and Simon Urbanek 6 IsoGeneGUI: Multiple Approaches for Dose-Response Analysis of Microarray Data Using R  Martin Otava, Rudradev Sengupta, Ziv Shkedy, Dan Lin, Setia Pramana, Tobias Verbeke, Philippe Haldermans, Ludwig A. Hothorn, Daniel Gerhard, Rebecca M. Kuiper, Florian Klinglmueller and Adetayo Kasim 14 Network Visualization with ggplot2  Sam Tyner, François Briatte and Heike Hofmann 27 OrthoPanels: An R Package for Estimating a Dynamic Panel Model with Fixed Effects Using the Orthogonal Reparameterization Approach  Mark Pickup, Paul Gustafson, Davor Cubranic and Geoffrey Evans 60 The mosaic Package: Helping Students to Think with Data Using R  Randall Pruim, Daniel T Kaplan and Nicholas J Horton 77 smoof: Single- and Multi-Objective Optimization Test Functions  Jakob Bossek 103 minval: An R package for MINimal VALidation of Stoichiometric Reactions  Daniel Osorio, Janneth González and Andrés Pinzón 114 Working with Daily Climate Model Output Data in R and the futureheatwaves Package  G. Brooke Anderson, Colin Eason and Elizabeth A. Barnes 124 alineR: an R Package for Optimizing Feature-Weighted Alignments […]
Genetics is a data science, right? One of my Summer of data science learning points was to play with out of the box prediction tools. So let’s try out a few genomic prediction methods. The code is on GitHub, and the simulated data are on Figshare. Genomic selection is the happy melding of quantitative and […]
We’re pleased to introduce a new package, sweep, now on CRAN! Think of it like broom for the forecast package. The forecast package is the most popular package for forecasting, and for good reason: it has a number of sophisticated forecast modeling f...
In my last post, I tried to train a deep neural net to detect brand logos in images. For that, I downloaded the Flickr27-dataset, containing 270 images of 27 different brands. As this dataset is rather small I turned to google image search and wrote a...
I gave a talk titled, “Parallel Computation in R:  What We Want, and How We (Might) Get It,” at last week’s useR! 2017 conference in Brussels. You can view my slides here, and I think the conference organizers said the videos would be placed online, not sure of that though. The goal of the talk … Continue reading My Presentation at useR! 2017, Etc. →
Statistics are often taught in school by and for people who like Mathematics. As a consequence, in those class emphasis is put on leaning equations, solving calculus problems and creating mathematics models instead of building an intuition for probabilistic problems. But, if you read this, you know a bit of R programming and have access […]
Related exercise sets:
Hacking statistics or: How I Learned to Stop Worrying About Calculus and Love Stats Exercises (Part-1) Density-Based Clustering Exercises Data Science for Doctors – Part 3 : Distributions Explore all our (__1000) R exercises
Find an R course using our R Course Finder directory
Some time ago I published a blog post with the title Know your data structures!. In this previous post I explained how I improved the running time of a genetic algorithm. I promised to go more into detail about other noteworthy things in the code in a separate article since not everything was straightforward when … Continue reading Learning Club 16: Genetic Algorithms
The post Learning Club 16: Genetic Algorithms appeared first on verenahaunschmid.
This is also a flexible and smooth technique which captures the Non linearities in the data and helps us to fit Non linear Models.In this article I am going to discuss the implementation of GAMs in R using the 'gam' package .Simply saying GAMs are just a Generalized version of Linear Models in which the […]
Related Post
Second step with non-linear regression: adding predictors
Weather forecast with regression models – part 4
Weather forecast with regression models – part 3
Weather forecast with regression models – part 2
Weather forecast with regression models – part 1
A new, and long overdue, release of xts is now on CRAN!  The major change is the completely new plot.xts() written by Michael Weylandt and Ross Bennett, and which is based on Jeff Ryan's quantmod::chart_Series code.Do note that the new plot.xts() includes breaking changes to the original (and rather limited) plot.xts().  However, we believe the new functionality more than compensates for the potential one-time inconvenience.  And I will no longer have to tell people that I use plot.zoo() on xts objects!This release also includes more bug fixes than you can shake a stick at.  We squashed several bugs that could have crashed your R session.  We also fixed some (always pesky and tricky) timezone issues.  We've also done more sanity checking (e.g. for NA in the index), and provide more informative errors when things aren't right.  And last, but not least, unit tests are running again!I'm sure you were hoping to see some examples of the new plot.xts() functionality.  Rather than clutter up this blog post with code, check out the basic examples, and the panel functionality examples that Ross Bennett created.I'm looking forward to your questions and feedback!  If you have a question, please ask on Stack  Overflow and use the [r] and [xts] tags.  Or you can send an email to the R-SIG-Financemailing list (you must subscribe to post).  Open an issue on GitHub if you find a bug or want to request a feature, but please read the contributing guide first!
Holiday weekend, schmoliday weekend. Ben Goodrich and Jonah Gabry shipped RStan 2.16.2 (their numbering is a little beyond base Stan, which is at 2.16.0). This reintroduces error reporting that got lost in the 2.15 refactor, so please upgrade if you want to debug your Stan programs! Joe Haupt translated the JAGS examples in the second […]
The post Stan Weekly Roundup, 7 July 2017 appeared first on Statistical Modeling, Causal Inference, and Social Science.
There has been some talk of adding native pipe notation to R (for example here, here, and here). I think a critical aspect of such an extension would be to treat such a notation as syntactic sugar and not insist such a pipe match magrittr semantics, or worse yet give a platform for authors to … Continue reading In praise of syntactic sugar
by Fang Zhou, Data Scientist; and Graham Williams, Director of Data Science, all at Microsoft Rattle — the R Analytical Tool To Learn Easily — is a popular open-source GUI for data mining using R. It presents statistical and visual summaries of data, transforms data that can be readily modelled, builds both unsupervised and supervised models from the data, presents the performance of models graphically, and scores new datasets. All of the underlying R code is presented as a script for learning R and for running independent of Rattle. Collaborating with IGD Data Insight Team and under the guidance of...
Introduction Shiny by RStudio is a web application framework targeted for R programmers. It allows our analysis to be extended as an interactive web application  that can be […]
The post Motor vehicle collisions in New York City - R / Shiny Data Visualization appeared first on NYC Data Science Academy Blog.
In our latest R and Big Data article we discuss replyr. Why replyr replyr stands for REmote PLYing of big data for R. Why should R users try replyr? Because it lets you take a number of common working patterns and apply them to remote data (such as databases or Spark). replyr allows users to … Continue reading Working With R and Big Data: Use Replyr
I’m very pleased to announce that my new package tidygraph is now available on CRAN. As the name suggests, tidygraph is an entry into the tidyverse that provides a tidy framework for all things relational (networks/graphs, trees, etc.). tidygraph
...
Part 5 of 5 in the series Set TheoryOrdered and Unordered Pairs A pair set is a set with two members, for example, , which can also be thought of as an unordered pair, in that . However, we seek a more a strict and rich object that tells us...
The post Set Theory Ordered Pairs and Cartesian Product with R appeared first on Aaron Schlegel.
R has a lot of tools to speed up computations making use of multiple CPU cores either on one computer, or on multiple machines. This series of exercises aims to introduce the basic techniques for implementing parallel computations using multiple CPU cores on one machine. The initial step in preparation for parallelizing computations is to […]
Related exercise sets:
Shiny Application Layouts Exercises (Part-5) Getting started with Plotly: basic Plots Density-Based Clustering Exercises Explore all our (__1000) R exercises
Find an R course using our R Course Finder directory
We are looking for a person with good skills and experience in Natural Language Processing for our project on the boundary of history and social network analysis. The project involves analysis of historical documents and written communication (e.g. letters) in Portuguese. Skills/experience: Skills and experience in text mining and natural language processing, especially solving Entity […]
Proposed solution to Euler Problem 29: Distinct Powers in the R language for statistical computing. Continue reading →
The post Euler Problem 29: Distinct Powers appeared first on The Devil is in the Data.
In an earlier post, I described in a fair amount of detail an algorithm to generate correlated binary or Poisson data. I mentioned that I would be updating simstudy with functions that would make generating these kind of data relatively painless. Well, I have managed to do that, and the updated package (version 0.1.3) is available for download from CRAN. There are now two additional functions to facilitate the generation of correlated data from binomial, poisson, gamma, and uniform distributions: genCorGen and addCorGen. Here’s a brief intro to these functions.
Generate generally correlated data
genCorGen is an extension of genCorData, which was provided in earlier versions of simstudy to generate multivariate normal data. In the first example below, we are generating data from a multivariate Poisson distribution. To do this, we need to specify the mean of the Poisson distribution for each new variable, and then we specify the correlation structure, just as we did with the normal distribution.
l
I’ll often keep non-sensitive data on Dropbox so that I can access it on all my machines without gumming up git. I just wrote a small script to find the Dropbox location on each of my computers automatically. The crucial information is available here, from Dropbox. My small snippet of code is the following: I haven’t […]
Data wrangling is a task of great importance in data analysis. Data wrangling, is the process of importing, cleaning and transforming raw data into actionable information for analysis. It is a time-consuming process which is estimated to take about 60-80% of analyst’s time. In this series we will go through this process. It will be […]
Related exercise sets:
Start plotting data! Data wrangling : I/O (Part-2) Select and Query Exercise Explore all our (__1000) R exercises
Find an R course using our R Course Finder directory
In the sponsor presentation for Microsoft at the useR!2017 conference in Brussels this morning, I thought I'd share how perceptions of R have changed over the years. Today, R known as is popular, comprehensive, accepted, scalable, production-ready and supported software environment for data analysis, but that wasn't always the case. You can find the slides for my presentation R, Then and Now below: (By the way, you can find links to the production case studies referenced on Slide 7, here.) The talk was also recording, and I'll add a link to the video in this post when it's available.
INTRODUCTION iPlots is a package which provides interactive statistical graphics, written in Java. You can find many interesting plots such as histograms, barcharts, scatterplots, boxplots, fluctuation diagrams, parallel coordinates plots and spineplots. The amazing part is that all of these plots support querying, linked highlighting, color brushing, and interactive changing of parameters. Before proceeding, please […]
Related exercise sets:
How to create visualizations with iPlots package in R Multiple Regression (Part 3) Diagnostics Building Shiny App exercises part 4 Explore all our (__1000) R exercises
Find an R course using our R Course Finder directory
Ordinary Least Squares (OLS) linear regression is a statistical technique used for the analysis and modelling of linear relationships between a response variable and one or more predictor variables. If the relationship between two variables appears to be linear, then a straight line can be fit to the data in order to model the relationship.… Continue reading Ordinary Least Squares (OLS) Linear Regression in R →
One of the nice features of the ggapproach to plotting is that one can save plots as R objects at any step and use later to render and/or modify. I used that feature extensively while creating maps with ggplot2 (see my previous posts: one, two, three, four, five). It is just convenient to first create a canvas with all the theme parameters appropriate for a map, and then overlay the map layer. At some point I decided to check if that workflow was computationally efficient or not. To my surprise, the usage of canvas reduces the rendering time of a ggplot quite a lot. To my further surprise, this finding holds for simple plots as well as maps.
BiclustGUI 1.1.1: BiBit Workflow for Noisy Biclusters
Ewoud De Troyer, University of Hasselt (CenStat)
Introduction - 1.1.1 Update
The 1.1.1 update of the BiclustGUI introduces two 2 new features:
The BiBit Workflow Tool for BiBit results:
Discov...
Here's a quick roundup of some case studies published recently on the Microsoft Customer Stories portal, with examples of companies running R in production environments using the Microsoft stack. eToro, the social trading network, uses Microsoft R Server and Azure Machine Learning so that investors can automatically duplicate market strategies and actual investment decisions of successful traders in the eToro network to use in their own portfolios. Acxiom, the marketing analytics company, is using SQL Server R Services to build more accurate models using larger data sets, which results in more precisely targeted marketing for customers. Merck, the pharmaceutical company,...
On the Riddler of last week, first a birthday puzzle: Given a group of 23 persons, what is the probability of observing three pairs of identical birthdays? which can be found by a quick simulation as ave=0 for (t in 1:1e6){ dupz=dates[duplicated(sample(1:365,23,rep=TRUE))] ave=ave+as.integer((length(dupz)==3)&amp; (length(unique(dupz))==3))}} ave/M returning a value of 0.0183, but which combinatoric resolution I […]
In the first part in a series on Tidy Time Series Analysis, we’ll use tidyquant to investigate CRAN downloads. You’re probably thinking, “Why tidyquant?” Most people think of tidyquant as purely a financial package and rightfully so. However, b...
The useR!2017 conference, the annual meeting of R users worldwide and the largest to date, is sold completely sold out. But for those that couldn't make it to Brussels, Microsoft will be live-streaming the conference at aka.ms/useRConference-live. Bookmark that page and follow along during the conference starting with the opening of the main program at 9:00 AM Brussels time (UTC+2hrs) on Wednesday, July 5. To help you plan your viewing schedule, you can find the complete useR!2017 schedule here. (You can also find links to keynotes and presentations from my Microsoft colleagues here.) As with last year's conference (you can...
This is the second part of the series on volatility modelling. For other parts of the series follow the tag volatility. In this exercise set we will use the dmbp dataset from part-1 and extend our analysis to GARCH (Generalized Autoregressive Conditional Heteroscedasticity) models. Answers to the exercises are available here. Exercise 1 Load the […]
Related exercise sets:
Volatility modelling in R exercises (Part-1) Introduction to copulas Exercises (Part-2) Evaluate your model with R Exercises Explore all our (__1000) R exercises
Find an R course using our R Course Finder directory
Intro
We live in a society heavily reliant upon communication, networking and information exchange. Think about all the chat apps that have recently popped out: WhatsApp, Messenger, Skype, Viber, Slack (not to mention Snapchat or Telegram). They have ...
Weighted effect coding is a technique for dummy coding that can have attractive properties, particularly when analysing observational data. In a new publication in the R Journal we explain the rationale of weighted effect coding, ...
Good news, everyone! [crayon-595a10c0c7b68967137438-i/] package has recently received a major update. The version on CRAN is now v2.0.0. I thought that this is a big deal, so I decided to pause for a moment and explain what has happened, and why this new version is interesting. First of all, there is a new function, [crayon-595a10c0c7b73732349536-i/], […]
There is an opportunity at IARC for a postdoctoral statistician to work on a challenging problem with a big impact. One of the important tasks of the International Agency for Research on Cancer (IARC) is to collate and publish global … Continue reading →
INTRODUCTION iPlots is a package which provides interactive statistical graphics, written in Java. You can find many interesting plots such as histograms, barcharts, scatterplots, boxplots, fluctuation diagrams, parallel coordinates plots and spineplots. The amazing part is that all of these plots support querying, linked highlighting, color brushing, and interactive changing of parameters. Furthermore, iPlots includes […]
Related exercise sets:
Replicating Plots – Boxplot Exercises Experimental Design Exercises Accessing Dataframe Objects Exercises Explore all our (__1000) R exercises
Find an R course using our R Course Finder directory
New Zealand’s Family Violence Death Review Committee recently released their Fifth Report Data: January 2009 to December 2015, some grim reading on what it describes rightly as “the devastation of family violence”.
I was interested by Figure 3 o...
Statistics are often taught in school by and for people who like Mathematics. As a consequence, in those class emphasis is put on leaning equations, solving calculus problems and creating mathematics models instead of building an intuition for probabilistic problems. But, if you read this, you know a bit of R programming and have access […]
Related exercise sets:
Density-Based Clustering Exercises Data Science for Doctors – Part 3 : Distributions Forecasting for small business Exercises (Part-2) Explore all our (__1000) R exercises
Find an R course using our R Course Finder directory
As I said before, I’m firmly siding with Andrew Gelman (see e.g. here) in that model checking is dangerously neglected in Bayesian practice. The philosophical criticism against “rejecting” models (double-using data etc. etc.) is all well, but when using Bayesian methods in practice, I see few sensible alternatives to residual checks (both guessing a model and…
In our latest installment of “R and big data” let’s again discuss the task of left joining many tables from a data warehouse using R and a system called &quot;a join controller&quot; (last discussed here). One of the great advantages to specifying complicated sequences of operations in data (rather than in code) is: it is … Continue reading Join Dependency Sorting
A few months ago rOpenSci’s Scott Chamberlain asked me for feedback about a new package of his called crul, an http client like httr, so basically something you use for e.g. writing a package interfacing an API. He told me that a great thing about cr...
The R core team announced today the release of R 3.4.1 (codename: Single Candle). This release fixes...
Here’s some things that have been going on with Stan since the last week’s roundup Stan® and the logo and were was granted U.S. Trademark Registrations No. 5,222,891 and U.S. Serial Number: 87,237,369. Hard to feel special when there were millions of products ahead of you. Trademarked names are case insensitive and they required a […]
The post Stan Weekly Roundup, 30 June 2017 appeared first on Statistical Modeling, Causal Inference, and Social Science.
Candlestick, Pie, Gauge, Intensity Charts In the fifth part of our journey we will meet some special but more and more usable types of charts that googleVis provides. More specifically you will learn about the features of Candlestick, Pie, Gauge and Intensity Charts. Read the examples below to understand the logic of what we are […]
Related exercise sets:
Data Visualization with googleVis exercises part 2 Data Visualization with googleVis exercises part 3 Data Visualization with googleVis exercises part 4 Explore all our (__1000) R exercises
Find an R course using our R Course Finder directory
In HexJSON HTMLWidget for R, Part 1 I described a basic HTMLwidget for rendering hexJSON maps using d3-hexJSON, and HexJSON HTMLWidget for R, Part 2 described updates for supporting colour. Having booked off today for emergency family cover that turned out not to be required, I had another stab at the package, so it now […]
Splines are a smooth and flexible way of fitting Non linear Models and learning the Non linear interactions from the data.In most of the methods in which we fit Non linear Models to data and learn Non linearities is by transforming the data or the variables by applying a Non linear transformation. Cubic Splines Cubic […]
Related Post
Chi-Squared Test – The Purpose, The Math, When and How to Implement?
Missing Value Treatment
R for Publication by Page Piccinini
Assessing significance of slopes in regression models with interaction
First steps with Non-Linear Regression in R
In the development of operational loss forecasting models, the Frequency-Severity modeling approach, which the frequency and the severity of a Unit of Measure (UoM) are modeled separately, has been widely employed in the banking industry. However, sometimes it also makes sense to model the operational loss directly, especially for UoMs with non-material losses. First of […]
In my previous post – HexJSON HTMLWidget for R, Part 1 – I described a first attempt at an HTMLwidget for displaying hexJSON maps using d3-hexJSON. I had another play today and added a few extra features, including the ability to: add a grid (as demonstrated in the original d3-hexJSON examples), modify the default colour […]
I was recently alerted (thanks Maëlle and Mikhail!) to an enlightening presentation from last years' useR! conference. (This year's useR! conference takes place next week in Belgium.) Paul H Schuette, Scientific Computing Coordinator at the FDA Center for Drug Evaluation and Research (CDER), talked about how R is used in the process of regulating and approving drugs at the FDA. In what has become a common theme of FDA presentations at R conferences, Schuette refutes the fallacy that SAS is the only software that can be used for FDA submissions, by sponsors such as pharmaceutical companies. On the contrary, he...
This last weekend I helped Danny Kaplan and Kathryn Kozak (Coconino Community College) put on a StatPREP workshop. We were also joined by Amelia McNamara (Smith College) and Joe Roith (St. Catherine’s University). The idea behind StatPREP is to work directly … Continue reading →
In the last set of exercise of data.table ,we saw some interesting features of data.table .In this set we will cover some of the advanced features like set operation ,join in data.table.You should ideally complete the first part before attempting this one . Answers to the exercises are available here. If you obtained a different […]
Related exercise sets:
Data Manipulation with Data Table -Part 1 Shiny Application Layouts Exercises (Part-2) Bioinformatics Tutorial with Exercises in R (part 1) Explore all our (__1000) R exercises
Find an R course using our R Course Finder directory
Proposed solution to Euler Problem 28 in the R language for statistical computing. What is the sum of the numbers on the diagonals in a 1001 by 1001 spiral? Continue reading →
The post The Ulam Spiral (Euler Problem 28) appeared first on The Devil is in the Data.
Introduction Having figured out how to perform walk-forward analysis in Python with backtrader, I want to have a look at evaluating a strategy’s performance. So far, I have cared about only one metric: the final value of the account at the end of a backtest relative. This should not be the only metric considered. Most…Read more Stock Trading Analytics and Optimization in Python with PyFolio, R’s PerformanceAnalytics, and backtrader
While going over some of the discussion related to my last post I came up with a really neat way to use wrapr::let() and rlang/tidyeval together. Please read on to see the situation and example.Suppose we want to parameterize over a couple of names, one denoting a variable coming from the current environment and one … Continue reading Using wrapr::let() with tidyeval
By Gabriel Vasconcelos Motivation In a late post I talked about inference after model selection showing that a simple double selection procedure is enough to solve the problem. In this post I’m going to talk about a generalization of the … Continue reading →
In advance of the recent UK general election, ODI Leeds published an interactive hexmap of constituencies to provide a navigation surface over various datasets relating to Westminster constituencies: As well as the interactive front end, ODI Leeds published a simple JSON format for sharing the hex data – hexjson that allows you to specify an […]
Mango Solutions are excited to announce that the venue and dates have now been confirmed for the third EARL Boston Conference. Delegates will enjoy a new vibe this year,...
Lately, I have been more and more taken with tidy principles of data analysis. They are elegant and make analyses clearer and easier to comprehend. Following the tidyverse and ggraph, I have been quite intrigued by applying tidy principles to text anal...
Hello R users! We have a big announcement this week as well, DataCamp just released our first Spark in R course: Introduction to Spark in R using sparklyr! This course is taught by Richie Cotton.
R is mostly optimized to help you write data analysis ...
This exercise set provides (further) practice in writing Excel documents using the xlsx package as well as importing and general data manipulation. Specifically we have loops in order for you to practice scaling. A previous exercise set focused on writing a simple sheet with the same package, see here. We will use a subset of […]
Related exercise sets:
Using the xlsx package to create an Excel file Building Shiny App exercises part 1 Building Shiny App exercises part 2 Explore all our (__1000) R exercises
Find an R course using our R Course Finder directory
Below is an example showing how to fit a Generalized Linear Model with H2O in R. The output is much more comprehensive than the one generated by the generic R glm().
I’m pleased to announce the release of the dbplyr package, which now contains all dplyr code related to connecting to databases. This shouldn’t affect you-as-a-user much, but it makes dplyr simpler, and makes it easier to release improvements just for database related code. You can install the latest version of dbplyr with: install.packages(&quot;dbplyr&quot;) DBI and […]
Written by Chen Trilnik and Jack Yip. To view the original source code, visit our Github repo here.   I. Introduction Business Objective In the last decade, […]
The post Forecasting Economic Risk in the EU into 2020 appeared first on NYC Data Science Academy Blog.
textDon’t worry about a thing ’cause every little thing gonna be alright (Three Little Birds, Bob Marley) One of my favourite books is The Computational Beauty of Nature by Gary William Flake where there is a fantastic chapter about fractals in which I discovered the L-Systems. L-Systems were conceived  in 1968 by Aristide Lindenmayer, a Hungarian … Continue reading A Shiny App to Draw Curves Based on L-System →
textRmarkdown is an enormously useful system for combining text, output and graphics generated by R into a single document. Images, in particular, are a powerful means of communication in a report, whether they be data visualizations, diagrams, or pictures. To maximize the power of those images, Zev Ross has created a comprehensive list of tips and tricks for working with images in R markdown documents. Those tips include: How to change the width and height of images created by R, and of imported image files. How to change the resolution of your images. (Usually not an issue for HTML output,...
text
DENSITY THE AVERAGE PERSON EXPERIENCES CLICK TO ENLARGE In Alaska, there’s about one person for each square mile of land. You might picture a typical Alaskan not being able to see the next house. But it’s not that way of course. Most of Alaska is uninhabited. People have crowded into a few areas. The average […]
The post Weighted population density appeared first on Decision Science News.
textVolatility modelling is typically used for high frequency financial data. Asset returns are typically uncorrelated while the variation of asset prices (volatility) tends to be correlated across time. In this exercise set we will use the rugarch package (package description: here) to implement the ARCH (Autoregressive Conditional Heteroskedasticity) model in R. Answers to the exercises […]
Related exercise sets:
Forecasting: Multivariate Regression Exercises (Part-4) Introduction to copulas Exercises (Part-2) Conditional execution exercises Explore all our (>1000) R exercises
Find an R course using our R Course Finder directory
textI’m pleased to announce that bigrquery 0.4.0 is now on CRAN. bigrquery makes it possible to talk to Google’s BigQuery cloud database. It provides both DBI and dplyr backends so you can interact with BigQuery using either low-level SQL or high-level dplyr verbs. Install the latest version of bigrquery with: install.packages(&quot;bigrquery&quot;) Basic usage Connect to […]
From dplyr issue 2916. The following appears to work. suppressPackageStartupMessages(library(&quot;dplyr&quot;)) COL &lt;- &quot;homeworld&quot; starwars %>% group_by(.data[[COL]]) %>% head(n=1) ## # A tibble: 1 x 14 ## # Groups: COL [1] ## name height mass hair_color skin_color eye_color birth_year ## &lt;chr> &lt;int> &lt;dbl> &lt;chr> &lt;chr> &lt;chr> &lt;dbl> ## 1 Luke Skywalker 172 77 blond fair blue … <a href="http://www.win-vector.com/blog/2017/06/please-consider-using-wraprlet-for-replacement-tasks/">Continue reading <span>Please Consider Using wrapr::let() for Replacement Tasks</span></a>
I’m very pleased to announce my DataCamp course on Visualizing Time Series Data in R. This course is also part of the  Time Series with R skills track. Feel free to have a look, the first chapter is free! Course Description As the saying goes, “A chart is worth a thousand words”. This is why visualization […]
I’ve caved in to the hex sticker craze, and produced some hex stickers for the forecast package for R. If you attend a workshop I teach, I’ll give you one. Otherwise you can order (in bulk) from hexi.pics.
The padr package was designed to prepare datetime data for analysis. That is, to take raw, timestamped data, and quickly convert it into a tidy format that can be analyzed with all the tidyverse tools. Recently, a colleague and I discovered a second use for the package that I had not anticipated: checking data quality. Every analysis should contain checking if data are as expected. In the case of timestamped data, observations are sometimes missing due to technical malfunction of the system that produced them. Here are two examples that show how pad and thicken can be leveraged to detect problems in timestamped data quickly.
textSince about October last year, I’ve been writing an introduction to R book. It’s been quite the experience. I’ve finally started making time to document some of the interesting things I’ve learned (about R, about writing, about how to bring...Continue Reading →
<p>I just discovered an interesting <code>ggplot2</code> extension, <code>geofacet</code>, that supports arranging facet panels that mimics geographic topoloty.</p>
<p>After playing with it, I realized that it is not only for visualizing <code>geo</code>-related data, but also can be fun for presenting data to mimics pixel art.</p>
<p></p>
<p></p>
<p>Here is an example using the Turkey shape:</p>
<pre><code>Turkey &lt;- read.csv(&quot;http://pages.iu.edu/~cdesante/turkey.csv&quot;)
Turkey = Turkey[, -3]
colnames(Turkey)[2:1] = c(&quot;row&quot;, &quot;col&quot;)
Turkey$row = max(Turkey$row) - Turkey$row +1
Turkey$name &lt;- Turkey$code &lt;- paste0('turkey', 1:nrow(Turkey))

require(ggplot2)
require(geofacet)
x &lt;- split(eu_gdp, eu_gdp$code)
x &lt;- x[sample.int(length(x), nrow(Turkey), replace=T)]
for (i in 1:length(x)) {
  x[[i]]$code = Turkey$code[i]
}
y &lt;-  do.call(rbind, x)

p1 &lt;- ggplot(y, aes(gdp_pc, year))+ geom_line(color=&quot;steelblue&quot;) +
  facet_geo(~code, grid=Turkey, scales='free')
print(p1)
</code></pre>
<p></p>
<pre><code>p1 + theme_void() + theme(strip.text.x = element_blank())
</code></pre>
<p></p>
Data visuals: notes for my talks in 2017
Supplementary notes for CJ Brown’s talks on dataviz in 2017 for Griffith
University’s honours students and the UQ Winterschool in
Bioinformatics.
Skip to the quiz
Visualsing sexual dimorphism in elephant ...
The importFile() function in H2O is extremely efficient due to the parallel reading. The benchmark comparison below shows that it is comparable to the read.df() in SparkR and significantly faster than the generic read.csv().
Adding Features to your Charts We saw in the previous charts some basic and well-known types of charts that googleVis offers to users. Before continuing with other, more sophisticated charts in the next parts we are going to “dig a little deeper” and see some interesting features of those we already know. Read the examples […]<div>
<h3>Related exercise sets:</h3>
<ol>
<li><a rel="bookmark" href="http://www.r-exercises.com/2017/06/11/data-visualization-with-googlevis-exercises-part-2/" title="Data Visualization with googleVis exercises part 2">Data Visualization with googleVis exercises part 2 </a></li>
<li><a rel="bookmark" href="http://www.r-exercises.com/2017/06/20/data-visualization-with-googlevis-exercises-part-3/" title="Data Visualization with googleVis exercises part 3">Data Visualization with googleVis exercises part 3 </a></li>
<li><a rel="bookmark" href="http://www.r-exercises.com/2017/06/04/data-visualization-with-googlevis-exercises-part-1/" title="Data Visualization with googleVis exercises part 1">Data Visualization with googleVis exercises part 1 </a></li>
<li><a href="http://www.r-exercises.com/start-here-to-learn-r/">Explore all our (>1000) R exercises</a></li>
<li>
<a href="http://www.r-exercises.com/r-courses/">Find an R course using our R Course Finder</a> directory</li>
</ol>
</div>
Stefan Feuerriegel This blog entry concerns our course on “Operations Reserch with R” that we teach as part of our study program. We hope that the materials are of value to lectures and everyone else working in the field of numerical optimiatzion. Course outline The course starts with a review of numerical and linear algebra … <a href="http://www.rblog.uni-freiburg.de/2017/06/25/operations-research-with-r/">Continue reading<span> &quot;Operations Research with R&quot;</span></a>
<p>This week’s R bulletin will cover topics on how to resolve some common errors in R. Hope you like this R weekly bulletin. Enjoy reading! Shortcut Keys 1. Find and Replace – Ctrl+F 2. Find Next – F3 3. Find Previous – Shift+F3 Problem Solving Ideas Resolving the ‘: cannot open the connection’ Error There...</p>
<p>The post <a rel="nofollow" href="https://www.quantinsti.com/blog/r-weekly-bulletin-vol-xii/">R Weekly Bulletin Vol – XII</a> appeared first on <a rel="nofollow" href="https://www.quantinsti.com/"></a>.</p>
In the development of operational loss models, it is important to identify which distribution should be used to model operational risk measures, e.g. frequency and severity. For instance, why should we use the Gamma distribution instead of the Inverse Gaussian distribution to model the severity? In my previous post https://statcompute.wordpress.com/2016/11/20/modified-park-test-in-sas, it is shown how to […]
Pooling the polls with Bayesian statistics
In an important 2005 article in the Australian Journal of Political Science, Simon Jackman set out a statistically-based approach to pooling polls in an election campaign. He describes the sensible intuitive...
R is incredible software for statistics and data science. But while the bits and bytes of software are an essential component of its usefulness, software needs a community to be successful. And that's an area where R really shines, as Shannon Ellis explains in this lovely ROpenSci blog post. For software, a thriving community offers developers, expertise, collaborators, writers and documentation, testers, agitators (to keep the community and software on track!), and so much more. Shannon provides links where you can find all of this in the R community: #rstats hashtag — a responsive, welcoming, and inclusive community of R...
So you are using this pipeline to have data treated by different functions in R. For example, you may be imputing some missing values using the simputation package. Let us first load the only realistic dataset in R > data(retailers, … <a href="http://www.markvanderloo.eu/yaRb/2017/06/23/track-changes-in-data-with-the-lumberjack/">Continue reading <span>→</span></a>
<p>Part 3 of 3 in the series Set TheoryThe union and intersection set operations were introduced in a previous post using two sets, and . These set operations can be generalized to accept any number of sets. Arbitrary Set Unions Operation Consider a set of infinitely many sets: It would...</p>
<p>The post <a rel="nofollow" href="http://www.aaronschlegel.com/set-theory-arbitrary-unions-intersections-r/">Set Theory Arbitrary Union and Intersection Operations with R</a> appeared first on <a rel="nofollow" href="http://www.aaronschlegel.com/">Aaron Schlegel</a>.</p>
Data wrangling is a task of great importance in data analysis. Data wrangling, is the process of importing, cleaning and transforming raw data into actionable information for analysis. It is a time-consuming process which is estimated to take about 60-80% of analyst’s time. In this series we will go through this process. It will be […]<div>
<h3>Related exercise sets:</h3>
<ol>
<li><a rel="bookmark" href="http://www.r-exercises.com/2016/07/06/data-shape-transformation-with-reshape/" title="Data Shape Transformation With Reshape()">Data Shape Transformation With Reshape() </a></li>
<li><a rel="bookmark" href="http://www.r-exercises.com/2017/04/11/building-shiny-app-exercises-part-10/" title="Building Shiny App exercises part 10">Building Shiny App exercises part 10 </a></li>
<li><a rel="bookmark" href="http://www.r-exercises.com/2016/08/26/reshape-2-exercises/" title="Reshape 2 Exercises">Reshape 2 Exercises </a></li>
<li><a href="http://www.r-exercises.com/start-here-to-learn-r/">Explore all our (>1000) R exercises</a></li>
<li>
<a href="http://www.r-exercises.com/r-courses/">Find an R course using our R Course Finder</a> directory</li>
</ol>
</div>
In this post, I demonstrate a few techniques for plotting information from a
relatively simple mixed-effects model fit in R. These plots can help us develop intuitions about what these models are doing and what “partial pooling” means.
The sleeps...
Face Recognition in R OpenCV is an incredibly powerful tool to have in your toolbox. I have had a lot of success using it in Python but very little success in R. I haven’t done too much other than searching Google but it seems as if “imager” and “videoplayR” provide a lot of the functionality […]
Which policy instruments should we use to cost-effectively reduce greenhouse gas emissions? For a given technological level there are many economic arguments in favour of tradeable emission certificates or a carbon tax: they generate static efficiency ...
Power BI has long had the capability to include custom R charts in dashboards and reports. But in sharp contrast to standard Power BI visuals, these R charts were static. While R charts would update when the report data was refreshed or filtered, it wasn't possible to interact with an R chart on the screen (to display tool-tips, for example). But in the latest update to Power BI, you can create create R custom visuals that embed interactive R charts, like this: The above chart was created with the plotly package, but you can also use htmlwidgets or any other...
By Yuri Resende   Today we will use an online convex optimization technique to build a very simple algorithm for portfolio allocation. Of course this is just an illustrative post and we are going to make some simplifying assumptions. The … <a href="https://insightr.wordpress.com/2017/06/22/online-portfolio-allocation-with-a-very-simple-algorithm/">Continue reading <span>→</span></a>
Among the many R packages, there is the outbreaks package. It contains datasets on epidemics, on of which is from the 2013 outbreak of influenza A H7N9 in China, as analysed by Kucharski et al (2014). I will be using their data as an example to test whether we can use Machine Learning algorithms for […]<ul></ul>
<strong><p>Related Post</p></strong><ol>
<li><a rel="bookmark" href="https://datascienceplus.com/graphical-presentation-of-missing-data-vim-package/" title="Permanent Link toGraphical Presentation of Missing Data; VIM Package">Graphical Presentation of Missing Data; VIM Package</a></li>
<li><a rel="bookmark" href="https://datascienceplus.com/creating-graphs-with-python-and-goopycharts/" title="Permanent Link toCreating Graphs with Python and GooPyCharts">Creating Graphs with Python and GooPyCharts</a></li>
<li><a rel="bookmark" href="https://datascienceplus.com/visualizing-tennis-grand-slam-winners-performances/" title="Permanent Link toVisualizing Tennis Grand Slam Winners Performances">Visualizing Tennis Grand Slam Winners Performances</a></li>
<li><a rel="bookmark" href="https://datascienceplus.com/plotting-data-online-via-plotly-and-python/" title="Permanent Link toPlotting Data Online via Plotly and Python">Plotting Data Online via Plotly and Python</a></li>
<li><a rel="bookmark" href="https://datascienceplus.com/using-postgresql-and-shiny-with-a-dynamic-leaflet-map-monitoring-trash-cans/" title="Permanent Link toUsing PostgreSQL and shiny with a dynamic leaflet map: monitoring trash cans">Using PostgreSQL and shiny with a dynamic leaflet map: monitoring trash cans</a></li>
</ol>
A new minor version 0.2.3 of RcppCCTZ is now on CRAN.
RcppCCTZ uses Rcpp to bring CCTZ to R. CCTZ is a C++ library for translating between absolute and civil times using the rules of a time zone. In fact, it is two libraries. One for dealing with civ...
<p>Solution to Euler Problem 25 in the R language. What is the index of the first term in the Fibonacci sequence to contain 1000 digits? <a href="http://r.prevos.net/euler-problem-25/">Continue reading <span>→</span></a></p>
<p>The post <a rel="nofollow" href="http://r.prevos.net/euler-problem-25/">Large integers in R: Fibonacci number with 1000 digits, Euler Problem 25</a> appeared first on <a rel="nofollow" href="http://r.prevos.net/">The Devil is in the Data</a>.</p>
The Windows edition of the Data Science Virtual Machine (DSVM), the all-in-one virtual machine image with a wide-collection of open-source and Microsoft data science tools, has been updated to the Windows Server 2016 platform. This update brings built-in support for Docker containers and GPU-based deep learning. GPU-based Deep Learning. While prior editions of the DSVM could access GPU-based capabilities by installing additional components, everything is now configured and ready at launch. The DSVM now includes GPU-enabled builds of popular deep learning frameworks including CNTK, Tensorflow, and MXNET. It also includes Microsoft R Server 9.1, and several machine-learning functions in the...
Neural network have become a corner stone of machine learning in the last decade. Created in the late 1940s with the intention to create computer programs who mimics the way neurons process information, those kinds of algorithm have long been believe to be only an academic curiosity, deprived of practical use since they require a […]<div>
<h3>Related exercise sets:</h3>
<ol>
<li><a rel="bookmark" href="http://www.r-exercises.com/2017/06/15/neural-networks-exercises-part-2/" title="Neural networks Exercises (Part-2)">Neural networks Exercises (Part-2) </a></li>
<li><a rel="bookmark" href="http://www.r-exercises.com/2017/06/08/neural-networks-exercises-part-1/" title="Neural networks Exercises (Part-1)">Neural networks Exercises (Part-1) </a></li>
<li><a rel="bookmark" href="http://www.r-exercises.com/2017/05/02/forecasting-for-small-business-exercises-part-4/" title="Forecasting for small business Exercises (Part-4)">Forecasting for small business Exercises (Part-4) </a></li>
<li><a href="http://www.r-exercises.com/start-here-to-learn-r/">Explore all our (>1000) R exercises</a></li>
<li>
<a href="http://www.r-exercises.com/r-courses/">Find an R course using our R Course Finder</a> directory</li>
</ol>
</div>
<p>Dear Fantasy Football Analytics Community, Four years ago, we released web apps to help people make better decisions in fantasy football based on the wisdom of the crowd.  Over the past[...]</p>
<p>The post <a rel="nofollow" href="http://fantasyfootballanalytics.net/2017/06/call-help-rshiny-developer.html">Call for Help: R/Shiny Developer</a> appeared first on <a rel="nofollow" href="http://fantasyfootballanalytics.net/">Fantasy Football Analytics</a>.</p>
Smoothing a time-series with a Bayesian model
Recently I looked at fitting a smoother to a
time-series using
Bayesian modelling.
Now I will look at how you can control the smoothness by using more or
less informative priors on the precision (1/varianc...
Nic Crane, Data Scientist
The agenda for EARL London has just been released, and with three streams running in parallel, I always like to pick which talks I'll be going to in advance, to make sure I get...
The first part of this post is a quick update on Tony Cooper’s of Double Digit Numerics’s volatility ETN momentum … <span><a href="https://quantstrattrader.wordpress.com/2017/06/20/an-out-of-sample-update-on-ddns-volatility-momentum-trading-strategy-and-beta-convexity/">Continue reading <span>→</span></a></span>
For the past four years, recruiting firm Burtch Works has conducted a simple survey of data scientists with just one question: &quot;Which do you prefer to use — SAS, R or Python&quot;. The results for this year's survey of 1,046 respondents are in: R: 40% (2016: 42%) SAS: 34% (2016: 39%) Python: 26% (2016: 20%) Compared to last year's results, Python has gained 6 percentage points, mainly at the expense of SAS. (2016 results do not add to 100% due to rounding.) The trend for the commercial tool compared to the open-source tools is apparent looking at the data from...
Scatter &amp; Bubble chart This is the third part of our data visualization series and at this part we will explore the features of two more of the charts that googleVis provides. Read the examples below to understand the logic of what we are going to do and then test yous skills with the exercise […]<div>
<h3>Related exercise sets:</h3>
<ol>
<li><a rel="bookmark" href="http://www.r-exercises.com/2017/06/11/data-visualization-with-googlevis-exercises-part-2/" title="Data Visualization with googleVis exercises part 2">Data Visualization with googleVis exercises part 2 </a></li>
<li><a rel="bookmark" href="http://www.r-exercises.com/2017/06/04/data-visualization-with-googlevis-exercises-part-1/" title="Data Visualization with googleVis exercises part 1">Data Visualization with googleVis exercises part 1 </a></li>
<li><a rel="bookmark" href="http://www.r-exercises.com/2017/04/30/shiny-application-layouts-exercises-part-3/" title="Shiny Application Layouts exercises part 3">Shiny Application Layouts exercises part 3 </a></li>
<li><a href="http://www.r-exercises.com/start-here-to-learn-r/">Explore all our (>1000) R exercises</a></li>
<li>
<a href="http://www.r-exercises.com/r-courses/">Find an R course using our R Course Finder</a> directory</li>
</ol>
</div>
Traditionally, statistics like wOBA (weighted on-base average) have been calculated using league averages. While building the baseballDBR package, I thought it would be interesting to group the American and National when making wOBA calculations. In th...
<p>Great painters can teach us a lot about how to compose a good palet for our plot. That is why we look here at a way to algorithmically derive a colour palette from any jpeg image. Application of derived palettes on ggplot is also showed.</p>
<p>The post <a rel="nofollow" href="http://www.milanor.net/blog/build-color-palette-from-image-with-paletter/">How to build a color palette from any image with R and k-means algo</a> appeared first on <a rel="nofollow" href="http://www.milanor.net/blog">MilanoR</a>.</p>
Back in February, I wrote some R code to analyse tweets covering the 2017 Lorne Genome conference. It worked pretty well. So I reused the code for two recent bioinformatics meetings held in Sydney: the Sydney Bioinformatics Research Symposium and the VIZBI 2017 meeting. So without further ado, here are the reports in markdown format, … <a href="https://nsaunders.wordpress.com/2017/06/20/visualising-twitter-coverage-of-recent-bioinformatics-conferences/">Continue reading <span>Visualising Twitter coverage of recent bioinformatics conferences</span></a>
<img width="300" alt="Partial Least Squares can be used find drivers of consumer preference" src="https://www.displayr.com/wp-content/uploads/2017/04/arrows-780-480-300x195.png" class="webfeedsFeaturedVisual wp-post-image" height="195">Partial Least Squares (PLS) is a popular method for relative importance analysis in fields where the data typically includes more predictors than observations. Relative importance analysis is a general term applied to any technique used for...
In this post I explore two different methods for computing the relative importance of predictors in regression: Johnson’s Relative Weights and Partial Least Squares (PLS) regression. Both techniques solve a problem with Multiple Linear Regression, which can perform poorly when there are correlations...
Below is the latest update to The Popularity of Data Science Software. It contains an analysis of the tools used in the most recent complete year of scholarly articles. The section is also integrated into the main paper itself. New … <a href="http://r4stats.com/2017/06/19/scholarly-articles/">Continue reading <span>→</span></a>
The sparklyr package (by RStudio) provides a high-level interface between R and Apache Spark. Among many other things, it allows you to filter and aggregate data in Spark using the dplyr syntax. In Microsoft R Server 9.1, you can now connect to a a Spark session using the sparklyr package as the interface, allowing you to combine the data-preparation capabilities of sparklyr and the data-analysis capabilities of Microsoft R Server in the same environment. In a presentation by at the Spark Summit (embedded below, and you can find the slides here), Ali Zaidi shows how to connect to a Spark...
A year ago, I used Google’s Vision API to detect brand logos in images. Since then the DIY deep learning possibilities in R have vastly improved. With the release of Keras for R, one of the key deep learning frameworks is now available at your R fin...
In this post we will see how to include the effect of predictors in non-linear regressions. In other words, letting the parameters of non-linear regressions vary according to some explanatory variables (or predictors). Be sure to check the first post on this if you are new to non-linear regressions. The example that I will use […]<ul></ul>
<strong><p>Related Post</p></strong><ol>
<li><a rel="bookmark" href="https://datascienceplus.com/weather-forecast-with-regression-models-part-4/" title="Permanent Link toWeather forecast with regression models – part 4">Weather forecast with regression models – part 4</a></li>
<li><a rel="bookmark" href="https://datascienceplus.com/weather-forecast-with-regression-models-part-3/" title="Permanent Link toWeather forecast with regression models – part 3">Weather forecast with regression models – part 3</a></li>
<li><a rel="bookmark" href="https://datascienceplus.com/weather-forecast-with-regression-models-part-2/" title="Permanent Link toWeather forecast with regression models – part 2">Weather forecast with regression models – part 2</a></li>
<li><a rel="bookmark" href="https://datascienceplus.com/weather-forecast-with-regression-models-part-1/" title="Permanent Link toWeather forecast with regression models – part 1">Weather forecast with regression models – part 1</a></li>
<li><a rel="bookmark" href="https://datascienceplus.com/weighted-linear-support-vector-machine/" title="Permanent Link toWeighted Linear Support Vector Machine">Weighted Linear Support Vector Machine</a></li>
</ol>
Bias vs Variance tradeoff is always encountered in applying supervised learning algorithms. Least squares regression provides a good fit for the training set but can suffer from high variance which lowers predictive ability. To counter this problem, we can regularize the beta coefficients by employing a penalization term. Ridge regression applies l2 penalty to the […]<div>
<h3>Related exercise sets:</h3>
<ol>
<li><a rel="bookmark" href="http://www.r-exercises.com/2017/06/12/lasso-regression-in-r-exercises/" title="LASSO regression in R exercises">LASSO regression in R exercises </a></li>
<li><a rel="bookmark" href="http://www.r-exercises.com/2017/05/31/evaluate-your-model-with-r-exercises/" title="Evaluate your model with R Exercises">Evaluate your model with R Exercises </a></li>
<li><a rel="bookmark" href="http://www.r-exercises.com/2017/06/05/quantile-regression-in-r-exercises/" title="Quantile Regression in R exercises">Quantile Regression in R exercises </a></li>
<li><a href="http://www.r-exercises.com/start-here-to-learn-r/">Explore all our (>1000) R exercises</a></li>
<li>
<a href="http://www.r-exercises.com/r-courses/">Find an R course using our R Course Finder</a> directory</li>
</ol>
</div>
When analyzing data, you can start by thinking about how to classify the data you have according four main scales of measurement.
Tradewatch is a dashboard, written in R using the flexdashboard framework. Visualizations are made possible thanks to highcharter and other packages.
The main purpose of the dashboard is to monitor trade, mostly from the Mexican perspective, of goods...
To post your R job on the next post Just visit this link and post a new R job to the R community. You can post a job for free (and there are also “featured job” options available for extra exposure). Current R jobs Job seekers: please follow the links below to learn more and apply for your R job of interest: Featured Jobs Full-Time Data analyst – infectious disease mapping John Drake (University of Georgia) – Posted by John Drake Athens Georgia, United States 16 Jun2017 Full-Time Kenya Product Innovations Analyst One Acre Fund – Posted by OAF Kakamega Kakamega County, Kenya 14 Jun2017 Full-Time Software Developer One Acre Fund – Posted by OAF Jinja Eastern Region, Uganda 8 Jun2017 Full-Time Senior Data Scientist Warner Bros. Entertainment Inc. – Posted by WBCareers Anywhere 5 Jun2017 Full-Time Manager, Statistical Consulting &amp; Data Science Warner Bros. Entertainment Inc. – Posted by WBCareers Burbank California, United States 5 Jun2017 Full-Time Financial Controller RStudio, Inc. – Posted by tkawaf Anywhere 5 Jun2017 Full-Time Software Engineer in Test for RStudio RStudio – Posted by agadrow Anywhere 30 May 2017 Full-Time Data Scientist / Analytics Consultant Hedera Insights – Posted by HederaInsights Antwerpen Vlaanderen, Belgium 18 May2017 Full-Time Data Analyst, Chicago Violence Reduction Strategy National Network for Safe Communities (NNSC) – Posted by nnsc Chicago Illinois, United States 5 […]
I started writing my first package as collection of various functions that I needed for (almost) daily work. Meanwhile, packages were growing and bit by bit I sourced out functions to put them into new packages. Although this means more work for CRAN members when they have more packages to manage on their network, from […]<img border="0" alt="" width="1" src="https://pixel.wp.com/b.gif?host=strengejacke.wordpress.com&blog=10592659&post=1360&subd=strengejacke&ref=&feed=1" height="1">
Tradewatch is a dashboard, written in R using the flexdashboard framework. Visualizations are made possible thanks to highcharter and other packages.
The main purpose of the dashboard is to monitor trade, mostly from the Mexican perspective, of goods...
Introduction The development version of our R helper function wrapr::let() has switched from string-based substitution to abstract syntax tree based substitution (AST based subsitution, or language based substitution). I am looking for some feedback from wrapr::let() users already doing substantial work with wrapr::let(). If you are already using wrapr::let() please test if the current development … <a href="http://www.win-vector.com/blog/2017/06/wrapr-implementation-update/">Continue reading <span>wrapr Implementation Update</span></a>
Bioinformatics is an amalgamation Biology and Computer science. Biological Data is manipulated using Computers and Computer software’s in Bioinformatics. Biological Data includes DNA; RNA &amp; Proteins. DNA &amp; RNA is made of Nucleotide which are our genetic material in which we are coded.Our Structure and Functions are done by protein, which are build of Amino […]<div>
<h3>Related exercise sets:</h3>
<ol>
<li><a rel="bookmark" href="http://www.r-exercises.com/2017/05/28/manipulate-biological-data-using-biostrings-package-exercises-part-2/" title="Manipulate Biological Data Using Biostrings Package Exercises(Part 2)">Manipulate Biological Data Using Biostrings Package Exercises(Part 2) </a></li>
<li><a rel="bookmark" href="http://www.r-exercises.com/2017/06/06/manipulate-biological-data-using-biostrings-package-exercisespart-3/" title="Manipulate Biological Data Using Biostrings Package Exercises(Part 3)">Manipulate Biological Data Using Biostrings Package Exercises(Part 3) </a></li>
<li><a rel="bookmark" href="http://www.r-exercises.com/2017/01/22/bioinformatics-lessons-in-r-1-biostrings-exercises/" title="Bioinformatics Tutorial with Exercises in R (part 1)">Bioinformatics Tutorial with Exercises in R (part 1) </a></li>
<li><a href="http://www.r-exercises.com/start-here-to-learn-r/">Explore all our (>1000) R exercises</a></li>
<li>
<a href="http://www.r-exercises.com/r-courses/">Find an R course using our R Course Finder</a> directory</li>
</ol>
</div>
Quantifying the magnitude of a population decline with Bayesian time-series modelling
Population abundances tend to vary year to year. This variation can make
it make it hard detect a change and hard to quantify exactly what that
change is.
Bayesian ...
I couldn’t let this stand unchallenged: The new Rasmussen Poll, one of the most accurate in the 2016 Election, just out with a Trump 50% Approval Rating.That's higher than O's #'s!— Donald J. Trump (@realDonaldTrump) June 18, 2017 Ramussen makes their Presidential polling data available for both 🍊 &amp; O. Why not compare their ratings... <a href="https://rud.is/b/2017/06/18/r%E2%81%B6-disproving-approval/">Continue reading <span>→</span></a>
After a (really long!) hiatus I am reactivating my statistical blog. The first article  concerns the clarification of a point made in the manual of our recently published statistical model for short RNA sequencing data. The background for this post, in case one wants to skip reading the manuscript (please do read it !), centers around the […]
My last entry introduces principal component analysis (PCA), one of many unsupervised learning tools. I concluded the post with a demonstration of principal component regression (PCR), which essentially is a ordinary least squares (OLS) fit using the first principal components (PCs) from the predictors. This brings about many advantages: There is virtually no limit for the … <a href="https://poissonisfish.wordpress.com/2017/06/17/partial-least-squares-in-r/">Continue reading <span>Partial least squares in R</span> <span>→</span></a>
Microsoft Excel is perhaps the most popular data anlysis tool out there. While arguably convenient, spreadsheet software is error prone and Excel code can be very hard to review and test. After successfully completing this exercise set, you will be able to prepare a basic Excel document using just R (no need to touch Excel […]<div>
<h3>Related exercise sets:</h3>
<ol>
<li><a rel="bookmark" href="http://www.r-exercises.com/2016/12/12/building-shiny-app-exercises-part-1/" title="Building Shiny App exercises part 1">Building Shiny App exercises part 1 </a></li>
<li><a rel="bookmark" href="http://www.r-exercises.com/2017/05/24/data-wrangling-part-1-io/" title="Data wrangling : I/O (Part-1)">Data wrangling : I/O (Part-1) </a></li>
<li><a rel="bookmark" href="http://www.r-exercises.com/2017/01/01/building-shiny-app-exercises-part-4/" title="Building Shiny App exercises part 4">Building Shiny App exercises part 4 </a></li>
<li><a href="http://www.r-exercises.com/start-here-to-learn-r/">Explore all our (>1000) R exercises</a></li>
<li>
<a href="http://www.r-exercises.com/r-courses/">Find an R course using our R Course Finder</a> directory</li>
</ol>
</div>
Continuing on the theme of updating older visualizations into newer formats, below is a clip of the impact of Superstorm Sandy on Delaware Estuary water surface elevations.  The original MS Excel and screen capture version is posted <a href="https://www.youtube.com/watch?v=LFioBnHs88I" target="_blank">here</a>.  The new version developed in R with the animation package is cleaner and more portable.<br><br><div><a href="https://4.bp.blogspot.com/-NPRwqK9YkK4/WUWJGQlaLUI/AAAAAAAAA9M/Y00fbCPgtgcb3FjjBoTwrMNt7M2jNwlTgCLcBGAs/s1600/DelawareEstuarySandy_short2.gif"></a></div>The graph shows measured and predicted water surface elevations provided by the <a href="https://tidesandcurrents.noaa.gov/ports/index.html?port=db" target="_blank">NOAA PORTS</a> system for the Delaware River and Bay.  The shape of the Delaware Estuary amplifies the tidal signal at the mouth of the estuary resulting in a wider tidal range at the upper end of the estuary, which is also the more urbanized densely populated end.  Storm surges are a particular concern since the potential exists for the surge to be amplified within the estuary.<br><br>The graph shows that the measured water surface elevations initially closely match those predicted by harmonic constituents (based on lunar and other cycles), but as the storm approaches the impact of the surge results in greater differences between the predicated and observed water levels.<br><br>E-mail me at JYagecic@gmail.com for the full length GIF or more details about construction of the animated graph.
The Apache Drill folks have a nice walk-through tutorial on how to analyze the Yelp Academic Dataset with Drill. It’s a bit out of date (the current Yelp data set structure is different enough that the tutorial will error out at various points), but it’s a great example of how to work with large, nested... <a href="https://rud.is/b/2017/06/17/replicating-the-apache-drill-yelp-academic-dataset-with-sergeant/">Continue reading <span>→</span></a>
<p>I mentioned in an earlier post about how I’m revamping my presentation slides process but that I hadn’t tackled the user experience of browsing my slides, which wasted lots of the effort I put in. To tackle this part of</p>
<p>The post <a rel="nofollow" href="http://itsalocke.com/presentation-site-hugo-walkthrough/">All my talks in one place (plus a Hugo walkthrough!)</a> appeared first on <a rel="nofollow" href="http://itsalocke.com/">Locke Data</a>. Locke Data are a data science consultancy aimed at helping organisations get ready and get started with data science.</p>
On Tuesday I gave a talk at a meetup of the R users group of Barcelona. I got to choose the topic of my talk, and decided I’d like to expand a bit on a recent tweet of mine. There are tools that help you improve your R packages, some of them are not ...
qualtRics version 2.0 is now available from CRAN. This version contains several additional features and simplifies the process of importing survey exports from Qualtrics. Below, I outline the most important changes. You can check the full changelog her...
In this article we will discuss composing standard-evaluation interfaces (SE) and composing non-standard-evaluation interfaces (NSE) in R. In R the package tidyeval/rlang is a tool for building domain specific languages intended to allow easier composition of NSE interfaces. To use it you must know some of its structure and notation. Here are some details paraphrased … <a href="http://www.win-vector.com/blog/2017/06/non-standard-evaluation-and-function-composition-in-r/">Continue reading <span>Non-Standard Evaluation and Function Composition in R</span></a>
Results so far obtained allow us to predict the RainTomorrow Yes/No variable. In the first part, we highlighted that such factor variable can be put in relationship with the Rainfall quantitative one by: all.equal(weather_data$Rainfall > 1, weather_data$RainToday == &quot;Yes&quot;) As a consequence, we are able so far to predict if tomorrow rainfall shall be above […]<ul></ul>
<strong><p>Related Post</p></strong><ol>
<li><a rel="bookmark" href="https://datascienceplus.com/weather-forecast-with-regression-models-part-3/" title="Permanent Link toWeather forecast with regression models – part 3">Weather forecast with regression models – part 3</a></li>
<li><a rel="bookmark" href="https://datascienceplus.com/weather-forecast-with-regression-models-part-2/" title="Permanent Link toWeather forecast with regression models – part 2">Weather forecast with regression models – part 2</a></li>
<li><a rel="bookmark" href="https://datascienceplus.com/weather-forecast-with-regression-models-part-1/" title="Permanent Link toWeather forecast with regression models – part 1">Weather forecast with regression models – part 1</a></li>
<li><a rel="bookmark" href="https://datascienceplus.com/weighted-linear-support-vector-machine/" title="Permanent Link toWeighted Linear Support Vector Machine">Weighted Linear Support Vector Machine</a></li>
<li><a rel="bookmark" href="https://datascienceplus.com/logistic-regression-regularized-with-optimization/" title="Permanent Link toLogistic Regression Regularized with Optimization">Logistic Regression Regularized with Optimization</a></li>
</ol>
The Integrated Data Infrastructure (IDI) is an amazing research tool
One of New Zealand’s most important data assets is the integrated data infrastructure:
“The Integrated Data Infrastructure (IDI) is a large research database containing micro...
The Mango team held their first instance of the EARL conference series in San Francisco last month, and it was a fantastic showcase of real-world applications of R. This was a smaller version of the EARL conferences in London and Boston, but with that came the opportunity to interact with R users from industry in a more intimate setting. Hopefully Mango will return to the venue again next year, and if so I'll definitely be back! As always with EARL events, the program featured many interesting presentations of how R is used to implement a data-driven (or data-informed) policy at...
<p>To view the original source code, you can visit my Github page here. If you are interested in learning more about my background, you can visit […]</p>
<p>The post <a rel="nofollow" href="http://blog.nycdatascience.com/student-works/retailers-heres-employees-saying-project-web-scraping-indeed-com/">Retailers: Here's What Your Employees Are Saying About You - A Project on Web Scraping Indeed.com</a> appeared first on <a rel="nofollow" href="http://blog.nycdatascience.com/">NYC Data Science Academy Blog</a>.</p>
Introduction Not so long ago there was a nice dataiku meetup with Pierre Gutierrez talking about transfer learning. RStudio recently released the keras package, an R interface to work with keras for deep learning and transfer learning. Both events inspired … <a href="https://longhowlam.wordpress.com/2017/06/16/a-poor-mans-video-analyzer/">Continue reading <span>→</span></a>
Neural network have become a corner stone of machine learning in the last decade. Created in the late 1940s with the intention to create computer programs who mimics the way neurons process information, those kinds of algorithm have long been believe to be only an academic curiosity, deprived of practical use since they require a […]<div>
<h3>Related exercise sets:</h3>
<ol>
<li><a rel="bookmark" href="http://www.r-exercises.com/2017/06/08/neural-networks-exercises-part-1/" title="Neural networks Exercises (Part-1)">Neural networks Exercises (Part-1) </a></li>
<li><a rel="bookmark" href="http://www.r-exercises.com/2017/05/31/evaluate-your-model-with-r-exercises/" title="Evaluate your model with R Exercises">Evaluate your model with R Exercises </a></li>
<li><a rel="bookmark" href="http://www.r-exercises.com/2016/12/02/model-evaluation-exercise-1/" title="Model Evaluation Exercises 1">Model Evaluation Exercises 1 </a></li>
<li><a href="http://www.r-exercises.com/start-here-to-learn-r/">Explore all our (>1000) R exercises</a></li>
<li>
<a href="http://www.r-exercises.com/r-courses/">Find an R course using our R Course Finder</a> directory</li>
</ol>
</div>
In my early post (https://statcompute.wordpress.com/2017/01/22/monotonic-binning-with-smbinning-package/), I wrote a monobin() function based on the smbinning package by Herman Jopia to improve the monotonic binning algorithm. The function works well and provides robust binning outcomes. However, there are a couple potential drawbacks due to the coarse binning. First of all, the derived Information Value for each binned […]
Partial Least Squares (PLS) is a popular method for relative importance analysis in fields where the data typically includes more predictors than observations. Relative importance analysis is a general term applied to any technique used for...
<p>Part 2 of 2 in the series Set TheoryThe set operations of unions and intersections should ring a bell for those who’ve worked with relational databases and Venn Diagrams. The ‘union’ of two of sets and represents a set that comprises all members of and (or both). One of the...</p>
<p>The post <a rel="nofollow" href="http://www.aaronschlegel.com/set-operations-unions-intersections-r/">Set Operations Unions and Intersections in R</a> appeared first on <a rel="nofollow" href="http://www.aaronschlegel.com/">Aaron Schlegel</a>.</p>
In the exercises below we cover the some useful features of data.table ,data.table is a library in R for fast manipulation of large data frame .Please see the data.table vignette before trying the solution .This first set is intended for the begineers of data.table package and does not cover set keywords, joins of data.table which […]<div>
<h3>Related exercise sets:</h3>
<ol>
<li><a rel="bookmark" href="http://www.r-exercises.com/2017/04/26/shiny-application-layout-exercises-part-2/" title="Shiny Application Layouts Exercises (Part-2)">Shiny Application Layouts Exercises (Part-2) </a></li>
<li><a rel="bookmark" href="http://www.r-exercises.com/2016/12/11/descriptive-analytics-part-5-data-visualisation-categorical-variables/" title="Descriptive Analytics-Part 5: Data Visualisation (Categorical variables)">Descriptive Analytics-Part 5: Data Visualisation (Categorical variables) </a></li>
<li><a rel="bookmark" href="http://www.r-exercises.com/2016/06/16/summary-statistics-with-aggregate/" title="Summary Statistics With Aggregate()">Summary Statistics With Aggregate() </a></li>
<li><a href="http://www.r-exercises.com/start-here-to-learn-r/">Explore all our (>1000) R exercises</a></li>
<li>
<a href="http://www.r-exercises.com/r-courses/">Find an R course using our R Course Finder</a> directory</li>
</ol>
</div>
At the R/Finance conference last month, I demonstrated how to operationalize models developed in Microsoft R Server as web services using the mrsdeploy package. Then, I used that deployed model to generate predictions for loan delinquency, using a Python script as the client. (You can see slides here, and a video of the presentation below.) With Microsoft R Server 9.1, there are now two ways to operationalize models as a Web service or as a SQL Server stored procedure: Flexible Operationalization: Deploy any R script or function. Real-Time Operationalization: Deploy model objects generated by specific functions in Microsoft R, but...
Here is an absolutely horrible way to confuse yourself and get an inflated reported R-squared on a simple linear regression model in R. We have written about this before, but we found a new twist on the problem (interactions with categorical variable encoding) which we would like to call out here. First let’s set up … <a href="http://www.win-vector.com/blog/2017/06/an-easy-way-to-accidentally-inflate-reported-r-squared-in-linear-regression-models/">Continue reading <span>An easy way to accidentally inflate reported R-squared in linear regression models</span></a>
So many things have been said about weighting, but on my personal view of statistical inference processes, you do have to weight. From a single statistic until a complex model, you have to weight, because of the probability measure that induces the var...
At DataCamp, we are proud to have the best instructors in the data science community teaching courses on our site. Today, close to 50 instructors have one or more DataCamp courses live on the platform, and many more have a course in development (so sta...
[…]
<p>Euler Problem 24 asks to develop lexicographic permutations which are ordered arrangements of objects in lexicographic order. Tushar Roy of Coding Made Simple has shared a great introduction on how to generate lexicographic permutations. Euler Problem 24 Definition A permutation is … <a href="http://r.prevos.net/euler-problem-24/">Continue reading <span>→</span></a></p>
<p>The post <a rel="nofollow" href="http://r.prevos.net/euler-problem-24/">Lexicographic Permutations: Euler Problem 24</a> appeared first on <a rel="nofollow" href="http://r.prevos.net/">The Devil is in the Data</a>.</p>
For almost a year now, a collection of researchers from around the world has been collaborating to develop the next generation of analysis tools for disease outbreak response using R. The R Epidemics Consortium (RECON) creates R packages for handling, visualizing, and analyzing outbreak data using cutting-edge statistical methods, along with general-purpose tools for data cleaning, versioning, and encryption, and system infrastructure. Like ROpenSci, the Epidemics Consortium is focused on developing efficient, reliable, and accessible open-source tools, but with a focus on epidemology as opposed to science generally. The Epidemics Consortium has already created several useful resources for epidemiology: outbreaks,...
This set of exercises is about exploratory factor analysis. We shall use some basic features of psych package. For quick introduction to exploratory factor analysis and psych package, we recommend this short “how to” guide. You can download the dataset here. The data is fictitious. Answers to the exercises are available here. If you have […]<div>
<h3>Related exercise sets:</h3>
<ol>
<li><a rel="bookmark" href="http://www.r-exercises.com/2016/11/29/repeated-measures-anova-in-r-exercises/" title="Repeated measures ANOVA in R Exercises">Repeated measures ANOVA in R Exercises </a></li>
<li><a rel="bookmark" href="http://www.r-exercises.com/2017/04/21/data-science-for-doctors-cluster-analysis-exercises/" title="Data science for Doctors: Cluster Analysis Exercises">Data science for Doctors: Cluster Analysis Exercises </a></li>
<li><a rel="bookmark" href="http://www.r-exercises.com/2015/12/28/factor-exercises/" title="Factor exercises">Factor exercises </a></li>
<li><a href="http://www.r-exercises.com/start-here-to-learn-r/">Explore all our (>1000) R exercises</a></li>
<li>
<a href="http://www.r-exercises.com/r-courses/">Find an R course using our R Course Finder</a> directory</li>
</ol>
</div>
By Gabriel Vasconcelos When the LASSO fails? The LASSO has two important uses, the first is forecasting and the second is variable selection. We are going to talk about the second. The variable selection objective is to recover the correct … <a href="https://insightr.wordpress.com/2017/06/14/when-the-lasso-fails/">Continue reading <span>→</span></a>
In MicrosoftML package comes – in my opinion – long anticipated function for installing R packages for SQL Server and Microsoft R Server. And, I am super happy. Last year, in one of my previous blog posts, I have been showing how to install R package from SSMS using sp_execute_external_script. Now, with new package MicrosoftML … <a href="https://tomaztsql.wordpress.com/2017/06/14/installing-r-packages-with-rxinstallpackages-in-microsoft-r-server/">Continue reading <span>Installing R packages with rxInstallPackages in Microsoft R Server</span></a>
If you find yourself often repeating the same scripts in R, you might come to the point where you want to turn them into reusable functions and create your own R package. I recently reached that point and wanted to learn how to build my own R package – as simple as possible. It took […]<ul></ul>
<strong><p>Related Post</p></strong><ol>
<li><a rel="bookmark" href="https://datascienceplus.com/introductory-data-analysis-with-python/" title="Permanent Link toIntroductory Data Analysis with Python">Introductory Data Analysis with Python</a></li>
<li><a rel="bookmark" href="https://datascienceplus.com/understanding-linear-svm-with-r/" title="Permanent Link toUnderstanding Linear SVM with R">Understanding Linear SVM with R</a></li>
<li><a rel="bookmark" href="https://datascienceplus.com/how-to-add-a-background-image-to-ggplot2-graphs/" title="Permanent Link toHow to add a background image to ggplot2 graphs">How to add a background image to ggplot2 graphs</a></li>
<li><a rel="bookmark" href="https://datascienceplus.com/streamline-your-analyses-linking-r-to-sas/" title="Permanent Link toStreamline your analyses linking R to SAS: the workfloweR experiment">Streamline your analyses linking R to SAS: the workfloweR experiment</a></li>
<li><a rel="bookmark" href="https://datascienceplus.com/r-programming-pitfalls-to-avoid-part-1/" title="Permanent Link toR Programming – Pitfalls to avoid (Part 1)">R Programming – Pitfalls to avoid (Part 1)</a></li>
</ol>
Hello, R users! Today we're launching Data Visualization in R with lattice by Deepayan Sarkar, the creator of the lattice package.
Visualization is an essential component of interactive data analysis in R. Traditional (base) graphics is powerful, bu...
This note describes a useful replyr tool we call a &quot;join controller&quot; (and is part of our &quot;R and Big Data&quot; series, please see here for the introduction, and here for one our big data courses). When working on real world predictive modeling tasks in production, the ability to join data and document how you … <a href="http://www.win-vector.com/blog/2017/06/use-a-join-controller-to-document-your-work/">Continue reading <span>Use a Join Controller to Document Your Work</span></a>
Putting R code into production generally involves orchestrating the execution of a series of R scripts. Even if much of the application logic is encoded into R packages, a run-time environment typically involves scripts to ingest and prepare data, run the application logic, validate the results, and operationalize the output. Managing those scripts, especially in the face of working with multiple R versions, can be a pain — and worse, very complex scripts are difficult to understand and reuse for future applications. That's where Syberia comes in: an open-source framework created by Robert Krzyzanowski and other engineers at the consumer...
I caught a mention of this project by Pete Warden on Four Short Links today. If his name sounds familiar, he’s the creator of the DSTK, an O’Reilly author, and now works at Google. A decidedly clever and decent chap. The project goal is noble: crowdsource and make a repository of open speech data for... <a href="https://rud.is/b/2017/06/13/keeping-users-safe-while-collecting-data/">Continue reading <span>→</span></a>
In my last two posts (Part 1 and Part 2), I explored time series forecasting with the timekit package.
In this post, I want to compare how Facebook’s prophet performs on the same dataset.
Predicting future events/sales/etc. isn’t trivial for a ...
I’m pleased to announce that dplyr 0.7.0 is now on CRAN! (This was dplyr 0.6.0 previously; more on that below.) dplyr provides a “grammar” of data transformation, making it easy and elegant to solve the most common data manipulation challenges. dplyr supports multiple backends: as well as in-memory data frames, you can also use it […]
<p>I wrote a little while back about using Microsoft Cognitive Services APIs with R to first of all detect the language of pieces of text and then do sentiment analysis on them. I wasn’t too happy with the some of</p>
<p>The post <a rel="nofollow" href="http://itsalocke.com/using-purrr-with-apis/">Using purrr with APIs – revamping my code</a> appeared first on <a rel="nofollow" href="http://itsalocke.com/">Locke Data</a>. Locke Data are a data science consultancy aimed at helping organisations get ready and get started with data science.</p>
<p>Last week I gave a talk at the San Francisco EARL Conference about monetizing R packages. The talk was well received, so this Thursday at 9am...</p>
<p>The post <a rel="nofollow" href="https://www.arilamstein.com/blog/2017/06/12/workshop-monetizing-r-packages/">Workshop on Monetizing R Packages</a> appeared first on <a rel="nofollow" href="https://www.arilamstein.com/">AriLamstein.com</a>.</p>
A new package! Or at least new on CRAN as the very initial version 0.1.0 had been available via the ghrr drat for over a year. But now we have version 0.1.1 to announce as a CRAN package.
RcppMspPack provides R with MessagePack header files for use v...
Art Owen has arXived a new version of his thinning MCMC paper, where he studies how thinning or subsampling can improve computing time in MCMC chains. I remember quite well the message set by Mark Berliner and Steve MacEachern in an early 1990’s paper that subsampling was always increasing the variance of the resulting estimators. […]
While R (and its package ecosystem) provides a wealth of functions for querying and analyzing data, in our cloud-enabled world there's now a plethora of online services with APIs you can use to augment R's capabilities. Many of these APIs use a RESTful interface, which means you will typically send/receive data encoded in the JSON format using HTTP commands. Fortunately, as Steph Locke explains in her most recent R Quick tip, the process is pretty simple using R: Obtain an authentication key for using the service Find the URL of the API service you wish to use Convert your input...
This post describes the single biggest time saving technique that I know about for highlighting significant results on a table. The table below, which shows the results of a MANOVA, illustrates the trick. The coloring...
Hello, everyone! I’ve been meaning to get a new blog post out for the past couple of weeks. During that time I’ve been messing around with clustering. Clustering, or cluster analysis, is a method of data mining that groups similar observations together. Classification and clustering are quite alike, but clustering is more concerned with exploration … <a href="https://realdataweb.wordpress.com/2017/06/12/clustering/">Continue reading <span>Clustering</span></a>
Lease Absolute Shrinkage and Selection Operator (LASSO) performs regularization and variable selection on a given model. Depending on the size of the penalty term, LASSO shrinks less relevant predictors to (possibly) zero. Thus, it enables us to consider a more parsimonious model. In this exercise set we will use the glmnet package (package description: here) […]<div>
<h3>Related exercise sets:</h3>
<ol>
<li><a rel="bookmark" href="http://www.r-exercises.com/2017/05/31/evaluate-your-model-with-r-exercises/" title="Evaluate your model with R Exercises">Evaluate your model with R Exercises </a></li>
<li><a rel="bookmark" href="http://www.r-exercises.com/2017/03/11/multiple-regression-part-3-diagnostics/" title="Multiple Regression (Part 3) Diagnostics">Multiple Regression (Part 3) Diagnostics </a></li>
<li><a rel="bookmark" href="http://www.r-exercises.com/2017/04/02/data-science-for-doctors-inferential-statistics-part4/" title="Data science for Doctors: Inferential Statistics Exercises(Part-4)">Data science for Doctors: Inferential Statistics Exercises(Part-4) </a></li>
<li><a href="http://www.r-exercises.com/start-here-to-learn-r/">Explore all our (>1000) R exercises</a></li>
<li>
<a href="http://www.r-exercises.com/r-courses/">Find an R course using our R Course Finder</a> directory</li>
</ol>
</div>
<p>Hello R-Users, we have a great news! We are going to host Nathan Stephens and Joseph Rickert from RStudio and R Consortium: they are coming to Milano just for us (from USA) to meet the MilanoR community and talk about the latest news from RStudio and R Consortium</p>
<p>The post <a rel="nofollow" href="http://www.milanor.net/blog/milanor-meeting-special-ed-rstudio-meets-milanor-june-29th-milan/">RStudio meets MilanoR! June 29th, Milan</a> appeared first on <a rel="nofollow" href="http://www.milanor.net/blog">MilanoR</a>.</p>
<p>Sberbank Russian Housing Market A Kaggle Competition on Predicting Realty Price in Russia Written by Haseeb Durrani, Chen Trilnik, and Jack Yip   Introduction In May […]</p>
<p>The post <a rel="nofollow" href="http://blog.nycdatascience.com/student-works/data-scientists-guide-predicting-housing-prices-russia/">A Data Scientist's Guide to Predicting Housing Prices in Russia</a> appeared first on <a rel="nofollow" href="http://blog.nycdatascience.com/">NYC Data Science Academy Blog</a>.</p>
In the previous part of this tutorial, we build several models based on logistic regression. One aspect to be further considered is the decision threshold tuning that may help in reaching a better accuracy. We are going to show a procedure able to determine an optimal value for such purpose. ROC plots will be introduced […]<ul></ul>
<strong><p>Related Post</p></strong><ol>
<li><a rel="bookmark" href="https://datascienceplus.com/weather-forecast-with-regression-models-part-2/" title="Permanent Link toWeather forecast with regression models – part 2">Weather forecast with regression models – part 2</a></li>
<li><a rel="bookmark" href="https://datascienceplus.com/weather-forecast-with-regression-models-part-1/" title="Permanent Link toWeather forecast with regression models – part 1">Weather forecast with regression models – part 1</a></li>
<li><a rel="bookmark" href="https://datascienceplus.com/weighted-linear-support-vector-machine/" title="Permanent Link toWeighted Linear Support Vector Machine">Weighted Linear Support Vector Machine</a></li>
<li><a rel="bookmark" href="https://datascienceplus.com/logistic-regression-regularized-with-optimization/" title="Permanent Link toLogistic Regression Regularized with Optimization">Logistic Regression Regularized with Optimization</a></li>
<li><a rel="bookmark" href="https://datascienceplus.com/analytical-and-numerical-solutions-to-linear-regression-problems/" title="Permanent Link toAnalytical and Numerical Solutions to Linear Regression Problems">Analytical and Numerical Solutions to Linear Regression Problems</a></li>
</ol>
When reading poetry, it feels as if you are reading emotion. A good poet can use words to bring to their reader whichever emotions they choose. To do this, these poets often have to look deep within themselves, using their own lives as fuel. For this r...
Animations of continuous data in GIF format offer some portability advantages over video files.  A few years ago, shortly after Superstorm Sandy, a colleague and I developed a video of animated water surface elevations from USGS gages in Barnegat Bay, NJ as the eye of the storm approached.  That version used video screen capture and MS Excel VBA - you can see it <a href="https://www.youtube.com/watch?v=_GRlg2gVfuI" target="_blank">here</a>.<br><br><div></div>
<div></div>
<div><a href="https://4.bp.blogspot.com/-Zv7XiC6XeUc/WT3DS_4fkvI/AAAAAAAAA8w/P8H-fRWO3iEvQwLi3H_kWg6MpLS1xz17wCLcB/s1600/BarnegatSandy_pixels_short.gif"></a></div>
<br><div></div>
<br>With the pending 5 year anniversary of Sandy and the R animation package, the time was right to revisit the animation as a GIF.<br><br>Sources of data include:<br>USGS 01409335 Little Egg Inlet near Tuckerton, NJ<br>USGS 01409146 East Thorofare at Ship Bottom, NJ<br>USGS 01409110 Barnegat Bay at Waretown, NJ<br>USGS 01408205 Barnegat Bay at Route 37 bridge near Bay Shore NJ<br>USGS 01408168 Barnegat Bay at Mantoloking , NJ<br>USGS 01408043 Point Pleasant Canal at Point Pleasant NJ<br>USGS 01408050 Manasquan River at Point Pleasant NJ<br>NOAA National Data Buoy Center Station 44009 (LLNR 168) - DELAWARE BAY 26 NM Southeast of Cape May, NJ<br><div>
<br>See and download the full length GIF at <a href="https://media.giphy.com/media/xUA7beSsKPUPI3V41a/giphy.gif" target="_blank">https://media.giphy.com/media/xUA7beSsKPUPI3V41a/giphy.gif</a>
</div>
We have spent previous six posts discussing basics of [crayon-593dadf5825e3102232597-i/] function (underlying models and their implementation). Now it is time to move forward. Starting from this post we will discuss common parameters, shared by all the forecasting functions implemented in smooth. This means that the topics that we discuss are not only applicable to [crayon-593dadf5825f4551770389-i/], […]
Area, Stepped &amp; Combo charts In the second part of our series we are going to meet three more googleVis charts. More specifically these charts are Area Chart, Stepped Area Chart and Combo Chart. Read the examples below to understand the logic of what we are going to do and then test yous skills with […]<div>
<h3>Related exercise sets:</h3>
<ol>
<li><a rel="bookmark" href="http://www.r-exercises.com/2017/06/04/data-visualization-with-googlevis-exercises-part-1/" title="Data Visualization with googleVis exercises part 1">Data Visualization with googleVis exercises part 1 </a></li>
<li><a rel="bookmark" href="http://www.r-exercises.com/2017/01/10/getting-started-with-plotly-basic-plots/" title="Getting started with Plotly: basic Plots">Getting started with Plotly: basic Plots </a></li>
<li><a rel="bookmark" href="http://www.r-exercises.com/2017/04/30/shiny-application-layouts-exercises-part-3/" title="Shiny Application Layouts exercises part 3">Shiny Application Layouts exercises part 3 </a></li>
<li><a href="http://www.r-exercises.com/start-here-to-learn-r/">Explore all our (>1000) R exercises</a></li>
<li>
<a href="http://www.r-exercises.com/r-courses/">Find an R course using our R Course Finder</a> directory</li>
</ol>
</div>
It has been a while since I posted anything here, but I can’t resist this one. Let me just give three numbers.  The first two are: 314, the number of seats predicted for the largest party (Conservatives) in the UK House of Commons, at 10pm in Thursday (i.e., before even a single vote had been […]
Metabolomic network analysis can be used to interpret experimental results within a variety of contexts including: biochemical relationships, structural and spectral similarity and empirical correlation. Machine learning is useful for modeling relationships in the context of pattern recognition, clustering, classification and regression based predictive modeling. The combination of developed metabolomic networks and machine learning based […]
    Hello, An example often used in the ABC literature is the g-and-k distribution (e.g. reference [1] below), which is defined through the inverse of its cumulative distribution function (cdf). It is easy to simulate from such distributions by drawing uniform variables and applying the inverse cdf to them. However, since there is no closed-form […]
Density-based clustering is a technique that allows to partition data into groups with similar characteristics (clusters) but does not require specifying the number of those groups in advance. In density-based clustering, clusters are defined as dense regions of data points separated by low-density regions. Density is measured by the number of data points within some […]<div>
<h3>Related exercise sets:</h3>
<ol>
<li><a rel="bookmark" href="http://www.r-exercises.com/2017/04/21/data-science-for-doctors-cluster-analysis-exercises/" title="Data science for Doctors: Cluster Analysis Exercises">Data science for Doctors: Cluster Analysis Exercises </a></li>
<li><a rel="bookmark" href="http://www.r-exercises.com/2016/12/14/hierarchical-clustering-exercises-beginner/" title="Hierarchical Clustering exercises (beginner)">Hierarchical Clustering exercises (beginner) </a></li>
<li><a rel="bookmark" href="http://www.r-exercises.com/2017/02/19/building-shiny-app-exercises-part-7/" title="Building Shiny App exercises part 7">Building Shiny App exercises part 7 </a></li>
<li><a href="http://www.r-exercises.com/start-here-to-learn-r/">Explore all our (>1000) R exercises</a></li>
<li>
<a href="http://www.r-exercises.com/r-courses/">Find an R course using our R Course Finder</a> directory</li>
</ol>
</div>
<p>This week’s R bulletin will cover topics on how to round to the nearest desired number, converting and comparing dates and how to remove last x characters from an element. Hope you like this R weekly bulletin. Enjoy reading! Shortcut Keys 1. Comment/uncomment current line/selection – Ctrl+Shift+C 2. Move Lines Up/Down – Alt+Up/Down 3. Delete...</p>
<p>The post <a rel="nofollow" href="https://www.quantinsti.com/blog/r-weekly-bulletin-vol-xi/">R Weekly Bulletin Vol – XI</a> appeared first on <a rel="nofollow" href="https://www.quantinsti.com/"></a>.</p>
As the reality of a hung parliament starts to sink in, economists, political scientists and commentators will begin their usual routine of “post mortem” analysis of the surprise result of the UK 2017 general election. My co-authors Sascha Becker and Dennis Novy have done a similar exercise studying the EU Referendum last year [see also here]  and … <a href="http://freigeist.devmag.net/r/1002-uk-2017-general-election-results-data.html">Continue reading <span>UK 2017 General Election Results Data</span> <span>→</span></a>
I caught the 0.7.0 release of dplyr on my home CRAN server early Friday morning and immediately set out to install it since I’m eager to finish up my sergeant package and get it on CRAN. “Tidyverse” upgrades aren’t trivial for me as I tinker quite a bit with the tidyverse and create packages that... <a href="https://rud.is/b/2017/06/10/engaging-the-tidyverse-clean-slate-protocol/">Continue reading <span>→</span></a>
Some times it is necessary to download really big csv files to deliver some analysis. When you hit file sizes in Gigabytes it is useful to use R instead of spreadsheets. This exercise teaches us to manipulate this kind of files. Answers to the exercises are available here. Exercise 1 Create a directory canada immigration/Work/Income […]<div>
<h3>Related exercise sets:</h3>
<ol>
<li><a rel="bookmark" href="http://www.r-exercises.com/2017/05/05/forecasting-arimax-model-exercises-part-5/" title="Forecasting: ARIMAX Model Exercises (Part-5)">Forecasting: ARIMAX Model Exercises (Part-5) </a></li>
<li><a rel="bookmark" href="http://www.r-exercises.com/2017/05/24/data-wrangling-part-1-io/" title="Data wrangling : I/O (Part-1)">Data wrangling : I/O (Part-1) </a></li>
<li><a rel="bookmark" href="http://www.r-exercises.com/2016/02/22/bind-exercises/" title="Bind exercises">Bind exercises </a></li>
<li><a href="http://www.r-exercises.com/start-here-to-learn-r/">Explore all our (>1000) R exercises</a></li>
<li>
<a href="http://www.r-exercises.com/r-courses/">Find an R course using our R Course Finder</a> directory</li>
</ol>
</div>
The full schedule of talks for useR!2017, the global R user conference, has now been posted. The conference will feature 16 tutorials, 6 keynotes, 141 full talks, and 86 lightning talks starting on July 5 in Brussels. That's a lot to fir into 4 days, but I'm especially looking forward to the keynote presentations: 20 years of CRAN (Uwe Ligges) Parallel Computation in R: What We Want, and How We (Might) Get It (Norm Matloff) Structural Equation Modeling: models, software and stories (Yves Roseel) Teaching data science to new useRs (Mine Cetinkaya-Rundel) Dose-response analysis: considering dose both as qualitative factor...
In our latest “R and big data” article we show how to manage intermediate results in non-trivial Apache Spark workflows using R, sparklyr, dplyr, and replyr. Handle management Many Sparklyr tasks involve creation of intermediate or temporary tables. This can be through dplyr::copy_to() and through dplyr::compute(). These handles can represent a reference leak and eat … <a href="http://www.win-vector.com/blog/2017/06/managing-intermediate-results-when-using-rsparklyr/">Continue reading <span>Managing intermediate results when using R/sparklyr</span></a>
<p>And finally, we end our series of <a href="https://ropensci.org/blog/blog/2017/06/02/unconf2017">unconf</a> project summaries (<a href="https://ropensci.org/blog/blog/2017/06/05/unconf_recap_1">day 1</a>, <a href="https://ropensci.org/blog/blog/2017/06/06/unconf_recap_2">day 2</a>, <a href="https://ropensci.org/blog/blog/2017/06/07/unconf_projects_3">day 3</a>, <a href="https://ropensci.org/blog/blog/2017/06/08/unconf_recap_4">day 4</a>).</p>
<h2><code>mwparser</code></h2>
<p><strong>Summary:</strong> Wikimarkup is the language used on Wikipedia and similar projects, and as such contains a lot of valuable data both for scientists studying collaborative systems and people studying things documented on or in Wikipedia. mwparser parses wikimarkup, allowing a user to filter down to specific types of tags such as links or templates, and then extract components of those tags.</p>
<p><strong>Team:</strong> Oliver Keyes</p>
<p><strong>Github:</strong> <a href="https://github.com/Ironholds/mwparser">https://github.com/Ironholds/mwparser</a></p>
<h2><code>Gargle</code></h2>
<p><strong>Summary:</strong> Gargle is a library that provides authentication for Google APIs but without all the agonizing pain. The package provides helper functions (for <code>httr</code>) to support automatic retries, paging, and progress bars for API calls.</p>
<p><strong>Team:</strong> Craig Citro</p>
<p><strong>Github:</strong> <a href="https://github.com/r-lib/gargle">https://github.com/r-lib/gargle</a></p>
<h2><code>arresteddev</code></h2>
<p><strong>Summary:</strong> This package is designed to help troubleshoot errors that come up during package and analysis development. As of now, the package helps track tracebacks and errors but more functionality is planned for the future.</p>
<p><strong>Team:</strong> Lucy D'Agostino McGowan, Karthik Ram, Miles McBain</p>
<p><strong>Github:</strong> <a href="https://github.com/ropenscilabs/arresteddev">https://github.com/ropenscilabs/arresteddev</a></p>
In my last post, I prepared and visually explored time series data.
Now, I will use this data to test the timekit package for time series forecasting with machine learning.
Forecasting
In time series forecasting, we use models to predict future ti...
SparkR sparklyr
<p>Sets define a ‘collection’ of objects, or things typically referred to as ‘elements’ or ‘members.’ The concept of sets arises naturally when dealing with any collection of objects, whether it be a group of numbers or anything else. Conceptually, the following examples can be defined as a ‘set’: {1, 2,...</p>
<p>The post <a rel="nofollow" href="http://www.aaronschlegel.com/introduction-set-theory-r/">Introduction to Set Theory and Sets with R</a> appeared first on <a rel="nofollow" href="http://www.aaronschlegel.com/">Aaron Schlegel</a>.</p>
At the EARL conference in San Francisco this week, JS Tan from Microsoft gave an update (PDF slides here) on the doAzureParallel package . As we've noted here before, this package allows you to easily distribute parallel R computations to an Azure cluster. The package was recently updated to support using automatically-scaling Azure Batch clusters with low-priority nodes, which can be used at a discount of up to 80% compared to the price of regular high-availability VMs. JS Tan using doAzureParallel #rstats package to run simulation on a cluster of 20 low-priority Azure VMs. Total cost: $0.02 #EARLConf2017 pic.twitter.com/Mpl3IUa9zY —...
Our free video course Campaign Response Testing is no longer published on Udemy. It remains available for free on YouTube with all source code available from GitHub. I’ll try to correct bad links as I find them. Please read on for the reasons. Udemy recently unilaterally instituted a new policy on free courses: “When a … <a href="http://www.win-vector.com/blog/2017/06/campaign-response-testing-no-longer-published-on-udemy/">Continue reading <span>Campaign Response Testing no longer published on Udemy</span></a>
Neural network have become a corner stone of machine learning in the last decade. Created in the late 1940s with the intention to create computer programs who mimics the way neurons process information, those kinds of algorithm have long been believe to be only an academic curiosity, deprived of practical use since they require a […]<div>
<h3>Related exercise sets:</h3>
<ol>
<li><a rel="bookmark" href="http://www.r-exercises.com/2017/05/31/evaluate-your-model-with-r-exercises/" title="Evaluate your model with R Exercises">Evaluate your model with R Exercises </a></li>
<li><a rel="bookmark" href="http://www.r-exercises.com/2017/01/10/getting-started-with-plotly-basic-plots/" title="Getting started with Plotly: basic Plots">Getting started with Plotly: basic Plots </a></li>
<li><a rel="bookmark" href="http://www.r-exercises.com/2016/10/20/network-analysis-part-3-exercises/" title="Network Analysis Part 3 Exercises">Network Analysis Part 3 Exercises </a></li>
<li><a href="http://www.r-exercises.com/start-here-to-learn-r/">Explore all our (>1000) R exercises</a></li>
<li>
<a href="http://www.r-exercises.com/r-courses/">Find an R course using our R Course Finder</a> directory</li>
</ol>
</div>
In May, some friends and I built Tagger News, a real-time automatic classifier of Hacker News articles based on their text (see here for more about how we built it). This process started me down some interesting paths, particularly analyzing trends in ...
My first R package has been released on CRAN recently. It is named MonteCarlo and aims to make simulation studies as easy as possible – including parallelization and the generation of tables. What Are Simulation Studies Good For? Monte Carlo simulations are an essential tool in statistics and related disciplines. They are routinely used to … <a href="https://firstdifferences.wordpress.com/2017/06/08/introducing-the-montecarlo-package/">Continue reading <span>Introducing the MonteCarlo Package</span></a>
Recently new tools for data science pop up constantly, so it is hard to keep up with the changes and choose those that promise to be useful in the long run.<br>Recently two new solutions were announced that look very promising: <a href="https://github.com/rstudio/reticulate" target="_blank">reticulate</a> package for R and <a href="https://github.com/Microsoft/CNTK" target="_blank">Microsoft Cognitive Toolkit 2.0</a> (CNTK). Together with <a href="https://www.linkedin.com/in/jakuczunwit/" target="_blank">Wit Jakuczun</a> we have decided to test drive them on <a href="https://docs.microsoft.com/en-us/azure/machine-learning/machine-learning-data-science-virtual-machine-overview" target="_blank">Azure Data Science Virtual Machine</a>.<br>CNTK is a very promising deep learning toolkit, backed by Microsoft. Unfortunately it provides bindings to Python and not to R. Therefore it is an ideal situation to test reticulate package, which provides R interface to Python. We wanted to test CNTK on GPUs, so we have decided to use Azure Data Science Virtual Machine using NV6 Instance with Tesla M60 GPU.<br>For our test-drive we have used classic MNIST dataset. You can find all the source codes and detailed results of the test on <a href="https://github.com/WLOGSolutions/microsoft_cntk2.0_from_r" target="_blank">GitHub</a>. The short conclusion is that all components worked and played together excellently:<br><br><ul>
<li>on CNTK we have build MLP network model, which has reported 2.3% classification error, without any special tuning, which is a pretty decent result;</li>
<li>calling Python library from R using reticulate was simple and stable;</li>
<li>Calculations using GPU give a significant speedup and configuration of Azure DSVM was really simple.</li>
</ul>
<div>In summary, we have found the combo MSFT CNTK + R/reticulate + Azure DSVM to be simple to set up and powerful. It if definitely worth checking in your next data science project.</div>
<br><br>
Over the next few months we’re running a number of R, Stan and Scala courses around the UK. June (London) Mon Jun 26 – Introduction to R Tue Jun 27 – Statistical Modelling with R Wed Jun 28 – Programming with R Thu Jun 29 (2-day course) – Advanced R Programming August (Belfast) Thu Aug 10 […]
For R users, there hasn’t been a production grade solution for deep learning (sorry MXNET). This post introduces the Keras interface for R and how it can be used to perform image classification. The post ends by providing some code snippets that show Keras is intuitive and powerful. Tensorflow Last January, Tensorflow for R was […]<ul></ul>
<strong><p>Related Post</p></strong><ol>
<li><a rel="bookmark" href="https://datascienceplus.com/unsupervised-learning-and-text-mining-of-emotion-terms-using-r/" title="Permanent Link toUnsupervised Learning and Text Mining of Emotion Terms Using R">Unsupervised Learning and Text Mining of Emotion Terms Using R</a></li>
<li><a rel="bookmark" href="https://datascienceplus.com/using-mca-and-variable-clustering-in-r-for-insights-in-customer-attrition/" title="Permanent Link toUsing MCA and variable clustering in R for insights in customer attrition">Using MCA and variable clustering in R for insights in customer attrition</a></li>
<li><a rel="bookmark" href="https://datascienceplus.com/web-scraping-and-applied-clustering-global-happiness-and-social-progress-index/" title="Permanent Link toWeb Scraping and Applied Clustering Global Happiness and Social Progress Index">Web Scraping and Applied Clustering Global Happiness and Social Progress Index</a></li>
<li><a rel="bookmark" href="https://datascienceplus.com/key-phrase-extraction-from-tweets/" title="Permanent Link toKey Phrase Extraction from Tweets">Key Phrase Extraction from Tweets</a></li>
<li><a rel="bookmark" href="https://datascienceplus.com/financial-time-series-forecasting-an-easy-approach/" title="Permanent Link toFinancial time series forecasting – an easy approach">Financial time series forecasting – an easy approach</a></li>
</ol>
[…]
<p>In two previous posts, I presented code to teach R to play the trivial game of Tic Tac Toe. I thought this was an unbeatable algorithm. Alas, a comment from Alberto shattered my pride as he was able to beat … <a href="http://r.prevos.net/tic-tac-toe-part-3-minimax-algorithm/">Continue reading <span>→</span></a></p>
<p>The post <a rel="nofollow" href="http://r.prevos.net/tic-tac-toe-part-3-minimax-algorithm/">Tic Tac Toe Part 3: The Minimax Algorithm</a> appeared first on <a rel="nofollow" href="http://r.prevos.net/">The Devil is in the Data</a>.</p>
Choropleths are a common approach to visualizing data on geographic maps. But choropleths — by design or necessity — aggregate individual data points into a single geographic region (like a country or census tract), which is all shaded a single colour. This can introduce interpretability issues (are we seeing changes in the variable of interest, or just population density?) and can fail to express the richness of the underlying data. For an alternative approach, take a look at the recent Culture of Insight blog post which provides a tutorial on creating dot-density maps in R. The chart below is based...
By Henrique Helfer Motivation Until very recently, only a very limited classes of feasible non Gaussian time series models were available. For example, one could use extensions of state space models to non Gaussian environments (see, for example, Durbin and … <a href="https://insightr.wordpress.com/2017/06/07/non-gaussian-time-series-lets-handle-it-with-score-driven-models/">Continue reading <span>→</span></a>
Let’s worry a bit about substitution in R. Substitution is very powerful, which means it can be both used and mis-used. However, that does not mean every use is unsafe or a mistake. From Advanced R : substitute: We can confirm the above code performs no substitution: a &lt;- 1 b &lt;- 2 substitute(a + … <a href="http://www.win-vector.com/blog/2017/06/more-on-safe-substitution-in-r/">Continue reading <span>More on safe substitution in R</span></a>
Data wrangling is a task of great importance in data analysis. Data wrangling, is the process of importing, cleaning and transforming raw data into actionable information for analysis. It is a time-consuming process which is estimated to take about 60-80% of analyst’s time. In this series we will go through this process. It will be […]<div>
<h3>Related exercise sets:</h3>
<ol>
<li><a rel="bookmark" href="http://www.r-exercises.com/2017/05/24/data-wrangling-part-1-io/" title="Data wrangling : I/O (Part-1)">Data wrangling : I/O (Part-1) </a></li>
<li><a rel="bookmark" href="http://www.r-exercises.com/2017/01/01/building-shiny-app-exercises-part-4/" title="Building Shiny App exercises part 4">Building Shiny App exercises part 4 </a></li>
<li><a rel="bookmark" href="http://www.r-exercises.com/2016/12/20/web-scraping-exercises/" title="Web Scraping Exercises">Web Scraping Exercises </a></li>
<li><a href="http://www.r-exercises.com/start-here-to-learn-r/">Explore all our (>1000) R exercises</a></li>
<li>
<a href="http://www.r-exercises.com/r-courses/">Find an R course using our R Course Finder</a> directory</li>
</ol>
</div>
Most of the time when we stumble across a code snippet online, we often blindly copy and paste it into the R console. I suspect almost everyone does this. After all, what’s the harm? Consider this simple piece of R code that performs simple linear regression # Generate data x = rnorm(10) y = rnorm(10) […]
<p>Locke Data will be up to some shenanigans of various stripes in the big smoke. We hope to see you at some of them! June 26th — Monday Introduction to R (Newcastle) I won’t be in London for this but</p>
<p>The post <a rel="nofollow" href="http://itsalocke.com/r-data-science-activities-london-june-27th-29th/">R and Data Science activities in London, June 27th – 29th</a> appeared first on <a rel="nofollow" href="http://itsalocke.com/">Locke Data</a>. Locke Data are a data science consultancy aimed at helping organisations get ready and get started with data science.</p>
To post your R job on the next post Just visit this link and post a new R job to the R community. You can post a job for free (and there are also “featured job” options available for extra exposure). Current R jobs Job seekers: please follow the links below to learn more and apply for your R job of interest: Featured Jobs Part-Time Big Data Discovery Automation in Digital Health (5-10 hours per week) MedStar Institutes for Innovation – Posted by Praxiteles Anywhere 22 May2017 Full-Time Data Scientist / Analytics Consultant Hedera Insights – Posted by HederaInsights Antwerpen Vlaanderen, Belgium 18 May2017 Full-Time Data Scientist (m/f) Meritocracy – Posted by arianna@meritocracy Hamburg Hamburg, Germany 11 May2017 Full-Time Data Scientist One Acre Fund – Posted by OAF Nairobi Nairobi County, Kenya 9 May2017 Full-Time Back-End Developer – Sacramento Kings (Sacramento, CA) Sacramento Kings – Posted by khavey Sacramento California, United States 9 May2017 Full-Time Data Analyst, Chicago Violence Reduction Strategy National Network for Safe Communities (NNSC) – Posted by nnsc Chicago Illinois, United States 5 May2017 More New Jobs Full-Time Phd opportunity @ Barcelona / Moorepark Teagasc (Ireland), Universitat Autònoma de Barcelona (Spain) and CEDRIC (Centre d’études et de recherché en informatique et communications) of CNAM (Paris) […]
In case you missed them, here are some articles from May of particular interest to R users. Many interesting presentations recorded at the R/Finance 2017 conference in Chicago are now available to watch. A review of some of the R packages and projects implemented at the 2017 ROpenSci Unconference. An example of applying Bayesian Learning with the &quot;bnlearn&quot; package to challenge stereotypical assumptions Data from the Billboard Hot 100 chart used to find the most popular words in the titles of pop hits. Microsoft R Open 3.4.0 is now available for Windows, Mac and Linux. How to use the &quot;tweenr&quot;...
Authors: Stefan Feuerriegel, Nicolas Pröllochs This report introduces sentiment analysis in R and shows how to use our package “SentimentAnalysis”. What is sentiment analysis? Sentiment analysis is a research branch located at the heart of natural language processing (NLP), computational linguistics and text mining. It refers to any measures by which subjective information is extracted … <a href="http://www.rblog.uni-freiburg.de/2017/06/06/package-sentimentanalysis-released-on-cran/">Continue reading<span> &quot;Package “SentimentAnalysis” released on CRAN&quot;</span></a>
Bioinformatics is an amalgamation Biology and Computer science.Biological Data is manipulated using Computers and Computer software’s in Bioinformatics. Biological Data includes DNA; RNA &amp; Proteins. DNA &amp; RNA is made of Nucleotide which are our genetic material in which we are coded.Our Structure and Functions are done by protein, which are build of Amino acids […]<div>
<h3>Related exercise sets:</h3>
<ol>
<li><a rel="bookmark" href="http://www.r-exercises.com/2017/05/28/manipulate-biological-data-using-biostrings-package-exercises-part-2/" title="Manipulate Biological Data Using Biostrings Package Exercises(Part 2)">Manipulate Biological Data Using Biostrings Package Exercises(Part 2) </a></li>
<li><a rel="bookmark" href="http://www.r-exercises.com/2017/01/22/bioinformatics-lessons-in-r-1-biostrings-exercises/" title="Bioinformatics Tutorial with Exercises in R (part 1)">Bioinformatics Tutorial with Exercises in R (part 1) </a></li>
<li><a rel="bookmark" href="http://www.r-exercises.com/2017/05/10/accessing-and-manipulating-biological-databases-exercises-part-3/" title="Accessing and Manipulating Biological Databases  Exercises (Part-3)">Accessing and Manipulating Biological Databases Exercises (Part-3) </a></li>
<li><a href="http://www.r-exercises.com/start-here-to-learn-r/">Explore all our (>1000) R exercises</a></li>
<li>
<a href="http://www.r-exercises.com/r-courses/">Find an R course using our R Course Finder</a> directory</li>
</ol>
</div>
There are several ways to model seasonality in a time series. Traditionally, trend-cycle decomposition such as the Holt-Winters procedure has been very popular. Also, until today applied researchers often try to account for seasonality by using seasonal dummy variables. But of course, in a stochastic process it seems unreasonable to assume that seasonal effects are … <a href="https://firstdifferences.wordpress.com/2017/06/05/the-case-against-seasonal-unit-roots-3/">Continue reading <span>The Case Against Seasonal Unit Roots</span></a>
The previous post described how the deeply nested JSON data on fligths were parsed and stored in an R-friendly database structure. However, looking into the data, the information is not yet ready for statistical analysis and visualization and some further processing is necessary before extracting insights and producing nice plots. In the parsed batch, it … <a href="http://r-posts.com/web-data-acquisition-from-database-to-dataframe-for-data-analysis-and-visualization-part-4/">Continue reading <span>Web data acquisition: from database to dataframe for data analysis and visualization (Part 4)</span></a>
I analyze the performance of my ABM solution to the Battle for Riddler Nation and properties of strategy submissions using clustering.
The future package defines the Future API, which is a unified, generic, friendly API for parallel processing. The Future API follows the principle of write code once and run anywhere - the developer chooses what to parallelize and the user how and wher...
Recently I came across the classical 1983 paper A note on screening regression equations by David Freedman. Freedman shows in an impressive way the dangers of data reuse in statistical analyses. The potentially dangerous scenarios include those where t...
The standard OLS (Ordinary Least Squares) model explains the relationship between independent variables and the conditional mean of the dependent variable. In contrast, quantile regression models this relationship for different quantiles of the dependent variable. In this exercise set we will use the quantreg package (package description: here) to implement quantile regression in R. Answers […]<div>
<h3>Related exercise sets:</h3><ol><li><a rel="bookmark" href="http://www.r-exercises.com/2017/05/01/forecasting-multivariate-regression-exercises-part-4/" title="Forecasting: Multivariate Regression Exercises (Part-4)">Forecasting: Multivariate Regression Exercises (Part-4) </a></li>
<li><a rel="bookmark" href="http://www.r-exercises.com/2017/05/29/instrumental-variables-in-r-exercises-part-3/" title="Instrumental Variables in R exercises (Part-3)">Instrumental Variables in R exercises (Part-3) </a></li>
<li><a rel="bookmark" href="http://www.r-exercises.com/2017/05/05/forecasting-arimax-model-exercises-part-5/" title="Forecasting: ARIMAX Model Exercises (Part-5)">Forecasting: ARIMAX Model Exercises (Part-5) </a></li>
<li><a href="http://www.r-exercises.com/start-here-to-learn-r/">Explore all our (>1000) R exercises</a></li><li><a href="http://www.r-exercises.com/r-courses/">Find an R course using our R Course Finder</a> directory</li></ol></div>
Prelude: When you start with R and try to estimate a standard ANOVA , which is relatively simple in commercial software like SPSS, R kind of sucks. Especially for unbalanced designs or designs with repeated-measures replicating the results from such software in base R may require considerable effort. For a newcomer (and even an old […]
Python has a fairly famous design principle (from “PEP 20 — The Zen of Python”): There should be one– and preferably only one –obvious way to do it. Frankly in R (especially once you add many packages) there is usually more than one way. As an example we will talk about the common R functions: … <a href="http://www.win-vector.com/blog/2017/06/there-is-usually-more-than-one-way-in-r/">Continue reading <span>There is usually more than one way in R</span></a>
I’ve been doing intermittent prep work for a follow-up to an earlier post on store closings and came across this CNN Money “article” on it. Said “article” is a deliberately obfuscated or lazily crafted series of GIF images that contain all the Radio Shack impending store closings. It’s the most comprehensive list I’ve found, but... <a href="https://rud.is/b/2017/06/05/r%E2%81%B6-scraping-images-to-pdfs/">Continue reading <span>→</span></a>
In the first part, I introduced the weather dataset and outlining its exploratory analysis. In the second part of our tutorial, we are going to build multiple logistic regression models to predict weather forecast. Specifically, we intend to produce the following forecasts: tomorrow’s weather forecast at 9am of the current day tomorrow’s weather forecast at […]<ul></ul><strong><p>Related Post</p></strong><ol><li><a rel="bookmark" href="https://datascienceplus.com/weather-forecast-with-regression-models-part-1/" title="Permanent Link toWeather forecast with regression models – part 1">Weather forecast with regression models – part 1</a></li><li><a rel="bookmark" href="https://datascienceplus.com/weighted-linear-support-vector-machine/" title="Permanent Link toWeighted Linear Support Vector Machine">Weighted Linear Support Vector Machine</a></li><li><a rel="bookmark" href="https://datascienceplus.com/logistic-regression-regularized-with-optimization/" title="Permanent Link toLogistic Regression Regularized with Optimization">Logistic Regression Regularized with Optimization</a></li><li><a rel="bookmark" href="https://datascienceplus.com/analytical-and-numerical-solutions-to-linear-regression-problems/" title="Permanent Link toAnalytical and Numerical Solutions to Linear Regression Problems">Analytical and Numerical Solutions to Linear Regression Problems</a></li><li><a rel="bookmark" href="https://datascienceplus.com/how-to-create-a-loop-to-run-multiple-regression-models/" title="Permanent Link toHow to create a loop to run multiple regression models">How to create a loop to run multiple regression models</a></li></ol>
To post your R job on the next post Just visit this link and post a new R job to the R community. You can post a job for free (and there are also “featured job” options available for extra exposure). Current R jobs Job seekers: please follow the links below to learn more and apply for your R job of interest: Featured Jobs Part-Time Big Data Discovery Automation in Digital Health (5-10 hours per week) MedStar Institutes for Innovation – Posted by Praxiteles Anywhere 22 May2017 Full-Time Data Scientist / Analytics Consultant Hedera Insights – Posted by HederaInsights Antwerpen Vlaanderen, Belgium 18 May2017 Full-Time Data Scientist (m/f) Meritocracy – Posted by arianna@meritocracy Hamburg Hamburg, Germany 11 May2017 Full-Time Data Scientist One Acre Fund – Posted by OAF Nairobi Nairobi County, Kenya 9 May2017 Full-Time Back-End Developer – Sacramento Kings (Sacramento, CA) Sacramento Kings – Posted by khavey Sacramento California, United States 9 May2017 Full-Time Data Analyst, Chicago Violence Reduction Strategy National Network for Safe Communities (NNSC) – Posted by nnsc Chicago Illinois, United States 5 May2017 More New Jobs Full-Time Phd opportunity @ Barcelona / Moorepark Teagasc (Ireland), Universitat Autònoma de Barcelona (Spain) and CEDRIC (Centre d’études et de recherché en informatique et communications) of CNAM (Paris) – Posted by emmanuel.serrano.ferron Anywhere 25 […]
Do you struggle with learning ggplot2? Do you have problems with understanding what aesthetics actually do and how manipulating them change your plots? Here is the solution! Explore 33 ggplot2 geoms on one website! I created this ggplot2 explorer to help all R learners to understand how to plot beautiful/useful charts using the most popular … <a href="http://r-posts.com/shiny-app-to-explore-ggplot2/">Continue reading <span>Shiny app to explore ggplot2</span></a>
Running R on the cloud isn’t very difficult. This demo shows how to get Rstudio running on Amazon Web Services. To run R on the cloud we need to initiate a machine/computer and install R – that’s all very simple. Where you might get caught up is in the settings and permissions. Step 1: Create... <a href="http://stanke.co/r-on-aws-cloud/">Continue reading <span>→</span></a>