<p><div><div><div><div data-show-faces="false" data-height="21" data-width="100" data-layout="button_count" data-send="true" data-href="https://www.r-bloggers.com/coder-an-lstm-that-writes-r-code/"></div></div><div></div></div><div></div></div>
<div>
(This article was first published on <strong><a href="https://www.statworx.com/de/blog/coder-an-lstm-that-writes-r-code/"> r-bloggers â€“ STATWORX</a></strong>, and kindly contributed to <a rel="nofollow" href="https://www.r-bloggers.com/">R-bloggers)</a> </div>
<p>Everybody talks about them, many people know how to use them, few people understand them: Long Short-Term Memory Neural Networks (LSTM). At <a rel="nofollow" href="https://www.statworx.com/de/" target="_blank">STATWORX</a>, with the beginning of the hype around AI and projects with large amounts of data, we also started using this powerful tool to solve business problems. </p>
<p>In short, an LSTM is a special type of recurrent neural network – i.e. a network able to access its internal state to process sequences of inputs – which is really handy if you want to exploit some time-like structure in your data. Use cases for recurrent networks range from guessing the next frame in a video to <a rel="nofollow" href="http://journals.plos.org/plosone/article?id=10.1371/journal.pone.0180944" target="_blank">stock prediction</a>, but you can also use them to learn and produce original text. And this shall already be enough information about LSTMs from my side. I won't bother you with yet another introduction into the theory of LSTMs, as there are more than enough great blog posts about their architecture (Kudos to Andrej Karpathy for <a rel="nofollow" href="http://karpathy.github.io/2015/05/21/rnn-effectiveness/" target="_blank">this</a> very informative piece of work which you should definitely read if you are not already bored by neural networks :)).</p>
<p>Especially inspired by the blog mentioned above, I thought about playing with a use case for LSTMs that actually has no intended use at all. LSTMs are good for learning text, so I thought it might be fun to let a character-level LSTM learn to write R code. It was not so important that the code is semantically correct or even solves a particular problem. Having a NN that is able to produce (more or less) syntactically correct code is already enough.</p>
<p>So, on my journey to <em>CodeR</em>, a NN that makes my workforce totally obsolete, I will let you participate in the three major steps of getting an RNN to write R code:</p>
<ol>
<li>Get enough text training data</li>
<li>Build and train <em>CodeR</em> with that data</li>
<li>Let <em>CodeR</em> write majestic R code</li>
</ol>
<p>If you like to try it yourself or follow the subsequent steps along, you can get the code from my GitHub <a rel="nofollow" href="https://github.com/tkrabel/rcoder" target="_blank">repository</a>.</p>
<h2>Step 1: Data Acquisition</h2>
<h3>Where to get enough Data?</h3>
<p>Ultimately, <em>CodeR</em> needs data; <em>a lot of</em> data. Plus, the data should be of good quality and not be too heterogeneous so that <em>CodeR</em> is able to learn the structure from the given text. Since R is open source, the first address to search for good R code is <a rel="nofollow" href="https://github.com/STATWORX" target="_blank">GitHub</a>. GitHub offers you an API to access information about its repositories, but for the flexibility and data I needed, I found the API too restrictive. That's why I decided to scrape the webpage myself using Hadley Wickham's <a rel="nofollow" href="https://cran.r-project.org/web/packages/rvest/rvest.pdf" target="_blank">rvest</a> package.</p>
<h3>Scrape GitHub</h3>
<p>The goal is simple: <em>Clone all R repositories from famous R users</em>. Of course, you could manually define R contributors that seem to be good programmers, but chances are you miss someone out that has some good and influential packages to offer. Remember that we need a lot of code and that it isn't much of a problem to reduce the data afterwards (which I in fact did).</p>
<h3>Get trending R user names</h3>
<p>So, let's start by getting the names of the trending users. If you visit <a rel="nofollow" href="https://github.com/trending/developers/r?since=monthly" target="_blank">https://github.com/trending/developers/r?since=monthly</a>, you see a list of all trending users. On June 14, 2018, it looked like this:</p>
<p><img alt="trending-r-users" src="https://i0.wp.com/www.statworx.com/wp-content/uploads/trending-r-users.png?w=450&ssl=1" class="wp-image-15053" data-recalc-dims="1"></p>
<p>If you inspect the HTML code, you quickly see that the actual user names are the <code>href</code> attribute of a link surrounded by <code><h2> tags, so we use <code>rvest</code> to dig us through that structure.
<pre><code>git_url &lt;- &quot;https://github.com&quot;

trending_user &lt;- glue(&quot;{git_url}/trending/developers/r?since=monthly&quot;) %>%
  read_html() %>%
  html_nodes(., &quot;h2&quot;) %>%
  html_nodes(., &quot;a&quot;) %>%
  html_attr(., &quot;href&quot;) %>% 
  gsub(&quot;/&quot;, &quot;&quot;, .)
</code></pre>
<pre><code>trending_user
[1] &quot;hadley&quot;        &quot;rstudio&quot;       &quot;yihui&quot;         ...
</code></pre>
<p>It's good to see that the names match the expected result from the webpage :). </p>
<h3>Get R repository names</h3>
<p>In the next step, taking user <img src="https://i1.wp.com/www.statworx.com/wp-content/ql-cache/quicklatex.com-695d9d59bd04859c6c99e7feb11daab6_l3.png?resize=6%2C12&ssl=1" alt="i" width="6" title="Rendered by QuickLaTeX.com" data-recalc-dims="1" height="12">, we need to get all repositories of <img src="https://i1.wp.com/www.statworx.com/wp-content/ql-cache/quicklatex.com-695d9d59bd04859c6c99e7feb11daab6_l3.png?resize=6%2C12&ssl=1" alt="i" width="6" title="Rendered by QuickLaTeX.com" data-recalc-dims="1" height="12"> that are her own (i.e. not forked) R repositories. When checking the url that lets you inspect all repos of a user (e.g. <a rel="nofollow" href="https://github.com/hadley?page=1&tab=repositories" target="_blank">https://github.com/hadley?page=1&amp;tab=repositories</a>), you realize that you need to go through all pages of a user's repository tab. I wrote a function that does that plus makes sure that:</p>
<ul>
<li>The repo's main language is R</li>
<li>If the repo is forked, the repo will be assigned to the original author</li>
</ul>
<p>With that function, it is easy to extract all R repo names from our trending users</p>
<pre><code>repos &lt;- list()
for (user in trending_user) {
  cat(&quot;User: &quot;, user, &quot;\n&quot;)
  repos[[user]] &lt;- get_r_repos(user)  # The actual magic
}
repos %&lt;>% unlist() %>% unique()
</code></pre>
<h3>Clone R repositories</h3>
<p>Now that we have a bunch of repository names, the last step is to clone all those repos and to clean them so that they only contain R files. I have decided to clean a repo directly after I have cloned it since I am going to download a lot of data and don't want to use too much space on my hard drive. The example code below clones the repo where you can find all of the code above (you are welcome ;)).</p>
<pre><code>repo &lt;- &quot;tkrabel/rcoder&quot;
system(glue(&quot;git clone https://github.com{repo}.git&quot;),
       wait = TRUE)
</code></pre>
<p>After having cloned all repos, I simply smash their content together in one big text file (<code>r_scripts_text.txt</code>).</p>
<h2>Step 2: Teach the Baby to Walk</h2>
<p>So, we have a big text file now that is ready to be inspected by <em>CodeR</em> so that it can learn to produce own good pieces of code. But how does the training actually work? There are a few steps that need to be taken care of here</p>
<ol>
<li>Prepare the data in a way it can actually be learned by an LSTM</li>
<li>Construct the network's architecture </li>
<li>The actual training step</li>
</ol>
<p>The general idea behind step 1 is to slice the text data in overlapping sequences of characters with a pre-specified size <img src="https://i1.wp.com/www.statworx.com/wp-content/ql-cache/quicklatex.com-ae1901659f469e6be883797bfd30f4f8_l3.png?resize=8%2C8&ssl=1" alt="s" width="8" title="Rendered by QuickLaTeX.com" data-recalc-dims="1" height="8"> corresponding to the &quot;time horizon&quot;. For example, imagine a text file containing the string <code>&quot;STATWORX ROCKS!&quot;</code> and let <img src="https://i1.wp.com/www.statworx.com/wp-content/ql-cache/quicklatex.com-555e124ef9a598df4606b976dfaa6955_l3.png?resize=41%2C12&ssl=1" alt="s = 3" width="41" title="Rendered by QuickLaTeX.com" data-recalc-dims="1" height="12">, meaning that you want the LSTM to use the last three characters to predict the fourth one. From this text file, you generate the data which looks like this.</p>
<figure>
<table>
<thead>
<tr>
<th>x1</th>
<th>x2</th>
<th>x3</th>
<th>y</th>
</tr>
</thead>
<tbody>
<tr>
<td>'S'</td>
<td>'T'</td>
<td>'A'</td>
<td>'T'</td>
</tr>
<tr>
<td>'T'</td>
<td>'A'</td>
<td>'T'</td>
<td>'W'</td>
</tr>
<tr>
<td>'A'</td>
<td>'T'</td>
<td>'W'</td>
<td>'O'</td>
</tr>
<tr>
<td>…</td>
<td>…</td>
<td>…</td>
<td>…</td>
</tr>
<tr>
<td>'C'</td>
<td>'K'</td>
<td>'S'</td>
<td>'!'</td>
</tr>
</tbody>
</table>
</figure>
<p>In a next step, you have to represent each character as a numeric object so that your model can actually work with it. The most popular way is to represent characters as unit vectors. Making it more tangible, remember that in the sentence above, we have 11 distinct characters (including the blank space and the exclamation mark). The so-called vocabulary <code>{'S', 'T', 'A', 'W', 'O', 'R', 'X', ' ', 'C', 'K', '!'}</code>is utilized to represent each character by a 11-dimensional unit vector with the 1 at its respective character position, e.g. <img src="https://i2.wp.com/www.statworx.com/wp-content/ql-cache/quicklatex.com-0a834d4f44f5b5b81861ae7329749338_l3.png?resize=133%2C19&ssl=1" alt="S = (1, 0, dots, 0)^top" width="133" title="Rendered by QuickLaTeX.com" data-recalc-dims="1" height="19"> (because <code>'S'</code> is the first character of the vocabulary), <img src="https://i1.wp.com/www.statworx.com/wp-content/ql-cache/quicklatex.com-41f5208656a8d3a13d7f4a21a7c1240d_l3.png?resize=151%2C19&ssl=1" alt="T = (0, 1, 0, dots, 0)^top" width="151" title="Rendered by QuickLaTeX.com" data-recalc-dims="1" height="19">, and so on. With these transformations, we finally have data our model can learn from.</p>
<p>Step 2 (building the model) is an ease with the R <code>keras</code> package, and it in fact took only 9 lines of code to build and LSTM with one input layer, 2 hidden LSTM layers with 128 units each and a softmax output layer, making it four layers in total. I kept the model that &quot;simple&quot; because I knew it is going to take a long time to learn. However, the learning results were not satisfying even after longer training times, so I decided to look out for ways of training networks on better (free) hardware in order to configure much more complex models. My search brought me to the <a rel="nofollow" href="https://colab.research.google.com/notebooks/welcome.ipynb" target="_blank">Google Colaboratory</a>, an environment that runs in the cloud and offers GPU support. Especially the GPU support gave training a huge time boost. However, for all R passionates out there, Google's Colab has a drawback: it is a Jupyter Notebook environment and therefore requires you to write Python code, which makes my use case somewhat cynical since I now use Python to train a network which writes R code. Well, in the end, I suppose, we all have to make some sacrifices :)!</p>
<p>As I started translating my code into Python, I found that there is a very useful package <a rel="nofollow" href="https://github.com/minimaxir/textgenrnn" target="_blank"><code>textgenrnn</code></a> that lets you very easily build and train a model. The advantage of the package is that its functions handle the whole data preparation step for you. The only thing you need to do is to specify the raw input text file from which the model learns and to configure the model, the rest is done for you (Credits go to <a rel="nofollow" href="https://minimaxir.com/" target="_blank">Max Woolf</a> for this great piece of work). </p>
<p>If you want to build your own version of <em>CodeR</em>, just copy <a rel="nofollow" href="https://colab.research.google.com/drive/1O8qFVJ1UDJ4NROufWCfWcb_Uwl-zkSq8" target="_blank">this notebook</a> to your Google Drive and follow the instructions.</p>
<h2>Step 3: Let CodeR Talk to Us</h2>
<p>After we have a trained version of <em>CodeR</em>, it is time to let it write some code. Starting with a blank sheet, <em>CodeR</em> is asked to sample the first character, which is a random one. In the next step, we feed that created character back to the model in order to write the next character. After that, we always use up to the last 40 characters as an input for the prediction of the next element in the text sequence. </p>
<p>There is a parameter in the corresponding textgenrnn function that can determine <em>CodeR</em>'s creativity while writing R code (so-called <strong>temperature</strong>). The higher the value, the more creative, i.e. diverse, the text. However, the results are not checked for syntactical correctness, so choosing too high of a temperature leads to more syntax errors. On the other hand, lower values in temperature (e.g. 0.5) make <em>CodeR</em> more conservative in its predictions, being closer to what it has learned. For a value of <code>temperature = 0.5</code>, <em>CodeR</em> knows how to pass any code review:</p>
<pre><code lang="r">partition_by = NULL,
                                                                                                                                                                                                                                                                                                                                                                                                  NA, NA, NA, NA, NA, NA, NA, NA, NA, NA, NA, NA, NA, NA, NA, NA, NA, NA, NA, NA, NA, NA, NA, NA, NA, NA, NA, NA, NA, NA, NA, NA, NA, NA, NA, NA, NA, NA, NA, NA, NA, NA, NA, NA, NA, NA, 
</code></pre>
<p>BATMAAAAAAAN!!! Looks like <em>CodeR</em> is lost in an infinite <code>NA</code> loop. Note the comma after the first statement. It is an artefact of the fact that we have some shiny code in the code base. This is, of course, an issue, as it leads to a very heterogenous text base, destabilizing RNN's learning process. Making the training data more homogenous is terms of syntax is definitely a topic for future refinements.</p>
<p>But let's look at some code now that is funny and, quite frankly, impressive. It will also teach us something about the learning set. Staying with a temerature value of 0.5, p<em>CodeR</em> mainly produces, well, <code>NA</code> loops, but also writes a lot of roxygen comments. </p>
<pre><code lang="r">#' @param x An object to transform to a string.
#'
#' @param x An object of column names
#' @param categorical_column A string with a model format.
#' @param ... Export to make setting to a selection of the length of the scale.
#' @param ordered If not supported values are stored in the data state on a post container.
#' @param conf.level character vector of structure (a, b, separate the second library in the first directory for the global environment for each vector of the same as a single argument.
#'
#' @param ... Additional arguments to exponentiate for the new name of the document, not a context
#'   the \code{searchPackages} returns a function that dependencies to create a character vector of the specified values. The name of the command packages in the top level normal for a single static when the default to the selection
</code></pre>
<p>Quite entertaining, isn't it? I like the way <em>CodeR</em> is using totally confusing argument names, such as <code>categorical_column</code> for a &quot;string with a model format&quot;.</p>
<p>With temperature values around 1, <em>CodeR</em> starts writing first syntactically correct functions (although the semantics might be a problem). Let's look at some snippets I found in the output. </p>
<pre><code lang="r">format.rdnore_configNames &lt;- function(x) {
  class(x) &lt;- y_string[[mull_bt]]
  x
}
addToN &lt;- function(packagePath, ..., recursive = FALSE) {
  assert_that(is_string(id))

  base &lt;- as.data.frame(names(purrr::packages$jitter))
  stop(&quot;parents has columns&quot;)
  layer_class = c(list(values, command), top = &quot;Tar&quot;)
}
counts.latex &lt;- function(x, ...) {
  if(is.null(x)) {
    stop(&quot;ROC Git classes:&quot;)
  }
}

spark_guess_value &lt;- function(pd, stringsAsFactors = FALSE)
{
  if (!is.null(varify))
    cat(&quot;Installing &quot;, cols, &quot;/&quot;, x$date)


  return(self$registry)
}
</code></pre>
<p>It is impressive that <em>CodeR</em> correctly sets blank spaces and braces most of the time. I only needed to mildly correct it when it set a backtick instead of a quotation mark. </p>
<p>It can also use <code>dplyr</code> functions and the <code>magrittr</code> pipe (which is great since I am a big fan of the pipe as you can read <a rel="nofollow" href="https://www.statworx.com/de/blog/show-me-your-pipe/" target="_blank">here</a>).</p>
<pre><code lang="r">rf_car &lt;- function(operator, input_col, config, default.uniques) %>%
  group_by(minorm) %>% summarise(week = 100)
</code></pre>
<p>Of course, this is just the tip of the iceberg, and there was a lot of unusable code <em>CodeR</em> produced. So to be fair, let's take a look a the lines that didn't make it into the hall of fame.</p>
<pre><code lang="r">#' @import knit_print.j installed file
#' @param each HTML pages (in numbers matching metadata),
      # in the seed location
      if(tf$item %in% cl) {
        unknown &lt;- 100 : mutate(contsList)
      v = integer(1)
    }

    unused(
      {retries_cred_, revdeps, message_format = &quot;,&quot;))
      

      if (flattenToBinour &amp;&amp; !renamed) {

    # define validations for way top aggregated instead, constants that does not want
#' arguments  \code{list}.
#'
#' @inheritParams \dontrun{
#' # Default environment is supplied
#'
#' @keywords internal
str_replace_all &lt;- function(pkg_lines, list(token = path),
                           list(using := force_init(), compare, installInfo$name, &quot;) %>%
  ` %>%
#'   modifyList(list(2, coord = FALSE))
#'
</code></pre>
<p>If you set the temperature to 2, it becomes wildly creative.</p>
<pre><code lang="r">x &lt;- x$p>$scenqNy89L'&lt;JW]
#' Clear tuning
#r
# verifican
ignore &lt;- wwMap.com:(.p/qafffs.tboods,4max LNh,	rmAR',5R}/6)  Y/AS_M(SB423eyt
mf(,9] **.4L2;3) # v1.3mDE); *}

g3 &lt;%yype_3X-C(r63,JAE)Zsd &lt;- 1
</code></pre>
<h2>Summary and outlook</h2>
<p>LSTMs offer many interesting and amusing use cases. As a free-time side project, I try to leverage the recurrent structure of an LSTM in order to train a model I call <em>CodeR</em> to write, well ..., R code. The results are truly entertaining and informative, as they reveal some of the training data set's structure. For example, we see that R code contains roxygen comments to a large extent, which makes sense as we included many R packages in the training set.</p>
<p>One point for further improvements of <em>CodeR</em> definitely is to remove all the shiny code from the training set in order to make the syntax more homogeneous and therefore to improve <em>CodeR</em>'s output text quality. Furthermore, it may be worthwhile to remove all roxygen comments. </p>
<p>If you have any ideas what to do next with <em>CodeR</em>, if you have any suggestions on how to improve my code or if you just want to leave a comment, please feel free to shoot me a message. Especially if you trained a version of <em>CodeR</em> yourself, don't hesitate to share your favorite lines of code with me. I would also be very curious if you could improve <em>CodeR</em>'s output quality by altering the training set (e.g. in the way described above).</p>
<div>
<h6>ÃƒÅ&quot;ber den Autor</h6>
<p><img src="https://i0.wp.com/www.statworx.com/wp-content/uploads/tobias-2-300x300.jpg?resize=180%2C180&ssl=1" width="180" alt="Tobias Krabel" class="wp-user-avatar wp-user-avatar-180" data-recalc-dims="1" height="180"></p>
<div>
<h4>Tobias Krabel</h4>
<p>Tobias ist im Data Science Team und absolviert im Moment seinen 2. Master in Informatik. In seiner Freizeit ist er sozial engagiert und geht gerne Wandern in der Natur.</p>
</div>
</div>
<p>Der Beitrag <a rel="nofollow" href="https://www.statworx.com/de/blog/coder-an-lstm-that-writes-r-code/" target="_blank">CodeR: an LSTM that writes R Code</a> erschien zuerst auf <a rel="nofollow" href="https://www.statworx.com/de/" target="_blank">STATWORX</a>.</p>
	<p>
<div>
<div>To <strong>leave a comment</strong> for the author, please follow the link and comment on their blog: <strong><a href="https://www.statworx.com/de/blog/coder-an-lstm-that-writes-r-code/"> r-bloggers â€“ STATWORX</a></strong>.</div>
<hr>
<a rel="nofollow" href="https://www.r-bloggers.com/">R-bloggers.com</a> offers <strong><a rel="nofollow" href="https://feedburner.google.com/fb/a/mailverify?uri=RBloggers">daily e-mail updates</a></strong> about <a rel="nofollow" href="https://www.r-project.org/" title="The R Project for Statistical Computing">R</a> news and <a rel="nofollow" href="https://www.r-bloggers.com/search/tutorial" title="R tutorials">tutorials</a> on topics such as: <a rel="nofollow" href="https://www.r-bloggers.com/search/data%20science" title="Data science">Data science</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/Big%20Data" title="Big Data">Big Data, <a rel="nofollow" href="https://www.r-users.com/" title="R jobs">R jobs</a>, visualization (<a rel="nofollow" href="https://www.r-bloggers.com/search/ggplot2" title="ggplot and ggplot2 tutorials">ggplot2</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/boxplot" title="Boxplots using lattice and ggplot2 tutorials">Boxplots</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/map" title="Maps and gis">maps</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/animation" title="Animation in R">animation</a>), programming (<a rel="nofollow" href="https://www.r-bloggers.com/search/RStudio" title="RStudio IDE for R">RStudio</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/sweave" title="Sweave and literate programming">Sweave</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/LaTeX" title="LaTeX in R">LaTeX</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/SQL" title="SQL and databases">SQL</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/eclipse" title="Eclipse IDE for R">Eclipse</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/git" title="git and github, Version Control System">git</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/hadoop" title="Large data in R using Hadoop">hadoop</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/Web+Scraping" title="Web Scraping of google, facebook, yahoo, twitter and more using R">Web Scraping</a>) statistics (<a rel="nofollow" href="https://www.r-bloggers.com/search/regression" title="Regressions and ANOVA analysis tutorials">regression</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/PCA" title="principal component analysis tutorial">PCA</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/time+series" title="Time series">time series</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/trading" title="finance trading">trading</a>) and more...
<img width="1" alt="" src="http://feeds.feedburner.com/~r/RBloggers/~4/HxkWmExOnBE" height="1"></a></div></h2></code>