<p><div><div><div><div data-show-faces="false" data-height="21" data-width="100" data-layout="button_count" data-send="true" data-href="https://www.r-bloggers.com/a-simple-intro-to-q-learning-in-r-floor-plan-navigation/"></div></div><div></div></div><div></div></div>
<div>
(This article was first published on <strong><a href="https://qualityandinnovation.com/2017/12/24/a-simple-intro-to-q-learning-in-r-floor-plan-navigation/"> R – Quality and Innovation</a></strong>, and kindly contributed to <a rel="nofollow" href="https://www.r-bloggers.com/">R-bloggers)</a> </div>
<p>This example is drawn from “A Painless Q-Learning Tutorial” at <a rel="nofollow" href="http://mnemstudio.org/path-finding-q-learning-tutorial.htm" target="_blank">http://mnemstudio.org/path-finding-q-learning-tutorial.htm</a> which explains how to manually calculate iterations using the updating equation for Q-Learning, based on the Bellman Equation (image from <a rel="nofollow" href="https://www.is.uni-freiburg.de/ressourcen/business-analytics/13_reinforcementlearning.pdf" target="_blank">https://www.is.uni-freiburg.de/ressourcen/business-analytics/13_reinforcementlearning.pdf</a>):</p>
<p><img data-orig-file="https://qualityandinnovation.files.wordpress.com/2017/12/qlearning-eqn-feuerriegel.jpg" data-orig-size="641,109" data-medium-file="https://qualityandinnovation.files.wordpress.com/2017/12/qlearning-eqn-feuerriegel.jpg?w=300" data-attachment-id="3465" data-image-meta="{" data-image-description="" src="https://qualityandinnovation.files.wordpress.com/2017/12/qlearning-eqn-feuerriegel.jpg?w=829&h=142&fit=456%2C456" data-permalink="https://qualityandinnovation.com/2017/12/24/reinforcement-learning-q-learning-with-the-hopping-robot/qlearning-eqn-feuerriegel/" data-large-file="https://qualityandinnovation.files.wordpress.com/2017/12/qlearning-eqn-feuerriegel.jpg?w=641" alt="" data-image-title="qlearning-eqn-feuerriegel" sizes="(max-width: 641px) 100vw, 641px" data-comments-opened="1" class="wp-image-3465" data-recalc-dims="1"></p>
<p>The example explores path-finding through a house:</p>
<p><img data-orig-file="https://qualityandinnovation.files.wordpress.com/2017/12/q-floorplan.gif?w=456" data-orig-size="576,334" data-medium-file="https://qualityandinnovation.files.wordpress.com/2017/12/q-floorplan.gif?w=456?w=300" data-attachment-id="3473" data-image-meta="{" data-image-description="" src="https://qualityandinnovation.files.wordpress.com/2017/12/q-floorplan.gif?w=456" data-permalink="https://qualityandinnovation.com/2017/12/24/a-simple-intro-to-q-learning-in-r-floor-plan-navigation/q-floorplan/" data-large-file="https://qualityandinnovation.files.wordpress.com/2017/12/q-floorplan.gif?w=456?w=576" alt="" data-image-title="q-floorplan" data-comments-opened="1" class="wp-image-3473" data-recalc-dims="1"></p>
<p>The question to be answered here is: <span><strong>What’s the best way to get from Room 2 to Room 5 (outside)?</strong></span> Notice that by answering this question using reinforcement learning, we will also know how to find optimal routes from any room to outside. And if we run the iterative algorithm again for a new target state, we can find out the optimal route from any room to that new target state.</p>
<p>Since Q-Learning is model-free, we don’t need to know how likely it is that our agent will move between any room and any other room (the transition probabilities). If you had observed the behavior in this system over time, you <em>might</em> be able to find that information, but it many cases it just isn’t available. So the key for this problem is to construct a Rewards Matrix that explains the benefit (or penalty!) of attempting to go from one state (room) to another.</p>
<p>Assigning the rewards is somewhat arbitrary, but you should give a large positive value to your target state and negative values to states that are impossible or highly undesirable. Here’s the guideline we’ll use for this problem:</p>
<ul>
<li>-1 if “you can’t get there from here”</li>
<li>0 if the destination is not the target state</li>
<li>100 if the destination <em>is</em> the target state</li>
</ul>
<p>We’ll start constructing our rewards matrix by listing the states we’ll come <span><strong>FROM down the rows</strong></span>, and the states we’ll go <span><strong>TO in the columns</strong></span>. First, let’s fill the diagonal with -1 rewards, because we don’t want our agent to stay in the same place (that is, move from Room 1 to Room 1, or from Room 2 to Room 2, and so forth).</p>
<p>Next, let’s move across the first row. Starting in Room 0, we only have one choice: go to Room 4. All other possibilities are blocked (-1):</p>
<p><img data-orig-file="https://qualityandinnovation.files.wordpress.com/2017/12/qlearn-1.jpg" data-orig-size="551,332" data-medium-file="https://qualityandinnovation.files.wordpress.com/2017/12/qlearn-1.jpg?w=300&amp;h=181&amp;fit=300%2C181&amp;resize=300%2C181" data-attachment-id="3478" data-image-meta="{" data-image-description="" src="https://qualityandinnovation.files.wordpress.com/2017/12/qlearn-1.jpg?w=300&h=181&fit=300%2C181&resize=300%2C181" data-permalink="https://qualityandinnovation.com/2017/12/24/a-simple-intro-to-q-learning-in-r-floor-plan-navigation/qlearn-1/" data-large-file="https://qualityandinnovation.files.wordpress.com/2017/12/qlearn-1.jpg?w=551" alt="" data-image-title="qlearn-1" sizes="(max-width: 300px) 100vw, 300px" data-comments-opened="1" class="wp-image-3478" data-recalc-dims="1"></p>
<p>Now let’s fill in the row labeled 1. From Room 1, you have two choices: go to Room 3 (which is not great but permissible, so give it a 0) or go to Room 5 (the target, worth 100)!</p>
<p><img data-orig-file="https://qualityandinnovation.files.wordpress.com/2017/12/qlearn-2.jpg" data-orig-size="546,335" data-medium-file="https://qualityandinnovation.files.wordpress.com/2017/12/qlearn-2.jpg?w=300&amp;h=184&amp;fit=300%2C184&amp;resize=300%2C184" data-attachment-id="3479" data-image-meta="{" data-image-description="" src="https://qualityandinnovation.files.wordpress.com/2017/12/qlearn-2.jpg?w=300&h=184&fit=300%2C184&resize=300%2C184" data-permalink="https://qualityandinnovation.com/2017/12/24/a-simple-intro-to-q-learning-in-r-floor-plan-navigation/qlearn-2/" data-large-file="https://qualityandinnovation.files.wordpress.com/2017/12/qlearn-2.jpg?w=546" alt="" data-image-title="qlearn-2" sizes="(max-width: 300px) 100vw, 300px" data-comments-opened="1" class="wp-image-3479" data-recalc-dims="1"></p>
<p>Continue moving row by row, determining if you can’t get there from here (-1), you can but it’s not the target (0), or it’s the target(100). You’ll end up with a final rewards matrix that looks like this:</p>
<p><img data-orig-file="https://qualityandinnovation.files.wordpress.com/2017/12/qlearn-3.jpg" data-orig-size="545,344" data-medium-file="https://qualityandinnovation.files.wordpress.com/2017/12/qlearn-3.jpg?w=300&amp;h=189&amp;fit=300%2C189&amp;resize=300%2C189" data-attachment-id="3480" data-image-meta="{" data-image-description="" src="https://qualityandinnovation.files.wordpress.com/2017/12/qlearn-3.jpg?w=300&h=189&fit=300%2C189&resize=300%2C189" data-permalink="https://qualityandinnovation.com/2017/12/24/a-simple-intro-to-q-learning-in-r-floor-plan-navigation/qlearn-3/" data-large-file="https://qualityandinnovation.files.wordpress.com/2017/12/qlearn-3.jpg?w=545" alt="" data-image-title="qlearn-3" sizes="(max-width: 300px) 100vw, 300px" data-comments-opened="1" class="wp-image-3480" data-recalc-dims="1"></p>
<p>Now create this rewards matrix in R:</p>
<p><img data-orig-file="https://qualityandinnovation.files.wordpress.com/2017/12/qlearn-4.jpg" data-orig-size="548,341" data-medium-file="https://qualityandinnovation.files.wordpress.com/2017/12/qlearn-4.jpg?w=300&amp;h=187&amp;fit=300%2C187&amp;resize=300%2C187" data-attachment-id="3481" data-image-meta="{" data-image-description="" src="https://qualityandinnovation.files.wordpress.com/2017/12/qlearn-4.jpg?w=300&h=187&fit=300%2C187&resize=300%2C187" data-permalink="https://qualityandinnovation.com/2017/12/24/a-simple-intro-to-q-learning-in-r-floor-plan-navigation/qlearn-4/" data-large-file="https://qualityandinnovation.files.wordpress.com/2017/12/qlearn-4.jpg?w=548" alt="" data-image-title="qlearn-4" sizes="(max-width: 300px) 100vw, 300px" data-comments-opened="1" class="wp-image-3481" data-recalc-dims="1"></p>
<pre>
R &lt;- matrix(c(-1, -1, -1, -1, 0, 1,
       -1, -1, -1, 0, -1, 0,
       -1, -1, -1, 0, -1, -1, 
       -1, 0, 0, -1, 0, -1,
        0, -1, -1, 0, -1, 0,
       -1, 100, -1, -1, 100, 100), nrow=6, ncol=6, byrow=TRUE)
</pre>
<p>And run the code. Notice that we’re calling the target state 6 instead of 5 because even though we have a room labeled with a zero, our matrix starts with a 1s so we have to adjust:</p>
<pre>
source(&quot;https://raw.githubusercontent.com/NicoleRadziwill/R-Functions/master/qlearn.R&quot;)

results &lt;- q.learn(R,10000,alpha=0.1,gamma=0.8,tgt.state=6) 
> round(results)
     [,1] [,2] [,3] [,4] [,5] [,6]
[1,]    0    0    0    0   64   80
[2,]    0    0    0   51    0   80
[3,]    0    0    0   51    0    0
[4,]    0   64   41    0   64    0
[5,]   64    0    0   51    0   80
[6,]    0   84    0    0   84  100
</pre>
<p>You can read this table of average value to obtain policies. A policy is a “path” through the states of the system:</p>
<ul>
<li>Start at Room 0 (first row, labeled 1): Choose Room 4 (80), then from Room 4 choose Room 5 (100)</li>
<li>Start at Room 1: Choose Room 5 (100)</li>
<li>Start at Room 2: Choose Room 3 (64), from Room 3 choose Room 1 or Room 4 (80); from 1 or 4 choose 5 (100)</li>
<li>Start at Room 3: Choose Room 1 or Room 4 (80), then Room 5 (100)</li>
<li>Start at Room 4: Choose Room 5 (100)</li>
<li>Start at Room 5: Stay at Room 5 (100)</li>
</ul>
<p><span><strong>To answer the original problem, we would take route 2-3-1-5 or 2-3-4-5 to get out the quickest if we started in Room 2.</strong> </span>This is easy to see with a simple map, but is much more complicated when the maps get bigger.</p>
<p> <a rel="nofollow" href="https://feeds.wordpress.com/1.0/gocomments/qualityandinnovation.wordpress.com/3472/" target="_blank"><img border="0" alt="" src="https://feeds.wordpress.com/1.0/comments/qualityandinnovation.wordpress.com/3472/"></a> <img border="0" alt="" src="https://i1.wp.com/pixel.wp.com/b.gif?resize=1%2C1&ssl=1" data-recalc-dims="1"></p>
	<p>
<div>
<div>To <strong>leave a comment</strong> for the author, please follow the link and comment on their blog: <strong><a href="https://qualityandinnovation.com/2017/12/24/a-simple-intro-to-q-learning-in-r-floor-plan-navigation/"> R – Quality and Innovation</a></strong>.</div>
<hr>
<a rel="nofollow" href="https://www.r-bloggers.com/">R-bloggers.com</a> offers <strong><a rel="nofollow" href="https://feedburner.google.com/fb/a/mailverify?uri=RBloggers">daily e-mail updates</a></strong> about <a rel="nofollow" href="https://www.r-project.org/" title="The R Project for Statistical Computing">R</a> news and <a rel="nofollow" href="https://www.r-bloggers.com/search/tutorial" title="R tutorials">tutorials</a> on topics such as: <a rel="nofollow" href="https://www.r-bloggers.com/search/data%20science" title="Data science">Data science</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/Big%20Data" title="Big Data">Big Data, <a rel="nofollow" href="https://www.r-users.com/" title="R jobs">R jobs</a>, visualization (<a rel="nofollow" href="https://www.r-bloggers.com/search/ggplot2" title="ggplot and ggplot2 tutorials">ggplot2</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/boxplot" title="Boxplots using lattice and ggplot2 tutorials">Boxplots</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/map" title="Maps and gis">maps</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/animation" title="Animation in R">animation</a>), programming (<a rel="nofollow" href="https://www.r-bloggers.com/search/RStudio" title="RStudio IDE for R">RStudio</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/sweave" title="Sweave and literate programming">Sweave</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/LaTeX" title="LaTeX in R">LaTeX</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/SQL" title="SQL and databases">SQL</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/eclipse" title="Eclipse IDE for R">Eclipse</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/git" title="git and github, Version Control System">git</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/hadoop" title="Large data in R using Hadoop">hadoop</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/Web+Scraping" title="Web Scraping of google, facebook, yahoo, twitter and more using R">Web Scraping</a>) statistics (<a rel="nofollow" href="https://www.r-bloggers.com/search/regression" title="Regressions and ANOVA analysis tutorials">regression</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/PCA" title="principal component analysis tutorial">PCA</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/time+series" title="Time series">time series</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/trading" title="finance trading">trading</a>) and more...
<img alt="" width="1" src="http://feeds.feedburner.com/~r/RBloggers/~4/rT9XoAKRQ1w" height="1"></a></div>