<p><div><div><div><div data-show-faces="false" data-height="21" data-width="100" data-layout="button_count" data-send="true" data-href="https://www.r-bloggers.com/generative-assessment-creation/"></div></div><div></div></div><div></div></div>
<div>
(This article was first published on <strong><a href="https://blog.ouseful.info/2018/05/14/generative-assessment-creation/"> Rstats – OUseful.Info, the blog…</a></strong>, and kindly contributed to <a rel="nofollow" href="https://www.r-bloggers.com/">R-bloggers)</a> </div>
<p>It’s coming round to that time of year where we have to create the assessment material for courses with an October start date. In many cases, we reuse question forms from previous presentations but change the specific details. If a question is suitably defined, then large parts of this process could be automated.</p>
<p>In the OU, automated question / answer option randomisation is used to provide iCMAs (interactive computer marked assessments) via the student VLE using <a rel="nofollow" href="http://www.open.ac.uk/openmarkexamples/" target="_blank">OpenMark</a>. As well as purely text based questions, questions can include tables or images as part of the question.</p>
<p><a rel="nofollow" href="http://www.open.ac.uk/openmarkexamples/numeric-response/two-entries" target="_blank"><img data-orig-file="https://ouseful.files.wordpress.com/2018/05/two_entries___openmark_examples.png?w=385&amp;h=367&amp;fit=385%2C367&amp;resize=385%2C367" data-orig-size="1608,1530" data-medium-file="https://ouseful.files.wordpress.com/2018/05/two_entries___openmark_examples.png?w=385&amp;h=367&amp;fit=385%2C367&amp;resize=385%2C367?w=300" data-attachment-id="17953" data-image-meta="{" data-image-description="" src="https://ouseful.files.wordpress.com/2018/05/two_entries___openmark_examples.png?w=385&h=367&fit=385%2C367&resize=385%2C367" data-permalink="https://blog.ouseful.info/2018/05/14/generative-assessment-creation/two_entries___openmark_examples/" data-large-file="https://ouseful.files.wordpress.com/2018/05/two_entries___openmark_examples.png?w=385&amp;h=367&amp;fit=385%2C367&amp;resize=385%2C367?w=830" alt="" data-image-title="Two_entries___OpenMark_Examples" sizes="(max-width: 385px) 100vw, 385px" width="385" data-comments-opened="1" class="wp-image-17953" data-recalc-dims="1" height="367"></a></p>
<p>One way of supporting such question types is to manually create a set of answer options, perhaps with linked media assets, and then allow randomisation of them.</p>
<p>Another way is to define the question in a <em>generative</em> way so that the correct and incorrect answers are automatically generated.<em>(This seems to be one of those use cases for why ‘everyone should learn to code’;-)</em></p>
<p>Pinching screenshots from an (old?) <a rel="nofollow" href="http://www.open.edu/openlearncreate/mod/oucontent/view.php?id=52747&section=2.1.2" target="_blank">OpenMark tutorial</a>, we can see how a dynamically generated question might be defined. For example, create a set of variables:</p>
<p><img data-orig-file="https://ouseful.files.wordpress.com/2018/05/vnvariables25.png?w=229&amp;h=231&amp;fit=229%2C231&amp;resize=229%2C231" data-orig-size="416,419" data-medium-file="https://ouseful.files.wordpress.com/2018/05/vnvariables25.png?w=229&amp;h=231&amp;fit=229%2C231&amp;resize=229%2C231?w=298" data-attachment-id="17954" data-image-meta="{" data-image-description="" src="https://ouseful.files.wordpress.com/2018/05/vnvariables25.png?w=229&h=231&fit=229%2C231&resize=229%2C231" data-permalink="https://blog.ouseful.info/2018/05/14/generative-assessment-creation/vnvariables25/" data-large-file="https://ouseful.files.wordpress.com/2018/05/vnvariables25.png?w=229&amp;h=231&amp;fit=229%2C231&amp;resize=229%2C231?w=416" alt="" data-image-title="vnvariables25" sizes="(max-width: 229px) 100vw, 229px" width="229" data-comments-opened="1" class="wp-image-17954" data-recalc-dims="1" height="231"></p>
<p>and then generate a templated question, and student feedback generator, around them:</p>
<p><img data-orig-file="https://ouseful.files.wordpress.com/2018/05/vngeneral25.png?w=271&amp;h=292&amp;fit=271%2C292&amp;resize=271%2C292" data-orig-size="500,537" data-medium-file="https://ouseful.files.wordpress.com/2018/05/vngeneral25.png?w=271&amp;h=292&amp;fit=271%2C292&amp;resize=271%2C292?w=279" data-attachment-id="17955" data-image-meta="{" data-image-description="" src="https://ouseful.files.wordpress.com/2018/05/vngeneral25.png?w=271&h=292&fit=271%2C292&resize=271%2C292" data-permalink="https://blog.ouseful.info/2018/05/14/generative-assessment-creation/vngeneral25/" data-large-file="https://ouseful.files.wordpress.com/2018/05/vngeneral25.png?w=271&amp;h=292&amp;fit=271%2C292&amp;resize=271%2C292?w=500" alt="" data-image-title="vngeneral25" sizes="(max-width: 271px) 100vw, 271px" width="271" data-comments-opened="1" class="wp-image-17955" data-recalc-dims="1" height="292"></p>
<p>Packages also exist for creating generative questions/answers more generally. For example, the R <a rel="nofollow" href="http://www.r-exams.org/intro/elearning/" target="_blank"><code>exams</code></a> package allows you to define question/answer templates in Rmd and then generate questions and solutions in a variety of output document formats.</p>
<p><a rel="nofollow" href="http://www.r-exams.org/templates/deriv/" target="_blank"><br>
<img data-orig-file="https://ouseful.files.wordpress.com/2018/05/deriv-rmd_pdf_exams.png?w=450&amp;h=223&amp;fit=495%2C223" data-orig-size="2333,1052" data-medium-file="https://ouseful.files.wordpress.com/2018/05/deriv-rmd_pdf_exams.png?w=450&amp;h=223&amp;fit=495%2C223?w=300" data-attachment-id="17956" data-image-meta="{" data-image-description="" src="https://ouseful.files.wordpress.com/2018/05/deriv-rmd_pdf_exams.png?w=450&h=223&fit=495%2C223" data-permalink="https://blog.ouseful.info/2018/05/14/generative-assessment-creation/deriv-rmd_pdf_exams/" data-large-file="https://ouseful.files.wordpress.com/2018/05/deriv-rmd_pdf_exams.png?w=450&amp;h=223&amp;fit=495%2C223?w=830" alt="" data-image-title="deriv-Rmd_pdf_exams" sizes="(max-width: 495px) 100vw, 495px" data-comments-opened="1" class="wp-image-17956" data-recalc-dims="1"></a></p>
<p>You can also write templates that include the creation of graphical assets such as charts:</p>
<p><a rel="nofollow" href="http://www.r-exams.org/templates/dist/" target="_blank"><img data-orig-file="https://ouseful.files.wordpress.com/2018/05/pythag-rmd_pdf_exams.png?w=450&amp;h=268&amp;fit=503%2C268" data-orig-size="2358,1258" data-medium-file="https://ouseful.files.wordpress.com/2018/05/pythag-rmd_pdf_exams.png?w=450&amp;h=268&amp;fit=503%2C268?w=300" data-attachment-id="17957" data-image-meta="{" data-image-description="" src="https://ouseful.files.wordpress.com/2018/05/pythag-rmd_pdf_exams.png?w=450&h=268&fit=503%2C268" data-permalink="https://blog.ouseful.info/2018/05/14/generative-assessment-creation/pythag-rmd_pdf_exams/" data-large-file="https://ouseful.files.wordpress.com/2018/05/pythag-rmd_pdf_exams.png?w=450&amp;h=268&amp;fit=503%2C268?w=830" alt="" data-image-title="pythag-Rmd_pdf_exams" sizes="(max-width: 503px) 100vw, 503px" data-comments-opened="1" class="wp-image-17957" data-recalc-dims="1"></a></p>
<p> </p>
<p>Via my feeds over the weekend, I noticed that this package now also supports the creation of more general diagrams created from a TikZ diagram template. For example, <a rel="nofollow" href="http://www.r-exams.org/templates/logic/" target="_blank">logic diagrams</a>:</p>
<p><a rel="nofollow" href="http://www.r-exams.org/templates/logic/" target="_blank"><img data-orig-file="https://ouseful.files.wordpress.com/2018/05/logic-rmd_-exams.png?w=450&amp;h=279&amp;fit=509%2C279" data-orig-size="2452,1342" data-medium-file="https://ouseful.files.wordpress.com/2018/05/logic-rmd_-exams.png?w=450&amp;h=279&amp;fit=509%2C279?w=300" data-attachment-id="17959" data-image-meta="{" data-image-description="" src="https://ouseful.files.wordpress.com/2018/05/logic-rmd_-exams.png?w=450&h=279&fit=509%2C279" data-permalink="https://blog.ouseful.info/2018/05/14/generative-assessment-creation/logic-rmd_-exams/" data-large-file="https://ouseful.files.wordpress.com/2018/05/logic-rmd_-exams.png?w=450&amp;h=279&amp;fit=509%2C279?w=830" alt="" data-image-title="logic-Rmd_-exams" sizes="(max-width: 509px) 100vw, 509px" data-comments-opened="1" class="wp-image-17959" data-recalc-dims="1"></a></p>
<p>Or <a rel="nofollow" href="http://www.r-exams.org/templates/automaton/" target="_blank">automata diagrams</a>:</p>
<p><a rel="nofollow" href="http://www.r-exams.org/templates/automaton/" target="_blank"><img data-orig-file="https://ouseful.files.wordpress.com/2018/05/automaton-rmd_exams.png?w=450&amp;h=286&amp;fit=533%2C286" data-orig-size="2450,1314" data-medium-file="https://ouseful.files.wordpress.com/2018/05/automaton-rmd_exams.png?w=450&amp;h=286&amp;fit=533%2C286?w=300" data-attachment-id="17958" data-image-meta="{" data-image-description="" src="https://ouseful.files.wordpress.com/2018/05/automaton-rmd_exams.png?w=450&h=286&fit=533%2C286" data-permalink="https://blog.ouseful.info/2018/05/14/generative-assessment-creation/automaton-rmd_exams/" data-large-file="https://ouseful.files.wordpress.com/2018/05/automaton-rmd_exams.png?w=450&amp;h=286&amp;fit=533%2C286?w=830" alt="" data-image-title="automaton-Rmd_exams" sizes="(max-width: 533px) 100vw, 533px" data-comments-opened="1" class="wp-image-17958" data-recalc-dims="1"></a></p>
<p><em>(You can see more <code>exam</code> templates here: <a rel="nofollow" href="http://www.r-exams.org/templates/" target="_blank">www.r-exams.org/templates</a>.)</em></p>
<p>As I’m still on a “we can do everything in Jupyter” kick, one of the things I’ve explored is various IPython/notebook magics that support diagram creation. At the moment, these are just generic magics that allow you to write TikZ diagrams, for example, that make use of various TikZ packages:</p>
<p><img data-orig-file="https://ouseful.files.wordpress.com/2018/05/gettingstarted_3_2_0_electronics_diagrams_packages_ipynb__ousefulinfo__-_microsoft_azure_notebooks.png?w=448&amp;h=244&amp;fit=448%2C244&amp;resize=448%2C244" data-orig-size="1530,832" data-medium-file="https://ouseful.files.wordpress.com/2018/05/gettingstarted_3_2_0_electronics_diagrams_packages_ipynb__ousefulinfo__-_microsoft_azure_notebooks.png?w=448&amp;h=244&amp;fit=448%2C244&amp;resize=448%2C244?w=300" data-attachment-id="17960" data-image-meta="{" data-image-description="" src="https://ouseful.files.wordpress.com/2018/05/gettingstarted_3_2_0_electronics_diagrams_packages_ipynb__ousefulinfo__-_microsoft_azure_notebooks.png?w=448&h=244&fit=448%2C244&resize=448%2C244" data-permalink="https://blog.ouseful.info/2018/05/14/generative-assessment-creation/gettingstarted_3_2_0_electronics_diagrams_packages_ipynb__ousefulinfo__-_microsoft_azure_notebooks/" data-large-file="https://ouseful.files.wordpress.com/2018/05/gettingstarted_3_2_0_electronics_diagrams_packages_ipynb__ousefulinfo__-_microsoft_azure_notebooks.png?w=448&amp;h=244&amp;fit=448%2C244&amp;resize=448%2C244?w=830" alt="" data-image-title="gettingstarted_3_2_0_Electronics_Diagrams_Packages_ipynb__OUsefulInfo__-_Microsoft_Azure_Notebooks" sizes="(max-width: 448px) 100vw, 448px" width="448" data-comments-opened="1" class="wp-image-17960" data-recalc-dims="1" height="244"></p>
<p>One the to do list is to create some example magics that template different question types.</p>
<p>I’m not sure if OpenCreate is following a similar model? (I seem to have lost access permissions again…)</p>
<p>FWIW, I’ve also started looking at my show’n’tell notebooks again, trying to get them working in Azure notebooks. (OU staff should be able to log in to <code>noteooks.azure.com</code> using <code>OUCU@open.ac.uk</code> credentials.) For the moment, I’m depositing them at <em><a rel="nofollow" href="https://notebooks.azure.com/OUsefulInfo/libraries/gettingstarted" target="_blank">https://notebooks.azure.com/OUsefulInfo/libraries/gettingstarted</a></em>, although some tidying may happen at some point. There are also several more basic demo notebooks I need to put together (e.g. on creating charts and using interactive widgets, digital humanities demos, R demos and (if they work!) polyglot R and python notebook demos, etc.). To use the notebooks interactively, log in and clone the library into your own user space.</p>
	<p>
<div>
<div>To <strong>leave a comment</strong> for the author, please follow the link and comment on their blog: <strong><a href="https://blog.ouseful.info/2018/05/14/generative-assessment-creation/"> Rstats – OUseful.Info, the blog…</a></strong>.</div>
<hr>
<a rel="nofollow" href="https://www.r-bloggers.com/">R-bloggers.com</a> offers <strong><a rel="nofollow" href="https://feedburner.google.com/fb/a/mailverify?uri=RBloggers">daily e-mail updates</a></strong> about <a rel="nofollow" href="https://www.r-project.org/" title="The R Project for Statistical Computing">R</a> news and <a rel="nofollow" href="https://www.r-bloggers.com/search/tutorial" title="R tutorials">tutorials</a> on topics such as: <a rel="nofollow" href="https://www.r-bloggers.com/search/data%20science" title="Data science">Data science</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/Big%20Data" title="Big Data">Big Data, <a rel="nofollow" href="https://www.r-users.com/" title="R jobs">R jobs</a>, visualization (<a rel="nofollow" href="https://www.r-bloggers.com/search/ggplot2" title="ggplot and ggplot2 tutorials">ggplot2</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/boxplot" title="Boxplots using lattice and ggplot2 tutorials">Boxplots</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/map" title="Maps and gis">maps</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/animation" title="Animation in R">animation</a>), programming (<a rel="nofollow" href="https://www.r-bloggers.com/search/RStudio" title="RStudio IDE for R">RStudio</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/sweave" title="Sweave and literate programming">Sweave</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/LaTeX" title="LaTeX in R">LaTeX</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/SQL" title="SQL and databases">SQL</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/eclipse" title="Eclipse IDE for R">Eclipse</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/git" title="git and github, Version Control System">git</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/hadoop" title="Large data in R using Hadoop">hadoop</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/Web+Scraping" title="Web Scraping of google, facebook, yahoo, twitter and more using R">Web Scraping</a>) statistics (<a rel="nofollow" href="https://www.r-bloggers.com/search/regression" title="Regressions and ANOVA analysis tutorials">regression</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/PCA" title="principal component analysis tutorial">PCA</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/time+series" title="Time series">time series</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/trading" title="finance trading">trading</a>) and more...
<img alt="" width="1" src="http://feeds.feedburner.com/~r/RBloggers/~4/RDbGIgwla3s" height="1"></a></div>