<p><div><div><div><div data-width="100" data-show-faces="false" data-send="true" data-layout="button_count" data-height="21" data-href="https://www.r-bloggers.com/text-processing-using-the-texttinyr-package/"></div></div><div></div></div><div></div></div>
<div>
(This article was first published on <strong><a href="http://mlampros.github.io/mlampros.github.io/2017/01/05/textTinyR_package/"> mlampros</a></strong>, and kindly contributed to <a rel="nofollow" href="https://www.r-bloggers.com/">R-bloggers)</a> </div>
<p>This blog post is about my recently released package on CRAN, <strong>textTinyR</strong>. The following notes and examples are based mainly on the package Vignette.</p>
<p>The advantage of the <em>textTinyR</em> package lies in its ability to process big text data files in batches efficiently. For this purpose, it offers functions for splitting, parsing, tokenizing and creating a vocabulary. Moreover, it includes functions for building either a document-term matrix or a term-document matrix and extracting information from those (term-associations, most frequent terms). Lastly, it embodies functions for calculating token statistics (collocations, look-up tables, string dissimilarities) and functions to work with sparse matrices. The source code is based mainly on C++11 and exported in R through the Rcpp, RcppArmadillo and BH packages.</p>
<p></p>
<p>The following <strong>classes</strong> (based on the R6 package) and <strong>functions</strong> are part of the package:</p>
<p></p>
<h4><strong>classes</strong></h4>
<p></p>
<table>
<thead>
<tr>
<th>big_tokenize_transform</th>
<th>sparse_term_matrix</th>
<th>token_stats</th>
</tr>
</thead>
<tbody>
<tr>
<td>big_text_splitter()</td>
<td>Term_Matrix()</td>
<td>path_2vector()</td>
</tr>
<tr>
<td>big_text_parser()</td>
<td>Term_Matrix_Adjust()</td>
<td>freq_distribution()</td>
</tr>
<tr>
<td>big_text_tokenizer()</td>
<td>term_associations()</td>
<td>print_frequency()</td>
</tr>
<tr>
<td>vocabulary_accumulator()</td>
<td>most_frequent_terms()</td>
<td>count_character()</td>
</tr>
<tr>
<td> </td>
<td> </td>
<td>print_count_character()</td>
</tr>
<tr>
<td> </td>
<td> </td>
<td>collocation_words()</td>
</tr>
<tr>
<td> </td>
<td> </td>
<td>print_collocations()</td>
</tr>
<tr>
<td> </td>
<td> </td>
<td>string_dissimilarity_matrix()</td>
</tr>
<tr>
<td> </td>
<td> </td>
<td>look_up_table()</td>
</tr>
<tr>
<td> </td>
<td> </td>
<td>print_words_lookup_tbl()</td>
</tr>
</tbody>
</table>
<p></p>
<h4><strong>functions</strong></h4>
<p></p>
<table>
<thead>
<tr>
<th>sparse_matrices</th>
<th>tokenization</th>
<th>utilities</th>
</tr>
</thead>
<tbody>
<tr>
<td>dense_2sparse()</td>
<td>tokenize_transform_text()</td>
<td>bytes_converter()</td>
</tr>
<tr>
<td>load_sparse_binary()</td>
<td>tokenize_transform_vec_docs()</td>
<td>cosine_distance()</td>
</tr>
<tr>
<td>matrix_sparsity()</td>
<td> </td>
<td>dice_distance()</td>
</tr>
<tr>
<td>save_sparse_binary()</td>
<td> </td>
<td>levenshtein_distance()</td>
</tr>
<tr>
<td>sparse_Means()</td>
<td> </td>
<td>read_characters()</td>
</tr>
<tr>
<td>sparse_Sums()</td>
<td> </td>
<td>read_rows()</td>
</tr>
<tr>
<td> </td>
<td> </td>
<td>text_file_parser()</td>
</tr>
<tr>
<td> </td>
<td> </td>
<td>utf_locale()</td>
</tr>
<tr>
<td> </td>
<td> </td>
<td>vocabulary_parser()</td>
</tr>
</tbody>
</table>
<h3><em>big_tokenize_transform</em> class</h3>
<p></p>
<p>The <em>big_tokenize_transform</em> class can be utilized to process big data files and I’ll illustrate this using the <a target="_blank" rel="nofollow" href="https://dumps.wikimedia.org/enwiki/latest/">english wikipedia pages and articles</a> (to download the data use the following web-address : https://dumps.wikimedia.org/enwiki/latest/enwiki-latest-pages-articles.xml.bz2). The size of the file (after downloading and extracting locally) is aproximalely 59.4 GB and it’s of type .xml (to reproduce the results one needs to have free hard drive space of approx. 200 GB). <br> <br>
<em>Xml</em> files have a tree structure and one should use queries to acquire specific information. First, I’ll observe the structure of the .xml file by using the utility function <em>read_rows()</em>. The <em>read_rows()</em> function takes a file as input and by specifying the <em>rows</em> argument it returns a subset of the file. It doesn’t load the entire file in memory, but it just opens the file and reads the specific number of rows,</p>
<p></p>
<div>
<pre><code><span>
</span><span>library</span><span>(</span><span>textTinyR</span><span>)</span><span>
</span><span>PATH</span><span> </span><span>=</span><span> </span><span>'enwiki-latest-pages-articles.xml'</span><span>
</span><span>subset</span><span> </span><span>=</span><span> </span><span>read_rows</span><span>(</span><span>input_file</span><span> </span><span>=</span><span> </span><span>PATH</span><span>,</span><span> </span><span>read_delimiter</span><span> </span><span>=</span><span> </span><span>&quot;\n&quot;</span><span>,</span><span>
</span><span>rows</span><span> </span><span>=</span><span> </span><span>100</span><span>,</span><span>
</span><span>write_2file</span><span> </span><span>=</span><span> </span><span>&quot;/subs_output.txt&quot;</span><span>)</span><span>
</span></code></pre>
</div>
<p></p>
<div>
<pre><code><span>
</span><span># data subset : subs_output.txt
</span><span>
</span><span>&lt;</span><span>mediawiki</span><span> </span><span>xmlns</span><span>=</span><span>&quot;http://www.mediawiki.org/xml/export-0.10/&quot;</span><span> </span><span>xmlns</span><span>:</span><span>xsi</span><span>=</span><span>&quot;http://www.w3.org/2001/XMLSchema-instance&quot;</span><span> </span><span>xsi</span><span>:</span><span>schemaLocation</span><span>=</span><span>&quot;http://www.mediawiki.org/xml/export-0.10/ http://www.mediawiki.org/xml/export-0.10.xsd&quot;</span><span> </span><span>version</span><span>=</span><span>&quot;0.10&quot;</span><span> </span><span>xml</span><span>:</span><span>lang</span><span>=</span><span>&quot;en&quot;</span><span>></span><span>
</span><span>&lt;</span><span>siteinfo</span><span>></span><span>
</span><span>&lt;</span><span>sitename</span><span>></span><span>Wikipedia</span><span>&lt;/</span><span>sitename</span><span>></span><span>
</span><span>&lt;</span><span>dbname</span><span>></span><span>enwiki</span><span>&lt;/</span><span>dbname</span><span>></span><span>
</span><span>&lt;</span><span>base</span><span>></span><span>https</span><span>://</span><span>en.wikipedia.org</span><span>/</span><span>wiki</span><span>/</span><span>Main_Page</span><span>&lt;/</span><span>base</span><span>></span><span>
</span><span>&lt;</span><span>generator</span><span>></span><span>MediaWiki</span><span> </span><span>1.28.0</span><span>-</span><span>wmf.23</span><span>&lt;/</span><span>generator</span><span>></span><span>
</span><span>&lt;</span><span>case</span><span>></span><span>first</span><span>-</span><span>letter</span><span>&lt;/</span><span>case</span><span>></span><span>
</span><span>&lt;</span><span>namespaces</span><span>></span><span>
</span><span>&lt;</span><span>namespace</span><span> </span><span>key</span><span>=</span><span>&quot;-2&quot;</span><span> </span><span>case</span><span>=</span><span>&quot;first-letter&quot;</span><span>></span><span>Media</span><span>&lt;/</span><span>namespace</span><span>></span><span>
</span><span>&lt;</span><span>namespace</span><span> </span><span>key</span><span>=</span><span>&quot;-1&quot;</span><span> </span><span>case</span><span>=</span><span>&quot;first-letter&quot;</span><span>></span><span>Special</span><span>&lt;/</span><span>namespace</span><span>></span><span>
</span><span>&lt;</span><span>namespace</span><span> </span><span>key</span><span>=</span><span>&quot;0&quot;</span><span> </span><span>case</span><span>=</span><span>&quot;first-letter&quot;</span><span> </span><span>/></span><span>
</span><span>&lt;</span><span>namespace</span><span> </span><span>key</span><span>=</span><span>&quot;1&quot;</span><span> </span><span>case</span><span>=</span><span>&quot;first-letter&quot;</span><span>></span><span>Talk</span><span>&lt;/</span><span>namespace</span><span>></span><span>
</span><span>&lt;</span><span>namespace</span><span> </span><span>key</span><span>=</span><span>&quot;2&quot;</span><span> </span><span>case</span><span>=</span><span>&quot;first-letter&quot;</span><span>></span><span>User</span><span>&lt;/</span><span>namespace</span><span>></span><span>
</span><span>&lt;</span><span>namespace</span><span> </span><span>key</span><span>=</span><span>&quot;3&quot;</span><span> </span><span>case</span><span>=</span><span>&quot;first-letter&quot;</span><span>></span><span>User</span><span> </span><span>talk</span><span>&lt;/</span><span>namespace</span><span>></span><span>
</span><span>&lt;</span><span>namespace</span><span> </span><span>key</span><span>=</span><span>&quot;4&quot;</span><span> </span><span>case</span><span>=</span><span>&quot;first-letter&quot;</span><span>></span><span>Wikipedia</span><span>&lt;/</span><span>namespace</span><span>></span><span>
</span><span>&lt;</span><span>namespace</span><span> </span><span>key</span><span>=</span><span>&quot;5&quot;</span><span> </span><span>case</span><span>=</span><span>&quot;first-letter&quot;</span><span>></span><span>Wikipedia</span><span> </span><span>talk</span><span>&lt;/</span><span>namespace</span><span>></span><span>
</span><span>&lt;</span><span>namespace</span><span> </span><span>key</span><span>=</span><span>&quot;6&quot;</span><span> </span><span>case</span><span>=</span><span>&quot;first-letter&quot;</span><span>></span><span>File</span><span>&lt;/</span><span>namespace</span><span>></span><span>
</span><span>&lt;</span><span>namespace</span><span> </span><span>key</span><span>=</span><span>&quot;7&quot;</span><span> </span><span>case</span><span>=</span><span>&quot;first-letter&quot;</span><span>></span><span>File</span><span> </span><span>talk</span><span>&lt;/</span><span>namespace</span><span>></span><span>
</span><span>&lt;</span><span>namespace</span><span> </span><span>key</span><span>=</span><span>&quot;8&quot;</span><span> </span><span>case</span><span>=</span><span>&quot;first-letter&quot;</span><span>></span><span>MediaWiki</span><span>&lt;/</span><span>namespace</span><span>></span><span>
</span><span>.</span><span>
</span><span>.</span><span>
</span><span>.</span><span>
</span><span>&lt;/</span><span>namespaces</span><span>></span><span>
</span><span>&lt;/</span><span>siteinfo</span><span>></span><span>
</span><span>&lt;</span><span>page</span><span>></span><span>
</span><span>&lt;</span><span>title</span><span>></span><span>AccessibleComputing</span><span>&lt;/</span><span>title</span><span>></span><span>
</span><span>&lt;</span><span>ns</span><span>></span><span>0</span><span>&lt;/</span><span>ns</span><span>></span><span>
</span><span>&lt;</span><span>id</span><span>></span><span>10</span><span>&lt;/</span><span>id</span><span>></span><span>
</span><span>&lt;</span><span>redirect</span><span> </span><span>title</span><span>=</span><span>&quot;Computer accessibility&quot;</span><span> </span><span>/></span><span>
</span><span>&lt;</span><span>revision</span><span>></span><span>
</span><span>&lt;</span><span>id</span><span>></span><span>631144794</span><span>&lt;/</span><span>id</span><span>></span><span>
</span><span>&lt;</span><span>parentid</span><span>></span><span>381202555</span><span>&lt;/</span><span>parentid</span><span>></span><span>
</span><span>&lt;</span><span>timestamp</span><span>></span><span>2014-10-26</span><span>T</span><span>04</span><span>:</span><span>50</span><span>:</span><span>23</span><span>Z</span><span>&lt;/</span><span>timestamp</span><span>></span><span>
</span><span>&lt;</span><span>contributor</span><span>></span><span>
</span><span>&lt;</span><span>username</span><span>></span><span>Paine</span><span> </span><span>Ellsworth</span><span>&lt;/</span><span>username</span><span>></span><span>
</span><span>&lt;</span><span>id</span><span>></span><span>9092818</span><span>&lt;/</span><span>id</span><span>></span><span>
</span><span>&lt;/</span><span>contributor</span><span>></span><span>
</span><span>&lt;</span><span>comment</span><span>></span><span>add</span><span> </span><span>[[</span><span>WP</span><span>:</span><span>RCAT</span><span>|</span><span>rcat</span><span>]]</span><span>s</span><span>&lt;/</span><span>comment</span><span>></span><span>
</span><span>&lt;</span><span>model</span><span>></span><span>wikitext</span><span>&lt;/</span><span>model</span><span>></span><span>
</span><span>&lt;</span><span>format</span><span>></span><span>text</span><span>/</span><span>x</span><span>-</span><span>wiki</span><span>&lt;/</span><span>format</span><span>></span><span>
</span><span>&lt;</span><span>text</span><span> </span><span>xml</span><span>:</span><span>space</span><span>=</span><span>&quot;preserve&quot;</span><span>></span><span>#REDIRECT [[Computer accessibility]]
</span><span>
</span><span>&lt;/</span><span>text</span><span>></span><span>
</span><span>&lt;</span><span>sha1</span><span>></span><span>4</span><span>ro7vvppa5kmm0o1egfjztzcwd0vabw</span><span>&lt;/</span><span>sha1</span><span>></span><span>
</span><span>&lt;/</span><span>revision</span><span>></span><span>
</span><span>&lt;/</span><span>page</span><span>></span><span>
</span><span>&lt;</span><span>page</span><span>></span><span>
</span><span>&lt;</span><span>title</span><span>></span><span>Anarchism</span><span>&lt;/</span><span>title</span><span>></span><span>
</span><span>&lt;</span><span>ns</span><span>></span><span>0</span><span>&lt;/</span><span>ns</span><span>></span><span>
</span><span>&lt;</span><span>id</span><span>></span><span>12</span><span>&lt;/</span><span>id</span><span>></span><span>
</span><span>&lt;</span><span>revision</span><span>></span><span>
</span><span>&lt;</span><span>id</span><span>></span><span>746687538</span><span>&lt;/</span><span>id</span><span>></span><span>
</span><span>&lt;</span><span>parentid</span><span>></span><span>744318951</span><span>&lt;/</span><span>parentid</span><span>></span><span>
</span><span>&lt;</span><span>timestamp</span><span>></span><span>2016-10-28</span><span>T</span><span>22</span><span>:</span><span>43</span><span>:</span><span>19</span><span>Z</span><span>&lt;/</span><span>timestamp</span><span>></span><span>
</span><span>&lt;</span><span>contributor</span><span>></span><span>
</span><span>&lt;</span><span>username</span><span>></span><span>Eduen</span><span>&lt;/</span><span>username</span><span>></span><span>
</span><span>&lt;</span><span>id</span><span>></span><span>7527773</span><span>&lt;/</span><span>id</span><span>></span><span>
</span><span>&lt;/</span><span>contributor</span><span>></span><span>
</span><span>&lt;</span><span>minor</span><span> </span><span>/></span><span>
</span><span>&lt;</span><span>comment</span><span>>/*</span><span> </span><span>Free</span><span> </span><span>love</span><span> </span><span>*/&lt;/</span><span>comment</span><span>></span><span>
</span><span>&lt;</span><span>model</span><span>></span><span>wikitext</span><span>&lt;/</span><span>model</span><span>></span><span>
</span><span>&lt;</span><span>format</span><span>></span><span>text</span><span>/</span><span>x</span><span>-</span><span>wiki</span><span>&lt;/</span><span>format</span><span>></span><span>
</span><span>&lt;</span><span>text</span><span> </span><span>xml</span><span>:</span><span>space</span><span>=</span><span>&quot;preserve&quot;</span><span>></span><span>
</span></code></pre>
</div>
<p></p>
<p>In that way one has a picture of the .xml tree structure and can continue by performing queries. The initial data file is too big to fit in the memory of a PC, thus it has to be split in smaller files, pre-processed and then returned as a single file. The main aim of the <em>big_text_splitter()</em> method is to split the data in smaller files of (approx.) equal size by either using the <em>batches</em> parameter or if the file has a structure by adding the <em>end_query</em> parameter too. Here I’ll take advantage of both the <em>batches</em> and the <em>end_query</em> parameters for this task, because I’ll use queries to extract the text tree-elements, so I don’t want that the file is split arbitrarily. Each sub-element in the file begins and ends with the same key-word, i.e. text,</p>
<p></p>
<div>
<pre><code><span>
</span><span>btt</span><span> </span><span>=</span><span> </span><span>big_tokenize_transform</span><span>$</span><span>new</span><span>(</span><span>verbose</span><span> </span><span>=</span><span> </span><span>TRUE</span><span>)</span><span>
</span><span>btt</span><span>$</span><span>big_text_splitter</span><span>(</span><span>input_path_file</span><span> </span><span>=</span><span> </span><span>PATH</span><span>,</span><span> </span><span># path to the enwiki data file
</span><span> </span><span>output_path_folder</span><span> </span><span>=</span><span> </span><span>&quot;/enwiki_spl_data/&quot;</span><span>,</span><span> </span><span># folder to save the files
</span><span> </span><span>end_query</span><span> </span><span>=</span><span> </span><span>'&lt;/text>'</span><span>,</span><span> </span><span># splits the file taking into account the key-word
</span><span> </span><span>batches</span><span> </span><span>=</span><span> </span><span>40</span><span>,</span><span> </span><span># split file in 40 batches (files)
</span><span> </span><span>trimmed_line</span><span> </span><span>=</span><span> </span><span>FALSE</span><span>)</span><span> </span><span># the lines will be trimmed
</span></code></pre>
</div>
<div>
<pre><code><span>
</span><span>approx.</span><span> </span><span>10</span><span> </span><span>% of data pre-processed
approx. 20 %</span><span> </span><span>of</span><span> </span><span>data</span><span> </span><span>pre</span><span>-</span><span>processed</span><span>
</span><span>approx.</span><span> </span><span>30</span><span> </span><span>% of data pre-processed
approx. 40 %</span><span> </span><span>of</span><span> </span><span>data</span><span> </span><span>pre</span><span>-</span><span>processed</span><span>
</span><span>approx.</span><span> </span><span>50</span><span> </span><span>% of data pre-processed
approx. 60 %</span><span> </span><span>of</span><span> </span><span>data</span><span> </span><span>pre</span><span>-</span><span>processed</span><span>
</span><span>approx.</span><span> </span><span>70</span><span> </span><span>% of data pre-processed
approx. 80 %</span><span> </span><span>of</span><span> </span><span>data</span><span> </span><span>pre</span><span>-</span><span>processed</span><span>
</span><span>approx.</span><span> </span><span>90</span><span> </span><span>% of data pre-processed
approx. 100 %</span><span> </span><span>of</span><span> </span><span>data</span><span> </span><span>pre</span><span>-</span><span>processed</span><span>
</span><span>It</span><span> </span><span>took</span><span> </span><span>42.7098</span><span> </span><span>minutes</span><span> </span><span>to</span><span> </span><span>complete</span><span> </span><span>the</span><span> </span><span>splitting</span><span>
</span></code></pre>
</div>
<p></p>
<p>After the data is split and saved in the <em>output_path_folder</em> (“/ewiki_spl_data/”) the next step is to extract the <strong>text</strong> tree-elements from the batches by using the <em>big_text_parser()</em> method. The latter takes as arguments the previously created <em>input_path_folder</em>, an <em>output_path_folder</em> to save the resulted text files, a <em>start_query</em>, an <em>end_query</em>, the <em>min_lines</em> (only subsets of text with more than or equal to this minimum will be kept) and the <em>trimmed_line</em> ( specifying if each line is already trimmed both-sides ),</p>
<p></p>
<div>
<pre><code><span>
</span><span>btt</span><span>$</span><span>big_text_parser</span><span>(</span><span>input_path_folder</span><span> </span><span>=</span><span> </span><span>&quot;/enwiki_spl_data/&quot;</span><span>,</span><span> </span><span># the previously created folder
</span><span> </span><span>output_path_folder</span><span> </span><span>=</span><span> </span><span>&quot;/enwiki_parse/&quot;</span><span>,</span><span> </span><span># folder to save the parsed files
</span><span> </span><span>start_query</span><span> </span><span>=</span><span> </span><span>&quot;&lt;text xml:space=\&quot;preserve\&quot;>&quot;</span><span>,</span><span> </span><span># starts to extract text
</span><span> </span><span>end_query</span><span> </span><span>=</span><span> </span><span>&quot;&lt;/text>&quot;</span><span>,</span><span> </span><span># stop to extract once here
</span><span> </span><span>min_lines</span><span> </span><span>=</span><span> </span><span>1</span><span>,</span><span> </span><span>trimmed_line</span><span> </span><span>=</span><span> </span><span>TRUE</span><span>)</span><span>
</span></code></pre>
</div>
<div>
<pre><code><span>
</span><span>====================</span><span>
</span><span>batch</span><span> </span><span>1</span><span> </span><span>begins</span><span> </span><span>...</span><span>
</span><span>====================</span><span>
</span><span>approx.</span><span> </span><span>10</span><span> </span><span>% of data pre-processed
approx. 20 %</span><span> </span><span>of</span><span> </span><span>data</span><span> </span><span>pre</span><span>-</span><span>processed</span><span>
</span><span>approx.</span><span> </span><span>30</span><span> </span><span>% of data pre-processed
approx. 40 %</span><span> </span><span>of</span><span> </span><span>data</span><span> </span><span>pre</span><span>-</span><span>processed</span><span>
</span><span>approx.</span><span> </span><span>50</span><span> </span><span>% of data pre-processed
approx. 60 %</span><span> </span><span>of</span><span> </span><span>data</span><span> </span><span>pre</span><span>-</span><span>processed</span><span>
</span><span>approx.</span><span> </span><span>70</span><span> </span><span>% of data pre-processed
approx. 80 %</span><span> </span><span>of</span><span> </span><span>data</span><span> </span><span>pre</span><span>-</span><span>processed</span><span>
</span><span>approx.</span><span> </span><span>90</span><span> </span><span>% of data pre-processed
approx. 100 %</span><span> </span><span>of</span><span> </span><span>data</span><span> </span><span>pre</span><span>-</span><span>processed</span><span>
</span><span>It</span><span> </span><span>took</span><span> </span><span>0.296151</span><span> </span><span>minutes</span><span> </span><span>to</span><span> </span><span>complete</span><span> </span><span>the</span><span> </span><span>preprocessing</span><span>
</span><span>It</span><span> </span><span>took</span><span> </span><span>0.0525948</span><span> </span><span>minutes</span><span> </span><span>to</span><span> </span><span>save</span><span> </span><span>the</span><span> </span><span>pre</span><span>-</span><span>processed</span><span> </span><span>data</span><span>
</span><span>.</span><span>
</span><span>.</span><span>
</span><span>.</span><span>
</span><span>.</span><span>
</span><span>====================</span><span>
</span><span>batch</span><span> </span><span>40</span><span> </span><span>begins</span><span> </span><span>...</span><span>
</span><span>====================</span><span>
</span><span>approx.</span><span> </span><span>10</span><span> </span><span>% of data pre-processed
approx. 20 %</span><span> </span><span>of</span><span> </span><span>data</span><span> </span><span>pre</span><span>-</span><span>processed</span><span>
</span><span>approx.</span><span> </span><span>30</span><span> </span><span>% of data pre-processed
approx. 40 %</span><span> </span><span>of</span><span> </span><span>data</span><span> </span><span>pre</span><span>-</span><span>processed</span><span>
</span><span>approx.</span><span> </span><span>50</span><span> </span><span>% of data pre-processed
approx. 60 %</span><span> </span><span>of</span><span> </span><span>data</span><span> </span><span>pre</span><span>-</span><span>processed</span><span>
</span><span>approx.</span><span> </span><span>70</span><span> </span><span>% of data pre-processed
approx. 80 %</span><span> </span><span>of</span><span> </span><span>data</span><span> </span><span>pre</span><span>-</span><span>processed</span><span>
</span><span>approx.</span><span> </span><span>90</span><span> </span><span>% of data pre-processed
approx. 100 %</span><span> </span><span>of</span><span> </span><span>data</span><span> </span><span>pre</span><span>-</span><span>processed</span><span>
</span><span>It</span><span> </span><span>took</span><span> </span><span>1.04127</span><span> </span><span>minutes</span><span> </span><span>to</span><span> </span><span>complete</span><span> </span><span>the</span><span> </span><span>preprocessing</span><span>
</span><span>It</span><span> </span><span>took</span><span> </span><span>0.0448579</span><span> </span><span>minutes</span><span> </span><span>to</span><span> </span><span>save</span><span> </span><span>the</span><span> </span><span>pre</span><span>-</span><span>processed</span><span> </span><span>data</span><span>
</span><span>It</span><span> </span><span>took</span><span> </span><span>40.9034</span><span> </span><span>minutes</span><span> </span><span>to</span><span> </span><span>complete</span><span> </span><span>the</span><span> </span><span>parsing</span><span>
</span></code></pre>
</div>
<p></p>
<p>Here, it’s worth mentioning that the <em>big_text_parser</em> is more efficient if it extracts big chunks of text, rather than one-liners. In case of one-line text queries it has to check line by line the whole file, which is inefficient especially for files equal to the enwiki size.</p>
<p></p>
<p>By extracting the text chunks from the data the .xml file size reduces to (approx.) 48.9 GB. One can now continue utilizing the <em>big_text_tokenizer()</em> method in order to tokenize and transform the data. This method takes the following parameters:</p>
<p><strong>batches</strong> (each file can be further split in batches during tokenization), <strong>to_lower</strong> (convert to lower case), <strong>to_upper</strong> (convert to upper case), <strong>utf_locale</strong> (change utf locale depending on the language), <strong>remove_char</strong> (remove specific characters from the text), <strong>remove_punctuation_string</strong> (remove punctuation before the data is split), <strong>remove_punctuation_vector</strong> (remove punctuation after the data is split), <strong>remove_numbers</strong> (remove numbers from the data), <strong>trim_token</strong> (trim the tokens both-sides), <strong>split_string</strong> (split the string), <strong>split_separator</strong> (token split seprator where multiple delimiters can be used), <strong>remove_stopwords</strong> (remove stopwords using one of the available languages or by providing a user defined vector of words), <strong>language</strong> (the language of use), <strong>min_num_char</strong> (the minimum number of characters to keep), <strong>max_num_char</strong> (the maximum number of characters to keep), <strong>stemmer</strong> (stemming of the words using either the porter_2steemer or n-gram stemming – those two methods will be explained in the tokenization function), <strong>min_n_gram</strong> (minimum n-grams), <strong>max_n_gram</strong> (maximum n-grams), <strong>skip_n_gram</strong> (skip n-gram), <strong>skip_distance</strong> (skip distance for n-grams), <strong>n_gram_delimiter</strong> (n-gram delimiter), <strong>concat_delimiter</strong> (concatenation of the data in case that one wants to save the file), <strong>path_2folder</strong> (specified folder to save the data), <strong>stemmer_ngram</strong> (in case of n-gram stemming the n-grams), <strong>stemmer_gamma</strong> (in case of n-gram stemming the gamma parameter), <strong>stemmer_truncate</strong> (in case of n-gram stemming the truncation parameter), <strong>stemmer_batches</strong> (in case of n-gram stemming the batches parameter ), <strong>threads</strong> (the number of cores to use in parallel ), <strong>save_2single_file</strong> (should the output data be saved in a single file), <strong>increment_batch_nr</strong> (the enumeration of the output files will start from this number), <strong>vocabulary_path_file</strong> (should a vocabulary be saved in a separate file). </p>
<p>More information about those parameters can be found in the package documentation.</p>
<p></p>
<p>In this vignette I’ll continue using the following transformations:</p>
<ul>
<li>conversion to lowercase</li>
<li>trim each line</li>
<li>split each line using multiple delimiters</li>
<li>remove the punctuation ( once splitting is taken place )</li>
<li>remove the numbers from the tokens</li>
<li>limit the output words to a specific number of characters</li>
<li>remove the english stopwords</li>
<li>and save both the data (to a single file) and the vocabulary files (to a folder).</li>
</ul>
<p>Each initial file will be split in additional batches to limit the memory usage during the tokenization and transformation phase,</p>
<p></p>
<div>
<pre><code><span>
</span><span>btt</span><span>$</span><span>big_text_tokenizer</span><span>(</span><span>input_path_folder</span><span> </span><span>=</span><span> </span><span>&quot;/enwiki_parse/&quot;</span><span>,</span><span> </span><span># the previously parsed data
</span><span> </span><span>batches</span><span> </span><span>=</span><span> </span><span>4</span><span>,</span><span> </span><span># each single file will be split further in 4 batches
</span><span> </span><span>to_lower</span><span> </span><span>=</span><span> </span><span>TRUE</span><span>,</span><span> </span><span>trim_token</span><span> </span><span>=</span><span> </span><span>TRUE</span><span>,</span><span>
</span><span>split_string</span><span>=</span><span>TRUE</span><span>,</span><span> </span><span>max_num_char</span><span> </span><span>=</span><span> </span><span>100</span><span>,</span><span>
</span><span>split_separator</span><span> </span><span>=</span><span> </span><span>&quot; \r\n\t.,;:()?!//&quot;</span><span>,</span><span>
</span><span>remove_punctuation_vector</span><span> </span><span>=</span><span> </span><span>TRUE</span><span>,</span><span>
</span><span>remove_numbers</span><span> </span><span>=</span><span> </span><span>TRUE</span><span>,</span><span>
</span><span>remove_stopwords</span><span> </span><span>=</span><span> </span><span>TRUE</span><span>,</span><span> </span><span>threads</span><span> </span><span>=</span><span> </span><span>4</span><span>,</span><span> </span><span>save_2single_file</span><span> </span><span>=</span><span> </span><span>TRUE</span><span>,</span><span> </span><span># save to a single file
</span><span> </span><span>vocabulary_path_folder</span><span> </span><span>=</span><span> </span><span>&quot;/enwiki_vocab/&quot;</span><span>,</span><span> </span><span># path to vocabulary folder
</span><span> </span><span>path_2folder</span><span>=</span><span>&quot;/enwiki_token/&quot;</span><span>)</span><span> </span><span># folder to save the transformed data
</span><span>
</span></code></pre>
</div>
<div>
<pre><code><span>
</span><span>====================================</span><span>
</span><span>transformation</span><span> </span><span>of</span><span> </span><span>file</span><span> </span><span>1</span><span> </span><span>starts</span><span> </span><span>...</span><span>
</span><span>====================================</span><span>
</span><span>-------------------</span><span>
</span><span>batch</span><span> </span><span>1</span><span> </span><span>begins</span><span> </span><span>...</span><span>
</span><span>-------------------</span><span>
</span><span>input</span><span> </span><span>of</span><span> </span><span>the</span><span> </span><span>data</span><span> </span><span>starts</span><span> </span><span>...</span><span>
</span><span>conversion</span><span> </span><span>to</span><span> </span><span>lower</span><span> </span><span>case</span><span> </span><span>starts</span><span> </span><span>...</span><span>
</span><span>removal</span><span> </span><span>of</span><span> </span><span>numeric</span><span> </span><span>values</span><span> </span><span>starts</span><span> </span><span>...</span><span>
</span><span>the</span><span> </span><span>string</span><span>-</span><span>trim</span><span> </span><span>starts</span><span> </span><span>...</span><span>
</span><span>the</span><span> </span><span>split</span><span> </span><span>of</span><span> </span><span>the</span><span> </span><span>character</span><span> </span><span>string</span><span> </span><span>and</span><span> </span><span>simultaneously</span><span> </span><span>the</span><span> </span><span>removal</span><span> </span><span>of</span><span> </span><span>the</span><span> </span><span>punctuation</span><span> </span><span>in</span><span> </span><span>the</span><span> </span><span>vector</span><span> </span><span>starts</span><span> </span><span>...</span><span>
</span><span>stop</span><span> </span><span>words</span><span> </span><span>of</span><span> </span><span>the</span><span> </span><span>english</span><span> </span><span>language</span><span> </span><span>will</span><span> </span><span>be</span><span> </span><span>used</span><span>
</span><span>the</span><span> </span><span>removal</span><span> </span><span>of</span><span> </span><span>stop</span><span>-</span><span>words</span><span> </span><span>starts</span><span> </span><span>...</span><span>
</span><span>character</span><span> </span><span>strings</span><span> </span><span>with</span><span> </span><span>more</span><span> </span><span>than</span><span> </span><span>or</span><span> </span><span>equal</span><span> </span><span>to</span><span> </span><span>1</span><span> </span><span>and</span><span> </span><span>less</span><span> </span><span>than</span><span> </span><span>100</span><span> </span><span>characters</span><span> </span><span>will</span><span> </span><span>be</span><span> </span><span>kept</span><span> </span><span>...</span><span>
</span><span>the</span><span> </span><span>vocabulary</span><span> </span><span>counts</span><span> </span><span>will</span><span> </span><span>be</span><span> </span><span>saved</span><span> </span><span>in</span><span>:</span><span> </span><span>/</span><span>enwiki_vocab</span><span>/</span><span>batch1.txt</span><span>
</span><span>the</span><span> </span><span>pre</span><span>-</span><span>processed</span><span> </span><span>data</span><span> </span><span>will</span><span> </span><span>be</span><span> </span><span>saved</span><span> </span><span>in</span><span> </span><span>a</span><span> </span><span>single</span><span> </span><span>file</span><span> </span><span>in</span><span>:</span><span> </span><span>/</span><span>enwiki_token</span><span>/</span><span>
</span><span>-------------------</span><span>
</span><span>batch</span><span> </span><span>2</span><span> </span><span>begins</span><span> </span><span>...</span><span>
</span><span>-------------------</span><span>
</span><span>input</span><span> </span><span>of</span><span> </span><span>the</span><span> </span><span>data</span><span> </span><span>starts</span><span> </span><span>...</span><span>
</span><span>conversion</span><span> </span><span>to</span><span> </span><span>lower</span><span> </span><span>case</span><span> </span><span>starts</span><span> </span><span>...</span><span>
</span><span>removal</span><span> </span><span>of</span><span> </span><span>numeric</span><span> </span><span>values</span><span> </span><span>starts</span><span> </span><span>...</span><span>
</span><span>the</span><span> </span><span>string</span><span>-</span><span>trim</span><span> </span><span>starts</span><span> </span><span>...</span><span>
</span><span>the</span><span> </span><span>split</span><span> </span><span>of</span><span> </span><span>the</span><span> </span><span>character</span><span> </span><span>string</span><span> </span><span>and</span><span> </span><span>simultaneously</span><span> </span><span>the</span><span> </span><span>removal</span><span> </span><span>of</span><span> </span><span>the</span><span> </span><span>punctuation</span><span> </span><span>in</span><span> </span><span>the</span><span> </span><span>vector</span><span> </span><span>starts</span><span> </span><span>...</span><span>
</span><span>stop</span><span> </span><span>words</span><span> </span><span>of</span><span> </span><span>the</span><span> </span><span>english</span><span> </span><span>language</span><span> </span><span>will</span><span> </span><span>be</span><span> </span><span>used</span><span>
</span><span>the</span><span> </span><span>removal</span><span> </span><span>of</span><span> </span><span>stop</span><span>-</span><span>words</span><span> </span><span>starts</span><span> </span><span>...</span><span>
</span><span>character</span><span> </span><span>strings</span><span> </span><span>with</span><span> </span><span>more</span><span> </span><span>than</span><span> </span><span>or</span><span> </span><span>equal</span><span> </span><span>to</span><span> </span><span>1</span><span> </span><span>and</span><span> </span><span>less</span><span> </span><span>than</span><span> </span><span>100</span><span> </span><span>characters</span><span> </span><span>will</span><span> </span><span>be</span><span> </span><span>kept</span><span> </span><span>...</span><span>
</span><span>the</span><span> </span><span>vocabulary</span><span> </span><span>counts</span><span> </span><span>will</span><span> </span><span>be</span><span> </span><span>saved</span><span> </span><span>in</span><span>:</span><span> </span><span>/</span><span>enwiki_vocab</span><span>/</span><span>batch1.txt</span><span>
</span><span>the</span><span> </span><span>pre</span><span>-</span><span>processed</span><span> </span><span>data</span><span> </span><span>will</span><span> </span><span>be</span><span> </span><span>saved</span><span> </span><span>in</span><span> </span><span>a</span><span> </span><span>single</span><span> </span><span>file</span><span> </span><span>in</span><span>:</span><span> </span><span>/</span><span>enwiki_token</span><span>/</span><span>
</span><span>.</span><span>
</span><span>.</span><span>
</span><span>.</span><span>
</span><span>.</span><span>
</span><span>====================================</span><span>
</span><span>transformation</span><span> </span><span>of</span><span> </span><span>file</span><span> </span><span>40</span><span> </span><span>starts</span><span> </span><span>...</span><span>
</span><span>====================================</span><span>
</span><span>-------------------</span><span>
</span><span>batch</span><span> </span><span>1</span><span> </span><span>begins</span><span> </span><span>...</span><span>
</span><span>-------------------</span><span>
</span><span>input</span><span> </span><span>of</span><span> </span><span>the</span><span> </span><span>data</span><span> </span><span>starts</span><span> </span><span>...</span><span>
</span><span>conversion</span><span> </span><span>to</span><span> </span><span>lower</span><span> </span><span>case</span><span> </span><span>starts</span><span> </span><span>...</span><span>
</span><span>removal</span><span> </span><span>of</span><span> </span><span>numeric</span><span> </span><span>values</span><span> </span><span>starts</span><span> </span><span>...</span><span>
</span><span>the</span><span> </span><span>string</span><span>-</span><span>trim</span><span> </span><span>starts</span><span> </span><span>...</span><span>
</span><span>the</span><span> </span><span>split</span><span> </span><span>of</span><span> </span><span>the</span><span> </span><span>character</span><span> </span><span>string</span><span> </span><span>and</span><span> </span><span>simultaneously</span><span> </span><span>the</span><span> </span><span>removal</span><span> </span><span>of</span><span> </span><span>the</span><span> </span><span>punctuation</span><span> </span><span>in</span><span> </span><span>the</span><span> </span><span>vector</span><span> </span><span>starts</span><span> </span><span>...</span><span>
</span><span>stop</span><span> </span><span>words</span><span> </span><span>of</span><span> </span><span>the</span><span> </span><span>english</span><span> </span><span>language</span><span> </span><span>will</span><span> </span><span>be</span><span> </span><span>used</span><span>
</span><span>the</span><span> </span><span>removal</span><span> </span><span>of</span><span> </span><span>stop</span><span>-</span><span>words</span><span> </span><span>starts</span><span> </span><span>...</span><span>
</span><span>character</span><span> </span><span>strings</span><span> </span><span>with</span><span> </span><span>more</span><span> </span><span>than</span><span> </span><span>or</span><span> </span><span>equal</span><span> </span><span>to</span><span> </span><span>1</span><span> </span><span>and</span><span> </span><span>less</span><span> </span><span>than</span><span> </span><span>100</span><span> </span><span>characters</span><span> </span><span>will</span><span> </span><span>be</span><span> </span><span>kept</span><span> </span><span>...</span><span>
</span><span>the</span><span> </span><span>vocabulary</span><span> </span><span>counts</span><span> </span><span>will</span><span> </span><span>be</span><span> </span><span>saved</span><span> </span><span>in</span><span>:</span><span> </span><span>/</span><span>enwiki_vocab</span><span>/</span><span>batch40.txt</span><span>
</span><span>the</span><span> </span><span>pre</span><span>-</span><span>processed</span><span> </span><span>data</span><span> </span><span>will</span><span> </span><span>be</span><span> </span><span>saved</span><span> </span><span>in</span><span> </span><span>a</span><span> </span><span>single</span><span> </span><span>file</span><span> </span><span>in</span><span>:</span><span> </span><span>/</span><span>enwiki_token</span><span>/</span><span>
</span><span>-------------------</span><span>
</span><span>batch</span><span> </span><span>2</span><span> </span><span>begins</span><span> </span><span>...</span><span>
</span><span>-------------------</span><span>
</span><span>input</span><span> </span><span>of</span><span> </span><span>the</span><span> </span><span>data</span><span> </span><span>starts</span><span> </span><span>...</span><span>
</span><span>conversion</span><span> </span><span>to</span><span> </span><span>lower</span><span> </span><span>case</span><span> </span><span>starts</span><span> </span><span>...</span><span>
</span><span>removal</span><span> </span><span>of</span><span> </span><span>numeric</span><span> </span><span>values</span><span> </span><span>starts</span><span> </span><span>...</span><span>
</span><span>the</span><span> </span><span>string</span><span>-</span><span>trim</span><span> </span><span>starts</span><span> </span><span>...</span><span>
</span><span>the</span><span> </span><span>split</span><span> </span><span>of</span><span> </span><span>the</span><span> </span><span>character</span><span> </span><span>string</span><span> </span><span>and</span><span> </span><span>simultaneously</span><span> </span><span>the</span><span> </span><span>removal</span><span> </span><span>of</span><span> </span><span>the</span><span> </span><span>punctuation</span><span> </span><span>in</span><span> </span><span>the</span><span> </span><span>vector</span><span> </span><span>starts</span><span> </span><span>...</span><span>
</span><span>stop</span><span> </span><span>words</span><span> </span><span>of</span><span> </span><span>the</span><span> </span><span>english</span><span> </span><span>language</span><span> </span><span>will</span><span> </span><span>be</span><span> </span><span>used</span><span>
</span><span>the</span><span> </span><span>removal</span><span> </span><span>of</span><span> </span><span>stop</span><span>-</span><span>words</span><span> </span><span>starts</span><span> </span><span>...</span><span>
</span><span>character</span><span> </span><span>strings</span><span> </span><span>with</span><span> </span><span>more</span><span> </span><span>than</span><span> </span><span>or</span><span> </span><span>equal</span><span> </span><span>to</span><span> </span><span>1</span><span> </span><span>and</span><span> </span><span>less</span><span> </span><span>than</span><span> </span><span>100</span><span> </span><span>characters</span><span> </span><span>will</span><span> </span><span>be</span><span> </span><span>kept</span><span> </span><span>...</span><span>
</span><span>the</span><span> </span><span>vocabulary</span><span> </span><span>counts</span><span> </span><span>will</span><span> </span><span>be</span><span> </span><span>saved</span><span> </span><span>in</span><span>:</span><span> </span><span>/</span><span>enwiki_vocab</span><span>/</span><span>batch40.txt</span><span>
</span><span>the</span><span> </span><span>pre</span><span>-</span><span>processed</span><span> </span><span>data</span><span> </span><span>will</span><span> </span><span>be</span><span> </span><span>saved</span><span> </span><span>in</span><span> </span><span>a</span><span> </span><span>single</span><span> </span><span>file</span><span> </span><span>in</span><span>:</span><span> </span><span>/</span><span>enwiki_token</span><span>/</span><span>
</span><span>.</span><span>
</span><span>.</span><span>
</span><span>.</span><span>
</span><span>.</span><span>
</span><span>It</span><span> </span><span>took</span><span> </span><span>111.689</span><span> </span><span>minutes</span><span> </span><span>to</span><span> </span><span>complete</span><span> </span><span>tokenization</span><span>
</span></code></pre>
</div>
<p></p>
<p>In total, it took approx. 195 minutes (or 3.25 hours) to pre-process (including tokenization, transformation and vocabulary extraction) the 59.4 GB of the enwiki data. </p>
<p></p>
<h4><em>word cloud</em></h4>
<p>Having a clean single text file of all the wikipedia pages and articles one can perform many tasks. For instance, one can build a <em>wordcloud</em> (using the wordcloud package) from the accumulated words (a word of caution : the memory consumption when running the <em>vocabulary_accumulator</em> method for this kind of data size can exceed the 10 GB),</p>
<p></p>
<div>
<pre><code><span>
</span><span>init</span><span>$</span><span>vocabulary_accumulator</span><span>(</span><span>input_path_folder</span><span> </span><span>=</span><span> </span><span>&quot;/enwiki_vocab/&quot;</span><span>,</span><span> </span><span>vocabulary_path_file</span><span> </span><span>=</span><span> </span><span>&quot;/VOCAB.txt&quot;</span><span>,</span><span>
</span><span>max_num_chars</span><span> </span><span>=</span><span> </span><span>50</span><span>)</span><span>
</span></code></pre>
</div>
<div>
<pre><code><span>
</span><span>vocabulary.of.batch</span><span> </span><span>40</span><span> </span><span>will.be.merged</span><span> </span><span>...</span><span>	</span><span>minutes.to.merge.sort.batches</span><span>:</span><span> </span><span>4.57273</span><span>
</span><span>minutes.to.save.data</span><span>:</span><span> </span><span>0.48584</span><span>
</span></code></pre>
</div>
<p></p>
<p>The following table shows the first rows of the vocabulary counts,</p>
<p></p>
<table>
<thead>
<tr>
<th>terms</th>
<th>frequency</th>
</tr>
</thead>
<tbody>
<tr>
<td>lt</td>
<td>111408435L</td>
</tr>
<tr>
<td>refgt</td>
<td>49197149L</td>
</tr>
<tr>
<td>quot</td>
<td>48688082L</td>
</tr>
<tr>
<td>gt</td>
<td>47466149L</td>
</tr>
<tr>
<td>user</td>
<td>32042007L</td>
</tr>
<tr>
<td>category</td>
<td>30619748L</td>
</tr>
<tr>
<td>www</td>
<td>25358252L</td>
</tr>
<tr>
<td>http</td>
<td>23008243L</td>
</tr>
</tbody>
</table>
<p></p>
<p>before plotting the wordcloud, I’ll limit the vocabulary to the first 200 words,</p>
<p></p>
<div>
<pre><code><span>
</span><span>rdr_vocab</span><span> </span><span>=</span><span> </span><span>textTinyR</span><span>::</span><span>read_rows</span><span>(</span><span>input_file</span><span> </span><span>=</span><span> </span><span>&quot;/VOCAB.txt&quot;</span><span>,</span><span> </span><span>read_delimiter</span><span> </span><span>=</span><span> </span><span>&quot;\n&quot;</span><span>,</span><span>
</span><span>rows</span><span> </span><span>=</span><span> </span><span>200</span><span>,</span><span> </span><span>write_2file</span><span> </span><span>=</span><span> </span><span>&quot;/vocab_subset_200terms.txt&quot;</span><span>)</span><span> </span><span># read the reduced data </span><span>
</span><span>vocab_sbs</span><span> </span><span>&lt;-</span><span> </span><span>readr</span><span>::</span><span>read_delim</span><span>(</span><span>&quot;/vocab_subset_200terms.txt&quot;</span><span>,</span><span> </span><span>&quot;\t&quot;</span><span>,</span><span>
</span><span>escape_double</span><span> </span><span>=</span><span> </span><span>FALSE</span><span>,</span><span> </span><span>col_names</span><span> </span><span>=</span><span> </span><span>FALSE</span><span>,</span><span> </span><span>trim_ws</span><span> </span><span>=</span><span> </span><span>TRUE</span><span>)</span><span>
</span><span># create the wordcloud
</span><span>
</span><span>pal2</span><span> </span><span>&lt;-</span><span> </span><span>RColorBrewer</span><span>::</span><span>brewer.pal</span><span>(</span><span>8</span><span>,</span><span> </span><span>&quot;Dark2&quot;</span><span>)</span><span>
</span><span>wordcloud</span><span>::</span><span>wordcloud</span><span>(</span><span>words</span><span> </span><span>=</span><span> </span><span>vocab_sbs</span><span>$</span><span>X</span><span>1</span><span>,</span><span> </span><span>freq</span><span> </span><span>=</span><span> </span><span>vocab_sbs</span><span>$</span><span>X</span><span>2</span><span>,</span><span> </span><span>scale</span><span> </span><span>=</span><span> </span><span>c</span><span>(</span><span>4.5</span><span>,</span><span> </span><span>0.8</span><span>),</span><span> </span><span>random.order</span><span> </span><span>=</span><span> </span><span>FALSE</span><span>,</span><span> </span><span>rot.per</span><span> </span><span>=</span><span> </span><span>.15</span><span>,</span><span> </span><span>colors</span><span> </span><span>=</span><span> </span><span>pal2</span><span>)</span><span>
</span></code></pre>
</div>
<p></p>
<p><img alt="Alt text" src="https://i0.wp.com/mlampros.github.io/images/Rplot_textTiny.png?w=456" data-recalc-dims="1"></p>
<p></p>
<h4><em>word vectors</em></h4>
<p>I’m currently also interested in word vectors and that’s why I also made R-wrappers for the <a target="_blank" rel="nofollow" href="http://nlp.stanford.edu/projects/glove/">GloVe</a> and the <a target="_blank" rel="nofollow" href="https://github.com/facebookresearch/fastText">fastText</a> word representation algorithms. The latter two reside in my Github account as separate repositories (<a target="_blank" rel="nofollow" href="https://github.com/mlampros/GloveR">GloveR</a> and <a target="_blank" rel="nofollow" href="https://github.com/mlampros/fastTextR">fastTextR</a>) and can be installed using the <em>install_github</em> function of the devtools package (devtools::install_github(‘mlampros/GloveR’), devtools::install_github(‘mlampros/fastTextR’)). <br>
“A <a target="_blank" rel="nofollow" href="http://www.iro.umontreal.ca/~lisa/pointeurs/turian-wordrepresentations-acl10.pdf">word representation</a> is a mathematical object associated with each word, often a vector. Each dimension’s value corresponds to a feature and might even have a semantic or grammatical interpretation, so we call it a word feature. Conventionally, supervised lexicalized NLP approaches take a word and convert it to a symbolic ID, which is then transformed into a feature vector using a one-hot representation: The feature vector has the same length as the size of the vocabulary, and only one dimension is on.” </p>
<p>Currently, there are many resources on the web on how to use <a target="_blank" rel="nofollow" href="https://blog.keras.io/using-pre-trained-word-embeddings-in-a-keras-model.html">pre-trained word vectors (embeddings) as input to neural networks</a>.</p>
<p>In this vignette I’ll use only the <em>fastTextR</em> word representation algorithm, however detailed documentation and system requirements on how to use either the <em>GloveR</em> or the <em>fastTextR</em> can be found in the corresponding repository. If I would train the whole data file (32.2 GB) using the <em>fastTextR</em> wrapper, it would take (approx.) 15 hours,</p>
<p></p>
<div>
<pre><code><span>
</span><span>library</span><span>(</span><span>fastTextR</span><span>)</span><span>
</span><span>skp</span><span> </span><span>=</span><span> </span><span>skipgram_cbow</span><span>(</span><span>input_path</span><span> </span><span>=</span><span> </span><span>&quot;/output_token_single_file.txt&quot;</span><span>,</span><span> </span><span>thread</span><span> </span><span>=</span><span> </span><span>4</span><span>,</span><span> </span><span>dim</span><span> </span><span>=</span><span> </span><span>50</span><span>,</span><span>
</span><span>output_path</span><span> </span><span>=</span><span> </span><span>&quot;/model&quot;</span><span>,</span><span> </span><span>method</span><span> </span><span>=</span><span> </span><span>&quot;skipgram&quot;</span><span>,</span><span> </span><span>verbose</span><span> </span><span>=</span><span> </span><span>2</span><span>)</span><span>
</span></code></pre>
</div>
<div>
<pre><code><span>
</span><span>Read</span><span> </span><span>4018</span><span>M</span><span> </span><span>words</span><span>
</span><span>Number</span><span> </span><span>of</span><span> </span><span>words</span><span>:</span><span> </span><span>12827221</span><span>
</span><span>Number</span><span> </span><span>of</span><span> </span><span>labels</span><span>:</span><span> </span><span>0</span><span>
</span><span>Progress</span><span>:</span><span> </span><span>0.2</span><span>%</span><span> </span><span>words</span><span>/</span><span>sec</span><span>/</span><span>thread</span><span>:</span><span> </span><span>89664</span><span> </span><span>lr</span><span>:</span><span> </span><span>0.099841</span><span> </span><span>loss</span><span>:</span><span> </span><span>1.055581</span><span> </span><span>eta</span><span>:</span><span> </span><span>15</span><span>h</span><span>32</span><span>m</span><span> </span><span>14</span><span>m</span><span>
</span></code></pre>
</div>
<p></p>
<p>thus, just for illustration purposes I’ll limit the train data to approx. 1 GB of the output file,</p>
<p></p>
<div>
<pre><code><span>
</span><span>reduced_data</span><span> </span><span>=</span><span> </span><span>read_characters</span><span>(</span><span>input_file</span><span> </span><span>=</span><span> </span><span>&quot;/output_token_single_file.txt&quot;</span><span>,</span><span> </span><span>characters</span><span> </span><span>=</span><span> </span><span>1000000000</span><span>,</span><span> </span><span># approx. 1 GB of the data
</span><span> </span><span>write_2file</span><span> </span><span>=</span><span> </span><span>&quot;/reduced_single_file.txt&quot;</span><span>)</span><span>
</span><span>skp</span><span> </span><span>=</span><span> </span><span>skipgram_cbow</span><span>(</span><span>input_path</span><span> </span><span>=</span><span> </span><span>&quot;/reduced_single_file.txt&quot;</span><span>,</span><span> </span><span># reduced data set
</span><span> </span><span>output_path</span><span> </span><span>=</span><span> </span><span>&quot;/model&quot;</span><span>,</span><span> </span><span># saves model and word vectors
</span><span> </span><span>dim</span><span> </span><span>=</span><span> </span><span>50</span><span>,</span><span> </span><span># 50-dimensional word vectors
</span><span> </span><span>method</span><span> </span><span>=</span><span> </span><span>&quot;skipgram&quot;</span><span>,</span><span> </span><span># method = 'skipgram' </span><span> </span><span>thread</span><span> </span><span>=</span><span> </span><span>4</span><span>,</span><span> </span><span>verbose</span><span> </span><span>=</span><span> </span><span>2</span><span>)</span><span>
</span></code></pre>
</div>
<div>
<pre><code><span>
</span><span>Read</span><span> </span><span>124</span><span>M</span><span> </span><span>words</span><span>
</span><span>Number</span><span> </span><span>of</span><span> </span><span>words</span><span>:</span><span> </span><span>5029370</span><span>
</span><span>Number</span><span> </span><span>of</span><span> </span><span>labels</span><span>:</span><span> </span><span>0</span><span>
</span><span>Progress</span><span>:</span><span> </span><span>100.0</span><span>%</span><span> </span><span>words</span><span>/</span><span>sec</span><span>/</span><span>thread</span><span>:</span><span> </span><span>94027</span><span> </span><span>lr</span><span>:</span><span> </span><span>0.000000</span><span> </span><span>loss</span><span>:</span><span> </span><span>0.186674</span><span> </span><span>eta</span><span>:</span><span> </span><span>0</span><span>h</span><span>0</span><span>m</span><span> </span><span>time</span><span> </span><span>to</span><span> </span><span>complete</span><span> </span><span>:</span><span> </span><span>33.53577</span><span> </span><span>mins</span><span> </span></code></pre>
</div>
<p></p>
<p>the following vector-subset is the example output of the “model.vec” file, which includes the 50-dimensional word vectors,</p>
<p></p>
<div>
<pre><code><span>
</span><span>lt</span><span> </span><span>0.12207</span><span> </span><span>0.16117</span><span> </span><span>0.4641</span><span> </span><span>0.73876</span><span> </span><span>0.43968</span><span> </span><span>0.63911</span><span> </span><span>-0.53747</span><span> </span><span>0.1398</span><span> </span><span>.....</span><span> </span><span>refgt</span><span> </span><span>-0.0038898</span><span> </span><span>-0.13976</span><span> </span><span>0.26077</span><span> </span><span>0.7775</span><span> </span><span>0.2228</span><span> </span><span>0.28169</span><span> </span><span>-0.48306</span><span> </span><span>.....</span><span>
</span><span>quot</span><span> </span><span>0.7295</span><span> </span><span>-0.12472</span><span> </span><span>0.32131</span><span> </span><span>0.46965</span><span> </span><span>0.45363</span><span> </span><span>0.85022</span><span> </span><span>-0.051471</span><span> </span><span>.....</span><span> </span><span>gt</span><span> </span><span>0.41287</span><span> </span><span>0.26584</span><span> </span><span>0.6612</span><span> </span><span>0.78185</span><span> </span><span>0.46692</span><span> </span><span>0.74092</span><span> </span><span>-0.23816</span><span> </span><span>.....</span><span>
</span><span>cite</span><span> </span><span>0.037943</span><span> </span><span>0.095684</span><span> </span><span>0.62832</span><span> </span><span>0.93794</span><span> </span><span>0.19776</span><span> </span><span>0.44592</span><span> </span><span>-0.21641</span><span> </span><span>.....</span><span>
</span><span>www</span><span> </span><span>-0.31855</span><span> </span><span>0.42268</span><span> </span><span>0.3875</span><span> </span><span>1.5457</span><span> </span><span>-0.23804</span><span> </span><span>0.34022</span><span> </span><span>-0.051849</span><span> </span><span>.....</span><span> </span><span>ref</span><span> </span><span>0.45236</span><span> </span><span>-0.21766</span><span> </span><span>0.6341</span><span> </span><span>0.76392</span><span> </span><span>0.53734</span><span> </span><span>0.66976</span><span> </span><span>-0.23162</span><span> </span><span>.....</span><span>
</span><span>http</span><span> </span><span>-0.42692</span><span> </span><span>0.48637</span><span> </span><span>0.28622</span><span> </span><span>1.7019</span><span> </span><span>-0.25739</span><span> </span><span>0.25948</span><span> </span><span>-0.026582</span><span> </span><span>.....</span><span> </span><span>namequot</span><span> </span><span>0.56828</span><span> </span><span>-0.30782</span><span> </span><span>0.45707</span><span> </span><span>0.78346</span><span> </span><span>0.53727</span><span> </span><span>0.62445</span><span> </span><span>.....</span><span> </span><span>–</span><span> </span><span>-0.010281</span><span> </span><span>0.25528</span><span> </span><span>0.04708</span><span> </span><span>0.49679</span><span> </span><span>0.043934</span><span> </span><span>0.33733</span><span> </span><span>-0.42706</span><span> </span><span>.....</span><span>
</span><span>amp</span><span> </span><span>0.06308</span><span> </span><span>0.11968</span><span> </span><span>0.11885</span><span> </span><span>0.67699</span><span> </span><span>-0.11448</span><span> </span><span>0.25183</span><span> </span><span>-0.48789</span><span> </span><span>.....</span><span>
</span><span>category</span><span> </span><span>-1.5705</span><span> </span><span>-0.40638</span><span> </span><span>0.61064</span><span> </span><span>2.5691</span><span> </span><span>-0.52987</span><span> </span><span>0.68096</span><span> </span><span>.....</span><span>
</span><span>county</span><span> </span><span>-0.85743</span><span> </span><span>0.071625</span><span> </span><span>-0.43393</span><span> </span><span>0.17157</span><span> </span><span>-0.32874</span><span> </span><span>1.771</span><span> </span><span>.....</span><span> </span><span>org</span><span> </span><span>-0.26974</span><span> </span><span>0.76983</span><span> </span><span>0.57599</span><span> </span><span>1.5939</span><span> </span><span>-0.1706</span><span> </span><span>0.21937</span><span> </span><span>-0.44645</span><span> </span><span>.....</span><span>
</span><span>states</span><span> </span><span>-0.40973</span><span> </span><span>-0.48528</span><span> </span><span>0.092905</span><span> </span><span>0.011603</span><span> </span><span>-0.035727</span><span> </span><span>0.52807</span><span> </span><span>.....</span><span>
</span><span>united</span><span> </span><span>-0.25079</span><span> </span><span>-0.49813</span><span> </span><span>0.070942</span><span> </span><span>0.16762</span><span> </span><span>0.069961</span><span> </span><span>0.56868</span><span> </span><span>.....</span><span>
</span><span>web</span><span> </span><span>-0.066578</span><span> </span><span>0.14837</span><span> </span><span>0.23088</span><span> </span><span>1.2919</span><span> </span><span>-0.252</span><span> </span><span>0.31441</span><span> </span><span>-0.3799</span><span> </span><span>.....</span><span> </span><span>census</span><span> </span><span>-0.29033</span><span> </span><span>-0.73695</span><span> </span><span>0.35474</span><span> </span><span>-0.5237</span><span> </span><span>-0.15206</span><span> </span><span>1.7089</span><span> </span><span>.....</span><span>
</span><span>.</span><span>
</span><span>.</span><span>
</span><span>.</span><span>
</span></code></pre>
</div>
<p></p>
<h3><em>sparse_term_matrix</em> class</h3>
<p></p>
<p>The <em>sparse_term_matrix</em> class includes methods for building a document-term or a term-document matrix and extracting information from those matrices (it relies on RcppArmadillo and can handle large sparse matrices too). I’ll explain all the different methods using a toy text file downloaded from wikipedia,</p>
<p></p>
<div>
<pre><code><span>
</span><span>The</span><span> </span><span>term</span><span> </span><span>planet</span><span> </span><span>is</span><span> </span><span>ancient</span><span>,</span><span> </span><span>with</span><span> </span><span>ties</span><span> </span><span>to</span><span> </span><span>history</span><span>,</span><span> </span><span>astrology</span><span>,</span><span> </span><span>science</span><span>,</span><span> </span><span>mythology</span><span>,</span><span> </span><span>and</span><span> </span><span>religion.</span><span> </span><span>Several</span><span> </span><span>planets</span><span> </span><span>in</span><span> </span><span>the</span><span> </span><span>Solar</span><span> </span><span>System</span><span> </span><span>can</span><span> </span><span>be</span><span> </span><span>seen</span><span> </span><span>with</span><span> </span><span>the</span><span> </span><span>naked</span><span> </span><span>eye.</span><span> </span><span>These</span><span> </span><span>were</span><span> </span><span>regarded</span><span> </span><span>by</span><span> </span><span>many</span><span> </span><span>early</span><span> </span><span>cultures</span><span> </span><span>as</span><span> </span><span>divine</span><span>,</span><span> </span><span>or</span><span> </span><span>as</span><span> </span><span>emissaries</span><span> </span><span>of</span><span> </span><span>deities.</span><span> </span><span>As</span><span> </span><span>scientific</span><span> </span><span>knowledge</span><span> </span><span>advanced</span><span>,</span><span> </span><span>human</span><span> </span><span>perception</span><span> </span><span>of</span><span> </span><span>the</span><span> </span><span>planets</span><span> </span><span>changed</span><span>,</span><span> </span><span>incorporating</span><span> </span><span>a</span><span> </span><span>number</span><span> </span><span>of</span><span> </span><span>disparate</span><span> </span><span>objects.</span><span> </span><span>In</span><span> </span><span>2006</span><span>,</span><span> </span><span>the</span><span> </span><span>International</span><span> </span><span>Astronomical</span><span> </span><span>Union</span><span> </span><span>(</span><span>IAU</span><span>)</span><span> </span><span>officially</span><span> </span><span>adopted</span><span> </span><span>a</span><span> </span><span>resolution</span><span> </span><span>defining</span><span> </span><span>planets</span><span> </span><span>within</span><span> </span><span>the</span><span> </span><span>Solar</span><span> </span><span>System.</span><span> </span><span>This</span><span> </span><span>definition</span><span> </span><span>is</span><span> </span><span>controversial</span><span> </span><span>because</span><span> </span><span>it</span><span> </span><span>excludes</span><span> </span><span>many</span><span> </span><span>objects</span><span> </span><span>of</span><span> </span><span>planetary</span><span> </span><span>mass</span><span> </span><span>based</span><span> </span><span>on</span><span> </span><span>where</span><span> </span><span>or</span><span> </span><span>what</span><span> </span><span>they</span><span> </span><span>orbit.</span><span> </span><span>Although</span><span> </span><span>eight</span><span> </span><span>of</span><span> </span><span>the</span><span> </span><span>planetary</span><span> </span><span>bodies</span><span> </span><span>discovered</span><span> </span><span>before</span><span> </span><span>1950</span><span> </span><span>remain</span><span> </span><span>planets</span><span> </span><span>under</span><span> </span><span>the</span><span> </span><span>modern</span><span> </span><span>definition</span><span>,</span><span> </span><span>some</span><span> </span><span>celestial</span><span> </span><span>bodies</span><span>,</span><span> </span><span>such</span><span> </span><span>as</span><span> </span><span>Ceres</span><span>,</span><span> </span><span>Pallas</span><span>,</span><span> </span><span>Juno</span><span> </span><span>and</span><span> </span><span>Vesta</span><span> </span><span>(</span><span>each</span><span> </span><span>an</span><span> </span><span>object</span><span> </span><span>in</span><span> </span><span>the</span><span> </span><span>solar</span><span> </span><span>asteroid</span><span> </span><span>belt</span><span>),</span><span> </span><span>and</span><span> </span><span>Pluto</span><span> </span><span>(</span><span>the</span><span> </span><span>first</span><span> </span><span>trans</span><span>-</span><span>Neptunian</span><span> </span><span>object</span><span> </span><span>discovered</span><span>),</span><span> </span><span>that</span><span> </span><span>were</span><span> </span><span>once</span><span> </span><span>considered</span><span> </span><span>planets</span><span> </span><span>by</span><span> </span><span>the</span><span> </span><span>scientific</span><span> </span><span>community</span><span>,</span><span> </span><span>are</span><span> </span><span>no</span><span> </span><span>longer</span><span> </span><span>viewed</span><span> </span><span>as</span><span> </span><span>such.</span><span>
</span><span>The</span><span> </span><span>planets</span><span> </span><span>were</span><span> </span><span>thought</span><span> </span><span>by</span><span> </span><span>Ptolemy</span><span> </span><span>to</span><span> </span><span>orbit</span><span> </span><span>Earth</span><span> </span><span>in</span><span> </span><span>deferent</span><span> </span><span>and</span><span> </span><span>epicycle</span><span> </span><span>motions.</span><span> </span><span>Although</span><span> </span><span>the</span><span> </span><span>idea</span><span> </span><span>that</span><span> </span><span>the</span><span> </span><span>planets</span><span> </span><span>orbited</span><span> </span><span>the</span><span> </span><span>Sun</span><span> </span><span>had</span><span> </span><span>been</span><span> </span><span>suggested</span><span> </span><span>many</span><span> </span><span>times</span><span>,</span><span> </span><span>it</span><span> </span><span>was</span><span> </span><span>not</span><span> </span><span>until</span><span> </span><span>the</span><span> </span><span>17</span><span>th</span><span> </span><span>century</span><span> </span><span>that</span><span> </span><span>this</span><span> </span><span>view</span><span> </span><span>was</span><span> </span><span>supported</span><span> </span><span>by</span><span> </span><span>evidence</span><span> </span><span>from</span><span> </span><span>the</span><span> </span><span>first</span><span> </span><span>telescopic</span><span> </span><span>astronomical</span><span> </span><span>observations</span><span>,</span><span> </span><span>performed</span><span> </span><span>by</span><span> </span><span>Galileo</span><span> </span><span>Galilei.</span><span> </span><span>At</span><span> </span><span>about</span><span> </span><span>the</span><span> </span><span>same</span><span> </span><span>time</span><span>,</span><span> </span><span>by</span><span> </span><span>careful</span><span> </span><span>analysis</span><span> </span><span>of</span><span> </span><span>pre</span><span>-</span><span>telescopic</span><span> </span><span>observation</span><span> </span><span>data</span><span> </span><span>collected</span><span> </span><span>by</span><span> </span><span>Tycho</span><span> </span><span>Brahe</span><span>,</span><span> </span><span>Johannes</span><span> </span><span>Kepler</span><span> </span><span>found</span><span> </span><span>the</span><span> </span><span>planets</span><span> </span><span>orbits</span><span> </span><span>were</span><span> </span><span>not</span><span> </span><span>circular</span><span> </span><span>but</span><span> </span><span>elliptical.</span><span> </span><span>As</span><span> </span><span>observational</span><span> </span><span>tools</span><span> </span><span>improved</span><span>,</span><span> </span><span>astronomers</span><span> </span><span>saw</span><span> </span><span>that</span><span>,</span><span> </span><span>like</span><span> </span><span>Earth</span><span>,</span><span> </span><span>the</span><span> </span><span>planets</span><span> </span><span>rotated</span><span> </span><span>around</span><span> </span><span>tilted</span><span> </span><span>axes</span><span>,</span><span> </span><span>and</span><span> </span><span>some</span><span> </span><span>shared</span><span> </span><span>such</span><span> </span><span>features</span><span> </span><span>as</span><span> </span><span>ice</span><span> </span><span>caps</span><span> </span><span>and</span><span> </span><span>seasons.</span><span> </span><span>Since</span><span> </span><span>the</span><span> </span><span>dawn</span><span> </span><span>of</span><span> </span><span>the</span><span> </span><span>Space</span><span> </span><span>Age</span><span>,</span><span> </span><span>close</span><span> </span><span>observation</span><span> </span><span>by</span><span> </span><span>space</span><span> </span><span>probes</span><span> </span><span>has</span><span> </span><span>found</span><span> </span><span>that</span><span> </span><span>Earth</span><span> </span><span>and</span><span> </span><span>the</span><span> </span><span>other</span><span> </span><span>planets</span><span> </span><span>share</span><span> </span><span>characteristics</span><span> </span><span>such</span><span> </span><span>as</span><span> </span><span>volcanism</span><span>,</span><span> </span><span>hurricanes</span><span>,</span><span> </span><span>tectonics</span><span>,</span><span> </span><span>and</span><span> </span><span>even</span><span> </span><span>hydrology.</span><span>
</span><span>Planets</span><span> </span><span>are</span><span> </span><span>generally</span><span> </span><span>divided</span><span> </span><span>into</span><span> </span><span>two</span><span> </span><span>main</span><span> </span><span>types</span><span>:</span><span> </span><span>large</span><span> </span><span>low</span><span>-</span><span>density</span><span> </span><span>giant</span><span> </span><span>planets</span><span>,</span><span> </span><span>and</span><span> </span><span>smaller</span><span> </span><span>rocky</span><span> </span><span>terrestrials.</span><span> </span><span>Under</span><span> </span><span>IAU</span><span> </span><span>definitions</span><span>,</span><span> </span><span>there</span><span> </span><span>are</span><span> </span><span>eight</span><span> </span><span>planets</span><span> </span><span>in</span><span> </span><span>the</span><span> </span><span>Solar</span><span> </span><span>System.</span><span> </span><span>In</span><span> </span><span>order</span><span> </span><span>of</span><span> </span><span>increasing</span><span> </span><span>distance</span><span> </span><span>from</span><span> </span><span>the</span><span> </span><span>Sun</span><span>,</span><span> </span><span>they</span><span> </span><span>are</span><span> </span><span>the</span><span> </span><span>four</span><span> </span><span>terrestrials</span><span>,</span><span> </span><span>Mercury</span><span>,</span><span> </span><span>Venus</span><span>,</span><span> </span><span>Earth</span><span>,</span><span> </span><span>and</span><span> </span><span>Mars</span><span>,</span><span> </span><span>then</span><span> </span><span>the</span><span> </span><span>four</span><span> </span><span>giant</span><span> </span><span>planets</span><span>,</span><span> </span><span>Jupiter</span><span>,</span><span> </span><span>Saturn</span><span>,</span><span> </span><span>Uranus</span><span>,</span><span> </span><span>and</span><span> </span><span>Neptune.</span><span> </span><span>Six</span><span> </span><span>of</span><span> </span><span>the</span><span> </span><span>planets</span><span> </span><span>are</span><span> </span><span>orbited</span><span> </span><span>by</span><span> </span><span>one</span><span> </span><span>or</span><span> </span><span>more</span><span> </span><span>natural</span><span> </span><span>satellites.</span><span>
</span></code></pre>
</div>
<p></p>
<p>The <em>sparse_term_matrix</em> class can be initialized using either a vector of documents or a text file. Assuming the downloaded file is saved as “planets.txt”, then a document-term-matrix can be created in the following way,</p>
<p></p>
<div>
<pre><code><span>
</span><span>init</span><span> </span><span>=</span><span> </span><span>sparse_term_matrix</span><span>$</span><span>new</span><span>(</span><span>vector_data</span><span> </span><span>=</span><span> </span><span>NULL</span><span>,</span><span> </span><span># in case of vector of documents
</span><span> </span><span>file_data</span><span> </span><span>=</span><span> </span><span>&quot;/planets.txt&quot;</span><span>,</span><span> </span><span># input the .txt data
</span><span> </span><span>document_term_matrix</span><span> </span><span>=</span><span> </span><span>TRUE</span><span>)</span><span> </span><span># document term matrix as output
</span><span>
</span><span>tm</span><span> </span><span>=</span><span> </span><span>init</span><span>$</span><span>Term_Matrix</span><span>(</span><span>sort_columns</span><span> </span><span>=</span><span> </span><span>TRUE</span><span>,</span><span> </span><span># initial terms are sorted
</span><span> </span><span>to_lower</span><span> </span><span>=</span><span> </span><span>TRUE</span><span>,</span><span> </span><span># convert to lower case
</span><span> </span><span>trim_token</span><span> </span><span>=</span><span> </span><span>TRUE</span><span>,</span><span> </span><span># trim token
</span><span> </span><span>split_string</span><span> </span><span>=</span><span> </span><span>TRUE</span><span>,</span><span> </span><span># split string
</span><span> </span><span>tf_idf</span><span> </span><span>=</span><span> </span><span>TRUE</span><span>,</span><span> </span><span># tf-idf will be returned
</span><span> </span><span>verbose</span><span> </span><span>=</span><span> </span><span>TRUE</span><span>)</span><span>
</span></code></pre>
</div>
<div>
<pre><code><span>
</span><span>minutes.to.tokenize.transform.data</span><span>:</span><span> </span><span>0.00001</span><span>	</span><span>total.time</span><span>:</span><span> </span><span>0.00001</span><span>
</span><span>Warning</span><span> </span><span>message</span><span>:</span><span>
</span><span>empty</span><span> </span><span>character</span><span> </span><span>strings</span><span> </span><span>present</span><span> </span><span>in</span><span> </span><span>the</span><span> </span><span>column</span><span> </span><span>names</span><span> </span><span>they</span><span> </span><span>will</span><span> </span><span>be</span><span> </span><span>replaced</span><span> </span><span>with</span><span> </span><span>proper</span><span> </span><span>characters</span><span> </span><span>5</span><span> </span><span>x</span><span> </span><span>214</span><span> </span><span>sparse</span><span> </span><span>Matrix</span><span> </span><span>of</span><span> </span><span>class</span><span> </span><span>&quot;dgCMatrix&quot;</span><span>
</span><span>[[</span><span> </span><span>suppressing</span><span> </span><span>214</span><span> </span><span>column</span><span> </span><span>names</span><span> </span><span>'X'</span><span>,</span><span> </span><span>'X.planets.'</span><span>,</span><span> </span><span>'X17th'</span><span> </span><span>...</span><span> </span><span>]]</span><span>
</span><span>[</span><span>1</span><span>,]</span><span> </span><span>-0.001939591</span><span> </span><span>.</span><span> </span><span>.</span><span> </span><span>.</span><span> </span><span>0.00974777</span><span> </span><span>0.0194955</span><span> </span><span>.</span><span> </span><span>.</span><span> </span><span>.</span><span> </span><span>.</span><span> </span><span>.</span><span> </span><span>[</span><span>2</span><span>,]</span><span> </span><span>-0.003255742</span><span> </span><span>0.0163623</span><span> </span><span>.</span><span> </span><span>0.0163623</span><span> </span><span>.</span><span> </span><span>.</span><span> </span><span>.</span><span> </span><span>.</span><span> </span><span>.</span><span> </span><span>.</span><span> </span><span>.</span><span> </span><span>[</span><span>3</span><span>,]</span><span> </span><span>-0.003440029</span><span> </span><span>.</span><span> </span><span>0.017288</span><span> </span><span>.</span><span> </span><span>.</span><span> </span><span>.</span><span> </span><span>.</span><span> </span><span>.</span><span> </span><span>.</span><span> </span><span>.</span><span> </span><span>.</span><span> </span><span>[</span><span>4</span><span>,]</span><span> </span><span>-0.002196645</span><span> </span><span>.</span><span> </span><span>.</span><span> </span><span>.</span><span> </span><span>.</span><span> </span><span>.</span><span> </span><span>.</span><span> </span><span>.</span><span> </span><span>.</span><span> </span><span>.</span><span> </span><span>.</span><span> </span><span>[</span><span>5</span><span>,]</span><span> </span><span>-0.002681199</span><span> </span><span>.</span><span> </span><span>.</span><span> </span><span>.</span><span> </span><span>.</span><span> </span><span>.</span><span> </span><span>.</span><span> </span><span>.</span><span> </span><span>.</span><span> </span><span>.</span><span> </span><span>.</span><span> </span></code></pre>
</div>
<p></p>
<p>The <em>Term_Matrix</em> method takes almost the same parameters as the ( already explained ) <em>big_text_tokenizer()</em> method. The only differences are:</p>
<ul>
<li><strong>sort_terms</strong> ( should the output terms – rows or columns depending on the <em>document_term_matrix</em> parameter – be sorted in alphabetical order )</li>
<li><strong>print_every_rows</strong> ( verbose output intervalls )</li>
<li><strong>normalize</strong> ( applies <em>l1</em> or <em>l2</em> normalization )</li>
<li><strong>tf_idf</strong> ( the term-frequency-inverse-document-frequency will be returned )</li>
</ul>
<p>Details about the parameters of the <em>Term_Matrix</em> method can be found in the package documentation. </p>
<p></p>
<p>To adjust the sparsity of the output matrix one can take advantage of the <em>Term_Matrix_Adjust</em> method, (by adjusting the sparsity_thresh towards 0.0 a proportion of the sparse terms will be removed)</p>
<p></p>
<div>
<pre><code><span>
</span><span>init</span><span>$</span><span>Term_Matrix_Adjust</span><span>(</span><span>sparsity_thresh</span><span> </span><span>=</span><span> </span><span>0.995</span><span>)</span><span> </span></code></pre>
</div>
<div>
<pre><code><span>
</span><span>5</span><span> </span><span>x</span><span> </span><span>9</span><span> </span><span>sparse</span><span> </span><span>Matrix</span><span> </span><span>of</span><span> </span><span>class</span><span> </span><span>&quot;dgCMatrix&quot;</span><span>
</span><span>are</span><span> </span><span>bodies</span><span> </span><span>discovered</span><span> </span><span>four</span><span> </span><span>giant</span><span> </span><span>object</span><span> </span><span>such</span><span> </span><span>terrestrials</span><span> </span><span>was</span><span>
</span><span>[</span><span>1</span><span>,]</span><span> </span><span>.</span><span> </span><span>.</span><span> </span><span>.</span><span> </span><span>.</span><span> </span><span>.</span><span> </span><span>.</span><span> </span><span>.</span><span> </span><span>.</span><span> </span><span>.</span><span> </span><span>[</span><span>2</span><span>,]</span><span> </span><span>0.009121886</span><span> </span><span>0.0327246</span><span> </span><span>0.0327246</span><span> </span><span>.</span><span> </span><span>.</span><span> </span><span>0.03272467</span><span> </span><span>0.01824377</span><span> </span><span>.</span><span> </span><span>.</span><span> </span><span>[</span><span>3</span><span>,]</span><span> </span><span>.</span><span> </span><span>.</span><span> </span><span>.</span><span> </span><span>.</span><span> </span><span>.</span><span> </span><span>.</span><span> </span><span>.</span><span> </span><span>.</span><span> </span><span>0.034577</span><span>
</span><span>[</span><span>4</span><span>,]</span><span> </span><span>.</span><span> </span><span>.</span><span> </span><span>.</span><span> </span><span>.</span><span> </span><span>.</span><span> </span><span>.</span><span> </span><span>0.01230905</span><span> </span><span>.</span><span> </span><span>.</span><span> </span><span>[</span><span>5</span><span>,]</span><span> </span><span>0.030048566</span><span> </span><span>.</span><span> </span><span>.</span><span> </span><span>0.026949</span><span> </span><span>0.02694973</span><span> </span><span>.</span><span> </span><span>.</span><span> </span><span>0.0269497</span><span> </span><span>.</span><span>
</span></code></pre>
</div>
<p></p>
<p>The <em>term_associations</em> method returns the correlation of specific terms (<em>Terms</em>) with all the other terms in the output matrix. In case that one of the terms is not present in the corpus then a warning will be printed in the console,</p>
<p></p>
<div>
<pre><code><span>
</span><span>init</span><span>$</span><span>term_associations</span><span>(</span><span>Terms</span><span> </span><span>=</span><span> </span><span>c</span><span>(</span><span>'giant'</span><span>,</span><span> </span><span>'terrestrials'</span><span>,</span><span> </span><span>'INVALID'</span><span>),</span><span> </span><span>keep_terms</span><span> </span><span>=</span><span> </span><span>NULL</span><span>,</span><span> </span><span>verbose</span><span> </span><span>=</span><span> </span><span>TRUE</span><span>)</span><span>
</span></code></pre>
</div>
<div>
<pre><code><span>
</span><span>the</span><span> </span><span>' INVALID '</span><span> </span><span>term</span><span> </span><span>does</span><span> </span><span>not</span><span> </span><span>exist</span><span> </span><span>in</span><span> </span><span>the</span><span> </span><span>terms</span><span> </span><span>vector</span><span> </span><span>total.number.variables.processed</span><span>:</span><span> </span><span>2</span><span>	</span><span>minutes.to.complete</span><span>:</span><span> </span><span>0.00002</span><span>
</span><span>$</span><span>giant</span><span>
</span><span>term</span><span> </span><span>correlation</span><span>
</span><span>1</span><span>:</span><span> </span><span>then</span><span> </span><span>1.0000000</span><span>
</span><span>2</span><span>:</span><span> </span><span>two</span><span> </span><span>1.0000000</span><span>
</span><span>3</span><span>:</span><span> </span><span>divided</span><span> </span><span>1.0000000</span><span>
</span><span>4</span><span>:</span><span> </span><span>distance</span><span> </span><span>1.0000000</span><span>
</span><span>5</span><span>:</span><span> </span><span>rocky</span><span> </span><span>1.0000000</span><span>
</span><span>---</span><span> </span><span>209</span><span>:</span><span> </span><span>to</span><span> </span><span>-0.4082483</span><span>
</span><span>210</span><span>:</span><span> </span><span>that</span><span> </span><span>-0.5590170</span><span>
</span><span>211</span><span>:</span><span> </span><span>as</span><span> </span><span>-0.5897678</span><span>
</span><span>212</span><span>:</span><span> </span><span>the</span><span> </span><span>-0.6123724</span><span>
</span><span>213</span><span>:</span><span> </span><span>were</span><span> </span><span>-1.0000000</span><span>
</span><span>$</span><span>terrestrials</span><span>
</span><span>term</span><span> </span><span>correlation</span><span>
</span><span>1</span><span>:</span><span> </span><span>there</span><span> </span><span>1.0000000</span><span>
</span><span>2</span><span>:</span><span> </span><span>jupiter</span><span> </span><span>1.0000000</span><span>
</span><span>3</span><span>:</span><span> </span><span>two</span><span> </span><span>1.0000000</span><span>
</span><span>4</span><span>:</span><span> </span><span>neptune</span><span> </span><span>1.0000000</span><span>
</span><span>5</span><span>:</span><span> </span><span>rocky</span><span> </span><span>1.0000000</span><span>
</span><span>---</span><span> </span><span>209</span><span>:</span><span> </span><span>astronomical</span><span> </span><span>-0.4082483</span><span>
</span><span>210</span><span>:</span><span> </span><span>that</span><span> </span><span>-0.5590170</span><span>
</span><span>211</span><span>:</span><span> </span><span>as</span><span> </span><span>-0.5897678</span><span>
</span><span>212</span><span>:</span><span> </span><span>the</span><span> </span><span>-0.6123724</span><span>
</span><span>213</span><span>:</span><span> </span><span>were</span><span> </span><span>-1.0000000</span><span>
</span></code></pre>
</div>
<p></p>
<p>Lastly, the <em>most_frequent_terms</em> method gives the frequency of the terms in the corpus. However, the function returns only if the <em>normalize</em> parameter is NULL and the <em>tf_idf</em> parameter is FALSE ( the latter two parameters belong to the <em>init$Term_Matrix()</em> method ),</p>
<p></p>
<div>
<pre><code><span>
</span><span>init</span><span>$</span><span>most_frequent_terms</span><span>(</span><span>keep_terms</span><span> </span><span>=</span><span> </span><span>10</span><span>,</span><span> </span><span>threads</span><span> </span><span>=</span><span> </span><span>1</span><span>,</span><span> </span><span>verbose</span><span> </span><span>=</span><span> </span><span>TRUE</span><span>)</span><span>
</span><span>minutes.to.complete</span><span>:</span><span> </span><span>0.00000</span><span>
</span><span>term</span><span> </span><span>frequency</span><span>
</span><span>1</span><span>:</span><span> </span><span>the</span><span> </span><span>28</span><span>
</span><span>2</span><span>:</span><span> </span><span>planets</span><span> </span><span>13</span><span>
</span><span>3</span><span>:</span><span> </span><span>and</span><span> </span><span>11</span><span>
</span><span>4</span><span>:</span><span> </span><span>by</span><span> </span><span>9</span><span>
</span><span>5</span><span>:</span><span> </span><span>of</span><span> </span><span>9</span><span>
</span><span>6</span><span>:</span><span> </span><span>as</span><span> </span><span>8</span><span>
</span><span>7</span><span>:</span><span> </span><span>in.</span><span> </span><span>6</span><span>
</span><span>8</span><span>:</span><span> </span><span>X</span><span> </span><span>5</span><span>
</span><span>9</span><span>:</span><span> </span><span>that</span><span> </span><span>5</span><span>
</span><span>10</span><span>:</span><span> </span><span>are</span><span> </span><span>5</span><span>
</span></code></pre>
</div>
<p></p>
<p>More information about the <em>sparse_term_matrix</em> class can be found in the package documentation.</p>
<p></p>
<h3><em>token_stats</em> class</h3>
<p></p>
<p>The <em>token_stats</em> class can be utilized to output corpus statistics. Each of the following methods can take either a <em>vector of terms</em>, a <em>text file</em> or <em>a folder of files</em> as input:</p>
<p></p>
<ul>
<li><strong>path_2vector</strong> : is a helper method which takes a path to a file or folder of files and returns the content in form of a vector, </li>
</ul>
<div>
<pre><code><span>
</span><span>init</span><span> </span><span>=</span><span> </span><span>token_stats</span><span>$</span><span>new</span><span>(</span><span>path_2file</span><span> </span><span>=</span><span> </span><span>&quot;/planets.txt&quot;</span><span>)</span><span> </span><span># input the 'planets.txt' file
</span><span>
</span><span>vec</span><span> </span><span>=</span><span> </span><span>init</span><span>$</span><span>path_2vector</span><span>()</span><span>
</span></code></pre>
</div>
<div>
<pre><code><span>
</span><span>[</span><span>1</span><span>]</span><span> </span><span>&quot;The term planet is ancient, with ties to history, astrology, science, mythology, and religion. Several planets in the Solar System can be seen with the naked eye. These were regarded by many early cultures as divine, or as emissaries of deities. As scientific knowledge advanced, human perception of the planets changed, incorporating a number of disparate objects&quot;</span><span> </span><span>....</span><span>
</span><span>[</span><span>2</span><span>]</span><span> </span><span>&quot;Although eight of the planetary bodies discovered before 1950 remain&quot;</span><span> </span><span>....</span><span>
</span><span>.</span><span>
</span><span>.</span><span>
</span></code></pre>
</div>
<p></p>
<ul>
<li><strong>freq_distribution</strong> : it returns a named-unsorted vector frequency distribution for a vocabulary file</li>
</ul>
<div>
<pre><code><span>
</span><span># assuming the following 'freq_vocab.txt'
</span><span>
</span><span>the</span><span>
</span><span>term</span><span>
</span><span>planet</span><span>
</span><span>is</span><span>
</span><span>ancient</span><span>
</span><span>with</span><span>
</span><span>ties</span><span>
</span><span>to</span><span>
</span><span>history</span><span>
</span><span>astrology</span><span>
</span><span>science</span><span>
</span><span>mythology</span><span>
</span><span>and</span><span>
</span><span>religion</span><span>
</span><span>several</span><span>
</span><span>planets</span><span>
</span><span>in</span><span>
</span><span>the</span><span>
</span><span>solar</span><span>
</span><span>system</span><span>
</span><span>can</span><span>
</span><span>be</span><span>
</span><span>seen</span><span>
</span><span>with</span><span>
</span><span>the</span><span>
</span><span>naked</span><span>
</span><span>eye</span><span>
</span><span>these</span><span>
</span><span>were</span><span>
</span></code></pre>
</div>
<p></p>
<p>this method would return,</p>
<p></p>
<div>
<pre><code><span>
</span><span>init</span><span> </span><span>=</span><span> </span><span>token_stats</span><span>$</span><span>new</span><span>(</span><span>path_2file</span><span> </span><span>=</span><span> </span><span>'freq_vocab.txt'</span><span>)</span><span>
</span><span>init</span><span>$</span><span>freq_distribution</span><span>()</span><span>
</span><span>init</span><span>$</span><span>print_frequency</span><span>(</span><span>subset</span><span> </span><span>=</span><span> </span><span>NULL</span><span>)</span><span>
</span></code></pre>
</div>
<div>
<pre><code><span> </span><span>words</span><span> </span><span>freq</span><span>
</span><span>1</span><span>:</span><span> </span><span>the</span><span> </span><span>3</span><span>
</span><span>2</span><span>:</span><span> </span><span>with</span><span> </span><span>2</span><span>
</span><span>3</span><span>:</span><span> </span><span>ancient</span><span> </span><span>1</span><span>
</span><span>4</span><span>:</span><span> </span><span>and</span><span> </span><span>1</span><span>
</span><span>5</span><span>:</span><span> </span><span>astrology</span><span> </span><span>1</span><span>
</span><span>6</span><span>:</span><span> </span><span>be</span><span> </span><span>1</span><span>
</span><span>7</span><span>:</span><span> </span><span>can</span><span> </span><span>1</span><span>
</span><span>8</span><span>:</span><span> </span><span>eye</span><span> </span><span>1</span><span>
</span><span>9</span><span>:</span><span> </span><span>history</span><span> </span><span>1</span><span>
</span><span>10</span><span>:</span><span> </span><span>in</span><span> </span><span>1</span><span>
</span><span>11</span><span>:</span><span> </span><span>is</span><span> </span><span>1</span><span>
</span><span>12</span><span>:</span><span> </span><span>mythology</span><span> </span><span>1</span><span>
</span><span>13</span><span>:</span><span> </span><span>naked</span><span> </span><span>1</span><span>
</span><span>14</span><span>:</span><span> </span><span>planet</span><span> </span><span>1</span><span>
</span><span>15</span><span>:</span><span> </span><span>planets</span><span> </span><span>1</span><span>
</span><span>16</span><span>:</span><span> </span><span>religion</span><span> </span><span>1</span><span>
</span><span>17</span><span>:</span><span> </span><span>science</span><span> </span><span>1</span><span>
</span><span>18</span><span>:</span><span> </span><span>seen</span><span> </span><span>1</span><span>
</span><span>19</span><span>:</span><span> </span><span>several</span><span> </span><span>1</span><span>
</span><span>20</span><span>:</span><span> </span><span>solar</span><span> </span><span>1</span><span>
</span><span>21</span><span>:</span><span> </span><span>system</span><span> </span><span>1</span><span>
</span><span>22</span><span>:</span><span> </span><span>term</span><span> </span><span>1</span><span>
</span><span>23</span><span>:</span><span> </span><span>these</span><span> </span><span>1</span><span>
</span><span>24</span><span>:</span><span> </span><span>ties</span><span> </span><span>1</span><span>
</span><span>25</span><span>:</span><span> </span><span>to</span><span> </span><span>1</span><span>
</span><span>26</span><span>:</span><span> </span><span>were</span><span> </span><span>1</span><span>
</span><span>words</span><span> </span><span>freq</span><span>
</span></code></pre>
</div>
<p></p>
<ul>
<li><strong>count_character</strong> : it returns the number of characters for each word of the corpus.</li>
</ul>
<p></p>
<p>for the previously mentioned <em>‘freq_vocab.txt’</em> it would output,</p>
<p></p>
<div>
<pre><code><span>
</span><span>init</span><span> </span><span>=</span><span> </span><span>token_stats</span><span>$</span><span>new</span><span>(</span><span>path_2file</span><span> </span><span>=</span><span> </span><span>'freq_vocab.txt'</span><span>)</span><span>
</span><span>vec_tmp</span><span> </span><span>=</span><span> </span><span>init</span><span>$</span><span>count_character</span><span>()</span><span>
</span><span>init</span><span>$</span><span>print_count_character</span><span>(</span><span>number</span><span> </span><span>=</span><span> </span><span>3</span><span>)</span><span>
</span></code></pre>
</div>
<div>
<pre><code><span>
</span><span># words with number of characters equal to 3
</span><span>
</span><span>[</span><span>1</span><span>]</span><span> </span><span>&quot;the&quot;</span><span> </span><span>&quot;and&quot;</span><span> </span><span>&quot;the&quot;</span><span> </span><span>&quot;can&quot;</span><span> </span><span>&quot;the&quot;</span><span> </span><span>&quot;eye&quot;</span><span>
</span></code></pre>
</div>
<p></p>
<ul>
<li><strong>collocation_words</strong> : it returns a co-occurence frequency table for n-grams. “A <a target="_blank" rel="nofollow" href="http://nlp.stanford.edu/fsnlp/promo/colloc.pdf">collocation</a> is defined as a sequence of two or more consecutive words, that has characteristics of a syntactic and semantic unit, and whose exact and unambiguous meaning or connotation cannot be derived directly from the meaning or connotation of its components”. The input to the function should be text n-grams separated by a delimiter ( for instance the <em>tokenize_transform_text()</em> function in the next code chunk will build n_grams of length 3 ),</li>
</ul>
<p></p>
<div>
<pre><code><span>
</span><span># the data needs to be n-grams thus first tokenize and build the n-grams using # the 'tokenize_transform_text' function ( the &quot;planets.txt&quot; file as input )
</span><span>
</span><span>tok</span><span> </span><span>=</span><span> </span><span>tokenize_transform_text</span><span>(</span><span>&quot;planets.txt&quot;</span><span>,</span><span> </span><span>to_lower</span><span> </span><span>=</span><span> </span><span>T</span><span>,</span><span> </span><span>split_string</span><span> </span><span>=</span><span> </span><span>T</span><span>,</span><span>
</span><span>min_n_gram</span><span> </span><span>=</span><span> </span><span>3</span><span>,</span><span> </span><span>max_n_gram</span><span> </span><span>=</span><span> </span><span>3</span><span>,</span><span> </span><span>n_gram_delimiter</span><span> </span><span>=</span><span> </span><span>&quot;_&quot;</span><span>)</span><span>
</span><span>init</span><span> </span><span>=</span><span> </span><span>token_stats</span><span>$</span><span>new</span><span>(</span><span>x_vec</span><span> </span><span>=</span><span> </span><span>tok</span><span>$</span><span>token</span><span>)</span><span> </span><span># vector data as input
</span><span>
</span><span>vec_tmp</span><span> </span><span>=</span><span> </span><span>init</span><span>$</span><span>collocation_words</span><span>()</span><span>
</span></code></pre>
</div>
<p></p>
<p>the example output of the <em>vec_tmp</em> vector is,</p>
<p></p>
<div>
<pre><code><span>
</span><span>[</span><span>1</span><span>]</span><span> </span><span>&quot;&quot;</span><span> </span><span>&quot;17th&quot;</span><span> </span><span>&quot;1950&quot;</span><span> </span><span>&quot;2006&quot;</span><span> </span><span>&quot;a&quot;</span><span> </span><span>&quot;about&quot;</span><span> </span><span>&quot;adopted&quot;</span><span> </span><span>&quot;advanced&quot;</span><span> </span><span>[</span><span>9</span><span>]</span><span> </span><span>&quot;age&quot;</span><span> </span><span>&quot;although&quot;</span><span> </span><span>....</span><span> </span><span>.</span><span>
</span><span>.</span><span>
</span><span>.</span><span>
</span></code></pre>
</div>
<p></p>
<p>and the <em>print_collocations</em> method returns the coolocations for the example word <em>ancient</em>,</p>
<p></p>
<div>
<pre><code><span>
</span><span>res</span><span> </span><span>=</span><span> </span><span>init</span><span>$</span><span>print_collocations</span><span>(</span><span>word</span><span> </span><span>=</span><span> </span><span>&quot;ancient&quot;</span><span>)</span><span>
</span></code></pre>
</div>
<div>
<pre><code><span>
</span><span>is</span><span> </span><span>with</span><span> </span><span>ties</span><span> </span><span>planet</span><span> </span><span>0.333</span><span> </span><span>0.333</span><span> </span><span>0.167</span><span> </span><span>0.167</span><span> </span></code></pre>
</div>
<p></p>
<ul>
<li><strong>string_dissimilarity_matrix</strong> : it returns a string-dissimilarity-matrix using either the <em>dice</em>, <em>levenshtein</em> or <em>cosine distance</em>. The input can be a <em>character string vector</em> only. In case that the method is dice then the dice-coefficient (similarity) is calculated between two strings for a specific number of character n-grams ( dice_n_gram ). The <em>dice</em> and <em>levenshtein</em> methods are applied to words, whereas the <em>cosine</em> distance to word-sentences.</li>
</ul>
<p></p>
<p>For illustration purposes I’ll use the previously mentioned <em>‘freq_vocab.txt’</em> file, but first I have to convert the text file to a character vector,</p>
<p></p>
<div>
<pre><code><span>
</span><span># first initialization of token_stats </span><span>
</span><span>init</span><span> </span><span>=</span><span> </span><span>token_stats</span><span>$</span><span>new</span><span>(</span><span>path_2file</span><span> </span><span>=</span><span> </span><span>'freq_vocab.txt'</span><span>)</span><span>
</span><span>tmp_vec</span><span> </span><span>=</span><span> </span><span>init</span><span>$</span><span>path_2vector</span><span>()</span><span> </span><span># convert to vector
</span><span>
</span><span># second initialization to compute the dissimilarity matrix </span><span>
</span><span>init_tok</span><span> </span><span>=</span><span> </span><span>token_stats</span><span>$</span><span>new</span><span>(</span><span>x_vec</span><span> </span><span>=</span><span> </span><span>tmp_vec</span><span>)</span><span>
</span><span>res</span><span> </span><span>=</span><span> </span><span>init_tok</span><span>$</span><span>string_dissimilarity_matrix</span><span>(</span><span>dice_n_gram</span><span> </span><span>=</span><span> </span><span>2</span><span>,</span><span> </span><span>method</span><span> </span><span>=</span><span> </span><span>&quot;dice&quot;</span><span>,</span><span> </span><span>split_separator</span><span> </span><span>=</span><span> </span><span>&quot; &quot;</span><span>,</span><span> </span><span>dice_thresh</span><span> </span><span>=</span><span> </span><span>1.0</span><span>,</span><span> </span><span>upper</span><span> </span><span>=</span><span> </span><span>TRUE</span><span>,</span><span> </span><span>diagonal</span><span> </span><span>=</span><span> </span><span>TRUE</span><span>,</span><span> </span><span>threads</span><span> </span><span>=</span><span> </span><span>1</span><span>)</span><span>
</span></code></pre>
</div>
<div>
<pre><code><span>
</span><span>the</span><span> </span><span>term</span><span> </span><span>planet</span><span> </span><span>is</span><span> </span><span>ancient</span><span> </span><span>with</span><span> </span><span>ties</span><span> </span><span>to</span><span> </span><span>history</span><span> </span><span>....</span><span>
</span><span>the</span><span> </span><span>0.0000000</span><span> </span><span>0.6000000</span><span> </span><span>1.00000000</span><span> </span><span>1</span><span> </span><span>1.0000000</span><span> </span><span>1.0000000</span><span> </span><span>0.6000000</span><span> </span><span>1</span><span> </span><span>1.0000000</span><span> </span><span>....</span><span>
</span><span>term</span><span> </span><span>0.6000000</span><span> </span><span>0.0000000</span><span> </span><span>1.00000000</span><span> </span><span>1</span><span> </span><span>1.0000000</span><span> </span><span>1.0000000</span><span> </span><span>0.6666667</span><span> </span><span>1</span><span> </span><span>1.0000000</span><span> </span><span>....</span><span>
</span><span>planet</span><span> </span><span>1.0000000</span><span> </span><span>1.0000000</span><span> </span><span>0.00000000</span><span> </span><span>1</span><span> </span><span>0.8181818</span><span> </span><span>1.0000000</span><span> </span><span>1.0000000</span><span> </span><span>1</span><span> </span><span>1.0000000</span><span> </span><span>....</span><span>
</span><span>is</span><span> </span><span>1.0000000</span><span> </span><span>1.0000000</span><span> </span><span>1.00000000</span><span> </span><span>0</span><span> </span><span>1.0000000</span><span> </span><span>1.0000000</span><span> </span><span>1.0000000</span><span> </span><span>0</span><span> </span><span>1.0000000</span><span> </span><span>....</span><span>
</span><span>ancient</span><span> </span><span>1.0000000</span><span> </span><span>1.0000000</span><span> </span><span>0.81818182</span><span> </span><span>1</span><span> </span><span>0.0000000</span><span> </span><span>1.0000000</span><span> </span><span>0.7777778</span><span> </span><span>1</span><span> </span><span>1.0000000</span><span> </span><span>....</span><span>
</span><span>with</span><span> </span><span>1.0000000</span><span> </span><span>1.0000000</span><span> </span><span>1.00000000</span><span> </span><span>1</span><span> </span><span>1.0000000</span><span> </span><span>0.0000000</span><span> </span><span>1.0000000</span><span> </span><span>1</span><span> </span><span>1.0000000</span><span> </span><span>....</span><span>
</span><span>ties</span><span> </span><span>0.6000000</span><span> </span><span>0.6666667</span><span> </span><span>1.00000000</span><span> </span><span>1</span><span> </span><span>0.7777778</span><span> </span><span>1.0000000</span><span> </span><span>0.0000000</span><span> </span><span>1</span><span> </span><span>1.0000000</span><span> </span><span>....</span><span>
</span><span>to</span><span> </span><span>1.0000000</span><span> </span><span>1.0000000</span><span> </span><span>1.00000000</span><span> </span><span>0</span><span> </span><span>1.0000000</span><span> </span><span>1.0000000</span><span> </span><span>1.0000000</span><span> </span><span>0</span><span> </span><span>1.0000000</span><span> </span><span>....</span><span>
</span><span>history</span><span> </span><span>1.0000000</span><span> </span><span>1.0000000</span><span> </span><span>1.00000000</span><span> </span><span>1</span><span> </span><span>1.0000000</span><span> </span><span>1.0000000</span><span> </span><span>1.0000000</span><span> </span><span>1</span><span> </span><span>0.0000000</span><span> </span><span>....</span><span>
</span><span>astrology</span><span> </span><span>1.0000000</span><span> </span><span>1.0000000</span><span> </span><span>1.00000000</span><span> </span><span>1</span><span> </span><span>0.8571429</span><span> </span><span>1.0000000</span><span> </span><span>1.0000000</span><span> </span><span>1</span><span> </span><span>0.8571429</span><span> </span><span>....</span><span>
</span><span>science</span><span> </span><span>1.0000000</span><span> </span><span>1.0000000</span><span> </span><span>1.00000000</span><span> </span><span>1</span><span> </span><span>0.3333333</span><span> </span><span>1.0000000</span><span> </span><span>0.7777778</span><span> </span><span>1</span><span> </span><span>1.0000000</span><span> </span><span>....</span><span>
</span><span>.</span><span>
</span><span>.</span><span>
</span><span>.</span><span>
</span></code></pre>
</div>
<p></p>
<p>here by adjusting (reducing ) the <em>dice_thresh</em> parameter we can force values close to 1.0 to become 1.0,</p>
<p></p>
<div>
<pre><code><span>
</span><span>the</span><span> </span><span>term</span><span> </span><span>planet</span><span> </span><span>is</span><span> </span><span>ancient</span><span> </span><span>with</span><span> </span><span>ties</span><span> </span><span>to</span><span> </span><span>history</span><span> </span><span>astrology</span><span> </span><span>science</span><span> </span><span>....</span><span>
</span><span>the</span><span> </span><span>0.0000000</span><span> </span><span>1</span><span> </span><span>1.00000000</span><span> </span><span>1</span><span> </span><span>1.0000000</span><span> </span><span>1</span><span> </span><span>1</span><span> </span><span>1</span><span> </span><span>1</span><span> </span><span>1</span><span> </span><span>1.0000000</span><span> </span><span>....</span><span>
</span><span>term</span><span> </span><span>1.0000000</span><span> </span><span>0</span><span> </span><span>1.00000000</span><span> </span><span>1</span><span> </span><span>1.0000000</span><span> </span><span>1</span><span> </span><span>1</span><span> </span><span>1</span><span> </span><span>1</span><span> </span><span>1</span><span> </span><span>1.0000000</span><span> </span><span>....</span><span>
</span><span>planet</span><span> </span><span>1.0000000</span><span> </span><span>1</span><span> </span><span>0.00000000</span><span> </span><span>1</span><span> </span><span>1.0000000</span><span> </span><span>1</span><span> </span><span>1</span><span> </span><span>1</span><span> </span><span>1</span><span> </span><span>1</span><span> </span><span>1.0000000</span><span> </span><span>....</span><span>
</span><span>is</span><span> </span><span>1.0000000</span><span> </span><span>1</span><span> </span><span>1.00000000</span><span> </span><span>0</span><span> </span><span>1.0000000</span><span> </span><span>1</span><span> </span><span>1</span><span> </span><span>0</span><span> </span><span>1</span><span> </span><span>1</span><span> </span><span>1.0000000</span><span> </span><span>....</span><span>
</span><span>ancient</span><span> </span><span>1.0000000</span><span> </span><span>1</span><span> </span><span>1.00000000</span><span> </span><span>1</span><span> </span><span>0.0000000</span><span> </span><span>1</span><span> </span><span>1</span><span> </span><span>1</span><span> </span><span>1</span><span> </span><span>1</span><span> </span><span>0.3333333</span><span> </span><span>....</span><span>
</span><span>with</span><span> </span><span>1.0000000</span><span> </span><span>1</span><span> </span><span>1.00000000</span><span> </span><span>1</span><span> </span><span>1.0000000</span><span> </span><span>0</span><span> </span><span>1</span><span> </span><span>1</span><span> </span><span>1</span><span> </span><span>1</span><span> </span><span>1.0000000</span><span> </span><span>....</span><span>
</span><span>ties</span><span> </span><span>1.0000000</span><span> </span><span>1</span><span> </span><span>1.00000000</span><span> </span><span>1</span><span> </span><span>1.0000000</span><span> </span><span>1</span><span> </span><span>0</span><span> </span><span>1</span><span> </span><span>1</span><span> </span><span>1</span><span> </span><span>1.0000000</span><span> </span><span>....</span><span>
</span><span>to</span><span> </span><span>1.0000000</span><span> </span><span>1</span><span> </span><span>1.00000000</span><span> </span><span>0</span><span> </span><span>1.0000000</span><span> </span><span>1</span><span> </span><span>1</span><span> </span><span>0</span><span> </span><span>1</span><span> </span><span>1</span><span> </span><span>1.0000000</span><span> </span><span>....</span><span>
</span><span>history</span><span> </span><span>1.0000000</span><span> </span><span>1</span><span> </span><span>1.00000000</span><span> </span><span>1</span><span> </span><span>1.0000000</span><span> </span><span>1</span><span> </span><span>1</span><span> </span><span>1</span><span> </span><span>0</span><span> </span><span>1</span><span> </span><span>1.0000000</span><span> </span><span>....</span><span>
</span><span>astrology</span><span> </span><span>1.0000000</span><span> </span><span>1</span><span> </span><span>1.00000000</span><span> </span><span>1</span><span> </span><span>1.0000000</span><span> </span><span>1</span><span> </span><span>1</span><span> </span><span>1</span><span> </span><span>1</span><span> </span><span>0</span><span> </span><span>1.0000000</span><span> </span><span>....</span><span>
</span><span>science</span><span> </span><span>1.0000000</span><span> </span><span>1</span><span> </span><span>1.00000000</span><span> </span><span>1</span><span> </span><span>0.3333333</span><span> </span><span>1</span><span> </span><span>1</span><span> </span><span>1</span><span> </span><span>1</span><span> </span><span>1</span><span> </span><span>0.0000000</span><span> </span><span>....</span><span> </span><span>.</span><span>
</span><span>.</span><span>
</span><span>.</span><span>
</span></code></pre>
</div>
<p></p>
<ul>
<li><strong>look_up_table</strong> : The idea here is to split the input words to n-grams using a numeric value and then retrieve the words which have a similar character n-gram. <br>
It returns a look-up-list where the list-names are the n-grams and the list-vectors are the words associated with those n-grams. The words for each n-gram can be retrieved using the <em>print_words_lookup_tbl</em> method. The input can be a character string vector only.</li>
</ul>
<p></p>
<div>
<pre><code><span>
</span><span># first initialization of token_stats </span><span>
</span><span>init</span><span> </span><span>=</span><span> </span><span>token_stats</span><span>$</span><span>new</span><span>(</span><span>path_2file</span><span> </span><span>=</span><span> </span><span>'freq_vocab.txt'</span><span>)</span><span>
</span><span>tmp_vec</span><span> </span><span>=</span><span> </span><span>init</span><span>$</span><span>path_2vector</span><span>()</span><span> </span><span># convert to vector
</span><span>
</span><span># second initialization to compute the look-up-table
</span><span>
</span><span>init_lk</span><span> </span><span>=</span><span> </span><span>token_stats</span><span>$</span><span>new</span><span>(</span><span>x_vec</span><span> </span><span>=</span><span> </span><span>tmp_vec</span><span>)</span><span>
</span><span>is_vec</span><span> </span><span>=</span><span> </span><span>init_lk</span><span>$</span><span>look_up_table</span><span>(</span><span>n_grams</span><span> </span><span>=</span><span> </span><span>3</span><span>)</span><span>
</span></code></pre>
</div>
<div>
<pre><code><span>
</span><span># example output for the 'is_vec' vector
</span><span>
</span><span>[</span><span>1</span><span>]</span><span> </span><span>&quot;&quot;</span><span> </span><span>&quot;ake&quot;</span><span> </span><span>&quot;_an&quot;</span><span> </span><span>&quot;anc&quot;</span><span> </span><span>&quot;ane&quot;</span><span> </span><span>&quot;_as&quot;</span><span> </span><span>&quot;ast&quot;</span><span> </span><span>&quot;cie&quot;</span><span> </span><span>&quot;eli&quot;</span><span> </span><span>&quot;enc&quot;</span><span> </span><span>&quot;era&quot;</span><span> </span><span>&quot;eve&quot;</span><span> </span><span>.....</span><span> </span><span>[</span><span>29</span><span>]</span><span> </span><span>&quot;net&quot;</span><span> </span><span>&quot;ola&quot;</span><span> </span><span>&quot;olo&quot;</span><span> </span><span>&quot;_pl&quot;</span><span> </span><span>&quot;pla&quot;</span><span> </span><span>&quot;_re&quot;</span><span> </span><span>&quot;rel&quot;</span><span> </span><span>&quot;rol&quot;</span><span> </span><span>&quot;_sc&quot;</span><span> </span><span>&quot;sci&quot;</span><span> </span><span>&quot;_se&quot;</span><span> </span><span>&quot;see&quot;</span><span> </span><span>.....</span><span>
</span><span>[</span><span>57</span><span>]</span><span> </span><span>&quot;tro&quot;</span><span> </span><span>&quot;ver&quot;</span><span> </span><span>&quot;_we&quot;</span><span> </span><span>&quot;wer&quot;</span><span> </span><span>&quot;_wi&quot;</span><span> </span><span>&quot;wit&quot;</span><span> </span><span>&quot;yst&quot;</span><span> </span><span>&quot;yth&quot;</span><span>
</span></code></pre>
</div>
<p></p>
<p>then retrieve words with same n-grams,</p>
<p></p>
<div>
<pre><code><span>
</span><span>init_lk</span><span>$</span><span>print_words_lookup_tbl</span><span>(</span><span>n_gram</span><span> </span><span>=</span><span> </span><span>&quot;log&quot;</span><span>)</span><span>
</span></code></pre>
</div>
<div>
<pre><code><span>
</span><span>&quot;_astrology_&quot;</span><span> </span><span>&quot;_mythology_&quot;</span><span>
</span></code></pre>
</div>
<p></p>
<p>the underscores are necessary to distinguish the begin and end of each word when computing the n-grams.</p>
<p>More information about the <em>token_stats</em> class can be found in the package documentation.</p>

<h3><em>helper functions for sparse_matrices</em></h3>
<p></p>
<p>The purpose of creating those functions is because I observed that they return faster in comparison to other R packages. The following code chunks explain each one of the functions,</p>
<p></p>
<div>
<pre><code><span>
</span><span>#---------------------------------
# conversion from dense to sparse
#---------------------------------
</span><span>
</span><span>library</span><span>(</span><span>textTinyR</span><span>)</span><span>
</span><span>set.seed</span><span>(</span><span>1</span><span>)</span><span>
</span><span>dsm</span><span> </span><span>=</span><span> </span><span>matrix</span><span>(</span><span>sample</span><span>(</span><span>0</span><span>:</span><span>1</span><span>,</span><span> </span><span>100</span><span>,</span><span> </span><span>replace</span><span> </span><span>=</span><span> </span><span>T</span><span>),</span><span> </span><span>10</span><span>,</span><span> </span><span>10</span><span>)</span><span>
</span><span>res_sp</span><span> </span><span>=</span><span> </span><span>dense_2sparse</span><span>(</span><span>dsm</span><span>)</span><span>
</span><span>res_sp</span><span>
</span></code></pre>
</div>
<div>
<pre><code><span>
</span><span>## 10 x 10 sparse Matrix of class &quot;dgCMatrix&quot;
## ## [1,] . . 1 . 1 . 1 . . .
## [2,] . . . 1 1 1 . 1 1 .
## [3,] 1 1 1 . 1 . . . . 1
## [4,] 1 . . . 1 . . . . 1
## [5,] . 1 . 1 1 . 1 . 1 1
## [6,] 1 . . 1 1 . . 1 . 1
## [7,] 1 1 . 1 . . . 1 1 .
## [8,] 1 1 . . . 1 1 . . .
## [9,] 1 . 1 1 1 1 . 1 . 1
## [10,] . 1 . . 1 . 1 1 . 1
</span><span>
</span></code></pre>
</div>
<p></p>
<div>
<pre><code><span>
</span><span>#-------------
# column- sums
#-------------
</span><span>
</span><span>sm_cols</span><span> </span><span>=</span><span> </span><span>sparse_Sums</span><span>(</span><span>res_sp</span><span>,</span><span> </span><span>rowSums</span><span> </span><span>=</span><span> </span><span>FALSE</span><span>)</span><span>
</span><span>sm_cols</span><span>
</span></code></pre>
</div>
<div>
<pre><code><span>
</span><span>## [1] 6 5 3 5 8 3 4 5 3 6
</span><span>
</span></code></pre>
</div>
<p></p>
<div>
<pre><code><span>
</span><span>#----------
# row-sums
#----------
</span><span>
</span><span>sm_rows</span><span> </span><span>=</span><span> </span><span>sparse_Sums</span><span>(</span><span>res_sp</span><span>,</span><span> </span><span>rowSums</span><span> </span><span>=</span><span> </span><span>TRUE</span><span>)</span><span>
</span><span>sm_rows</span><span>
</span></code></pre>
</div>
<div>
<pre><code><span>
</span><span>## [1] 3 5 5 3 6 5 5 4 7 5
</span><span>
</span></code></pre>
</div>
<p></p>
<div>
<pre><code><span>
</span><span>#---------------
# column- means
#---------------
</span><span>
</span><span>mn_cols</span><span> </span><span>=</span><span> </span><span>sparse_Means</span><span>(</span><span>res_sp</span><span>,</span><span> </span><span>rowMeans</span><span> </span><span>=</span><span> </span><span>FALSE</span><span>)</span><span>
</span><span>mn_cols</span><span>
</span></code></pre>
</div>
<div>
<pre><code><span>
</span><span>## [1] 0.6 0.5 0.3 0.5 0.8 0.3 0.4 0.5 0.3 0.6
</span><span>
</span></code></pre>
</div>
<p></p>
<div>
<pre><code><span>
</span><span>#-----------
# row-means
#-----------
</span><span>
</span><span>mn_rows</span><span> </span><span>=</span><span> </span><span>sparse_Means</span><span>(</span><span>res_sp</span><span>,</span><span> </span><span>rowMeans</span><span> </span><span>=</span><span> </span><span>TRUE</span><span>)</span><span>
</span><span>mn_rows</span><span>
</span></code></pre>
</div>
<div>
<pre><code><span>
</span><span>## [1] 0.3 0.5 0.5 0.3 0.6 0.5 0.5 0.4 0.7 0.5
</span><span>
</span></code></pre>
</div>
<p></p>
<div>
<pre><code><span>
</span><span>#-------------------------------------
# sparsity of a matrix (as percentage)
#-------------------------------------
</span><span>
</span><span>matrix_sparsity</span><span>(</span><span>res_sp</span><span>)</span><span>
</span></code></pre>
</div>
<div>
<pre><code><span>
</span><span>## 51.9999 %
</span><span>
</span></code></pre>
</div>
<p></p>
<div>
<pre><code><span>
</span><span>#------------------------------------------------------
# saving and loading sparse matrices (in binary format)
#------------------------------------------------------
</span><span>
</span><span>save_sp</span><span> </span><span>=</span><span> </span><span>save_sparse_binary</span><span>(</span><span>res_sp</span><span>,</span><span> </span><span>file_name</span><span> </span><span>=</span><span> </span><span>&quot;save_sparse.mat&quot;</span><span>)</span><span>
</span><span>load_sp</span><span> </span><span>=</span><span> </span><span>load_sparse_binary</span><span>(</span><span>file_name</span><span> </span><span>=</span><span> </span><span>&quot;save_sparse.mat&quot;</span><span>)</span><span>
</span></code></pre>
</div>
<p></p>
<p>More information about the helper functions for sparse matrices can be found in the package documentation.</p>

<h3><em>tokenization</em></h3>
<p></p>
<p>The <strong>tokenize_transform_text()</strong> function applies tokenization and transformation in a similar way to the <em>big_text_tokenizer()</em> method, however for small to medium data sets. The input can be either a character string (text data) or a path to a file. This method takes as input a single character string (character-string == one line). The parameters for the <em>tokenize_transform_text()</em> function are the same to the (already explained) <em>big_text_tokenizer()</em> method with the only exception being the input data type.</p>
<p></p>
<p>The <strong>tokenize_transform_vec_docs()</strong> function works in the same way to the <em>Term_Matrix()</em> method and it targets small to medium data sets. It takes as input a vector of documents and retains their order after tokenization and transformation has taken place. Both the <em>tokenize_transform_text()</em> and <em>tokenize_transform_vec_docs()</em> share the same parameters, with the following two exceptions,</p>
<ul>
<li>the <strong>object</strong> is a character vector</li>
<li>the <strong>as_token</strong> parameter : if TRUE then the output of the function is a list of (split) token. Otherwise it’s a vector of character strings (sentences)</li>
</ul>
<p></p>
<p>The following code chunks give an overview of the mentioned functions,</p>
<p></p>
<div>
<pre><code><span>
</span><span>#------------------------
# tokenize_transform_text
#------------------------
</span><span>
</span><span># example input : &quot;planets.txt&quot;
</span><span>
</span><span>res_txt</span><span> </span><span>=</span><span> </span><span>tokenize_transform_text</span><span>(</span><span>object</span><span> </span><span>=</span><span> </span><span>&quot;/planets.txt&quot;</span><span>,</span><span> </span><span>to_lower</span><span> </span><span>=</span><span> </span><span>TRUE</span><span>,</span><span>
</span><span>utf_locale</span><span> </span><span>=</span><span> </span><span>&quot;&quot;</span><span>,</span><span> </span><span>trim_token</span><span> </span><span>=</span><span> </span><span>TRUE</span><span>,</span><span>
</span><span>split_string</span><span> </span><span>=</span><span> </span><span>TRUE</span><span>,</span><span>
</span><span>remove_stopwords</span><span> </span><span>=</span><span> </span><span>TRUE</span><span>,</span><span> </span><span>language</span><span> </span><span>=</span><span> </span><span>&quot;english&quot;</span><span>,</span><span>
</span><span>stemmer</span><span> </span><span>=</span><span> </span><span>&quot;porter2_stemmer&quot;</span><span>,</span><span> </span><span>threads</span><span> </span><span>=</span><span> </span><span>1</span><span>)</span><span>
</span></code></pre>
</div>
<p></p>
<p>the output is a vector of <em>tokens</em> after the english stopwords were removed and the terms were stemmed (<em>porter2-stemming</em>),</p>
<p></p>
<div>
<pre><code><span>
</span><span># example output :
</span><span>
</span><span>$</span><span>token</span><span>
</span><span>[</span><span>1</span><span>]</span><span> </span><span>&quot;term&quot;</span><span> </span><span>&quot;planet&quot;</span><span> </span><span>&quot;ancient&quot;</span><span> </span><span>&quot;tie&quot;</span><span> </span><span>&quot;histori&quot;</span><span> </span><span>...</span><span>
</span><span>[</span><span>9</span><span>]</span><span> </span><span>&quot;religion&quot;</span><span> </span><span>&quot;planet&quot;</span><span> </span><span>&quot;solar&quot;</span><span> </span><span>&quot;system&quot;</span><span> </span><span>&quot;nake&quot;</span><span> </span><span>....</span><span>
</span><span>[</span><span>17</span><span>]</span><span> </span><span>&quot;cultur&quot;</span><span> </span><span>&quot;divin&quot;</span><span> </span><span>&quot;emissari&quot;</span><span> </span><span>&quot;deiti&quot;</span><span> </span><span>&quot;scientif&quot;</span><span> </span><span>...</span><span> </span><span>[</span><span>25</span><span>]</span><span> </span><span>&quot;percept&quot;</span><span> </span><span>&quot;planet&quot;</span><span> </span><span>&quot;chang&quot;</span><span> </span><span>&quot;incorpor&quot;</span><span> </span><span>&quot;number&quot;</span><span> </span><span>.....</span><span>
</span><span>.</span><span>
</span><span>.</span><span>
</span><span>.</span><span>
</span><span>attr</span><span>(,</span><span>&quot;class&quot;</span><span>)</span><span>
</span><span>[</span><span>1</span><span>]</span><span> </span><span>&quot;tokenization and transformation&quot;</span><span>
</span></code></pre>
</div>
<p></p>
<div>
<pre><code><span>
</span><span>#----------------------------
# tokenize_transform_vec_docs
#----------------------------
</span><span>
</span><span># the input should be a vector of documents
</span><span>
</span><span>init</span><span> </span><span>=</span><span> </span><span>token_stats</span><span>$</span><span>new</span><span>(</span><span>path_2file</span><span> </span><span>=</span><span> </span><span>&quot;/planets.txt&quot;</span><span>)</span><span>
</span><span>inp</span><span> </span><span>=</span><span> </span><span>init</span><span>$</span><span>path_2vector</span><span>()</span><span> </span><span># convert text file to character vector
</span><span>
</span><span># run the function using the input-vector
</span><span>
</span><span>res_dct</span><span> </span><span>=</span><span> </span><span>tokenize_transform_vec_docs</span><span>(</span><span>object</span><span> </span><span>=</span><span> </span><span>inp</span><span>,</span><span> </span><span>as_token</span><span> </span><span>=</span><span> </span><span>FALSE</span><span>,</span><span> </span><span># return character vector of documents
</span><span> </span><span>to_lower</span><span> </span><span>=</span><span> </span><span>TRUE</span><span>,</span><span>
</span><span>utf_locale</span><span> </span><span>=</span><span> </span><span>&quot;&quot;</span><span>,</span><span> </span><span>trim_token</span><span> </span><span>=</span><span> </span><span>TRUE</span><span>,</span><span>
</span><span>split_string</span><span> </span><span>=</span><span> </span><span>TRUE</span><span>,</span><span>
</span><span>remove_stopwords</span><span> </span><span>=</span><span> </span><span>TRUE</span><span>,</span><span> </span><span>language</span><span> </span><span>=</span><span> </span><span>&quot;english&quot;</span><span>,</span><span>
</span><span>stemmer</span><span> </span><span>=</span><span> </span><span>&quot;ngram_sequential&quot;</span><span>,</span><span>
</span><span>stemmer_ngram</span><span> </span><span>=</span><span> </span><span>3</span><span>,</span><span>
</span><span>threads</span><span> </span><span>=</span><span> </span><span>1</span><span>)</span><span>
</span></code></pre>
</div>
<p></p>
<p>the output is a vector of <em>transformed documents</em> after the english stopwords were removed and the terms were stemmed (<em>ngram_sequential</em> of length 3),</p>
<p></p>
<div>
<pre><code><span>
</span><span>$</span><span>token</span><span>
</span><span>[</span><span>1</span><span>]</span><span> </span><span>&quot;term planet anci ties hist astro scien mythol relig planet solar system naked eye .....&quot;</span><span>
</span><span>[</span><span>2</span><span>]</span><span> </span><span>&quot;planet bodies discove 1950 remain \&quot;plane modern definit celest bodies ceres .....&quot;</span><span>
</span><span>[</span><span>3</span><span>]</span><span> </span><span>&quot;plan thou ptol orbit earth defer epicy moti idea plan orbit sun sugges times .....&quot;</span><span>
</span><span>[</span><span>4</span><span>]</span><span> </span><span>&quot;time care analy pre-telesco observat data collec tycho brahe johan kepler .....&quot;</span><span>
</span><span>[</span><span>5</span><span>]</span><span> </span><span>&quot;plan genera divi main types large low-dens giant plan smal rocky terrestri .....&quot;</span><span>
</span><span>attr</span><span>(,</span><span>&quot;class&quot;</span><span>)</span><span>
</span><span>[</span><span>1</span><span>]</span><span> </span><span>&quot;tokenization and transformation&quot;</span><span>
</span></code></pre>
</div>
<p></p>
<p>The documents can be returned as a list of character vectors by specifying, <em>as_token</em> = TRUE,</p>
<p></p>
<div>
<pre><code><span>
</span><span># run the function using the input-vector
</span><span>
</span><span>res_dct_tok</span><span> </span><span>=</span><span> </span><span>tokenize_transform_vec_docs</span><span>(</span><span>object</span><span> </span><span>=</span><span> </span><span>inp</span><span>,</span><span> </span><span>as_token</span><span> </span><span>=</span><span> </span><span>TRUE</span><span>,</span><span>
</span><span>to_lower</span><span> </span><span>=</span><span> </span><span>TRUE</span><span>,</span><span>
</span><span>utf_locale</span><span> </span><span>=</span><span> </span><span>&quot;&quot;</span><span>,</span><span> </span><span>trim_token</span><span> </span><span>=</span><span> </span><span>TRUE</span><span>,</span><span>
</span><span>split_string</span><span> </span><span>=</span><span> </span><span>TRUE</span><span>,</span><span>
</span><span>remove_stopwords</span><span> </span><span>=</span><span> </span><span>TRUE</span><span>,</span><span> </span><span>language</span><span> </span><span>=</span><span> </span><span>&quot;english&quot;</span><span>,</span><span>
</span><span>stemmer</span><span> </span><span>=</span><span> </span><span>&quot;ngram_sequential&quot;</span><span>,</span><span>
</span><span>stemmer_ngram</span><span> </span><span>=</span><span> </span><span>3</span><span>,</span><span>
</span><span>threads</span><span> </span><span>=</span><span> </span><span>1</span><span>)</span><span>
</span></code></pre>
</div>
<p></p>
<div>
<pre><code><span>
</span><span>$</span><span>token</span><span>
</span><span>$</span><span>token</span><span>[[</span><span>1</span><span>]]</span><span>
</span><span>[</span><span>1</span><span>]</span><span> </span><span>&quot;term&quot;</span><span> </span><span>&quot;planet&quot;</span><span> </span><span>&quot;anci&quot;</span><span> </span><span>&quot;ties&quot;</span><span> </span><span>&quot;hist&quot;</span><span> </span><span>.....</span><span> </span><span>$</span><span>token</span><span>[[</span><span>2</span><span>]]</span><span>
</span><span>[</span><span>1</span><span>]</span><span> </span><span>&quot;planet&quot;</span><span> </span><span>&quot;bodies&quot;</span><span> </span><span>&quot;discove&quot;</span><span> </span><span>&quot;1950&quot;</span><span> </span><span>.....</span><span> </span><span>$</span><span>token</span><span>[[</span><span>3</span><span>]]</span><span>
</span><span>[</span><span>1</span><span>]</span><span> </span><span>&quot;plan&quot;</span><span> </span><span>&quot;thou&quot;</span><span> </span><span>&quot;ptol&quot;</span><span> </span><span>&quot;orbit&quot;</span><span> </span><span>&quot;earth&quot;</span><span> </span><span>.....</span><span> </span><span>attr</span><span>(,</span><span>&quot;class&quot;</span><span>)</span><span>
</span><span>[</span><span>1</span><span>]</span><span> </span><span>&quot;tokenization and transformation&quot;</span><span>
</span></code></pre>
</div>
<p></p>
<p>A few words about the <strong>utf_locale</strong>, <strong>remove_stopwords</strong> and <strong>stemmer</strong> parameters.</p>
<p></p>
<ul>
<li>The <strong>utf_locale</strong> can take as input either an empty string (“”) or a character string (for instance “el_GR.UTF-8”). It should be a non-empty string if the text input is other than english. However, currently for the windows OS only english character strings or files can be input and pre-processed.</li>
</ul>
<p></p>
<ul>
<li>The <strong>remove_stopwords</strong> parameter can be either a boolean (TRUE, FALSE) or a character vector of user defined stop-words. The available languages are specified by the parameter <strong>language</strong>. Currently, there is no support for chinese, japanese, korean, thai or languages with ambiguous word boundaries.</li>
</ul>
<p></p>
<ul>
<li>The <strong>stemmer</strong> parameter can take as input one of the <strong>porter2_stemmer</strong>, <strong>ngram_sequential</strong> or <strong>ngram_overlap</strong>.
<ul>
<li>The <a target="_blank" rel="nofollow" href="https://github.com/smassung/porter2_stemmer"><em>porter2_stemmer</em></a> is a C++ implementation of the <a target="_blank" rel="nofollow" href="http://snowball.tartarus.org/algorithms/english/stemmer.html">snowball-porter2</a> stemming algorithm.</li>
<li>On the other hand, <em>n-gram stemming</em> is <em>“language independent”</em> and supported by the <strong>ngram_sequential</strong> and <strong>ngram_overlap</strong> functions.
<ul>
<li><a target="_blank" rel="nofollow" href="http://clef.isti.cnr.it/2007/working_notes/mcnameeCLEF2007.pdf"><em>ngram_overlap</em></a> : The ngram_overlap stemming method is based on N-Gram Morphemes for Retrieval, Paul McNamee and James Mayfield</li>
<li><a target="_blank" rel="nofollow" href="https://arxiv.org/pdf/1312.4824.pdf"><em>ngram_sequential</em></a> : The ngram_sequential stemming method is a modified version based on Generation, Implementation and Appraisal of an N-gram based Stemming Algorithm, B. P. Pande, Pawan Tamta, H. S. Dhami</li>
</ul>
</li>
</ul>
</li>
</ul>
<h3><em>utility</em> functions</h3>
<p></p>
<p>The following code chunks illustrate the <em>utility functions</em> of the package (besides the <strong>read_characters()</strong> and <strong>read_rows()</strong> which used in the previous code chunks),</p>
<p></p>
<div>
<pre><code><span>
</span><span>#---------------------------------------
# cosine distance between word sentences
#---------------------------------------
</span><span>
</span><span>s</span><span>1</span><span> </span><span>=</span><span> </span><span>'sentence with two words'</span><span>
</span><span>s</span><span>2</span><span> </span><span>=</span><span> </span><span>'sentence with three words'</span><span>
</span><span>sep</span><span> </span><span>=</span><span> </span><span>&quot; &quot;</span><span>
</span><span>cosine_distance</span><span>(</span><span>s</span><span>1</span><span>,</span><span> </span><span>s</span><span>2</span><span>,</span><span> </span><span>split_separator</span><span> </span><span>=</span><span> </span><span>sep</span><span>)</span><span>
</span></code></pre>
</div>
<div>
<pre><code><span>
</span><span>## [1] 0.75
</span><span>
</span></code></pre>
</div>
<p></p>
<div>
<pre><code><span>
</span><span>#------------------------------------------------------------------------
# dice distance between two words (using n-grams -- the lower the better)
#------------------------------------------------------------------------
</span><span>
</span><span>w</span><span>1</span><span> </span><span>=</span><span> </span><span>'word_one'</span><span>
</span><span>w</span><span>2</span><span> </span><span>=</span><span> </span><span>'word_two'</span><span>
</span><span>n</span><span> </span><span>=</span><span> </span><span>2</span><span>
</span><span>dice_distance</span><span>(</span><span>w</span><span>1</span><span>,</span><span> </span><span>w</span><span>2</span><span>,</span><span> </span><span>n_grams</span><span> </span><span>=</span><span> </span><span>n</span><span>)</span><span>
</span></code></pre>
</div>
<div>
<pre><code><span>
</span><span>## [1] 0.2857143
</span><span>
</span></code></pre>
</div>
<p></p>
<div>
<pre><code><span>
</span><span>#---------------------------------------
# levenshtein distance between two words
#---------------------------------------
</span><span>
</span><span>w</span><span>1</span><span> </span><span>=</span><span> </span><span>'word_two'</span><span>
</span><span>w</span><span>2</span><span> </span><span>=</span><span> </span><span>'word_one'</span><span>
</span><span>levenshtein_distance</span><span>(</span><span>w</span><span>1</span><span>,</span><span> </span><span>w</span><span>2</span><span>)</span><span>
</span></code></pre>
</div>
<div>
<pre><code><span>
</span><span>## [1] 3
</span><span>
</span></code></pre>
</div>
<p></p>
<div>
<pre><code><span>
</span><span>#---------------------------------------------
# bytes converter (returns the size of a file)
#---------------------------------------------
</span><span>
</span><span>PATH</span><span> </span><span>=</span><span> </span><span>&quot;/planets.txt&quot;</span><span>
</span><span>bytes_converter</span><span>(</span><span>input_path_file</span><span> </span><span>=</span><span> </span><span>PATH</span><span>,</span><span> </span><span>unit</span><span> </span><span>=</span><span> </span><span>&quot;KB&quot;</span><span> </span><span>)</span><span>
</span></code></pre>
</div>
<div>
<pre><code><span>
</span><span>## [1] 2.213867
</span><span>
</span></code></pre>
</div>
<p></p>
<div>
<pre><code><span>
</span><span>#---------------------------------------------------
# returns the utf-locale for the available languages
#---------------------------------------------------
</span><span>
</span><span>utf_locale</span><span>(</span><span>language</span><span> </span><span>=</span><span> </span><span>&quot;english&quot;</span><span>)</span><span>
</span></code></pre>
</div>
<div>
<pre><code><span>
</span><span>## [1] &quot;en.UTF-8&quot;
</span><span>
</span></code></pre>
</div>
<p></p>
<div>
<pre><code><span>
</span><span>#-----------------
# text file parser
#-----------------
</span><span>
</span><span># The text file should have a structure (such as an xml-structure), so that # subsets can be extracted using the &quot;start_query&quot; and &quot;end_query&quot; parameters.
# (it works similarly to the big_text_parser() method, however for small to medium sized files)
</span><span>
</span><span># example input &quot;example_file.xml&quot; file :
</span><span>
</span><span>&lt;?</span><span>xml</span><span> </span><span>version</span><span>=</span><span>&quot;1.0&quot;</span><span>?></span><span>
</span><span>&lt;</span><span>sked</span><span>></span><span>
</span><span>&lt;</span><span>version</span><span>></span><span>2</span><span>&lt;/</span><span>version</span><span>></span><span>
</span><span>&lt;</span><span>flight</span><span>></span><span>
</span><span>&lt;</span><span>carrier</span><span>></span><span>BA</span><span>&lt;/</span><span>carrier</span><span>></span><span>
</span><span>&lt;</span><span>number</span><span>></span><span>4001</span><span>&lt;/</span><span>number</span><span>></span><span>
</span><span>&lt;</span><span>date</span><span>></span><span>2011-07-21</span><span>&lt;/</span><span>date</span><span>></span><span>
</span><span>&lt;/</span><span>flight</span><span>></span><span>
</span><span>&lt;</span><span>flight</span><span> </span><span>cancelled</span><span>=</span><span>&quot;true&quot;</span><span>></span><span>
</span><span>&lt;</span><span>carrier</span><span>></span><span>BA</span><span>&lt;/</span><span>carrier</span><span>></span><span>
</span><span>&lt;</span><span>number</span><span>></span><span>4002</span><span>&lt;/</span><span>number</span><span>></span><span>
</span><span>&lt;</span><span>date</span><span>></span><span>2011-07-21</span><span>&lt;/</span><span>date</span><span>></span><span>
</span><span>&lt;/</span><span>flight</span><span>></span><span>
</span><span>&lt;/</span><span>sked</span><span>></span><span>
</span></code></pre>
</div>
<p></p>
<div>
<pre><code><span>
</span><span>fp</span><span> </span><span>=</span><span> </span><span>text_file_parser</span><span>(</span><span>input_path_file</span><span> </span><span>=</span><span> </span><span>&quot;example_file.xml&quot;</span><span>,</span><span> </span><span>output_path_file</span><span> </span><span>=</span><span> </span><span>&quot;/output_folder/example_output_file.txt&quot;</span><span>,</span><span> </span><span>start_query</span><span> </span><span>=</span><span> </span><span>'&lt;number>'</span><span>,</span><span> </span><span>end_query</span><span> </span><span>=</span><span> </span><span>'&lt;/number>'</span><span>,</span><span>
</span><span>min_lines</span><span> </span><span>=</span><span> </span><span>1</span><span>,</span><span> </span><span>trimmed_line</span><span> </span><span>=</span><span> </span><span>FALSE</span><span>)</span><span>
</span></code></pre>
</div>
<div>
<pre><code><span>
</span><span>&quot;example_output_file.txt&quot;</span><span> </span><span>:</span><span>
</span><span>4001</span><span>
</span><span>4002</span><span>
</span></code></pre>
</div>
<p></p>
<div>
<pre><code><span>
</span><span>#------------------
# vocabulary parser
#------------------
</span><span>
</span><span># the 'vocabulary_parser' function extracts a vocabulary from a structured text (such as # an .xml file) and works in the exact same way as the 'big_tokenize_transform' class, # however for small to medium sized data files
</span><span>
</span><span>pars_dat</span><span> </span><span>=</span><span> </span><span>vocabulary_parser</span><span>(</span><span>input_path_file</span><span> </span><span>=</span><span> </span><span>'/folder/input_data.txt'</span><span>,</span><span>
</span><span>start_query</span><span> </span><span>=</span><span> </span><span>'start_word'</span><span>,</span><span> </span><span>end_query</span><span> </span><span>=</span><span> </span><span>'end_word'</span><span>,</span><span>
</span><span>vocabulary_path_file</span><span> </span><span>=</span><span> </span><span>'/folder/vocab.txt'</span><span>,</span><span> </span><span>to_lower</span><span> </span><span>=</span><span> </span><span>TRUE</span><span>,</span><span> </span><span>split_string</span><span> </span><span>=</span><span> </span><span>TRUE</span><span>,</span><span>
</span><span>remove_stopwords</span><span> </span><span>=</span><span> </span><span>TRUE</span><span>)</span><span>
</span></code></pre>
</div>
<p></p>
<p>An updated version of the textTinyR package can be found in my <a target="_blank" rel="nofollow" href="https://github.com/mlampros/textTinyR">Github repository</a> and to report bugs/issues please use the following link, <a target="_blank" rel="nofollow" href="https://github.com/mlampros/textTinyR/issues">https://github.com/mlampros/textTinyR/issues</a>.</p>
<p></p>
<p>
<div>
<div>To <strong>leave a comment</strong> for the author, please follow the link and comment on their blog: <strong><a href="http://mlampros.github.io/mlampros.github.io/2017/01/05/textTinyR_package/"> mlampros</a></strong>.</div>
<hr>
<a rel="nofollow" href="https://www.r-bloggers.com/">R-bloggers.com</a> offers <strong><a rel="nofollow" href="https://feedburner.google.com/fb/a/mailverify?uri=RBloggers">daily e-mail updates</a></strong> about <a title="The R Project for Statistical Computing" rel="nofollow" href="https://www.r-project.org/">R</a> news and <a title="R tutorials" rel="nofollow" href="https://www.r-bloggers.com/search/tutorial">tutorials</a> on topics such as: <a title="Data science" rel="nofollow" href="https://www.r-bloggers.com/search/data%20science">Data science</a>, <a title="Big Data" rel="nofollow" href="https://www.r-bloggers.com/search/Big%20Data">Big Data, <a title="R jobs" rel="nofollow" href="https://www.r-users.com/">R jobs</a>, visualization (<a title="ggplot and ggplot2 tutorials" rel="nofollow" href="https://www.r-bloggers.com/search/ggplot2">ggplot2</a>, <a title="Boxplots using lattice and ggplot2 tutorials" rel="nofollow" href="https://www.r-bloggers.com/search/boxplot">Boxplots</a>, <a title="Maps and gis" rel="nofollow" href="https://www.r-bloggers.com/search/map">maps</a>, <a title="Animation in R" rel="nofollow" href="https://www.r-bloggers.com/search/animation">animation</a>), programming (<a title="RStudio IDE for R" rel="nofollow" href="https://www.r-bloggers.com/search/RStudio">RStudio</a>, <a title="Sweave and literate programming" rel="nofollow" href="https://www.r-bloggers.com/search/sweave">Sweave</a>, <a title="LaTeX in R" rel="nofollow" href="https://www.r-bloggers.com/search/LaTeX">LaTeX</a>, <a title="SQL and databases" rel="nofollow" href="https://www.r-bloggers.com/search/SQL">SQL</a>, <a title="Eclipse IDE for R" rel="nofollow" href="https://www.r-bloggers.com/search/eclipse">Eclipse</a>, <a title="git and github, Version Control System" rel="nofollow" href="https://www.r-bloggers.com/search/git">git</a>, <a title="Large data in R using Hadoop" rel="nofollow" href="https://www.r-bloggers.com/search/hadoop">hadoop</a>, <a title="Web Scraping of google, facebook, yahoo, twitter and more using R" rel="nofollow" href="https://www.r-bloggers.com/search/Web+Scraping">Web Scraping</a>) statistics (<a title="Regressions and ANOVA analysis tutorials" rel="nofollow" href="https://www.r-bloggers.com/search/regression">regression</a>, <a title="principal component analysis tutorial" rel="nofollow" href="https://www.r-bloggers.com/search/PCA">PCA</a>, <a title="Time series" rel="nofollow" href="https://www.r-bloggers.com/search/time+series">time series</a>, <a title="finance trading" rel="nofollow" href="https://www.r-bloggers.com/search/trading">trading</a>) and more...
<img height="1" alt="" width="1" src="http://feeds.feedburner.com/~r/RBloggers/~4/mvCKq49Yda4"></a></div>