<h1>Introducing text2vec 0.4</h1>
<p>Today I’m pleased to announce new major release of <a href="https://github.com/dselivanov/text2vec">text2vec</a> - text2vec 0.4 which is already on CRAN.</p>
<p>For those readers who is not familiar with text2vec - it is an R package which provides an efficient framework with a concise API for text analysis and natural language processing.</p>
<p>With this release I also launched project homepage - <a href="http://text2vec.org/">http://text2vec.org</a> where you can find up-to-date documents and tutorials.</p>
<h2>Functionality</h2>
<p>The core functionality at the moment includes:</p>
<ol><li>Fast text vectorization (creation of document-term matrices) on arbitrary n-grams, using vocabulary or feature hashing</li>
<li><a href="http://www-nlp.stanford.edu/projects/glove/">GloVe</a> word embeddings</li>
<li>Topic modeling with:
<ul><li>Latent Dirichlet Allocation</li>
<li>Latent Sematic Analysis</li>
</ul></li>
<li>Similarities/distances between matrices (documents in vector space)
<ul><li>Cosine</li>
<li>Jaccard</li>
<li><a href="http://vene.ro/blog/word-movers-distance-in-python.html">Relaxed Word Mover’s Distance</a></li>
<li>Euclidean</li>
</ul></li>
</ol><h2>What’s new?</h2>
<p>First of all, I would like to express special thanks to project contributors - <a href="https://github.com/lmullen">Lincoln Mullen</a>, <a href="https://github.com/qinwf">Qin Wenfeng</a>, <a href="https://github.com/zachmayer">Zach Mayer</a> and <a href="https://github.com/dselivanov/text2vec/graphs/contributors">others</a> (and of course for all of those who reported bugs on the issue tracker!).</p>
<p>A lot of work was done in the last 6 months. Most notable changes are:</p>
<ul><li><strong>Immutable iterators</strong>. Most frustrating and annoying thing in 0.3 was that <code>create_*</code> functions modified input objects (in contrast to usual R behavior with copy-on-modify semantics). So I received a <a href="https://github.com/dselivanov/text2vec/issues/107">lot of bug reports</a> on that. People just didn’t understand why they getting empty Document-Term matrices. That was my big mistake, R users assume that function can’t modify argument. So I rewrote iterators with <code>R6</code> classes (thanks to @hadley for suggestion). Learned a lot.</li>
<li>Now text2vec have <strong>consistent pipe-friendly interface for models</strong>. User should remember few main verbs - <code>fit</code>, <code>transform</code>, <code>fit_transform</code>. More details will be available soon in a separate blog post. Stay tuned.</li>
<li>Started to work on <strong>models</strong> which can be useful for NLP:
<ul><li><strong>Latent Dirichlet Allocation</strong>. Code for fast Collapsed Gibbs Sampling is based on <a href="https://cran.r-project.org/package=lda">lda</a> package by Jonathan Chang, but with a few tweaks (which will be incorporated into <code>lda</code> package in next release). It happened that LDA from text2vec ~ 2x faster that original (and ~10x faster than <a href="https://cran.r-project.org/package=topicmodels">topicmodels</a>!)</li>
<li><strong>Latent Semantic Analysis</strong> (based on updated <a href="https://cran.r-project.org/package=irlba">irlba</a> package)</li>
<li><strong>Tf-Idf</strong> is also rewritten to be consistent with other models interface</li>
</ul></li>
<li>Now text2vec contains functions for fast calculation of <strong>similarity between documents</strong> (actually similarities and distances between matrices):
<ul><li>Cosine distance</li>
<li>Jaccard distance</li>
<li>Relaxed Word Mover’s Distance (which was demonstrated to be very useful on latest kaggle competitions - <a href="http://blog.kaggle.com/2016/08/31/avito-duplicate-ads-detection-winners-interview-2nd-place-team-the-quants-mikel-peter-marios-sonny/">1</a>, <a href="http://blog.kaggle.com/2016/07/27/avito-duplicate-ads-detection-winners-interview-3rd-place-mario-gerard-kele-praveen-gilberto/">2</a>). Dedicated post/tutorial on that will be available soon. Stay tuned</li>
</ul></li>
<li><strong>GloVe</strong> word embeddings also significantly updated:
<ul><li>Even faster now - <strong>got ~2-3x performance boost</strong> from code optimizations and usage of single precision <code>float</code> arithmetic (don’t forget to enable <code>-ffast-math</code> option for your C++ compiler)</li>
<li><strong><code>L1</code> regularization</strong> - our new feature (I didn’t see implementations our papers where researchers tried to add regularization). Higher quality word embedding for small data sets. More details will be available in separate blog post. Stay tuned</li>
</ul></li>
</ul><h1>Updated tutorials</h1>
<p>Check out tutorials on <a href="http://text2vec.org/">text2vec.org</a> where I’ll be updating documentation on a regular basis.</p>
<p>Below is the updated <strong>introduction to text mining with text2vec</strong>. No fancy word clouds. No Jane Austen. Enjoy.</p>
<h2>Text analysis pipeline</h2>
<p>Most text mining and NLP modeling use <a href="https://en.wikipedia.org/wiki/Bag-of-words_model">bag of words</a> or <a href="https://en.wikipedia.org/wiki/N-gram">bag of n-grams</a> methods. Despite their simplicity, these models usually demonstrate good performance on text categorization and classification tasks. But in contrast to their theoretical simplicity and practical efficiency building bag-of-words models involves technical challenges. This is especially the case in R because of its copy-on-modify semantics.</p>
<p>Let’s briefly review some of the steps in a typical text analysis pipeline:</p>
<ol><li>The researcher usually begins by constructing a <a href="https://en.wikipedia.org/wiki/Document-term_matrix">document-term matrix</a> (DTM) or term-co-occurrence matrix (TCM) from input documents. In other words, the first step is to vectorize text by creating a map from words or n-grams to a <a href="https://en.wikipedia.org/wiki/Vector_space_model">vector space</a>.</li>
<li>The researcher fits a model to that DTM. These models might include text classification, topic modeling, similarity search, etc. Fitting the model will include tuning and validating the model.</li>
<li>Finally the researcher applies the model to new data.</li>
</ol><p>In this vignette we will primarily discuss the first step. Texts themselves can take up a lot of memory, but vectorized texts usually do not, because they are stored as sparse matrices. Because of R’s copy-on-modify semantics, it is not easy to iteratively grow a DTM. Thus constructing a DTM, even for a small collections of documents, can be a serious bottleneck for analysts and researchers. It involves reading the whole collection of text documents into RAM and processing it as single vector, which can easily increase memory use by a factor of 2 to 4. The text2vec package solves this problem by providing a better way of constructing a document-term matrix.</p>
<p>Let’s demonstrate package core functionality by applying it to a real case problem - sentiment analysis.</p>
<p>text2vec package provides the <code>movie_review</code> dataset. It consists of 5000 movie reviews, each of which is marked as positive or negative. We will also use the <a href="https://cran.r-project.org/package=data.table">data.table</a> package for data wrangling.</p>
<p>First of all let’s split out dataset into two parts - train and test. We will show how to perform data manipulations on train set and then apply exactly the same manipulations on the test set:</p>
<pre><code><span>library</span><span>(</span><span>text2vec</span><span>)</span><span>
</span><span>library</span><span>(</span><span>data.table</span><span>)</span><span>
</span><span>data</span><span>(</span><span>&quot;movie_review&quot;</span><span>)</span><span>
</span><span>setDT</span><span>(</span><span>movie_review</span><span>)</span><span>
</span><span>setkey</span><span>(</span><span>movie_review</span><span>,</span><span> </span><span>id</span><span>)</span><span>
</span><span>set.seed</span><span>(</span><span>2016L</span><span>)</span><span>
</span><span>all_ids</span><span> </span><span>=</span><span> </span><span>movie_review</span><span>$</span><span>id</span><span>
</span><span>train_ids</span><span> </span><span>=</span><span> </span><span>sample</span><span>(</span><span>all_ids</span><span>,</span><span> </span><span>4000</span><span>)</span><span>
</span><span>test_ids</span><span> </span><span>=</span><span> </span><span>setdiff</span><span>(</span><span>all_ids</span><span>,</span><span> </span><span>train_ids</span><span>)</span><span>
</span><span>train</span><span> </span><span>=</span><span> </span><span>movie_review</span><span>[</span><span>J</span><span>(</span><span>train_ids</span><span>)]</span><span>
</span><span>test</span><span> </span><span>=</span><span> </span><span>movie_review</span><span>[</span><span>J</span><span>(</span><span>test_ids</span><span>)]</span></code></pre><h1>Vectorization</h1>
<p>To represent documents in vector space, we first have to create mappings from terms to term IDS. We call them terms instead of words because they can be arbitrary n-grams not just single words. We represent a set of documents as a sparse matrix, where each row corresponds to a document and each column corresponds to a term. This can be done in 2 ways: using the vocabulary itself or by <a href="https://en.wikipedia.org/wiki/Feature_hashing">feature hashing</a>.</p>
<h2>Vocabulary-based vectorization</h2>
<p>Let’s first create a vocabulary-based DTM. Here we collect unique terms from all documents and mark each of them with a unique ID using the <code>create_vocabulary()</code> function. We use an iterator to create the vocabulary.</p>
<pre><code><span># define preprocessing function and tokenization fucntion
</span><span>prep_fun</span><span> </span><span>=</span><span> </span><span>tolower</span><span>
</span><span>tok_fun</span><span> </span><span>=</span><span> </span><span>word_tokenizer</span><span>
</span><span>it_train</span><span> </span><span>=</span><span> </span><span>itoken</span><span>(</span><span>train</span><span>$</span><span>review</span><span>,</span><span> </span><span>preprocessor</span><span> </span><span>=</span><span> </span><span>prep_fun</span><span>,</span><span> </span><span>tokenizer</span><span> </span><span>=</span><span> </span><span>tok_fun</span><span>,</span><span> </span><span>ids</span><span> </span><span>=</span><span> </span><span>train</span><span>$</span><span>id</span><span>,</span><span> </span><span>progressbar</span><span> </span><span>=</span><span> </span><span>FALSE</span><span>)</span><span>
</span><span>vocab</span><span> </span><span>=</span><span> </span><span>create_vocabulary</span><span>(</span><span>it_train</span><span>)</span></code></pre><p>What was done here?</p>
<ol><li>We created an iterator over tokens with the <code>itoken()</code> function. All functions prefixed with <code>create_</code> work with these iterators. R users might find this idiom unusual, but the iterator abstraction allows us to hide most of details about input and to process data in memory-friendly chunks.</li>
<li>We built the vocabulary with the <code>create_vocabulary()</code> function.</li>
</ol><p>Alternatively, we could create list of tokens and reuse it in further steps. Each element of the list should represent a document, and each element should be a character vector of tokens.</p>
<pre><code><span>train_tokens</span><span> </span><span>=</span><span> </span><span>train</span><span>$</span><span>review</span><span> </span><span>%>%</span><span> </span><span>prep_fun</span><span> </span><span>%>%</span><span> </span><span>tok_fun</span><span>
</span><span>it_train</span><span> </span><span>=</span><span> </span><span>itoken</span><span>(</span><span>train_tokens</span><span>,</span><span> </span><span>ids</span><span> </span><span>=</span><span> </span><span>train</span><span>$</span><span>id</span><span>,</span><span>
</span><span># turn off progressbar because it won't look nice in rmd
</span><span> </span><span>progressbar</span><span> </span><span>=</span><span> </span><span>FALSE</span><span>)</span><span>
</span><span>vocab</span><span> </span><span>=</span><span> </span><span>create_vocabulary</span><span>(</span><span>it_train</span><span>)</span><span>
</span><span>vocab</span></code></pre><pre>
Number of docs: 4000 
0 stopwords:  ... 
ngram_min = 1; ngram_max = 1 
Vocabulary: 
                terms terms_counts doc_counts
    1:     overturned            1          1
    2: disintegration            1          1
    3:         vachon            1          1
    4:     interfered            1          1
    5:      michonoku            1          1
   ---                                       
35592:        penises            2          2
35593:        arabian            1          1
35594:       personal          102         94
35595:            end          921        743
35596:        address           10         10
</pre>
<p>Note that text2vec provides a few tokenizer functions (see <code>?tokenizers</code>). These are just simple wrappers for the <code>base::gsub()</code> function and are not very fast or flexible. If you need something smarter or faster you can use the <a href="https://cran.r-project.org/package=tokenizers">tokenizers</a> package which will cover most use cases, or write your own tokenizer using the <a href="https://cran.r-project.org/package=stringi">stringi</a> package.</p>
<p>Now that we have a vocabulary, we can construct a document-term matrix.</p>
<pre><code><span>vectorizer</span><span> </span><span>=</span><span> </span><span>vocab_vectorizer</span><span>(</span><span>vocab</span><span>)</span><span>
</span><span>t</span><span>1</span><span> </span><span>=</span><span> </span><span>Sys.time</span><span>()</span><span>
</span><span>dtm_train</span><span> </span><span>=</span><span> </span><span>create_dtm</span><span>(</span><span>it_train</span><span>,</span><span> </span><span>vectorizer</span><span>)</span><span>
</span><span>print</span><span>(</span><span>difftime</span><span>(</span><span>Sys.time</span><span>(),</span><span> </span><span>t</span><span>1</span><span>,</span><span> </span><span>units</span><span> </span><span>=</span><span> </span><span>'sec'</span><span>))</span></code></pre><pre>
Time difference of 0.800817 secs
</pre>
<p>Now we have a DTM and can check its dimensions.</p>
<pre><code><span>dim</span><span>(</span><span>dtm_train</span><span>)</span></code></pre><pre>
[1]  4000 35596
</pre>
<pre><code><span>identical</span><span>(</span><span>rownames</span><span>(</span><span>dtm_train</span><span>),</span><span> </span><span>train</span><span>$</span><span>id</span><span>)</span></code></pre><pre>
[1] TRUE
</pre>
<p>As you can see, the DTM has rows, equal to the number of documents, and columns, equal to the number of unique terms.</p>
<p>Now we are ready to fit our first model. Here we will use the <a href="https://cran.r-project.org/package=glmnet">glmnet</a> package to fit a logistic regression model with an L1 penalty and 4 fold cross-validation.</p>
<pre><code><span>library</span><span>(</span><span>glmnet</span><span>)</span><span>
</span><span>NFOLDS</span><span> </span><span>=</span><span> </span><span>4</span><span>
</span><span>t</span><span>1</span><span> </span><span>=</span><span> </span><span>Sys.time</span><span>()</span><span>
</span><span>glmnet_classifier</span><span> </span><span>=</span><span> </span><span>cv.glmnet</span><span>(</span><span>x</span><span> </span><span>=</span><span> </span><span>dtm_train</span><span>,</span><span> </span><span>y</span><span> </span><span>=</span><span> </span><span>train</span><span>[[</span><span>'sentiment'</span><span>]],</span><span> </span><span>family</span><span> </span><span>=</span><span> </span><span>'binomial'</span><span>,</span><span> </span><span># L1 penalty
</span><span> </span><span>alpha</span><span> </span><span>=</span><span> </span><span>1</span><span>,</span><span>
</span><span># interested in the area under ROC curve
</span><span> </span><span>type.measure</span><span> </span><span>=</span><span> </span><span>&quot;auc&quot;</span><span>,</span><span>
</span><span># 5-fold cross-validation
</span><span> </span><span>nfolds</span><span> </span><span>=</span><span> </span><span>NFOLDS</span><span>,</span><span>
</span><span># high value is less accurate, but has faster training
</span><span> </span><span>thresh</span><span> </span><span>=</span><span> </span><span>1e-3</span><span>,</span><span>
</span><span># again lower number of iterations for faster training
</span><span> </span><span>maxit</span><span> </span><span>=</span><span> </span><span>1e3</span><span>)</span><span>
</span><span>print</span><span>(</span><span>difftime</span><span>(</span><span>Sys.time</span><span>(),</span><span> </span><span>t</span><span>1</span><span>,</span><span> </span><span>units</span><span> </span><span>=</span><span> </span><span>'sec'</span><span>))</span></code></pre><pre>
Time difference of 3.485586 secs
</pre>
<pre><code><span>plot</span><span>(</span><span>glmnet_classifier</span><span>)</span></code></pre><p></p>
<pre><code><span>print</span><span>(</span><span>paste</span><span>(</span><span>&quot;max AUC =&quot;</span><span>,</span><span> </span><span>round</span><span>(</span><span>max</span><span>(</span><span>glmnet_classifier</span><span>$</span><span>cvm</span><span>),</span><span> </span><span>4</span><span>)))</span></code></pre><pre>
[1] &quot;max AUC = 0.923&quot;
</pre>
<p>We have successfully fit a model to our DTM. Now we can check the model’s performance on test data.
Note that we use exactly the same functions from prepossessing and tokenization. Also we reuse/use the same <code>vectorizer</code> - function which maps terms to indices.</p>
<pre><code><span># Note that most text2vec functions are pipe friendly!
</span><span>it_test</span><span> </span><span>=</span><span> </span><span>test</span><span>$</span><span>review</span><span> </span><span>%>%</span><span> </span><span>prep_fun</span><span> </span><span>%>%</span><span> </span><span>tok_fun</span><span> </span><span>%>%</span><span> </span><span>itoken</span><span>(</span><span>ids</span><span> </span><span>=</span><span> </span><span>test</span><span>$</span><span>id</span><span>,</span><span> </span><span># turn off progressbar because it won't look nice in rmd
</span><span> </span><span>progressbar</span><span> </span><span>=</span><span> </span><span>FALSE</span><span>)</span><span>
</span><span>dtm_test</span><span> </span><span>=</span><span> </span><span>create_dtm</span><span>(</span><span>it_test</span><span>,</span><span> </span><span>vectorizer</span><span>)</span><span>
</span><span>preds</span><span> </span><span>=</span><span> </span><span>predict</span><span>(</span><span>glmnet_classifier</span><span>,</span><span> </span><span>dtm_test</span><span>,</span><span> </span><span>type</span><span> </span><span>=</span><span> </span><span>'response'</span><span>)[,</span><span>1</span><span>]</span><span>
</span><span>glmnet</span><span>:::</span><span>auc</span><span>(</span><span>test</span><span>$</span><span>sentiment</span><span>,</span><span> </span><span>preds</span><span>)</span></code></pre><pre>
[1] 0.916697
</pre>
<p>As we can see, performance on the test data is roughly the same as we expect from cross-validation.</p>
<h3>Pruning vocabulary</h3>
<p>We can note, however, that the training time for our model was quite high. We can reduce it and also significantly improve accuracy by pruning the vocabulary.</p>
<p>For example, we can find words “a”, “the”, “in”, “I”, “you”, “on”, etc in almost all documents, but they do not provide much useful information. Usually such words are called <a href="https://en.wikipedia.org/wiki/Stop_words">stop words</a>. On the other hand, the corpus also contains very uncommon terms, which are contained in only a few documents. These terms are also useless, because we don’t have sufficient statistics for them. Here we will remove pre-defined stopwords, very common and very unusual terms.</p>
<pre><code><span>stop_words</span><span> </span><span>=</span><span> </span><span>c</span><span>(</span><span>&quot;i&quot;</span><span>,</span><span> </span><span>&quot;me&quot;</span><span>,</span><span> </span><span>&quot;my&quot;</span><span>,</span><span> </span><span>&quot;myself&quot;</span><span>,</span><span> </span><span>&quot;we&quot;</span><span>,</span><span> </span><span>&quot;our&quot;</span><span>,</span><span> </span><span>&quot;ours&quot;</span><span>,</span><span> </span><span>&quot;ourselves&quot;</span><span>,</span><span> </span><span>&quot;you&quot;</span><span>,</span><span> </span><span>&quot;your&quot;</span><span>,</span><span> </span><span>&quot;yours&quot;</span><span>)</span><span>
</span><span>t</span><span>1</span><span> </span><span>=</span><span> </span><span>Sys.time</span><span>()</span><span>
</span><span>vocab</span><span> </span><span>=</span><span> </span><span>create_vocabulary</span><span>(</span><span>it_train</span><span>,</span><span> </span><span>stopwords</span><span> </span><span>=</span><span> </span><span>stop_words</span><span>)</span><span>
</span><span>print</span><span>(</span><span>difftime</span><span>(</span><span>Sys.time</span><span>(),</span><span> </span><span>t</span><span>1</span><span>,</span><span> </span><span>units</span><span> </span><span>=</span><span> </span><span>'sec'</span><span>))</span></code></pre><pre>
Time difference of 0.439589 secs
</pre>
<pre><code><span>pruned_vocab</span><span> </span><span>=</span><span> </span><span>prune_vocabulary</span><span>(</span><span>vocab</span><span>,</span><span> </span><span>term_count_min</span><span> </span><span>=</span><span> </span><span>10</span><span>,</span><span> </span><span>doc_proportion_max</span><span> </span><span>=</span><span> </span><span>0.5</span><span>,</span><span>
</span><span>doc_proportion_min</span><span> </span><span>=</span><span> </span><span>0.001</span><span>)</span><span>
</span><span>vectorizer</span><span> </span><span>=</span><span> </span><span>vocab_vectorizer</span><span>(</span><span>pruned_vocab</span><span>)</span><span>
</span><span># create dtm_train with new pruned vocabulary vectorizer
</span><span>t</span><span>1</span><span> </span><span>=</span><span> </span><span>Sys.time</span><span>()</span><span>
</span><span>dtm_train</span><span> </span><span>=</span><span> </span><span>create_dtm</span><span>(</span><span>it_train</span><span>,</span><span> </span><span>vectorizer</span><span>)</span><span>
</span><span>print</span><span>(</span><span>difftime</span><span>(</span><span>Sys.time</span><span>(),</span><span> </span><span>t</span><span>1</span><span>,</span><span> </span><span>units</span><span> </span><span>=</span><span> </span><span>'sec'</span><span>))</span></code></pre><pre>
Time difference of 0.6738439 secs
</pre>
<pre><code><span>dim</span><span>(</span><span>dtm_train</span><span>)</span></code></pre><pre>
[1] 4000 6585
</pre>
<p>Note that the new DTM has many fewer columns than the original DTM. This usually leads to both accuracy improvement (because we removed “noise”) and reduction of the training time.</p>
<p>Also we need to create DTM for test data with the same vectorizer:</p>
<pre><code><span>dtm_test</span><span> </span><span>=</span><span> </span><span>create_dtm</span><span>(</span><span>it_test</span><span>,</span><span> </span><span>vectorizer</span><span>)</span><span>
</span><span>dim</span><span>(</span><span>dtm_test</span><span>)</span></code></pre><pre>
[1] 1000 6585
</pre>
<h2>N-grams</h2>
<p>Can we improve the model? Definitely - we can use n-grams instead of words. Here we will use up to 2-grams:</p>
<pre><code><span>t</span><span>1</span><span> </span><span>=</span><span> </span><span>Sys.time</span><span>()</span><span>
</span><span>vocab</span><span> </span><span>=</span><span> </span><span>create_vocabulary</span><span>(</span><span>it_train</span><span>,</span><span> </span><span>ngram</span><span> </span><span>=</span><span> </span><span>c</span><span>(</span><span>1L</span><span>,</span><span> </span><span>2L</span><span>))</span><span>
</span><span>print</span><span>(</span><span>difftime</span><span>(</span><span>Sys.time</span><span>(),</span><span> </span><span>t</span><span>1</span><span>,</span><span> </span><span>units</span><span> </span><span>=</span><span> </span><span>'sec'</span><span>))</span></code></pre><pre>
Time difference of 1.47972 secs
</pre>
<pre><code><span>vocab</span><span> </span><span>=</span><span> </span><span>vocab</span><span> </span><span>%>%</span><span> </span><span>prune_vocabulary</span><span>(</span><span>term_count_min</span><span> </span><span>=</span><span> </span><span>10</span><span>,</span><span> </span><span>doc_proportion_max</span><span> </span><span>=</span><span> </span><span>0.5</span><span>)</span><span>
</span><span>bigram_vectorizer</span><span> </span><span>=</span><span> </span><span>vocab_vectorizer</span><span>(</span><span>vocab</span><span>)</span><span>
</span><span>dtm_train</span><span> </span><span>=</span><span> </span><span>create_dtm</span><span>(</span><span>it_train</span><span>,</span><span> </span><span>bigram_vectorizer</span><span>)</span><span>
</span><span>t</span><span>1</span><span> </span><span>=</span><span> </span><span>Sys.time</span><span>()</span><span>
</span><span>glmnet_classifier</span><span> </span><span>=</span><span> </span><span>cv.glmnet</span><span>(</span><span>x</span><span> </span><span>=</span><span> </span><span>dtm_train</span><span>,</span><span> </span><span>y</span><span> </span><span>=</span><span> </span><span>train</span><span>[[</span><span>'sentiment'</span><span>]],</span><span> </span><span>family</span><span> </span><span>=</span><span> </span><span>'binomial'</span><span>,</span><span> </span><span>alpha</span><span> </span><span>=</span><span> </span><span>1</span><span>,</span><span>
</span><span>type.measure</span><span> </span><span>=</span><span> </span><span>&quot;auc&quot;</span><span>,</span><span>
</span><span>nfolds</span><span> </span><span>=</span><span> </span><span>NFOLDS</span><span>,</span><span>
</span><span>thresh</span><span> </span><span>=</span><span> </span><span>1e-3</span><span>,</span><span>
</span><span>maxit</span><span> </span><span>=</span><span> </span><span>1e3</span><span>)</span><span>
</span><span>print</span><span>(</span><span>difftime</span><span>(</span><span>Sys.time</span><span>(),</span><span> </span><span>t</span><span>1</span><span>,</span><span> </span><span>units</span><span> </span><span>=</span><span> </span><span>'sec'</span><span>))</span></code></pre><pre>
Time difference of 2.973802 secs
</pre>
<pre><code><span>plot</span><span>(</span><span>glmnet_classifier</span><span>)</span></code></pre><p></p>
<pre><code><span>print</span><span>(</span><span>paste</span><span>(</span><span>&quot;max AUC =&quot;</span><span>,</span><span> </span><span>round</span><span>(</span><span>max</span><span>(</span><span>glmnet_classifier</span><span>$</span><span>cvm</span><span>),</span><span> </span><span>4</span><span>)))</span></code></pre><pre>
[1] &quot;max AUC = 0.9217&quot;
</pre>
<p>Seems that usage of n-grams improved our model a little bit more. Let’s check performance on test dataset:</p>
<pre><code><span># apply vectorizer
</span><span>dtm_test</span><span> </span><span>=</span><span> </span><span>create_dtm</span><span>(</span><span>it_test</span><span>,</span><span> </span><span>bigram_vectorizer</span><span>)</span><span>
</span><span>preds</span><span> </span><span>=</span><span> </span><span>predict</span><span>(</span><span>glmnet_classifier</span><span>,</span><span> </span><span>dtm_test</span><span>,</span><span> </span><span>type</span><span> </span><span>=</span><span> </span><span>'response'</span><span>)[,</span><span>1</span><span>]</span><span>
</span><span>glmnet</span><span>:::</span><span>auc</span><span>(</span><span>test</span><span>$</span><span>sentiment</span><span>,</span><span> </span><span>preds</span><span>)</span></code></pre><pre>
[1] 0.9268974
</pre>
<p>Further tuning is left up to the reader.</p>
<h2>Feature hashing</h2>
<p>If you are not familiar with feature hashing (the so-called “hashing trick”) I recommend you start with the <a href="https://en.wikipedia.org/wiki/Feature_hashing">Wikipedia article</a>, then read the <a href="http://alex.smola.org/papers/2009/Weinbergeretal09.pdf">original paper</a> by a Yahoo! research team. This technique is very fast because we don’t have to perform a lookup over an associative array. Another benefit is that it leads to a very low memory footprint, since we can map an arbitrary number of features into much more compact space. This method was popularized by Yahoo! and is widely used in <a href="https://github.com/JohnLangford/vowpal_wabbit/">Vowpal Wabbit</a>.</p>
<p>Here is how to use feature hashing in text2vec.</p>
<pre><code><span>h_vectorizer</span><span> </span><span>=</span><span> </span><span>hash_vectorizer</span><span>(</span><span>hash_size</span><span> </span><span>=</span><span> </span><span>2</span><span> </span><span>^</span><span> </span><span>14</span><span>,</span><span> </span><span>ngram</span><span> </span><span>=</span><span> </span><span>c</span><span>(</span><span>1L</span><span>,</span><span> </span><span>2L</span><span>))</span><span>
</span><span>t</span><span>1</span><span> </span><span>=</span><span> </span><span>Sys.time</span><span>()</span><span>
</span><span>dtm_train</span><span> </span><span>=</span><span> </span><span>create_dtm</span><span>(</span><span>it_train</span><span>,</span><span> </span><span>h_vectorizer</span><span>)</span><span>
</span><span>print</span><span>(</span><span>difftime</span><span>(</span><span>Sys.time</span><span>(),</span><span> </span><span>t</span><span>1</span><span>,</span><span> </span><span>units</span><span> </span><span>=</span><span> </span><span>'sec'</span><span>))</span></code></pre><pre>
Time difference of 1.51502 secs
</pre>
<pre><code><span>t</span><span>1</span><span> </span><span>=</span><span> </span><span>Sys.time</span><span>()</span><span>
</span><span>glmnet_classifier</span><span> </span><span>=</span><span> </span><span>cv.glmnet</span><span>(</span><span>x</span><span> </span><span>=</span><span> </span><span>dtm_train</span><span>,</span><span> </span><span>y</span><span> </span><span>=</span><span> </span><span>train</span><span>[[</span><span>'sentiment'</span><span>]],</span><span> </span><span>family</span><span> </span><span>=</span><span> </span><span>'binomial'</span><span>,</span><span> </span><span>alpha</span><span> </span><span>=</span><span> </span><span>1</span><span>,</span><span>
</span><span>type.measure</span><span> </span><span>=</span><span> </span><span>&quot;auc&quot;</span><span>,</span><span>
</span><span>nfolds</span><span> </span><span>=</span><span> </span><span>5</span><span>,</span><span>
</span><span>thresh</span><span> </span><span>=</span><span> </span><span>1e-3</span><span>,</span><span>
</span><span>maxit</span><span> </span><span>=</span><span> </span><span>1e3</span><span>)</span><span>
</span><span>print</span><span>(</span><span>difftime</span><span>(</span><span>Sys.time</span><span>(),</span><span> </span><span>t</span><span>1</span><span>,</span><span> </span><span>units</span><span> </span><span>=</span><span> </span><span>'sec'</span><span>))</span></code></pre><pre>
Time difference of 4.494137 secs
</pre>
<pre><code><span>plot</span><span>(</span><span>glmnet_classifier</span><span>)</span></code></pre><p></p>
<pre><code><span>print</span><span>(</span><span>paste</span><span>(</span><span>&quot;max AUC =&quot;</span><span>,</span><span> </span><span>round</span><span>(</span><span>max</span><span>(</span><span>glmnet_classifier</span><span>$</span><span>cvm</span><span>),</span><span> </span><span>4</span><span>)))</span></code></pre><pre>
[1] &quot;max AUC = 0.8937&quot;
</pre>
<pre><code><span>dtm_test</span><span> </span><span>=</span><span> </span><span>create_dtm</span><span>(</span><span>it_test</span><span>,</span><span> </span><span>h_vectorizer</span><span>)</span><span>
</span><span>preds</span><span> </span><span>=</span><span> </span><span>predict</span><span>(</span><span>glmnet_classifier</span><span>,</span><span> </span><span>dtm_test</span><span> </span><span>,</span><span> </span><span>type</span><span> </span><span>=</span><span> </span><span>'response'</span><span>)[,</span><span> </span><span>1</span><span>]</span><span>
</span><span>glmnet</span><span>:::</span><span>auc</span><span>(</span><span>test</span><span>$</span><span>sentiment</span><span>,</span><span> </span><span>preds</span><span>)</span></code></pre><pre>
[1] 0.9036685
</pre>
<p>As you can see our AUC is a bit worse but DTM construction time is considerably lower. On large collections of documents this can be a significant advantage.</p>
<h1>Basic transformations</h1>
<p>Before doing analysis it usually can be useful to transform DTM. For example lengths of the documents in collection can significantly vary. In this case it can be useful to apply normalization.</p>
<h2>Normalization</h2>
<p>By “normalization” we assume transformation of the rows of DTM so we adjust values measured on different scales to a notionally common scale. For the case when length of the documents vary we can apply “L1” normalization. It means we will transform rows in a way that <code>sum</code> of the row values will be equal to <code>1</code>:</p>
<pre><code><span>dtm_train_l1_norm</span><span> </span><span>=</span><span> </span><span>normalize</span><span>(</span><span>dtm_train</span><span>,</span><span> </span><span>&quot;l1&quot;</span><span>)</span></code></pre><p>By this transformation we should improve the quality of data preparation.</p>
<h2>TF-IDF</h2>
<p>Another popular technique is <a href="https://en.wikipedia.org/wiki/Tf%E2%80%93idf">TF-IDF</a> transformation.
We can (and usually should) apply it to our DTM. It will not only normalize DTM, but also increase the weight of terms which are specific to a single document or handful of documents and decrease the weight for terms used in most documents:</p>
<pre><code><span>vocab</span><span> </span><span>=</span><span> </span><span>create_vocabulary</span><span>(</span><span>it_train</span><span>)</span><span>
</span><span>vectorizer</span><span> </span><span>=</span><span> </span><span>vocab_vectorizer</span><span>(</span><span>vocab</span><span>)</span><span>
</span><span>dtm_train</span><span> </span><span>=</span><span> </span><span>create_dtm</span><span>(</span><span>it_train</span><span>,</span><span> </span><span>vectorizer</span><span>)</span><span>
</span><span># define tfidf model
</span><span>tfidf</span><span> </span><span>=</span><span> </span><span>TfIdf</span><span>$</span><span>new</span><span>()</span><span>
</span><span># fit model to train data and transform train data with fitted model
</span><span>dtm_train_tfidf</span><span> </span><span>=</span><span> </span><span>fit_transform</span><span>(</span><span>dtm_train</span><span>,</span><span> </span><span>tfidf</span><span>)</span><span>
</span><span># tfidf modified by fit_transform() call!
# apply pre-trained tf-idf transformation to test data
</span><span>dtm_test_tfidf</span><span> </span><span>=</span><span> </span><span>create_dtm</span><span>(</span><span>it_test</span><span>,</span><span> </span><span>vectorizer</span><span>)</span><span> </span><span>%>%</span><span> </span><span>transform</span><span>(</span><span>tfidf</span><span>)</span></code></pre><p>Note that here we first time touched model object in text2vec. At this moment the user should remember several important things about text2vec models:</p>
<ol><li>Models can be fitted on a given data (train) and applied to unseen data (test)</li>
<li><strong>Models are mutable</strong> - once you will pass model to <code>fit()</code> or <code>fit_transform()</code> function, model will be modifed by it.</li>
<li>After model is fitted, it can be applied to a new data with <code>transform(new_data, fitted_model)</code> method.</li>
</ol><p>More detailed overview of models and models API will be available soon in a separate vignette.</p>
<p>Once we have tf-idf reweighted DTM we can fit our linear classifier again:</p>
<pre><code><span>t</span><span>1</span><span> </span><span>=</span><span> </span><span>Sys.time</span><span>()</span><span>
</span><span>glmnet_classifier</span><span> </span><span>=</span><span> </span><span>cv.glmnet</span><span>(</span><span>x</span><span> </span><span>=</span><span> </span><span>dtm_train_tfidf</span><span>,</span><span> </span><span>y</span><span> </span><span>=</span><span> </span><span>train</span><span>[[</span><span>'sentiment'</span><span>]],</span><span> </span><span>family</span><span> </span><span>=</span><span> </span><span>'binomial'</span><span>,</span><span> </span><span>alpha</span><span> </span><span>=</span><span> </span><span>1</span><span>,</span><span>
</span><span>type.measure</span><span> </span><span>=</span><span> </span><span>&quot;auc&quot;</span><span>,</span><span>
</span><span>nfolds</span><span> </span><span>=</span><span> </span><span>NFOLDS</span><span>,</span><span>
</span><span>thresh</span><span> </span><span>=</span><span> </span><span>1e-3</span><span>,</span><span>
</span><span>maxit</span><span> </span><span>=</span><span> </span><span>1e3</span><span>)</span><span>
</span><span>print</span><span>(</span><span>difftime</span><span>(</span><span>Sys.time</span><span>(),</span><span> </span><span>t</span><span>1</span><span>,</span><span> </span><span>units</span><span> </span><span>=</span><span> </span><span>'sec'</span><span>))</span></code></pre><pre>
Time difference of 3.033687 secs
</pre>
<pre><code><span>plot</span><span>(</span><span>glmnet_classifier</span><span>)</span></code></pre><p></p>
<pre><code><span>print</span><span>(</span><span>paste</span><span>(</span><span>&quot;max AUC =&quot;</span><span>,</span><span> </span><span>round</span><span>(</span><span>max</span><span>(</span><span>glmnet_classifier</span><span>$</span><span>cvm</span><span>),</span><span> </span><span>4</span><span>)))</span></code></pre><pre>
[1] &quot;max AUC = 0.9146&quot;
</pre>
<p>Let’s check the model performance on the test dataset:</p>
<pre><code><span>preds</span><span> </span><span>=</span><span> </span><span>predict</span><span>(</span><span>glmnet_classifier</span><span>,</span><span> </span><span>dtm_test_tfidf</span><span>,</span><span> </span><span>type</span><span> </span><span>=</span><span> </span><span>'response'</span><span>)[,</span><span>1</span><span>]</span><span>
</span><span>glmnet</span><span>:::</span><span>auc</span><span>(</span><span>test</span><span>$</span><span>sentiment</span><span>,</span><span> </span><span>preds</span><span>)</span></code></pre><pre>
[1] 0.9053246
</pre>
<p>Usually tf-idf transformation <strong>significantly</strong> improve performance on most of the dowstream tasks.</p>
<h1>What’s next</h1>
<p>Try <code>text2vec</code>, share your thoughts in comments. I’m waiting for feedback.</p>