<p><div><div><div><div data-show-faces="false" data-height="21" data-width="100" data-layout="button_count" data-send="true" data-href="https://www.r-bloggers.com/t-sne-dimension-reduction-on-spotify-mp3-samples/"></div></div><div></div></div><div></div></div>
<div>
(This article was first published on <strong><a href="https://longhowlam.wordpress.com/2018/01/31/t-sne-dimension-reduction-on-spotify-mp3-samples/"> R – Longhow Lam's Blog</a></strong>, and kindly contributed to <a rel="nofollow" href="https://www.r-bloggers.com/">R-bloggers)</a> </div>
<p><img data-orig-file="https://longhowlam.files.wordpress.com/2018/01/schermafdruk-van-2018-01-30-12-25-08.png?w=450&amp;h=361&amp;fit=470%2C361" data-orig-size="385,296" data-medium-file="https://longhowlam.files.wordpress.com/2018/01/schermafdruk-van-2018-01-30-12-25-08.png?w=450&amp;h=361&amp;fit=470%2C361?w=300" data-attachment-id="2575" data-image-meta="{" data-image-description="" src="https://longhowlam.files.wordpress.com/2018/01/schermafdruk-van-2018-01-30-12-25-08.png?w=450&h=361&fit=470%2C361" data-permalink="https://longhowlam.wordpress.com/2018/01/31/t-sne-dimension-reduction-on-spotify-mp3-samples/schermafdruk-van-2018-01-30-12-25-08/" data-large-file="https://longhowlam.files.wordpress.com/2018/01/schermafdruk-van-2018-01-30-12-25-08.png?w=450&amp;h=361&amp;fit=470%2C361?w=385" alt="Schermafdruk van 2018-01-30 12-25-08" data-image-title="Schermafdruk van 2018-01-30 12-25-08" sizes="(max-width: 470px) 100vw, 470px" data-comments-opened="1" class="wp-image-2575" data-recalc-dims="1"></p>
<h2>Introduction</h2>
<p>Not long ago I was reading on t-Distributed Stochastic Neighbor Embedding (t-sne), a very interesting dimension reduction technique, and on Mel frequency cepstrum a sound processing technique. Details of both techniques can be found <a rel="nofollow" href="https://en.wikipedia.org/wiki/T-distributed_stochastic_neighbor_embedding" target="_blank">here</a> and <a rel="nofollow" href="https://en.wikipedia.org/wiki/Mel-frequency_cepstrum" target="_blank">here</a>. Can we combine the two in a data analysis exercise? Yes, and with not too much R code you can already quickly create some visuals to get <strong>‘musical’ insights</strong>.</p>
<h2>Spotify Data</h2>
<p>Where can you get some sample audio files? Spotify! There is a <a rel="nofollow" href="https://developer.spotify.com/web-api/" target="_blank">Spotify API</a> which allows you to get information on playlists, artists, tracks, etc. Moreover, for many songs (not all though) Spotify provides downloadable preview mp3’s of <strong>30 seconds</strong>. The link to the preview mp3 can be retrieved from the API. I am going to use some of these mp3’s for analysis.</p>
<p>In the web interface of Spotify you can look for interesting playlists. In the search field type in for example ‘<strong>Bach</strong>‘ (my favorite classical composer). In the search results go to the playlists tab, you’ll find many ‘Bach’ playlists from different users, including the ‘user’ Spotify itself. Now, given the user_id (<strong>spotify</strong>) and the specific playlist_id (<strong>37i9dQZF1DWZnzwzLBft6A</strong> for the Bach playlist from Spotify) we can extract all the songs using the API:</p>
<pre> GET https:<span>//</span>api.spotify.com<span>/v1/users/</span>{user_id}<span>/playlists/</span>{playlist_id}
</pre>
<p>You will get the 50 Bach songs from the playlist, most of them have a preview mp3. Let’s also get the songs from a <strong>Heavy Metal</strong> play list, and a <strong>Michael Jackson</strong> play list. In total I have 146 songs with preview mp3’s in three ‘categories’:</p>
<ul>
<li>Bach,</li>
<li>Heavy Metal,</li>
<li>Michael Jackson.</li>
</ul>
<h2>Transforming audio mp3’s to features</h2>
<p>The mp3 files need to be transformed to data that I can use for machine learning, I am going to use the Python <a rel="nofollow" href="https://github.com/librosa/librosa" target="_blank">librosa</a> package to do this. It is easy to call it from R using the reticulate package.</p>
<pre>library(reticulate)
librosa = <span>import</span>(<span>&quot;librosa&quot;</span>)

#### python environment <span>with</span> librosa <span>module</span> installed
use_python(python = <span>&quot;/usr/bin/python3&quot;</span>)
</pre>
<p>The downloaded preview mp3’s have a sample rate of 22.050. So a 30 second audio file has in total 661.500 raw audio data points.</p>
<pre>onemp3 = librosa$load(&quot;mp3songs/bach1.mp3&quot;)

length(onemp3[[1]])
length(onemp3[[1]])/onemp3[[2]]  # ~30 seconds sound

<span>## 5 seconds plot</span>
pp = 5*onemp3[[2]]
plot(onemp3[<span>[1</span>]][1:pp], type=&quot;l&quot;)
</pre>
<p>A line plot of the raw audio values will look like.</p>
<div><img src="https://i2.wp.com/media-exp2.licdn.com/mpr/mpr/AAMABADGAAgAAQAAAAAAABHrAAAAJGIyZmYwMjk2LTg4YzctNGY5ZC1hMDBhLWRhM2JiN2M3ZmI4MA.png?w=456&ssl=1" data-recalc-dims="1"></div>
<p> </p>
<p>For sound processing, features extraction on the raw audio signal is often applied first. A commonly used feature extraction method is Mel-Frequency Cepstral Coefficients (MFCC). We can calculate the MFCC for a song with librosa.</p>
<pre>ff = librosa<span>$feature</span>
mel = librosa<span>$logamplitude</span>(
  ff<span>$melspectrogram</span>(
    onemp3[[1]], 
    sr = onemp3[[2]],
    n_mels=96
  ),
  ref_power=1.0
)
image(mel)
</pre>
<div><img src="https://i0.wp.com/media-exp2.licdn.com/mpr/mpr/AAMABADGAAgAAQAAAAAAAA8aAAAAJDk5NTY5MzM4LTFkZTktNDc5ZS1hMzMzLWQ5NjZkM2JkNjAxMA.png?w=450&ssl=1" data-recalc-dims="1"></div>
<p> </p>
<p>Each mp3 is now a matrix of MFC Coefficients as shown in the figure above. We have less data points than the original 661.500 data points but still quit a lot. In our example the MFCC are a 96 by 1292 matrix, so 124.032 values. We apply a the t-sne dimension reduction on the MFCC values.</p>
<h2>Calculating t-sne</h2>
<p>A simple and easy approach, each matrix is just flattened. So a song becomes a vector of length 124.032. The data set on which we apply t-sne consist of 146 records with 124.032 columns, which we will reduce to 3 columns with the Rtsne package:</p>
<pre>tsne_out = Rtsne(AllSongsMFCCMatrix, dims=3) 
</pre>
<p>The output object contains the 3 columns, I have joined it back with the data of the artists and song names so that I can create an interactive 3D scatter plot with R plotly. Below is a screen shot, the interactive one can be found <a rel="nofollow" href="https://longhowlam.github.io/songs.html" target="_blank">here</a>.</p>
<div><img src="https://i1.wp.com/media-exp2.licdn.com/mpr/mpr/AAMABADGAAgAAQAAAAAAABHaAAAAJDUzMTBjYjliLTIyZWEtNDg5Mi04OGYxLWEyNzEwMmZlNWQzMQ.png?w=456&ssl=1" data-recalc-dims="1"></div>
<h2>Conclusion</h2>
<p>It is obvious that Bach music, heavy metal and Michael Jackson are different, you don’t need machine learning to hear that. So as expected, it turns out that a straight forward dimension reduction on these songs with MFCC and t-sne clearly shows the differences in a 3D space. Some Michael Jackson songs are very close to heavy metal <img alt="🙂" src="https://i0.wp.com/s0.wp.com/wp-content/mu-plugins/wpcom-smileys/twemoji/2/72x72/1f642.png?w=456&ssl=1" class="wp-smiley" data-recalc-dims="1"> The complete R code can be found <a rel="nofollow" href="https://github.com/longhowlam/BitsAndPieces/blob/master/Playlists_tsne.R" target="_blank">here</a>.</p>
<p>Cheers, Longhow</p>
	<p>
<div>
<div>To <strong>leave a comment</strong> for the author, please follow the link and comment on their blog: <strong><a href="https://longhowlam.wordpress.com/2018/01/31/t-sne-dimension-reduction-on-spotify-mp3-samples/"> R – Longhow Lam's Blog</a></strong>.</div>
<hr>
<a rel="nofollow" href="https://www.r-bloggers.com/">R-bloggers.com</a> offers <strong><a rel="nofollow" href="https://feedburner.google.com/fb/a/mailverify?uri=RBloggers">daily e-mail updates</a></strong> about <a rel="nofollow" href="https://www.r-project.org/" title="The R Project for Statistical Computing">R</a> news and <a rel="nofollow" href="https://www.r-bloggers.com/search/tutorial" title="R tutorials">tutorials</a> on topics such as: <a rel="nofollow" href="https://www.r-bloggers.com/search/data%20science" title="Data science">Data science</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/Big%20Data" title="Big Data">Big Data, <a rel="nofollow" href="https://www.r-users.com/" title="R jobs">R jobs</a>, visualization (<a rel="nofollow" href="https://www.r-bloggers.com/search/ggplot2" title="ggplot and ggplot2 tutorials">ggplot2</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/boxplot" title="Boxplots using lattice and ggplot2 tutorials">Boxplots</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/map" title="Maps and gis">maps</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/animation" title="Animation in R">animation</a>), programming (<a rel="nofollow" href="https://www.r-bloggers.com/search/RStudio" title="RStudio IDE for R">RStudio</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/sweave" title="Sweave and literate programming">Sweave</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/LaTeX" title="LaTeX in R">LaTeX</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/SQL" title="SQL and databases">SQL</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/eclipse" title="Eclipse IDE for R">Eclipse</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/git" title="git and github, Version Control System">git</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/hadoop" title="Large data in R using Hadoop">hadoop</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/Web+Scraping" title="Web Scraping of google, facebook, yahoo, twitter and more using R">Web Scraping</a>) statistics (<a rel="nofollow" href="https://www.r-bloggers.com/search/regression" title="Regressions and ANOVA analysis tutorials">regression</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/PCA" title="principal component analysis tutorial">PCA</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/time+series" title="Time series">time series</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/trading" title="finance trading">trading</a>) and more...
<img alt="" width="1" src="http://feeds.feedburner.com/~r/RBloggers/~4/IMayOPDb4eI" height="1"></a></div>