The visualization of complex data sets is of essential importance in communicating your data products. Beyond pie charts, histograms, line graphs and other common forms of visual communication begins the reign of data sets that encompass too much information to be easily captured by these simple data displays. A typical context that abounds with complexity is found in the areas of text mining, natural language processing, and cognitive computing in general; such a complex context for data presentation is pervasive in an attempt to build a visual interface for products like semantic search engines or recommendation engines. For example, statistical models like LDA (Latent Dirichlet Allocation) enable for a thorough insight into the similarity structure across textual documents or vocabulary terms used to describe them. But as the number of pairwise similarities between terms of documents to be presented to the end users increases, the problem of effective data visualization becomes harder. In this and the following blog posts, I wish to present and discuss some of the well-known approaches to visualize difficult data sets from the viewpoint of pure ergonomics, as well as to suggest some improvements and new ideas here and there.
A simple similarity map for iris
In this first blog on this topic, the idea is to focus on the basics of ergonomics in the visualization of similarity information. We will compare two ideas: the usage of similarity maps following a dimensionality reduction of the dataset - which can be taken as a sort of baseline idea in the visualization of similarity data - and the usage of directed graphs from higher-dimensional spaces directly, which is less common. I would like to put forward an argument in support of the later approach which I started using some time ago.
But first, let us have some data. I will use a well known, rather simple data set, merely for the purposes of the illustration here: namely, the famous iris:
# - the iris dataset
data(iris)head(iris)
Sepal.Length Sepal.Width Petal.Length Petal.Width Species
1          5.1         3.5          1.4         0.2  setosa
2          4.9         3.0          1.4         0.2  setosa
3          4.7         3.2          1.3         0.2  setosa
4          4.6         3.1          1.5         0.2  setosa
5          5.0         3.6          1.4         0.2  setosa
6          5.4         3.9          1.7         0.4  setosa
The iris dataset with its four numerical and one categorical variable is, as you will see, complicated quite enough to illustrate the main problem that I wish to discuss. Now, let’s illustrate the problem. The classical approach to assess the similarity between the different observations (rows) in this dataset would be to derive some metric from the numerical variables, perform dimensionality reduction, and then plot the data points in a lower-dimensional space. I will work with Euclidean distance and a non-linear dimensionality reduction approach by performing ordinal multidimensional scaling (MDS) with {smacof} in R:
# - distances in iris:dIris