Author(s): Francesco Fumagalli Natural Language Processing Natural Language Clustering — Part 1 Help chatbots deal with FAQs, with Python code for Tokenization, GloVe, and TF-IDF Classifying things comes quite natural to us: our books, movies and music all have genres; the things we study are split between different subjects and even the food we eat belongs to different cuisines!In recent years we’ve been able to develop better and better algorithms to classify text: models like BERT-ITPT-FiT (BERT + withIn-Task Pre-Training + Fine-Tuning) or XL-NET seem to be reigning champions in this category, at least in the 29 benchmark datasets available on PapersWithCode. In recent years we’ve been able to develop better and better algorithms to classify text: models like BERT-ITPT-FiT (BERT + withIn-Task Pre-Training + Fine-Tuning) or XL-NET seem to be reigning champions in this category, at least in the 29 benchmark datasets available on PapersWithCode. image by author But what if we don’t know the available categories for the texts we want to analyze? Take for example a corpus of conversations or a collection of books or articles that all belong to different specializations within the same subject: labels aren’t always as clear cut as spam / not spam, we may not have any idea of how many or what kind of labels to expect, or normal pre-trained classification methods wouldn’t have the in-depth domain knowledge required not to classify them as all the same, while not enough material, time or computational power is available to fine-tune a Transformer model. Perhaps one of the most obvious examples is categorizing incoming messages into FAQs: not every message can be answered via a FAQ, but we’d like to associate as many as possible to the right answer, to minimize the amount we leave unanswered or address manually, and dynamically improve our FAQs adding new questions that become relevant. Let’s see how! From words to numerical values: Tokenization and Embedding The first thing we have to do, as always when dealing with Natural Language, is transforming the text into something our algorithms can understand. Paragraphs and sentences are split into their basic components, words, in a process called tokenization. But words themselves can’t be fed to a computational model since mathematical operations can’t be applied to them, which explains the need to have them mapped onto vectors of real numbers: basically transformed into sequences of numbers which represent them in a mathematical space, also preserving the relationships between them (a process known as word embedding, a term coming from mathematics to identify an injective and structure-preserving map). It sounds complex, but there are simple ways to do this, such as the one shown below: starting from three sentences (on the left) their tokens are extracted and a very simple embedding is performed (on the right) by counting the occurrences of each token (word) in the original texts Tokenization and a simple Count Vector Embedding from three short texts. In the example above the word startup is embedded as [1,0,1] because it appears once in the first and third text but not in the second one, whereas the word “in” is embedded as [0,2,0] because it only appears in the second sentence, twice. Note that it’s not a very practical example: given the small number of inputs different words end up having the same embedding (for example “specialized” and “in” both appear in the second text only, so they share the same embedding [0,1,0]), but we hope it helps you get a better understanding of how the process works. Also note that the tokens are all lowercase and the punctuation has been removed: that’s common practice, although it depends on the kind of text we’re dealing with. Another common practice not shown in the example above is the removal of stopwords, words so common they aren’t relevant for our purpose and very likely to be found an abundant number of times in most texts we come across (such as “a”, “the” or “is”). This approach, however, provides the advantage of embedding documents themselves at the same time as words: reading the table vertically, Text1 can be read as [1,1,1,1,1,1,0,0,0,0]. This makes it easy to confront different sources by introducing a distance metric. Tokenization Code Example in Python using nltk: # !pip install nltk from nltk.tokenize import word_tokenize text = “Digitiamo is a Startup from Italy” tokenized_text = nltk.word_tokenize(text) print(tokenized_text) # OUTPUT:# [‘Digitiamo’, ‘is’, ‘a’, ‘Startup’, ‘from’, ‘Italy’] Embedding Code Example in Python using gensim’s Word2Vec # !pip install –upgrade gensim from gensim.models import Word2Vecmodel = Word2Vec([tokenized_text], min_count=1)””” min_count (int, optional) – Ignores all words with total frequency lower than this. Word2Vec is optimized to work with multiple texts at the same time in the form of a list of texts. When working with a single text, it should be put into a list, which explains the square brackets around tokenized_text “”” print(list(model.wv.vocab)) # OUTPUT:# [‘Digitiamo’, ‘is’, ‘a’, ‘Startup’, ‘from’, ‘Italy’] Showing the embedding for a word. Word2Vec’s default parameters use a 100-dimensional embedding Commonly used Embeddings: TF-IDF and GloVe TF-IDF stands for Term Frequency–Inverse Document Frequency and it emphasizes the importance of each term based on how many times it appears in other texts from the corpus. It can be very useful to embed texts within a specialized domain for the purpose of clusterization. The words that appear in every document, no matter how infrequent they are in everyday usage, will be given very little weight, whereas words that only appear in some documents will be given more weight, making it easier to recognize possible clusters.For example, if we were trying to embed a bunch of papers about endocrinology, the word “endocrinology” itself would likely be mentioned multiple times in each paper, making its presence not very helpful for the purpose of distinguishing them. GloVe (Global Vectors for word representation) is an unsupervised algorithm that links words based on how often they occur together, trying to map the underlying meaning. You can see some examples of relationships between words obtained through GloVe in the image below: “man” is approximately as distant from “woman” as “king” is from “queen” going in the same direction, as they have the same meaning for different genders (and vice versa, of course). The same happens for comparatives and superlatives in the fourth panel: the vector that takes you from “clear” to “clearer”, if applied to “soft”, brings you to “softer”.Depending on the size of your corpus the Mittens library could be a good complement to GloVe: according to their GitHub, it’s “useful for domains that require specialized representations but lack sufficient data to train them from scratch. Mittens starts with the general-purpose pretrained representations and tunes them to a specialized domain”, it’s also compatible with Numpy and TensorFlow. (image from the official GloVe source: https://nlp.stanford.edu/projects/glove/ ) Excellent! Now that we’ve transformed our words into numerical vectors and mapped them onto a space, if we’ve used an embedding that only works for words and not full texts it’s time to go back to sentences: remember our original goal? We don’t just want to map words, but try to cluster the texts or sentences they’re in. There are many way to do this and some may work better than others, depending on the context. Perhaps the easiest one, that’s also quite effective for short text, would
