Fun with word embeddings

In this tutorial, you will learn more about word embeddings and how they work using the german language and using byte-pair encodings (BPEmb). For this I will do some experiments to give you a little more insight, about how word-embeddings work.

In case you ask, word embeddings are a vectorized representation of words or word-tokens. They grew from the idea, that usign One-Hot encoding, you can only tell if a word appears in a sequence, but you have no information about its context and also about it meaning compared to others words.

Lets take the sentence “Peter likes to eat marmalade on his bread” and we have a task of finding other words, you could replace marmalade with. We would need a system, that understood the context, in which marmalade is used. Therefore, words that are used in the same context as marmalade will be in the same vector space as marmalade, they could be also things you can put on your bread! Lets look at this further.

Into the python code!

As mentioned before, we use the byte-pair encodings of BPEmb. You can find more about them on https://github.com/bheinzerling/bpemb.
So we need to get the package first by using in terminal:

Then we can jump immediatly into the python code!

We import bpemb and numpy and define, that we want to use the german model with a vocabulary size of 25.000 and 25 dimensions.

But what does this mean? Let me show you a simple example, where we code the encodings manually and to vizualize the meaning of dimensions.

Now we just need to call _encode and see what happens:

What we can read from this dictionary is, that “Mann” (Key:0) is defined by beeing grownup (1) and having a relation-value of -1

“Frau” (Key:1) is defined by beeing grownup also (1) and having a relation-value of 1.

“Tochter” (Key:2) is defined by beeing not grownup (0) and having a relation-value of 0.

Numpy allows us to do array-wise calculations, so we can do mathematic operations on these word-vectors. Lets find out what happens, when “Mann” and “Frau” are combined. You might have an idea, whats coming next.

This combined word vector contains the values of the combination of the words “Mann” and “Frau” which are the same as the values of the word “Tochter”. We teached this simple, hard coded model an association, that man and women combined point at daughter.

Word embeddings in bpemb

Now we know about the dimension-parameter of the bpemb import. But let some words be said about the vocab_size parameter. Generally, each vocab in NLP should use tokens, that represent rare or unknown word-tokens. Depending on your hardware, the vocab size will restrict you at some point in your model building, since it will run out of memory. Simply said, having a vocab_size of 25.000 word-tokens means having 24.999 real-word tokens and 1 token, that represents the rare ones. (There might also be other tokens, like for special encodings, or other special word parts). Increasing the vocab size to 50.000 will result in more different word_tokens, that would be hidden in the 25.000 vs model in the rare token.

In bpemb, we have way more complex pretrained word vectors, that allow us to do some experiments to understand them better. What happens, when we do the same as above in bpemb?

Stop, lets go through each line to see what happens

We use the .embed function of our bpemb model, which tokenizes our input ‘Mann’ into subtokens, that are used by bpemb and then returns the corresponding word vector

The result is a list, containing lists of 25 elements (as the dimensions, we choose). Since “Mann” only results in one token, we only have one list element of 25 values.

We can add numpy arrays in python, so the lists are merged

What we have now, is a new vector containing the following Information: We want a word, that has the information of adding “Mann” and “Frau” together. For this we use .most_similar, with getting the top 6 results

“_frau” and “_mann” are still near our start vector, but we also find other words like “_freund” (friend) or mother and father. At the 6.th position we find “_kind”.

When we teached our predefined _embed method, which information each dimension contains, the bpemb model had to learn it on his own by alot of wikipedia articles. So how about we try to find out more about the meaning of each dimension? For this, we iterate through the dimensions of our embed_add vector and set the to 1 and print the top 3 results. Lets see if we can “kid” or even “daughter” to the top with this.

As we can see, we get alot of different results, which are rather more distant to having “kid” in the top three than before. This is because the whole learnt language in the model had to be squashed into 25 dimensions. Thats far too less for each dimension to have a clear context. We just dont know, what each dimension represents.

Lets try something else, restrict the information in the word vector more by adding extra information

Oh look! We just got “_kind” (children) on the second place thorugh adding the word vector of “birth” That is pretty cool, right? But what happens, when we swap “Mann” and “Frau”?

Mathematicans wont be surprised, it is the same. Simply, because 1+2 is the same as 2+1. But, lets see if we can “_kind” even higher through weighting the word_vectors

This is better, but we still did not got “children” to the top. Maybe Adding an extra information about children will raise it more. So lets puts Spielzeug (toys) in it.

We did it! Through adding the information “Spielzeug” we got the model to understand, that we talk about children. What do you think, can we change one word, to get the word “baby” to the top?

Lets try changing “Mann” to Windeln (diapers)

“kind” is still on the top. Maybe the vocabulary is too small, to contain the word “baby”, lets try it again with 50.000 words.

Weird, isn’t it? Still we don’t get “Baby” to the top. Lets see, what is similar to baby.

And there we find out explanation. The word baby has in this 50k vocab model different meanings but the most important difference is, that baby is seen as an english word, which is basically similar to other english words, which were also collected in the 50k vocab model. This might be a hint, that a model with more dimensions could resolve that, since its data might be better differentiated. The model learned, that the word baby appears only in the context of other english words. So by using german words, we will not the word baby easily with this model.

In this post you learned how to use the bpemb model, to extract word embeddings to analyze them. You also learned how to find other similar words and how bpemb stores them in vectors. I hope you found it interesting! There is alot to tell about word-embeddings, but for beginners, this might be a good way of starting to use them. Why dont you share your thoughts in the comments? Thanks for reading!