---
title: "Using a GPT2 transformer model to get word predictability"
bibliography: '`r system.file("REFERENCES.bib", package="pangoling")`'
vignette: >
  %\VignetteIndexEntry{Using a GPT2 transformer model to get word predictability}
  %\VignetteEngine{knitr::rmarkdown}
  %\VignetteEncoding{UTF-8}
---



<!-- https://ropensci.org/blog/2019/12/08/precompute-vignettes/ -->

Transformer models are a type of neural network architecture used for natural
language processing tasks such as language translation and text generation. They
were introduced in the @vaswani2017attention paper "Attention Is All You Need". 

Large Language Models (LLMs) are a specific type of pre-trained transformer 
models. These models have been trained on massive amounts of text data and can 
be fine-tuned to perform a variety of NLP tasks such as text classification, 
named entity recognition, question answering, etc.

A causal language model (also called GPT-like, auto-regressive, or decoder 
model) is a type of large language model usually used for text-generation that 
can predict the next word (or more accurately, the next token) based on a 
preceding context. GPT-2 (Generative Pre-trained Transformer 2) developed by OpenAI is an example of a causal language model 
[see also @radford2019language].

One interesting side-effect of causal language models is that the (log) 
probability of a word given a certain context can be extracted from the models. 

Load the following packages first:


``` r
library(pangoling)
library(tidytable) # fast alternative to dplyr
library(tictoc) # measure time
```

Then let's examine which continuation GPT-2 predicts following a specific 
context. [Hugging Face](https://huggingface.co/) provide access to pre-trained
models, including freely available versions of different sizes of 
[GPT-2](https://huggingface.co/gpt2). The function 
`causal_next_tokens_pred_tbl()` will, by default, use the smallest version of
[GPT-2](https://huggingface.co/gpt2), but this can be modified with the argument
`model`.

Let's see what GPT-2 predicts following "The apple doesn't fall far from the".


``` r
tic()
(df_pred <- causal_next_tokens_pred_tbl("The apple doesn't fall far from the"))
#> Processing using causal model 'gpt2/' ...
#> # A tidytable: 50,257 × 2
#>    token     pred
#>    <chr>    <dbl>
#>  1 Ġtree   -0.281
#>  2 Ġtrees  -3.60 
#>  3 Ġapple  -4.29 
#>  4 Ġtable  -4.50 
#>  5 Ġhead   -4.83 
#>  6 Ġmark   -4.86 
#>  7 Ġcake   -4.91 
#>  8 Ġground -5.08 
#>  9 Ġtruth  -5.31 
#> 10 Ġtop    -5.36 
#> # ℹ 50,247 more rows
toc()
#> 5.438 sec elapsed
```

(The pretrained models and tokenizers will be downloaded from 
https://huggingface.co/ the first time they are used.)

The most likely continuation is "tree", which makes sense.
The first time a model is run, it will download some files that will be 
available for subsequent runs. However, every time we start a new R session and 
we run a model, it will take some time to store it in memory. Next runs in the 
same session are much faster. We can also preload a model with 
`causal_preload()`.


``` r
tic()
(df_pred <- causal_next_tokens_pred_tbl("The apple doesn't fall far from the"))
#> Processing using causal model 'gpt2/' ...
#> # A tidytable: 50,257 × 2
#>    token     pred
#>    <chr>    <dbl>
#>  1 Ġtree   -0.281
#>  2 Ġtrees  -3.60 
#>  3 Ġapple  -4.29 
#>  4 Ġtable  -4.50 
#>  5 Ġhead   -4.83 
#>  6 Ġmark   -4.86 
#>  7 Ġcake   -4.91 
#>  8 Ġground -5.08 
#>  9 Ġtruth  -5.31 
#> 10 Ġtop    -5.36 
#> # ℹ 50,247 more rows
toc()
#> 0.773 sec elapsed
```

Notice that the tokens--that is, the way GPT-2 interprets words-- that are 
predicted start with `Ġ`, this indicates that they are not the first word of a
sentence.

In fact this is the way GPT-2 interprets our context:


``` r
tokenize_lst("The apple doesn't fall far from the")
#> [[1]]
#> [1] "The"    "Ġapple" "Ġdoesn" "'t"     "Ġfall"  "Ġfar"   "Ġfrom"  "Ġthe"
```

Also notice that GPT-2 tokenizer interprets differently initial tokens from 
tokens that follow a space. A space in  a token is indicated with "Ġ".


``` r
tokenize_lst("This is different from This")
#> [[1]]
#> [1] "This"       "Ġis"        "Ġdifferent" "Ġfrom"      "ĠThis"
```

It's also possible to decode the tokens to get "pure" text:


``` r
tokenize_lst("This is different from This", decode = TRUE)
#> [[1]]
#> [1] "This"       " is"        " different" " from"      " This"
```

Going back to the initial example, because `causal_next_tokens_pred_tbl()` 
returns by default log natural probabilities, if we exponentiate them and we 
sum them, we should get 1:


``` r
sum(exp(df_pred$pred))
#> [1] 1.000017
```

Because of approximation errors, this is not exactly one.


When doing tests, 
[`sshleifer/tiny-gpt2`](https://huggingface.co/sshleifer/tiny-gpt2) is quite 
useful since it's much faster because it's a tiny model. But notice that the
predictions are quite bad.


``` r
causal_preload("sshleifer/tiny-gpt2")
#> Preloading causal model sshleifer/tiny-gpt2...
tic()
causal_next_tokens_pred_tbl("The apple doesn't fall far from the",
  model = "sshleifer/tiny-gpt2"
)
#> Processing using causal model 'sshleifer/tiny-gpt2/' ...
#> # A tidytable: 50,257 × 2
#>    token          pred
#>    <chr>         <dbl>
#>  1 Ġstairs       -10.7
#>  2 Ġvendors      -10.7
#>  3 Ġintermittent -10.7
#>  4 Ġhauled       -10.7
#>  5 ĠBrew         -10.7
#>  6 Rocket        -10.7
#>  7 dit           -10.7
#>  8 ĠHabit        -10.7
#>  9 ĠJr           -10.7
#> 10 ĠRh           -10.7
#> # ℹ 50,247 more rows
toc()
#> 0.095 sec elapsed
```


All in all, the package `pangoling` would be most useful in the following 
situation. (And see also the [worked-out example vignette](example.html).)

Given a (toy) dataset where sentences are organized with one word or short 
phrase in each row:

``` r
sentences <- c(
  "The apple doesn't fall far from the tree.",
  "Don't judge a book by its cover."
)
df_sent <- strsplit(x = sentences, split = " ") |>
  map_dfr(.f = ~ data.frame(word = .x), .id = "sent_n")
df_sent
#> # A tidytable: 15 × 2
#>    sent_n word   
#>     <int> <chr>  
#>  1      1 The    
#>  2      1 apple  
#>  3      1 doesn't
#>  4      1 fall   
#>  5      1 far    
#>  6      1 from   
#>  7      1 the    
#>  8      1 tree.  
#>  9      2 Don't  
#> 10      2 judge  
#> 11      2 a      
#> 12      2 book   
#> 13      2 by     
#> 14      2 its    
#> 15      2 cover.
```

One can get the natural log-transformed probability of each word based on 
GPT-2 as follows:


``` r
df_sent <- df_sent |>
  mutate(lp = causal_words_pred(word, by = sent_n))
#> Processing using causal model 'gpt2/' ...
#> Processing a batch of size 1 with 10 tokens.
#> Processing a batch of size 1 with 9 tokens.
#> Text id: 1
#> `The apple doesn't fall far from the tree.`
#> Text id: 2
#> `Don't judge a book by its cover.`
#> ***
df_sent
#> # A tidytable: 15 × 3
#>    sent_n word         lp
#>     <int> <chr>     <dbl>
#>  1      1 The      NA    
#>  2      1 apple   -10.9  
#>  3      1 doesn't  -5.50 
#>  4      1 fall     -3.60 
#>  5      1 far      -2.91 
#>  6      1 from     -0.745
#>  7      1 the      -0.207
#>  8      1 tree.    -1.58 
#>  9      2 Don't    NA    
#> 10      2 judge    -6.27 
#> 11      2 a        -2.33 
#> 12      2 book     -1.97 
#> 13      2 by       -0.409
#> 14      2 its      -0.257
#> 15      2 cover.   -1.38
```

Notice that the `by` is inside the `causal_words_pred()` function. It's also 
possible to use `by` in the mutate call, or `group_by()`, but it will be slower.


The attentive reader might have noticed that the log-probability of "tree" here 
is not the same as the one presented before. This is because the actual word is 
`" tree."` (notice the space), which contains two tokens:


``` r
tokenize_lst(" tree.")
#> [[1]]
#> [1] "Ġtree" "."
```

The log-probability of `" tree."` is the sum of the log-probability of `" tree"`
given its context and `"."` given its context.

We can verify this in the following way.


``` r
df_token_lp <- causal_tokens_pred_lst(
  "The apple doesn't fall far from the tree.") |>
  # convert the list into a data frame
  map_dfr(~ data.frame(token = names(.x), pred = .x))
#> Processing using causal model 'gpt2/' ...
#> Processing a batch of size 1 with 10 tokens.
df_token_lp
#> # A tidytable: 10 × 2
#>    token        pred
#>    <chr>       <dbl>
#>  1 The     NA       
#>  2 Ġapple -10.9     
#>  3 Ġdoesn  -5.50    
#>  4 't      -0.000828
#>  5 Ġfall   -3.60    
#>  6 Ġfar    -2.91    
#>  7 Ġfrom   -0.745   
#>  8 Ġthe    -0.207   
#>  9 Ġtree   -0.281   
#> 10 .       -1.30

(tree_lp <- df_token_lp |> 
  # requires a Ġ because there is a space before
  filter(token == "Ġtree") |>
  pull())
#> [1] -0.2808024

(dot_lp <- df_token_lp |>
  # doesn't require a Ġ because there is no space before
  filter(token == ".") |>
  pull())
#> [1] -1.300929

tree._lp <- df_sent |>
  filter(word == "tree.") |>
  pull()

# Test whether it is equal
all.equal(
  tree_lp + dot_lp,
  tree._lp
)
#> [1] TRUE
```

In a scenario as the one above, when one has a word-by-word text, and one wants 
to know the log-probability of each word, one doesn't have to worry about the
encoding or tokens, since the function `causal_words_pred()` takes care of it.

# References
