---
title: "Introduction to tidyEmoji"
author: 'Youzhi Yu
University of Chicago'
date: "`r Sys.Date()`"
output:
rmarkdown::html_vignette:
toc: true
toc_depth: 3
vignette: >
%\VignetteIndexEntry{Introduction to tidyEmoji}
%\VignetteEngine{knitr::rmarkdown}
%\VignetteEncoding{UTF-8}
---
```{r, include = FALSE}
knitr::opts_chunk$set(
collapse = TRUE,
message = FALSE,
warning = FALSE,
fig.width = 8,
fig.height = 5,
comment = "#>"
)
```
## Overview
Emoji are everywhere in modern text β social-media posts, product reviews, chat
and support logs, survey free-text β and they carry information that plain words
do not. Yet summarising emoji from a corpus is surprisingly awkward. Unicode does
not interact cleanly with regular expressions, not every code point is an emoji,
and a single visible emoji is often built from several code points joined
together. Counting "how many posts contain an emoji" or "which emoji are most
common" by hand quickly becomes painful.
**tidyEmoji** removes that friction. It provides a small family of verbs that
take a data frame and the name of a text column, and return tidy data frames
that drop straight into a `dplyr`/`ggplot2` workflow:
| Task | Function(s) |
|------|-------------|
| Summarise / filter | `emoji_summary()`, `emoji_filter()` |
| Extract | `emoji_extract_nest()`, `emoji_extract_unnest()`, `emoji_tokens()` |
| Count | `emoji_frequency()`, `top_n_emojis()` |
| Categorise | `emoji_categorize()` |
| Score sentiment | `emoji_sentiment()` |
Two design choices are worth highlighting:
* **Grapheme-aware detection.** Detection is performed on whole grapheme
clusters, so skin-tone modifiers (ππ½) and zero-width-joiner sequences such
as the family emoji (π¨βπ©βπ§βπ¦) are treated as a *single* emoji rather than
being split into their component parts. This is illustrated in the
[extraction section](#a-note-on-grapheme-aware-detection).
* **Tidy by default.** Every verb returns a tibble, follows the
`verb(data, text_column)` convention, and supports unquoted column names, so
the functions compose naturally with the pipe.
```{r setup}
library(tidyEmoji)
library(dplyr)
library(ggplot2)
```
## Example data
Throughout this vignette we use a sample of 10,000 tweets collected in Atlanta,
Georgia. Tweets are a convenient, emoji-rich example, but nothing below is
specific to Twitter β any data frame with a text column will do.
```{r}
ata_tweets <- readr::read_csv("ata_tweets.rda", show_col_types = FALSE)
ata_tweets
```
The actual text lives in the `full_text` column, which is the column we pass to
each tidyEmoji verb.
## Detecting and summarising emoji
### `emoji_summary()`
`emoji_summary()` answers the first question one usually asks of a new corpus:
*how much emoji is in here?* It returns a one-row tibble with the number of
entries that contain at least one emoji and the total number of entries. An
entry is counted once regardless of how many emoji it holds.
```{r}
summary_tbl <- ata_tweets %>%
emoji_summary(full_text)
summary_tbl
```
Here, `r format(summary_tbl$emoji_tweets, big.mark = ",")` of the
`r format(summary_tbl$total_tweets, big.mark = ",")` tweets
(`r round(100 * summary_tbl$emoji_tweets / summary_tbl$total_tweets, 1)`%)
contain at least one emoji.
### `emoji_filter()`
`emoji_filter()` keeps only the rows whose text contains at least one emoji,
preserving every original column. This is useful when you want to compare
emoji-bearing and emoji-free text, or restrict an analysis to the emoji subset.
(`emoji_tweets()` is a synonym retained for backward compatibility.)
```{r}
ata_tweets %>%
emoji_filter(full_text)
```
## Extracting emoji
tidyEmoji offers three complementary ways to pull the emoji out of text,
depending on the shape of output you want.
### `emoji_extract_nest()`
`emoji_extract_nest()` leaves the data unchanged except for an added
list-column, `.emoji_unicode`, holding the emoji found in each row. The original
data structure is preserved, which makes this convenient as an intermediate
step.
```{r}
ata_tweets %>%
emoji_extract_nest(full_text) %>%
select(full_text, .emoji_unicode)
```
### `emoji_extract_unnest()`
`emoji_extract_unnest()` returns a long, tidy table with one row per
(entry, emoji) pair: `row_number` records the position of the entry in the data,
`.emoji_unicode` is the emoji, and `.emoji_count` is how many times that emoji
occurs in that entry. Entries without emoji are dropped.
```{r}
emoji_per_tweet <- ata_tweets %>%
emoji_extract_unnest(full_text)
emoji_per_tweet
```
We can use this to plot how many emoji each emoji-bearing tweet contains:
```{r, fig.alt = "Bar chart of the number of emoji per emoji-bearing tweet. The vast majority of tweets contain a single emoji, with a long, thin tail of more emoji-heavy tweets."}
emoji_per_tweet %>%
group_by(row_number) %>%
summarise(n_emoji = sum(.emoji_count)) %>%
ggplot(aes(n_emoji)) +
geom_bar() +
scale_x_continuous(breaks = seq(1, 15)) +
labs(x = "Number of emoji in the tweet",
y = "Number of tweets",
title = "Most emoji tweets contain a single emoji")
```
The overwhelming majority of emoji tweets carry just one emoji, with a long, thin
tail of more emoji-heavy tweets.
### `emoji_tokens()`
`emoji_tokens()` produces a "one row per emoji occurrence" table β the emoji
analogue of a tidy-text token table. It keeps the original columns and adds the
glyph (`.emoji`) together with its name (`.emoji_name`), category
(`.emoji_category`) and sentiment score (`.emoji_sentiment`). This single call
gives you everything needed for counting, joining and plotting.
```{r}
ata_tweets %>%
emoji_tokens(full_text)
```
### A note on grapheme-aware detection
Modern emoji are frequently composed of several code points: a base emoji plus a
skin-tone modifier, or several emoji joined by zero-width joiners. tidyEmoji
detects emoji at the level of grapheme clusters, so these stay intact. The
example below contains exactly two emoji β one family and one thumbs-up β and
tidyEmoji counts them as such rather than splitting the family into four people
or separating the thumb from its skin tone:
```{r}
demo <- data.frame(
text = c("our family \U0001F468β\U0001F469β\U0001F467β\U0001F466",
"great work \U0001F44D\U0001F3FD")
)
demo %>%
emoji_extract_unnest(text)
```
## Counting emoji across the corpus
### `emoji_frequency()`
`emoji_frequency()` counts how often each emoji appears across the whole text
column (an entry containing the same emoji twice contributes 2) and returns the
result sorted by descending count, annotated with each emoji's name, shortcode
and category.
```{r}
ata_tweets %>%
emoji_frequency(full_text)
```
### `top_n_emojis()`
When you only need the leaders, `top_n_emojis()` is a convenience wrapper around
`emoji_frequency()` that returns the `n` most frequent emoji (default `n = 20`).
```{r}
top_20_emojis <- ata_tweets %>%
top_n_emojis(full_text)
top_20_emojis
```
Plotting the top 20, coloured by category, gives an immediate sense of how the
community expresses itself:
```{r, fig.alt = "Horizontal bar chart of the 20 most frequent emoji in the corpus, coloured by Unicode category."}
top_20_emojis %>%
mutate(emoji_name = stringr::str_replace_all(emoji_name, "_", " "),
emoji_name = forcats::fct_reorder(emoji_name, n)) %>%
ggplot(aes(n, emoji_name, fill = emoji_category)) +
geom_col() +
labs(x = "Count",
y = NULL,
fill = "Category",
title = "The 20 most frequent emoji")
```
The `unicode` column holds the actual glyph, should you wish to render the emoji
themselves on a plot (this requires a graphics device with an emoji-capable
font). You can also request a different number of emoji:
```{r, fig.alt = "Horizontal bar chart of the 10 most frequent emoji in the corpus, coloured by Unicode category."}
ata_tweets %>%
top_n_emojis(full_text, n = 10) %>%
mutate(emoji_name = stringr::str_replace_all(emoji_name, "_", " "),
emoji_name = forcats::fct_reorder(emoji_name, n)) %>%
ggplot(aes(n, emoji_name, fill = emoji_category)) +
geom_col() +
labs(x = "Count", y = NULL, fill = "Category",
title = "The 10 most frequent emoji")
```
## Categorising emoji
The Unicode standard organises emoji into 10 categories (see
`?category_unicode_crosswalk`). `emoji_categorize()` keeps the emoji-bearing rows
and adds a `.emoji_category` column listing the distinct categories present in
each row, separated by `|` when a row spans more than one.
```{r}
ata_emoji_category <- ata_tweets %>%
emoji_categorize(full_text) %>%
select(.emoji_category)
ata_emoji_category
```
We can tally the most common category combinations:
```{r, fig.alt = "Horizontal bar chart of the most common emoji category combinations that appear in more than 20 tweets."}
ata_emoji_category %>%
count(.emoji_category, sort = TRUE) %>%
filter(n > 20) %>%
mutate(.emoji_category = forcats::fct_reorder(.emoji_category, n)) %>%
ggplot(aes(n, .emoji_category)) +
geom_col() +
labs(x = "Number of tweets", y = NULL,
title = "Most common emoji category combinations")
```
To count the 10 individual categories rather than their combinations, split the
`.emoji_category` strings on `|` with `tidyr::separate_rows()`:
```{r, fig.alt = "Horizontal bar chart of how often each individual Unicode emoji category is used, dominated by Smileys & Emotion followed by People & Body."}
ata_emoji_category %>%
tidyr::separate_rows(.emoji_category, sep = "\\|") %>%
count(.emoji_category, sort = TRUE) %>%
mutate(.emoji_category = forcats::fct_reorder(.emoji_category, n)) %>%
ggplot(aes(n, .emoji_category)) +
geom_col() +
labs(x = "Number of tweets", y = NULL,
title = "Emoji category usage")
```
"Smileys & Emotion" dominates, followed by "People & Body". Note that a tweet
spanning several categories is counted once in each, so these counts can exceed
the number of emoji tweets.
## Scoring emoji sentiment
### `emoji_sentiment()`
Emoji are a strong sentiment signal, and `emoji_sentiment()` surfaces it
directly. It adds `.emoji_n` (the number of emoji in the entry) and
`.emoji_sentiment` (the mean sentiment of those emoji, from -1 for negative to
+1 for positive). Scores come from the bundled `emoji_sentiment_lexicon`
(described below); entries with no emoji, or whose emoji are not in the lexicon,
receive `NA`.
```{r}
ata_sentiment <- ata_tweets %>%
emoji_sentiment(full_text)
ata_sentiment %>%
select(.emoji_n, .emoji_sentiment)
```
### Sentiment distribution
Looking across the tweets that contain at least one scored emoji:
```{r, fig.alt = "Histogram of the mean emoji sentiment per tweet, which is concentrated on the positive side of the scale."}
ata_sentiment %>%
filter(!is.na(.emoji_sentiment)) %>%
ggplot(aes(.emoji_sentiment)) +
geom_histogram(binwidth = 0.1) +
labs(x = "Mean emoji sentiment",
y = "Number of tweets",
title = "Emoji sentiment skews positive")
```
As is typical of social-media text, emoji sentiment leans strongly positive.
### Sentiment by category
Because `emoji_tokens()` attaches a sentiment score to every emoji occurrence, we
can summarise average sentiment by category in a couple of lines:
```{r, fig.alt = "Horizontal bar chart of the average emoji sentiment within each Unicode category."}
ata_tweets %>%
emoji_tokens(full_text) %>%
group_by(.emoji_category) %>%
summarise(mean_sentiment = mean(.emoji_sentiment, na.rm = TRUE),
n_scored = sum(!is.na(.emoji_sentiment))) %>%
filter(n_scored > 0) %>%
mutate(.emoji_category = forcats::fct_reorder(.emoji_category, mean_sentiment)) %>%
ggplot(aes(mean_sentiment, .emoji_category)) +
geom_col() +
labs(x = "Mean sentiment", y = NULL,
title = "Average emoji sentiment by category")
```
### The sentiment lexicon
The scores come from `emoji_sentiment_lexicon`, the *Emoji Sentiment Ranking* of
Kralj Novak et al. (2015), computed from around 70,000 tweets annotated in 13
European languages. You can work with it directly β for instance, to find the
most positive and most negative reasonably common emoji:
```{r}
emoji_sentiment_lexicon %>%
filter(occurrences >= 500) %>%
slice_max(sentiment_score, n = 8) %>%
select(emoji, unicode_name, occurrences, sentiment_score)
emoji_sentiment_lexicon %>%
filter(occurrences >= 500) %>%
slice_min(sentiment_score, n = 8) %>%
select(emoji, unicode_name, occurrences, sentiment_score)
```
## Bundled datasets
tidyEmoji ships three datasets, each documented with its own help page:
* **`emoji_sentiment_lexicon`** β emoji sentiment scores from the Emoji
Sentiment Ranking (see `?emoji_sentiment_lexicon`).
* **`emoji_unicode_crosswalk`** β one row per emoji name, mapping names /
shortcodes to glyphs and categories.
* **`category_unicode_crosswalk`** β one row per Unicode category, listing its
emoji.
These are regenerated from the current Unicode emoji list by the scripts in the
package's `data-raw/` directory.
## References
Kralj Novak P, SmailoviΔ J, Sluban B, MozetiΔ I (2015). Sentiment of Emojis.
*PLoS ONE* 10(12): e0144296.
. The Emoji Sentiment Ranking is
distributed under the Creative Commons Attribution-ShareAlike 4.0 International
(CC BY-SA 4.0) licence.