---
title: "Tidy Text & Tokenization"
subtitle: "Week 4 — Codealong Notes"
author: "YOUR NAME HERE"
date: "2026-09-14"
format: pdf
execute:
  echo: true
  warning: false
  message: false
  error: true
---

```{r}
library(tidyverse)
library(tidytext)
library(janeaustenr)
```

# 1.1 — Tidy text vs. other formats

## Recap: what makes data "tidy"?

From Weeks 1–2:

- Each _______ is a column
- Each _______ is a row
- Each _______ is a cell

**Tidy text**: one _______ (usually a word) per _______

## Four ways to store text

* _______ = Raw character vectors
* _______ = Raw strings + metadata
* _______ = One row per document, one column per term, sparse matrix
* _______ = One row per token


# 1.2 — The `unnest_tokens()` function

## Our example text

```{r}
text <- c("Because I could not stop for Death -",
          "He kindly stopped for me -",
          "The Carriage held but just Ourselves -",
          "and Immortality")

text_df <- tibble(line = 1:4, text = text)
text_df
```

## Your turn

Tokenize `text_df` — split it into one word per row.

```{r}
text_df |>
  ______(______, ______)
```

## Removing stop words
## Your turn

Add a step to your pipe that removes stop words.

```{r}
text_df |>
  unnest_tokens(word, text) |>
  ______(______, by = "word")
```


## Customize your dictionary

1. Look at the output of `text_df |> unnest_tokens(word, text)` and pick a word.
2. Add this word, in quotations, to the blank in the `word = c(...)` vector below.
3. Fill in the rest of the blanks to build your own stop word list and use it to filter `text_df`.

```{r}
my_stopwords <- tibble(
  word = c("um", "uh", "like", "okay", "______"),
  lexicon = "custom"
)

my_all_stopwords <- bind_rows(stop_words, ______)

text_df |>
  unnest_tokens(word, text) |>
  ______(my_all_stopwords, by = "word")
```


# 1.3 — Tidying the works of Jane Austen
## Meet `janeaustenr`

```{r}
______ <- austen_books()
head(abooks)
```

## Adding structure
Let's add a line number for each book and a variable to identify the chapters.

```{r}
original_books <- abooks |>
  group_by(book) |>
  mutate(linenumber = row_number(),
         chapter = cumsum(str_detect(text,
                      regex("^chapter [\\divxlc]", ignore_case = TRUE)))) |>
  ungroup()

head(original_books)
```

## Tokenize at scale

Note that metadata variables are still retained.

```{r}
tidy_books <- ______ |>
  ______(word, text)

tidy_books
```

## Your turn

Remove stop words from `tidy_books`, then count and sort word frequencies.

```{r}
______ |>
  ______(stop_words, by = "word") |>
  ______(word, sort = ______)
```

## Payoff

```{r}
tidy_books_clean <- tidy_books |>
  anti_join(stop_words, by = "word") 
```


## Visualize it

Create a horizontal barplot of the most frequent words.

```{r}
tidy_books_clean |>
  ______(word, sort = TRUE) |>
  ______(n > 600) |>
  mutate(word = ______(word, n)) |>
  ggplot(aes(y = word, x = n)) +
  ______()
```

## Ask further questions

Is there a difference in the most frequent words across books?

```{r}
tidy_books_clean |>
  ______(book) |>
  count(______, sort = TRUE) |>
  filter(n > ______) |>
  mutate(word = fct_reorder(word, n)) |>
  ggplot(aes(y = ______, x = ______, fill = book)) +
  geom_col() +
  facet_wrap(~______, scales = "free", nrow = 1) +
  theme(legend.position = "none")
```

> Don't forget to remove `error: true` from the YAML header. Then render again before you submit.