Homework 3: Tidy Text & Sentiment
Introduction
In this assignment you will apply the tokenization skills from code along 03 and TTM chapter 1, and the sentiment analysis skills from code along 04 and TTM Chapter 2 to selected questions from Amy’s research data.
Setup
The data set is called criminal_justice_short.csv and is available in our Jupyterhub Wicked/data folder. Copy this file to your own home directory.
Load the tidyverse, tidytext and sentimentr packages in this code chunk.
The cj_data data has been pre-filtered to drop all non-consenting students, and contains the following 4 variables: Q7, a mark all that apply program of study variable, with follow up questions why_cj, why_pols_legal, why_paralegal_cert that ask “Why did you decide to major or minor in [PLAN]? Please provide as much detail and description as possible.”
Since Q7 is a ‘mark all that apply’, variable, and only those that marked that specific plan were asked follow up “why_” questions, it would be helpful if we had a single indicator variable for each selected major. This way if we wanted to compare responses between programs we can use those variables as filters.
We will use the str_detect() (read: string detect) function for this. It searches for a pattern of characters in a variable, and returns a TRUE if it is found, otherwise it will show FALSE.
cj <- cj_data |>
mutate(
cj_major = str_detect(Q7, "Criminal Justice major"),
cj_minor = str_detect(Q7, "Criminal Justice minor"),
pols_legal = str_detect(Q7, "Political Science major"),
para_cert = str_detect(Q7, "Paralegal Certificate"),
un_consider = str_detect(Q7, "STRONGLY CONSIDERING"),
un_not_consider =str_detect(Q7, "NOT CONSIDERING")
)So for example, the new variable cj_major is TRUE for anyone that chose that major, by itself or in combination with other majors.
1. Interpret this output in 1-2 sentences.
128 students in total selected the CJ major, 4 of which are double majors with political science/legal studies, and 11 added on the Paralegal certificate program.
Part A — Tidy it
2. Tokenize and remove stop words
Let’s explore why students selected the Criminal Justice major. I have started this code for you by writing some data cleaning steps - specifically filtering to only keep rows where the student answered the question, then only selecting the why_cj variable to keep your results simple. Finish this process by tokenizing why_cj and removing stop words. Save the result as cj_words.
3. Identify and describe the most common words.
Use count() to count how often each word appears in cj_words, and save the result as cj_word_counts. Then create a barplot of the 10 most frequent words by following the example in the slides from codealong 03. In 2-3 sentences, describe the plot. Which words seem meaningful for understanding students’ reasons, and which words might simply reflect the wording of the survey question?
Words like criminal, justice, and major dominate — unsurprising, since the question itself is “why did you choose Criminal Justice.” Behind those, words like system, enforcement, and people start to show what’s actually motivating students.
4. Remove some custom stop words
Criminal, justice, decided, and major are real words, but they’re baked into the question and don’t tell us anything about why a student chose the major. Create a custom stop-word list called custom_stop. Then re-run the anti_join() on cj_words, saving the results into a new data set called cj_words_clean. Next count() the words, and create a barplot that shows the words that show up more than 15 times. Comment on the results.
custom_stop <- tibble(
word = c("criminal", "justice", "major", "decided"),
lexicon = "custom"
)
cj_words_clean <- cj_words |>
anti_join(custom_stop, by = "word")
cj_words_clean |>
count(word, sort = TRUE) |>
filter(n > 15) |>
mutate(word = fct_reorder(word, n)) |>
ggplot(aes(y=word, x=n)) + geom_col()
With the domain-specific words removed, the real motivations surface: words about the system itself, career goals (career, law), and personal connections (people, community) rise toward the top.
5. Investigate reasons for choosing another program.
Choose either why_pols_legal or why_paralegal_cert and repeat the above process. That is: tokenize, remove stop words, visualize, remove custom stop words, visualize, interpret.
Part B — Sentiment Analysis
6. Positive and negative words
Using the bing sentiment lexicon, create a barplot grouped by sentiment that shows the 5 most common positive and negative words in cj_word_counts_clean.
bing_cj <- cj_words_clean |>
inner_join(get_sentiments("bing"))|>
count(word, sentiment, sort = TRUE)
bing_cj |>
group_by(sentiment) |>
slice_max(n, n = 5) |>
ungroup() |>
mutate(word = reorder_within(word, n, sentiment)) |>
ggplot(aes(n, word, fill = sentiment)) +
geom_col(show.legend = FALSE) +
facet_wrap(~sentiment, scales = "free_y") +
scale_y_reordered() +
labs(x = "Contribution to sentiment", y = NULL)
Words such as injustice, lack and abuse top the negative sentiment, and words like love and enjoy lead the positive side.
7. Look closer at one emotion
Filter the nrc lexicon to only pull out a single emotion category. Then use inner_join() to pull the words in cj_words_clean that fall under that emotion category. Show the count of those words and comment on the results.
nrc_trust <- get_sentiments("nrc") |>
filter(sentiment == "trust")
cj_words_clean |>
inner_join(nrc_trust) |>
count(word, sentiment, sort = TRUE)# A tibble: 78 × 3
word sentiment n
<chr> <chr> <int>
1 system trust 76
2 law trust 49
3 school trust 22
4 police trust 19
5 officer trust 14
6 continue trust 7
7 found trust 7
8 legal trust 7
9 true trust 7
10 attorney trust 6
# ℹ 68 more rowsI picked trust, since this data is about why students chose a field. Words tied to systems show up the most, specifically system, law, and school.
8. Sentence-level sentiment
Choose one of the three open-ended response variables:
why_cjwhy_pols_legalwhy_paralegal_cert
Use sentimentr to score the responses at the sentence level. Then inspect and comment on the top 5 most positive and 5 most negative sentences. IMPORTANT Due to the way long sentences word wrap, you must pipe in knitr::kable(row.names = FALSE) after you pull(sentence) to format the results into a readable table.
chosen_responses <- cj |>
select(why_cj) |>
filter(!is.na(why_cj))
chosen_sentences <- get_sentences(chosen_responses) |>
unlist()
chosen_sentences <- tibble(
sentence_id = seq_along(chosen_sentences),
sentence = chosen_sentences
)
sentence_scores <- sentiment(chosen_sentences$sentence) |>
mutate(sentence_id = element_id) |>
select(sentence_id, word_count, sentence_score = sentiment)
chosen_sentence_scores <- chosen_sentences |>
left_join(sentence_scores, by = "sentence_id")
chosen_sentence_scores |>
arrange(desc(sentence_score)) |>
slice(1:5) |>
pull(sentence) |>
knitr::kable(row.names = FALSE)| x |
|---|
| Restorative justice is something I’ve always believed in, and wanted to learn more about as well. |
| I’m considering going the law school path, but I would also love to work with organizations like the innocence project and other nonprofit organizations. |
| Love and passion for justice |
| Growing up I always enjoyed learning about the criminal justice system. |
| I loved the ethics involved in criminal justice respect accountability integrity etc… |
| x |
|---|
| I want to help individuals who have not received justice as of being detained unlawfully. |
| I got into watching true crime shows and listening to true crime podcasts when I was in high school and I was obsessed ever since. |
| There were general fears of instability and potential collapse in 2020. |
| and years of domestic abuse. |
| The police later dropped all charges due to my former partner being under the influence at the time, although there was more than enough physical evidence on my body that showed the abuse I had faced that night. |
The positive sentiment reasons students choose to enter this field is because it aligns with their values, and to protect others. The negative sentiment reasons seem to be reactions to personal or family harm.
9. Reflection
Based on your word-level and sentence-level sentiment results, explain one limitation of using a general-purpose sentiment tool on this kind of qualitative data.
Your answer should not just say whether the tool was “right” or “wrong.” Instead, explain what kinds of meaning the tool can detect, and what kinds of meaning still require human interpretation.