NOTE: Before running this notebook, place a copy of your downloaded RomeoAndJuliet.txt file in the same folder with this notebook.

12.3 Visualizing Word Frequencies with Bar Charts and Word Clouds

  • Can enhance your corpus analyses
    • A bar chart quantitatively visualizes the top 20 words in Romeo and Juliet as bars representing each word and its frequency.
    • A word cloud qualitatively visualizes more frequently occurring words in larger fonts and less frequently occurring words in smaller fonts.

12.3.1 Visualizing Word Frequencies with Pandas

  • Visualize Romeo and Juliet’s top 20 words that are not stop words, using features from TextBlob, NLTK and pandas.

  • Pandas visualization capabilities are based on Matplotlib, so launch IPython with the following command for this session:

    ipython --matplotlib

  • Or enable matplotlib in Jupyter

In [1]:
%matplotlib inline

Loading the Data

In [2]:
from pathlib import Path
In [3]:
from textblob import TextBlob
In [4]:
blob = TextBlob(Path('RomeoAndJuliet.txt').read_text())
  • Load NLTK stop words
In [5]:
from nltk.corpus import stopwords
In [6]:
stop_words = stopwords.words('english')

Getting the Word Frequencies

  • Get word frequency tuples
In [7]:
items = blob.word_counts.items()

Eliminating the Stop Words

  • The expression item[0] gets the word from each tuple so we can check whether it’s in stop_words
In [8]:
items = [item for item in items if item[0] not in stop_words]

Sorting the Words by Frequency

  • Sort the tuples in items in descending order by frequency
  • To specify the tuple element to sort by, use the itemgetter function from the Python Standard Library’s operator module
In [9]:
from operator import itemgetter
In [10]:
sorted_items = sorted(items, key=itemgetter(1), reverse=True)

Getting the Top 20 Words

  • TextBlob tokenizaton splits all contractions at their apostrophes and counts the total number of apostrophes as one of the “words”
  • Romeo and Juliet has many contractions
    • If you display sorted_items[0], you’ll see that they are the most frequently occurring “word” with 867 of them
    • (In some locales this does not happen and element 0 is indeed 'romeo')
    • We ignore element 0
In [11]:
top20 = sorted_items[1:21]

Convert top20 to a DataFrame

In [12]:
import pandas as pd
In [13]:
df = pd.DataFrame(top20, columns=['word', 'count'])
In [14]:
df
Out[14]:
word count
0 romeo 315
1 thou 278
2 juliet 190
3 thy 170
4 capulet 163
5 nurse 149
6 love 148
7 thee 138
8 lady 117
9 shall 110
10 friar 105
11 come 94
12 mercutio 88
13 lawrence 82
14 good 80
15 benvolio 79
16 tybalt 79
17 enter 75
18 go 75
19 night 73

Visualizing the DataFrame

  • bar method of the DataFrame’s plot property creates and displays a Matplotlib bar chart
In [15]:
axes = df.plot.bar(x='word', y='count', legend=False)

import matplotlib.pyplot as plt

plt.gcf().tight_layout()

©1992–2020 by Pearson Education, Inc. All Rights Reserved. This content is based on Chapter 5 of the book Intro to Python for Computer Science and Data Science: Learning to Program with AI, Big Data and the Cloud.

DISCLAIMER: The authors and publisher of this book have used their best efforts in preparing the book. These efforts include the development, research, and testing of the theories and programs to determine their effectiveness. The authors and publisher make no warranty of any kind, expressed or implied, with regard to these programs or to the documentation contained in these books. The authors and publisher shall not be liable in any event for incidental or consequential damages in connection with, or arising out of, the furnishing, performance, or use of these programs.