{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Text Analysis:  fuzzy matching, sentiment analysis"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<U>Notes if you are using Jupyter Notebook</U>:  to call <B>exit()</B> from a notebook, please use <B>sys.exit()</B> (requires <B>import sys</B>); if a strange error occurs, it may be because Jupyter retains variables from all executed cells.  To reset the notebook's variables, click 'Restart Kernel' (the circular arrow) -- this will not undo any text changes.  "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "FUZZY MATCHING\n",
    "\n",
    "(Note:  these examples were taken from a <A HREF=\"https://medium.com/@categitau/fuzzy-string-matching-in-python-68f240d910fe\">blog tutorial by Catherine Gitau</A>)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    " "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<pre>\n",
    "pip install fuzzywuzzy\n",
    "pip install python-Levenshtein\n",
    "</pre>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from fuzzywuzzy import fuzz, process\n",
    "from fuzzywuzzy import process"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "fuzz.ratio(\"Catherine M Gitau\", \"Catherine Gitau\")\n",
    "\n",
    "fuzz.partial_ratio(\"Catherine M. Gitau\", \"Catherine Gitau\")  #100"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "fuzz.ratio(\"Catherine M Gitau\", \"Gitau Catherine\")           #55\n",
    "\n",
    "fuzz.partial_ratio(\"Catherine M. Gitau\", \"Gitau Catherine\")  #60"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "str1 = \"Los Angeles Lakers\"\n",
    "str2 = \"Lakers\"\n",
    "\n",
    "ratio = fuzz.ratio(str1.lower(), str2.lower())\n",
    "\n",
    "partial_ratio = fuzz.partial_ratio(str1.lower(), str2.lower())\n",
    "\n",
    "print(ratio)\n",
    "print(partial_ratio)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "fuzz.token_sort_ratio(\"Catherine Gitau M.\", \"Gitau Catherine\") #94\n",
    "\n",
    "str1 = \"united states v. nixon\"\n",
    "str2 = \"Nixon v. United States\"\n",
    "\n",
    "ratio = fuzz.ratio(str1.lower(), str2.lower())\n",
    "partial_ratio = fuzz.partial_ratio(str1.lower(), str2.lower())\n",
    "\n",
    "token_sort_ratio = fuzz.token_sort_ratio(str1, str2)\n",
    "\n",
    "print(ratio)\n",
    "print(partial_ratio)\n",
    "print(token_sort_ratio)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "query = 'geeks for geeks'\n",
    "choices = ['geek for geek', 'geek geek', 'g. for geeks']\n",
    "\n",
    "# get a list of matches ordered by score, default limit to 5\n",
    "process.extract(query, choices)\n",
    "[('geeks geeks', 95), ('g. for geeks', 95), ('geek for geek', 93)]\n",
    "\n",
    "# if we want only the top one\n",
    "process.extractOne(query, choices)\n",
    "('geeks geeks', 95)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "fuzz.WRatio('geeks for geeks', 'Geeks For Geeks')        # 100\n",
    "fuzz.WRatio('geeks for geeks!!!','geeks for geeks')      # 100\n",
    "\n",
    "# whereas simple ratio will give for above case\n",
    "fuzz.ratio('geeks for geeks!!!','geeks for geeks')       # 91"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    " "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "NATURAL LANGUAGE ANALYSIS\n",
    "\n",
    "Note:  these examples are taken from <A HREF=\"https://www.pearson.com/us/higher-education/program/Deitel-Intro-to-Python-for-Computer-Science-and-Data-Science-Learning-to-Program-with-AI-Big-Data-and-The-Cloud/PGM2392788.html\">Intro to Python for the Computer and Data Sciences from Pearson Publishing."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    " "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<B>10.1:  install textblob and supporting libraries"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<pre>\n",
    "# TextBlob:  simple interface to NLTK\n",
    "<B>conda install -c conda-forge textblob</B>\n",
    "\n",
    "# wordcloud - to create word cloud graphics\n",
    "<B>conda install -c conda-forge wordcloud</B>\n",
    "\n",
    "# spaCy - additional analysis\n",
    "<B>conda install -c conda-forge spacy</B>\n",
    "</pre>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    " "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<B>10.2:  download NLTK corpora"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<pre>\n",
    "python -m textblob.download_corpora\n",
    "\n",
    "# download NLTK stop words\n",
    "import nltk; nltk.download('stopwords')\n",
    "\n",
    "# spaCy English components\n",
    "python -m spacy download en\n",
    "\n",
    "python -m spacy download en_core_web_lg       # 827MB!\n",
    "</pre>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    " "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<B>10.3:  Create a TextBlob object.  "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from textblob import TextBlob\n",
    "\n",
    "text = 'Today is a beautiful day.  Tomorrow looks like bad weather.'\n",
    "\n",
    "blob = TextBlob(text)   # TextBlob object\n",
    "\n",
    "print(blob)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "blob_from_file = TextBlob(open('data/shakespeare/hamlet.txt').read())\n",
    "\n",
    "print(len(blob_from_file))       # number of chars"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    " "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<B>10.4:  Tokenizing text:  separating into words, sentences, phrases"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "text = 'Today is a beautiful day.  Tomorrow looks like bad weather.'\n",
    "\n",
    "blob = TextBlob(text)      # TextBlob object\n",
    "\n",
    "print('words:')\n",
    "print(blob.words)          # WordList object\n",
    "print()\n",
    "\n",
    "print('sentences:')\n",
    "print(blob.sentences)      # Sentences object\n",
    "print()\n",
    "\n",
    "print('noun phrases:')\n",
    "print(blob.noun_phrases)   # WordList object\n",
    "print()                    # \"beautiful day\", \"bad weather\""
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    " "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<B>10.5:  POS (\"parts of speech\" tagging"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "blob.tags"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<pre>\n",
    "nouns\n",
    "pronouns\n",
    "verbs\n",
    "adjectives\n",
    "adverbs\n",
    "prepositions\n",
    "conjunctions\n",
    "interjections\n",
    "</pre>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<pre>\n",
    "NN:   singular noun, mass noun\n",
    "VBZ:  third person singular present verb\n",
    "DT:   determiner\n",
    "JJ:   adjective\n",
    "NNP:  poper singular noun\n",
    "IN:   subordinating conjunction or preposition\n",
    "</pre>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "https://www.clips.uantwerpen.be/pattern\n",
    "https://www.clips.uantwerpen.be/pages/MBSP-tags"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    " "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<B>10.6:  Determine sentiment.  "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "text = ('Today is a beautiful day.  '          # single string (implicit concatenation)\n",
    "        'Tomorrow looks like bad weather.  '\n",
    "        'The food was not good.  '\n",
    "        'The movie was not bad.  ')\n",
    "\n",
    "blob = TextBlob(text)\n",
    "\n",
    "# Sentiment object, inludes 'polarity' and 'subjectivity' attributes\n",
    "smt = blob.sentiment\n",
    "\n",
    "# smt.polarity     (-1 to 1, positive/negative)\n",
    "# smt.subjectivity (0 to 1, objective to subjective)\n",
    "\n",
    "for sentence in blob.sentences:\n",
    "    print(f'\"{sentence}\"')\n",
    "    print(f'Polarity (-1 to 1):  {sentence.sentiment.polarity}')\n",
    "    print(f'Subjectivity:        {sentence.sentiment.subjectivity}')\n",
    "    print()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    " "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<B>10.7:  Analyze sentiment using <I>Naive Bayes</I> Analysis"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<B>Naive Bayes</B> is a classification system used in machine learning, that assumes that any individual predictor is unrelated to other predictors when evaluating a candidate.\n",
    "\n",
    "For example, an orange may be identified by its roundness, its orangeness and its size.  A Bayes predictor takes each of these features into account when evaluating a candidate to see if it is an orange, but it does not assume they are related to one another (thus \"naive\").\n",
    "\n",
    "The TextBlob NaiveBayesAnalyzer was trained on movie reviews.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from textblob.sentiments import NaiveBayesAnalyzer\n",
    "\n",
    "text = ('Today is a beautiful day.  '          # single string (implicit concatenation)\n",
    "        'Tomorrow looks like bad weather.  '\n",
    "        'The food was not good.  '\n",
    "        'The movie was not bad.  ')\n",
    "\n",
    "blob = TextBlob(text, analyzer=NaiveBayesAnalyzer())\n",
    "\n",
    "for sentence in blob.sentences:\n",
    "    print(f'\"{sentence}\"')\n",
    "    print(f'classifiction:    {sentence.sentiment.classification}')\n",
    "    print(f'p_pos:            {sentence.sentiment.p_pos}')\n",
    "    print(f'p_neg:            {sentence.sentiment.p_neg}')\n",
    "    print()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    " "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<B>10.8:  Work with inflections.  "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Inflections allow us to access an alternate version of a word.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import textblob\n",
    "from textblob import Word\n",
    "\n",
    "windex = textblob.Word('index')\n",
    "\n",
    "print(windex.pluralize())          # 'indices'\n",
    "\n",
    "\n",
    "wcacti = textblob.Word('cacti')\n",
    "\n",
    "print(wcacti.singularize())        # 'cactus'\n",
    "\n",
    "\n",
    "animals = TextBlob('dog cat fish bird').words\n",
    "\n",
    "print(animals.pluralize())         # WordList(['dogs', 'cats', 'fish', 'birds'])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    " "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<B>10.9:  Work with stemming and lemmatization.  "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "When preparing a text for analysis we may wish to normalize the text.  'Normalizing' in this context means dismissing word variations:\n",
    "<UL>\n",
    "  <LI>lowercasing</LI>\n",
    "  <LI>stemming:  finding the word stem of all word variations</LI>\n",
    "  <LI>lemmatizing:  finding a single base word, for example 'program' can stand for 'programming', 'programmer', etc.</LI>\n",
    "</UL>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "word = Word('varieties')\n",
    "\n",
    "# stem():  common stem to all word variations (may not be a word)\n",
    "print(word.stem())        # varieti\n",
    "\n",
    "# lemmatize():  create \"base\" word\n",
    "print(word.lemmatize())   # variety"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    " "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<B>10.10:  Delete stop words."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "\"Stop words\" are common words like 'a', 'an', 'as', etc. that are very common and usually excluded from analysis."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import nltk\n",
    "\n",
    "#nltk.download('stopwords')           # one time operation, loads words\n",
    "#                                     # into module for future use\n",
    "\n",
    "from nltk.corpus import stopwords\n",
    "\n",
    "stops = stopwords.words('english')\n",
    "\n",
    "print(f'{len(stops)} words in stop words collection')\n",
    "\n",
    "for word in stops:\n",
    "    print(word)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# quick and easy exclusion of stop words from any text\n",
    "\n",
    "blob = TextBlob('Today is a beautiful day.')\n",
    "\n",
    "words = [ word for word in blob.words if word not in stops ]\n",
    "\n",
    "print(words)       # ['Today', 'beuatiful', 'day']"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    " "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<B>10.11:  Analyze \"n-grams\""
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "An <B>ngram</B> is a sequence of text items -- letters or words that appear sequentially in a text.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "text = 'Today is a beautiful day.  Tomorrow looks like bad weather.'\n",
    "\n",
    "blob = TextBlob(text)\n",
    "\n",
    "for wordlist in blob.ngrams():\n",
    "    print(wordlist)\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "for wordlist in blob.ngrams(n=5):\n",
    "    print(wordlist)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    " "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<B>10.12:  Count word and phrase frequency.  "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "fname = 'data/shakespeare/romeo_and_juliet.txt'\n",
    "\n",
    "romeotext = open(fname).read()\n",
    "\n",
    "blob = TextBlob(romeotext)\n",
    "\n",
    "\n",
    "print(blob.word_counts['juliet'])\n",
    "\n",
    "print(blob.words.count('juliet'))   # only if TextBlob has been tokenized into a WordList (or Words object?)\n",
    "\n",
    "print(blob.noun_phrases.count('lady capulet'))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    " "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<B>10.13:  Use spaCy to identify proper nouns:  names, dates, places, etc."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import spacy\n",
    "\n",
    "nlp = spacy.load('en')\n",
    "\n",
    "document = nlp('In 1994, Tim Berners-Lee founded the '\n",
    "               'World Wide Web Consortium (W3C), '\n",
    "               'devoted to developing web technologies.')\n",
    "\n",
    "for entity in document.ents:\n",
    "    print(f'{entity.text}:  {entity.label_}')\n",
    "\n",
    "        # 1994:  DATE\n",
    "        # Tim Berners-Lee:  PERSON\n",
    "        # the World Wide Web Consortium:  ORG"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    " "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<B>10.14:  Use spaCy to detect textual similarities.  "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import spacy\n",
    "\n",
    "# nlp = spacy.load('en')                # small model - comes with spacy\n",
    "# nlp = spacy.load('en_core_web_sm')\n",
    "nlp = spacy.load('en_core_web_lg')      # has to be downloaded:\n",
    "                                        # python -m spacy download en_core_web_lg\n",
    "                                        # 827MB!\n",
    "\n",
    "# sh_:  Shakespeare\n",
    "# ma_:  Christopher Marlowe\n",
    "# be_:  Beaumont and Fletcher\n",
    "\n",
    "ws_romeo_juliet = open('data/shakespeare/romeo_and_juliet.txt').read()\n",
    "ws_hamlet =       open('data/shakespeare/hamlet.txt').read()\n",
    "cm_dr_faustus =   open('data/not_shakespeare/marlowe_dr_faustus.txt').read()\n",
    "bf_maids =        open('data/not_shakespeare/beaumont_fletcher_the_maids_tragedy.txt').read()\n",
    "tw_streetcar =    open('data/not_shakespeare/williams_streetcar_named_desire.txt').read()\n",
    "\n",
    "ws_romeo_text = nlp(ws_romeo_juliet)\n",
    "ws_hamlet_text = nlp(ws_hamlet)\n",
    "\n",
    "cm_faustus_text = nlp(cm_dr_faustus)\n",
    "\n",
    "bf_maids_text = nlp(bf_maids)\n",
    "tw_streetcar_text = nlp(tw_streetcar)\n",
    "\n",
    "\n",
    "print(f'romeo to hamlet:     {ws_romeo_text.similarity(ws_hamlet_text)}')        # Shakespeare -> Shakespeare\n",
    "print(f'romeo to faustus:    {ws_romeo_text.similarity(cm_faustus_text)}')       # Shakespeare -> Marlowe\n",
    "print(f\"romeo to maid's:     {ws_romeo_text.similarity(bf_maids_text)}\")         # Shakespeare -> Beaumont & Fletcher\n",
    "\n",
    "print(f\"faustus to maid's:   {cm_faustus_text.similarity(bf_maids_text)}\")      # Marlowe -> Beaumont & Fletcher\n",
    "print(f'romeo to streetcar:  {ws_romeo_text.similarity(tw_streetcar_text)}')  # Shakespeare -> Williams"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    " "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<B>10.15:  Detect language and translate"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "TextBlob is integrated with Google Translate, allows it to detect language and perform tralsations."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from textblob import TextBlob\n",
    "\n",
    "text = ('Today is a beautiful day.  '          # single string (implicit concatenation)\n",
    "        'Tomorrow looks like bad weather.  '\n",
    "        'The food was not good.  '\n",
    "        'The movie was not bad.  ')\n",
    "\n",
    "blob = TextBlob(text)\n",
    "\n",
    "print(blob.detect_language())           # 'en' (English)\n",
    "\n",
    "spanish_blob = blob.translate(from_lang='en', to='es')  # defaul from_lang='auto'\n",
    "print(spanish_blob)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "language codes:  https://en.wikipedia.org/wiki/List\\_of\\_ISO\\_639-1\\_codes\n",
    "\n",
    "supported langs:  https://cloud.google.com/translate/docs/languages"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    " "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<B>10.16:  Perform a spell check.  "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from textblob import Word\n",
    "\n",
    "word = Word('theyr')\n",
    "\n",
    "# suggest corrections\n",
    "print(word.spellcheck())     # [('they', 0.57), ('their', 0.43)]\n",
    "\n",
    "# choose most common correction\n",
    "print(word.correct())        # 'they'\n",
    "\n",
    "\n",
    "sentence = TextBlob('Ths sentense has svral mispellings.')\n",
    "\n",
    "blob = sentence.correct()\n",
    "\n",
    "print(blob)                  # The sentence has several misspellings."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    " "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<B>10.17:  Visualize word freqency.  "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from pathlib import Path\n",
    "from textblob import TextBlob\n",
    "import pandas as pd\n",
    "import matplotlib.pyplot as plt\n",
    "from nltk.corpus import stopwords\n",
    "\n",
    "\n",
    "blob = TextBlob(open('data/shakespeare/the_merry_wives_of_windsor.txt').read())\n",
    "\n",
    "# get list of stop words ('a', 'an', etc.)\n",
    "stop_words = stopwords.words('english')\n",
    "\n",
    "# tuples of (word, count)\n",
    "items = blob.word_counts.items()\n",
    "\n",
    "# exclude stop words\n",
    "items = [ item for item in items if item[0] not in stop_words ]\n",
    "\n",
    "# sort by 2nd item (the count)\n",
    "sorted_items = sorted(items, key=lambda x: x[1], reverse=True)\n",
    "\n",
    "# slice first 20 of sorted list\n",
    "top20 = sorted_items[1:21]\n",
    "\n",
    "df = pd.DataFrame(top20, columns=['word', 'count'])\n",
    "\n",
    "axes = df.plot.bar(x='word', y='count', legend=False)\n",
    "\n",
    "plt.gcf().tight_layout()\n",
    "\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    " "
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.7"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}
