{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Exploring, Analyzing and Visualizing Data"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Pandas' rich feature set can do all these things and more.  In this notebook we'll explore and work with a simple dataset to highlight many of these features.  "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<br>\n",
    "\n",
    "## Notebook Setup\n",
    "---"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Libraries"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "## import modules\n",
    "import numpy as np\n",
    "import pandas as pd\n",
    "\n",
    "import matplotlib\n",
    "import matplotlib.pyplot as plt\n",
    "import seaborn as sns"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Notebook Settings"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "## allow multiple statements to show output (i.e., not just the last)\n",
    "from IPython.core.interactiveshell import InteractiveShell\n",
    "InteractiveShell.ast_node_interactivity = \"all\"\n",
    "\n",
    "## disable the Pandas \"setting a copy of a slice\" warning\n",
    "# pd.options.mode.chained_assignment = None\n",
    "\n",
    "## disable the \"convert inf values to NaN\" warning\n",
    "import warnings\n",
    "warnings.filterwarnings(\"ignore\", \"use_inf_as_na\")\n",
    "\n",
    "### set default number of DataFrame rows printed to 8\n",
    "#pd.set_option('display.max_rows', 8)\n",
    "\n",
    "# set the size of matplotlib \"figures\" (i.e. images)\n",
    "matplotlib.rcParams['figure.figsize'] = [7.5,5.0]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Quick Views of Data"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# import a large dataset, set the date column (EST) to the index and set the type to datetime64\n",
    "dfny = pd.read_csv('data/weather_newyork.csv', index_col='EST', parse_dates=['EST'], dayfirst=False)\n",
    "dfny.loc['2016-01': '2016-01-05']"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Viewing Large Tables in Searchable HTML"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Large tables are not easily viewable in Jupyter; the 'to_page()' function writes any dataframe to an HTML page."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "def to_page(df, filename='dataview.html'):\n",
    "    import os\n",
    "    wfh = open(filename, 'w')\n",
    "    if os.path.isfile('table_stylesheet.html'):  \n",
    "        wfh.write(open('table_stylesheet.html').read() + '\\n')\n",
    "    wfh.write(df.to_html())\n",
    "    wfh.close()\n",
    "\n",
    "to_page(dfny)       # <-- anywhere below this cell, call this function \n",
    "                    #     with any dataframe to see it in dataview.html "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "After running the above code, look in the folder this notebook is in to find <B>dataview.html</B>, and double-click to view (or open in your browser).  Text can be searched with a browser find (usually <B>Cmd-F/Ctrl-F</B>).  \n",
    "\n",
    "Anytime you wish to view the whole of a DataFrame, call <B>to_page(<I>df</I>)</B> (assuming your DataFrame is named <B><I>df</I></B>) and refresh the browser window.  "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Plotting a simple visualization"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "It's important to check in on your data from time to time in a way that eyeballing a table won't allow.  Anomalies, outliers, and the general \"character\" of the data can be conveyed with a simple chart.  "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# view mean temp column in a line chart\n",
    "df = pd.DataFrame({ 'Mean Temp': dfny['Mean TemperatureF'], 'mean': dfny['Mean TemperatureF'].mean()});\n",
    "df.plot()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# group mean temp by month\n",
    "monthly_mean = dfny.groupby([dfny.index.month])['Mean TemperatureF'].mean()\n",
    "\n",
    "# plot a bar chart of monthly mean temp\n",
    "monthly_mean.plot(kind='bar', rot=45);"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# delete previous data - we don't need it now\n",
    "del dfny; del monthly_mean"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Data\n",
    "\n",
    "This notebook uses the following datasets, which represent a record of sales as well as corresponding customer data:\n",
    "\n",
    "`data/customer_sales.csv`\n",
    "\n",
    "`data/customers.csv`\n",
    "\n",
    "We begin by converting all our string columns to type 'string' -- by default they are assigned the generic dtype `object`.  This helps clarify the intended type for these columns, ensures that numeric values like 'PotentialCustomerID' will not be seen as integers (and respond to methods like `.sum()`), and is more efficient in terms of storage and string operations we might want to apply."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# read sales data\n",
    "df_sales_all = pd.read_csv(\"data/customer_sales.csv\")\n",
    "\n",
    "# convert string columns to string type\n",
    "# numeric values should be excluded from aggregations and other numeric summaries\n",
    "df_sales_all.PotentialCustomerID = df_sales_all.PotentialCustomerID.astype('string')\n",
    "df_sales_all.region              = df_sales_all.region.astype('string')\n",
    "df_sales_all.marketing_exposure  = df_sales_all.marketing_exposure.astype('string')\n",
    "df_sales_all.num_accounts        = df_sales_all.num_accounts.astype('string')\n",
    "\n",
    "# only interested in existing sales  \n",
    "df_sales = df_sales_all.loc[ (df_sales_all.sale_amount > 0) ]\n",
    "df_sales\n",
    "\n",
    "# .query() method:  alternative to .loc[] with condition\n",
    "df_sales2 = df_sales_all.query('sale_amount > 0')\n",
    "\n",
    "to_page(df_sales)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "## import 'clean' customer data (same as that we cleaned \n",
    "## and saved in the 'data cleaning' notebook)\n",
    "df_customer = pd.read_csv('data/customers_clean.csv')\n",
    "df_customer"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<br>\n",
    "\n",
    "## Exploratory Data Analysis\n",
    "---\n",
    "Let's get to know this data by:\n",
    "* looking at and visualizing the data\n",
    "* identifying patterns, trends, and outliers\n",
    "* generating hypotheses for further investigation. \n",
    "\n",
    "Exploratory analysis helps in uncovering the underlying structure of the data, identifying data quality issues, and guiding the selection of appropriate data preprocessing and modeling techniques. By performing this analysis, we can make informed decisions, validate assumptions, and derive actionable insights from the data before proceeding to more advanced analysis or modeling tasks.\n",
    "\n",
    "We will take a two-fold approach to this analysis, which is to examine our data both **numerically and visually.** When we are looking at one variable at a time, we call that **univariate** data analysis. When we start to examine relationships between multiple variables, we call that **multivariate** data analysis. \n",
    "\n",
    "Data may also be categorized into **continuous** data (numeric data marking values on a spectrum) and **categorical** data (data expressed as one of a discrete set of values).  \n",
    "\n",
    "The approach we take to this analysis depends on the underlying data type of the variable(s) of interest. \n",
    "\n",
    "<br>\n",
    "\n",
    "## Univariate Data Analysis: Continuous Variables\n",
    "---\n",
    "We examine continuous variables by visually looking at their **distribution,** or calculating **summary statistics** of their distribution. Let's begin with the `CreditScore` variable from the customer file as an example. \n",
    "\n",
    "### Initial Summary Statistics\n",
    "\n",
    "We can begin by getting many summary statistics using `describe()`. One thing to look for at this point is whether the data all fits into a plausible theoretical range. What would this range be for credit scores?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "df_customer.CreditScore.describe()\n",
    "\n",
    "# [1, 1, 2,     3, 6, 11,     22, 29, 30,     31, 33, 33]\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Given that FICO credit scores range from 300 to 850, do you see anything funny in any of the above summary statistics?"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Histogram\n",
    "We might be interested in the creditworthiness of our customers, and ask:\n",
    "\n",
    "> <B>\"What is the distribution of FICO scores among customers?\"</B>\n",
    "\n",
    "We can use Pandas plotting methods to quickly get plots like the following histogram:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "# build a histogram\n",
    "df_customer.CreditScore.hist(bins=25);"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Overall, this does appear somewhat symmetric, we could even say normally distributed.\n",
    "    \n",
    "However, it would appear that we have some outliers in the upper region of the plot! We could try to deal with this now, but will revisit it later...     "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Although a histogram graphs a general answer to this question, we might also ask this question:\n",
    "\n",
    "> <B>\"What is the \"middle\" of the data?\"</B>\n",
    "\n",
    "The \"middle\" could be described as either the <I><B>median</B></I> (the centermost value in a sorted list of values, i.e. same number of values on either side of this value) or the <I><B>mean</B></I> (the average of all values).  In an even distribution, these would be very similar.  In this case however, because of the outliers, the median and the mean are not so close to each other. \n",
    "\n",
    "The \"middle\" of a continuous distribution is an important idea that we often use when describing and comparing distributions. Be clear about which statistical measures you employ when you want to convey an idea like \"the middle.\""
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "df_customer['CreditScore'].median()\n",
    "df_customer['CreditScore'].mean()"
   ]
  },
  {
   "cell_type": "raw",
   "metadata": {},
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Always Visualize!\n",
    "\n",
    "We might have missed this data problem if we never visualized this particular column! If you only calculate a couple basic summary statistics (like the mean and standard deviation) in your initial data examination/cleaning steps, you are not doing enough...\n",
    "\n",
    "Put your eyes on each variable, especially anything you will use in your reports or models. "
   ]
  },
  {
   "cell_type": "raw",
   "metadata": {},
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Seaborn Continuous Plots\n",
    "\n",
    "We can also use the seaborn module to examine distribution plots. The seaborn module provides a convenient interface for building plots that have very nice looking default settings."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "# Using seaborn to produce a histogram\n",
    "sns.displot(df_customer['CreditScore'], kde=True, rug=True)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Univariate Data Analysis: Categorical Variables\n",
    "---\n",
    "\n",
    "We take a different approach to this analysis for categorical data. We consider the distribution of categorical data to be the counts of the unique values or levels, or alternatively the observed probabilities for each value or level. \n",
    "\n",
    "Let's examine the `df_sales` table with respect to the distribution of `UrbanRural`. \n",
    "\n",
    "Remember, the goal is to understand the relative probabilities of each value or level occurring.  So we might ask this questions:\n",
    "\n",
    "> <B>\"What percent of sales occured for customers that live in urban vs. rural areas?\"</B>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "df_sales.head()"
   ]
  },
  {
   "cell_type": "raw",
   "metadata": {},
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "If we use the `.describe()` method, how well does that answer the question?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "df_sales.UrbanRural.info()\n",
    "\n",
    "# Use the .describe() method on a DataFrame:\n",
    "# summarize numeric columns\n",
    "#df_sales.describe()"
   ]
  },
  {
   "cell_type": "raw",
   "metadata": {},
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Given that there are 2 unique values in UrbanRural, let's see how they are distributed, with `.value_counts()`:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "# get counts by level\n",
    "df_sales.UrbanRural.value_counts()"
   ]
  },
  {
   "cell_type": "raw",
   "metadata": {},
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "You can get \"probabilities\" (percentages) instead of counts with <B>normalize=True</B>:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "# Get percentages/probabilities instead of counts\n",
    "df_sales.UrbanRural.value_counts(normalize=True)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "By default, value_counts does not report any missing values. If you want to see them, you must set a flag."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "# Show missing values\n",
    "df_sales.UrbanRural.value_counts(normalize=True,          # also try this without 'normalize=True'\n",
    "                                    dropna=False)           "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In this data, we don't have any missing data in the `UrbanRural` column, so it is equivalent to the above table."
   ]
  },
  {
   "cell_type": "raw",
   "metadata": {},
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Bar Plots\n",
    "\n",
    "Bar plots usually require two steps:  first we get the value counts (or some other form of .groupby() aggregation), then take those summaries and feed them into the `plot()` function."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "# bar plot with counts\n",
    "( df_sales.UrbanRural.value_counts(sort=False)\n",
    "                     .plot(kind='bar', rot=45) );"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "(I'm using parentheses around the entire value so I can shorten the length of each line.)  "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<BR>Here is the same data but in a horizontal bar.  "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "# bar plot showing proportions, also horizontal\n",
    "df_sales.UrbanRural.value_counts(sort=False, normalize=True)   \\\n",
    "                   .plot(kind='barh');"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "(Here I'm using another method of splitting up a line:  the line continuation character or <B>\\\\</B>.  "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<BR>\n",
    "Alternatively we can use `countplot()` with seaborn. This saves us from performing the `value_counts()` step. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "sns.countplot(x='UrbanRural', data=df_sales, color='salmon'); "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<br>\n",
    "\n",
    "## Multivariate Data Analysis: Continuous by Categorical\n",
    "---\n",
    "If we are interested in looking at the relationship between a continuous variable and a categorical variable, we look at the distribution of the continuous variable across the different levels of the categorical variable. We can do so visually, or by using a group-by analysis.\n",
    "\n",
    "Let's check out sales related to other variables such as age. The question we are asking is: \n",
    "\n",
    "> <B>\"Is the distribution of `age` different when considered against 'Urban' vs. 'Rural'?\"</B>\n",
    "\n",
    "### Boxplots\n",
    "\n",
    "One appropriate plot here is a box plot.  This plot shows the <I>quartile</I> distribution.  "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "## visual representation by boxplots\n",
    "sns.boxplot(x=\"UrbanRural\",\n",
    "            y=\"age\",\n",
    "            data=df_sales);"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "One useful property of boxplots is that they can help you identify outliers. Plotting the quartiles of 'sale_amount' shows some rather high prices comapred to the others. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "sns.boxplot(x=\"UrbanRural\",\n",
    "            y=\"sale_amount\",\n",
    "            data=df_sales);"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Boxplots are built around quartiles of the data, and outliers are often defined as any value that lies more than 1.5 times the \"interquartile range\", or the 75th percentile value minus the 25th percentile value.  For a more detailed explanation see:\n",
    "\n",
    "> [Box Plot - Wikipedia: Example with Outliers](https://en.wikipedia.org/wiki/Box_plot#Example_with_outliers)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Colored Histograms\n",
    "\n",
    "Another way to visualize how distributions of a continuous variable are different across levels of a categorical variable is to use a colored  histogram or density plot.\n",
    "\n",
    "The \"hue\" value allows us a 2nd plot, similar to our 'y=' addition to the box plot."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "# colored histogram\n",
    "sns.displot(x=\"sale_amount\", hue=\"UrbanRural\", data=df_sales, bins=25);        # try 'y=' instead of 'hue='"
   ]
  },
  {
   "cell_type": "raw",
   "metadata": {},
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Kernel Density Plots\n",
    "\n",
    "Kernel density plots are yet another way to visualize the distribution of a continuous variable across the levels of a categorical variable.  The curve is \"smoothed\" by inferring the probability of intermediate values along the range of existing values.  "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "# kernel density plot\n",
    "sns.kdeplot(x=\"sale_amount\", hue=\"UrbanRural\", data=df_sales);"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "## sale amount grouped by Urban vs. Rural\n",
    "df_sales.groupby('UrbanRural').sale_amount.mean()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can even add another dimension to the boxplot. In this case we will use color as an additional signifier to allow us to look for relationships across three variables at once."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "# add a third dimension to this plot\n",
    "sns.boxplot(x=\"UrbanRural\", \n",
    "            y=\"sale_amount\",\n",
    "            hue='current_customer',\n",
    "            data=df_sales,\n",
    "            showmeans=True);          # marked by the green triangle"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Revisit Credit Scores\n",
    "Let's revisit those credit scores from the customer data set. There was a set of outliers than had implausibly high values. What if those values are related to another variable that we have measured?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "## are the outliers related to sign up program?\n",
    "sns.displot(x='CreditScore', \n",
    "            hue='SignUpProgramClean', \n",
    "            data=df_customer, \n",
    "            kind='kde');"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Okay... It's not the sign up program."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "What other factors might help identify this issue?  Could it be related to state of residence?  We can check how many customers come from each state:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "df_customer.State.value_counts()"
   ]
  },
  {
   "cell_type": "raw",
   "metadata": {},
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "## are the outliers related to customer's state of residence?\n",
    "sns.displot(x='CreditScore', \n",
    "            hue='State', \n",
    "            data=df_customer, \n",
    "            kind='kde', \n",
    "            legend=True,           # the legend just added clutter here\n",
    "            warn_singular=False     # some states have only a single customer\n",
    "           ); "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We got it! All the customers with strange credit scores are from the same state.\n",
    "\n",
    "Which state was it???"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# df_customer.loc[ df_customer['CreditScore'] > 850, 'State' ].unique()\n",
    "\n",
    "df_customer.query('CreditScore > 850').State.unique()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "It appears that all the strange credit scores are from customers in New Jersey."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<br>\n",
    "\n",
    "## Multivariate Data Analysis: Continuous by Continuous Data\n",
    "---\n",
    "\n",
    "So how can we identify the relationship between two continuous variables? This is going to be the same idea as what we just went over with continuous/categorical data.  We might ask a question like this one: \n",
    "\n",
    "> <B>\"How does age correlate with sale amount?\"</B>\n",
    "\n",
    "So what does that look like when both variables are continuous? Let's first consider `age.`"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "df_sales.age.describe()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "df_sales.age.hist(bins=25);"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "sns.boxplot(x=\"age\",\n",
    "            data=df_sales);"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Scatterplots\n",
    "\n",
    "The most common method for examining associations between two continuous variables is to make a scatterplot.\n",
    "\n",
    "Here we access the Pandas DataFrame plotting module, where there is a `scatter()` function we can use the specify which columns from the DataFrame should be used in constructing the plot."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "# cont by cont\n",
    "df_sales.plot.scatter(x='age',\n",
    "                      y='sale_amount');"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This is simulated data, so in this case it may look a little unusual. Our young spenders seem similar to our older spenders.\n",
    "\n",
    "Do you see a shape to this data? How would you describe this shape?\n",
    "\n",
    "There is an issue here with data density and data points on top of each other (\"occlusion.\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "# Too much data! Adjust the alpha (opacity) to see better.  (0.2 is a good number)\n",
    "df_sales.plot.scatter(x='age',\n",
    "                      y='sale_amount',\n",
    "                      alpha=0.2);"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Regression Line\n",
    "\n",
    "We can also try to summarize the relationship between two variables using a *line of best fit*. In the next example we try to fit a linear regression line to this data.  We can use the seaborn `.regplot()` function, an analogue to `.scatter()`, along with a line that should hopefully track a trend in the relationship bewteen the two factors."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "# Show a regression line\n",
    "sns.regplot(x=\"age\", \n",
    "            y=\"sale_amount\",\n",
    "            data=df_sales,\n",
    "            scatter_kws={'alpha':0.3, 'marker':\"v\"},\n",
    "            line_kws={'color':'magenta'});"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "What is your conclusion... is there any relationship between age and sale amount? \n",
    "\n",
    "> Do you think a straight line is the \"best fit\" for this data?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "## try a more flexible line-fitting approach\n",
    "sns.regplot(x=\"age\", \n",
    "            y=\"sale_amount\",\n",
    "            data=df_sales,\n",
    "            lowess=True,                            # <--- note this addition\n",
    "            scatter_kws={'alpha':0.3, 'marker':\"v\"},\n",
    "            line_kws={'color':'magenta'});"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "LOWESS (\"locally weighted scatterplot smoothing\" or \"locally weighted linear regression\") uses an algorithm from the `statsmodels` module to create a smooth line through a timeplot or scatter plot to help you to see relationship between variables and to foresee trends.  "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Line Plots\n",
    "Another way to examine the relationship between two continuous variables is to make a line plot. This type of plot uses an aggregating function to summarize how one variable changes across the levels of another. It's a very similar idea to the regression lines we were previously fitting on top of the scatterplots.\n",
    "\n",
    "Line plots are particularly common when you have a measure of time as a variable, but let's make a line plot here to continue examining the relationship between age and sales amounts. In this case the `lineplot()` function is performing an aggregation using the mean function as an estimator. What we are viewing in the following plot is the mean sale amounts across different ages.\n",
    "\n",
    "The lightly colored 'halo' around the line is the \"confidence interval band\" or \"error band\".  This represents the variability in the mean of the data at each point.  (The \"confidence interval\" is typically set at 95% -- that is, it depicts where 95% of the values lie in relation to the mean.)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# mean sale amount across age\n",
    "sns.lineplot(x='age',\n",
    "             y='sale_amount',\n",
    "             data=df_sales,\n",
    "             estimator='mean');"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# median sale amount across age\n",
    "sns.lineplot(x='age',\n",
    "             y='sale_amount',\n",
    "             data=df_sales,\n",
    "             estimator=np.median);     # <-- same as above, but 'median' instead of 'mean'"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can use color (or 'hue') and line types to see how this differs across levels of another categorical variable.  (It is a best practice to adjust both the hue and line type to make your plots color-blind friendly.)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# mean sale amount across age stratified by place of residence\n",
    "sns.lineplot(x='age',\n",
    "             y='sale_amount',\n",
    "             hue='UrbanRural',\n",
    "             style='UrbanRural',\n",
    "             data=df_sales,\n",
    "             estimator='mean');"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<br>\n",
    "\n",
    "## Multivariate EDA: Categorical by Categorical\n",
    "---\n",
    "Another example of pairwise data analysis is to examine if a categorical variable is related to another categorical variable. This is traditionally done by looking at frequency tables. \n",
    "\n",
    "We can start by looking at the overall probability of a variable taking on a specific value."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "df_sales.activity.value_counts()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "df_sales.activity.value_counts(normalize=True)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Do you think `activity` and `UrbanRural` are related in this dataset? If `activity` is not related to `UrbanRural`, then the probability of someone living in an urban area should be the same across all the different `activity` levels. If there is a relationship, it will deviate.\n",
    "\n",
    "We can check this by constructing a frequency table using the `pd.crosstab()` function. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "# Frequency table with counts\n",
    "pd.crosstab(df_sales['activity'],\n",
    "            df_sales['UrbanRural'])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can add the Location percentages by identifying the correct dimension, and adding the `normalize='columns'` argument. If we wanted the row-wise percentages, we could choose `normalize='index'`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "pd.crosstab(df_sales['activity'],\n",
    "            df_sales['UrbanRural'],\n",
    "            normalize='columns')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Since the likelihood of being <B>Rural</B> or <B>Urban</B> is the same for each activity level, it appears that there is no relationship between the two factors.  What do you think?\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Additional Types of Plots\n",
    "---\n",
    "There are a couple of additional types of plots that are useful to know about when doing exploratory analysis.\n",
    "\n",
    "### Pairs Plot\n",
    "One quick and easy way to look at all the continuous (not discrete) variables in the dataset is to create a pairs plot. This plots the histogram of each variable on the diagonal, with a scatterplot on the off-diagonals."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "## WARNING, this might take a couple of minutes to run\n",
    "sns.pairplot(df_sales);"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The only really interesting trend we observe here is the age by sales amount relationship.\n",
    "\n",
    "### Small Multiples\n",
    "\n",
    "Sometimes it helps to break up a visualization into many smaller plots for visual clarity.\n",
    "\n",
    "Other times it helps to \"facet\" a plot across a variable, to provide side-by-side comparisons. \n",
    "\n",
    "Seaborn provides many variations on this. We will use the `lmplot()` function which builds scatterplots and adds a best fitting regression line. Specifying the `col=` and `row=` arguments tells the plotting function how to stratify the subplots."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "sns.lmplot(data=df_sales,\n",
    "           x='age',\n",
    "           y='sale_amount',\n",
    "           row='UrbanRural',\n",
    "           col='activity',\n",
    "           lowess=True,\n",
    "           line_kws={'color':'red'});"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The relationship between age and sale amount seems very consistent between all the combinations of levels of `activity` and 'UrbanRural`."
   ]
  }
 ],
 "metadata": {
  "anaconda-cloud": {},
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.14.2"
  },
  "toc": {
   "base_numbering": 1,
   "nav_menu": {
    "height": "585.994px",
    "width": "429.219px"
   },
   "number_sections": true,
   "sideBar": true,
   "skip_h1_title": false,
   "title_cell": "Table of Contents",
   "title_sidebar": "Contents",
   "toc_cell": false,
   "toc_position": {
    "height": "calc(100% - 180px)",
    "left": "10px",
    "top": "150px",
    "width": "229.6px"
   },
   "toc_section_display": true,
   "toc_window_display": false
  },
  "varInspector": {
   "cols": {
    "lenName": 16,
    "lenType": 16,
    "lenVar": 40
   },
   "kernels_config": {
    "python": {
     "delete_cmd_postfix": "",
     "delete_cmd_prefix": "del ",
     "library": "var_list.py",
     "varRefreshCmd": "print(var_dic_list())"
    },
    "r": {
     "delete_cmd_postfix": ") ",
     "delete_cmd_prefix": "rm(",
     "library": "var_list.r",
     "varRefreshCmd": "cat(var_dic_list()) "
    }
   },
   "types_to_exclude": [
    "module",
    "function",
    "builtin_function_or_method",
    "instance",
    "_Feature"
   ],
   "window_display": false
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}
