Python 3

home

All Slides on One Page




Web Scraping

Web Scraping with Beautiful Soup (bs4)

Beautiful Soup parses XML or HTML documents, making text and attribute extraction a snap.


Here we are passing the text of a web page (obtained by requests) to the BS parser:

from bs4 import BeautifulSoup
import requests

response = requests.get('http://www.nytimes.com')

soup = BeautifulSoup(response.text, 'html.parser')

# show HTML in "pretty" form
print(soup.prettify())

# show all plain text in a page
print(soup.get_text())

The result is a BeautifulSoup object which we can use to search for tags and data.


For the following examples, let's use the HTML provided on the Beatiful Soup Quick Start page:

<!doctype html>
<html>
  <head>
    <title>The Dormouse's story</title>
  </head>
  <body>
    <p class="story_title"><b>The Dormouse's story</b></p>

    <p class="story">Once upon a time there were three little sisters; and their names were
        <a href="http://example.com/elsie" class="sister eldest" id="link1">Elsie</a>,
        <a href="http://example.com/lacie" class="sister" id="link2">Lacie</a> and
        <a href="http://example.com/tillie" class="sister" id="link3">Tillie</a>;
    and they lived at the bottom of a well.</p>

    <p class="story">They were happy, and eventually died.  The End.</p>
  </body>
</html>

bs4: finding a tag with attributes, find() and find_all()

Finding the first tag by name using soup.attribute


The BeautifulSoup object's attributes can be used to search for a tag. The first tag with a name will be returned.

# first (and only) <title> tag
print(soup.title)              # <title>The Dormouse's story</title>

# first (of several) <p> tags
print(soup.p)                  # <p class="title"><b>The Dormouse's story</b></p>

Attributes can be chained to drill down to a particular tag:

print(soup.body.p.b)           # <b>The Dormouse's story</b>

However keep in mind these represent the first of each tag listed.


Finding the first tag by name: find()


find() works similarly to an attribute, but filters can be applied (discussed shortly).

print soup.find('a')          <a class="sister eldest" href="http://example.com/elsie" id="link1">Elsie</a>

Finding all tags by name: find_all()


findall() retrieves a list of all tags with a particular name.

tags = soup.find_all('a')

bs4: finding a tag using varying criteria

Tag criteria can focus on a tag's name, its attributes, or text within the tag.


SEARCHING NAME, ATTRIBUTE OR TEXT Finding a tag by name


Links in a page are marked with the <A> tag (usually seen as <A HREF="">). This call pulls out all links from a page:

link_tags = soup.find_all('a')

Finding a tag by tag attribute and/or name and tag attribute


# all <a> tags with an 'id' attribute of link1
link1_a_tags = soup.find_all('a', id="link1")

# all tags (of any name) with an 'id' attribute of link1
link1_tags = soup.find_all(id="link1")

"multi-value" tag attribute


CSS allows multiple values in an attribute:

<a href="http://example.com/elsie" class="sister eldest" id="link1">Elsie</a>

If we'd like to find a tag through this value, we pass a list:

link1_elsie_tag = soup.find(class_=['sister', 'eldest'])

Finding a tag by string within the tag's text


All <a> tags containing text 'Dormouse'

elsie_tags = soup.find_all('a', text='Dormouse')

FILTER TYPES: STRING, LIST, REGEXP, FUNCTION string: filter on the tag's name


tags = soup.find_all('a')          # return a list of all <a> tags

list: filter on tag names


tags = soup.find_all(['a', 'b'])   # return a list of all <a> or <b> tags

regexp: filter on pattern match against name


import re
tags = soup.find_all(re.compile('^b'))      # a list of all tags whose names start with 'b'

re.compile() produces a pattern object that is applied to tag names using re.match()


function: filter if function returns True


soup.find_all(lambda tag: tag.name == 'a' and 'mysite.com' in tag.get('href'))

This lambda is a special function in the form lambda arg: return value. It acts just like a function except that the 2nd half (the return value on the right side of the colon) must be a single statement. The findall() above is saying 'for each tag in the soup, give me only those where the name of the tag is "a" and also has an "href" parameter with a value that contains mysite.com'.


bs4: the Tag object

Tags' attributes and contents can be read; they can also be queried for tags and text within


body_text = """
    <BODY class="someclass otherclass">
        <H1 id='mytitle'<This is a headings</H1>
        <A href="mysite.com"<This is a link</A>
    </BODY>
"""

An HTML tag has four types of data: 1. The tag's name ('BODY' 'H1' or 'A') 2. The tag's attributes (<BODY class=, H1 id= or <A href=) 3. The tag's text ('This is a header' or 'This is a link') 4. The tag's contents (i.e., tags within it -- for <BODY>, the <H1> and <A> tags)


from bs4 import BeautifulSoup
soup = BeautifulSoup(body_text, 'html.parser')

h1 = soup.body.h1        # h1 is a Tag object
print(h1.name)            # u'h1'
print(h1.get('id'))       # u'mytitle'
print(h1.attrs)           # {u'id': u'mytitle'}
print(h1.text)            # u'This is a heading'

body = soup.body         # body is a Tag object
print(body.name)          # u'body'
print(body.get('class'))  # ['someclass', 'otherclass']
print(body.attrs)         # {'class': ['someclass', 'otherclass']}
print(body.text)          # u'\nThis is a heading\nThis is a link\n'

A tag's child tags can be searched the same as the BeautifulSoup object

body = soup.body         # find the <body> tag in this document

atag = body.find('a')    # find first <a> tag in this <body> tag

Sidebar: Sending email with smtplib

Sending mail is simple when you have an SMTP server running and available on your host computer. Python's smtplib module makes this easy:


#!/usr/bin/env python

# Import smtplib for the actual sending function
import smtplib

# Import the email modules we'll need
from email.mime.text import MIMEText

# Create a text/plain message formatted for email
msg = MIMEText('Hello, email.')

from_address = 'dbb212@nyu.edu'
to_address = 'david.beddoe@gmail.com'
subject = 'Test message from a Python script'

msg['Subject'] = subject
msg['From'] =    from_address
msg['To'] =      to_address

s = smtplib.SMTP('localhost')
s.sendmail(from_address, [to_address], msg.as_string())
s.quit()



Packages

Introduction: Packages

A package is a directory of files that work together as a Python application or library module.


Many applications or library modules consist of more than one file. A script may require configuration files or data files; some applications combine several .py files that work together. In addition, programs need unit tests to ensure reliability. A package groups all of these files (scripts, supporting files and tests) together as one entity. In this unit we'll discover Python's structure and procedures for creating packages. Some of the steps here were taken from this very good tutorial on packages: https://python-packaging.readthedocs.io/en/latest/minimal.html


Package Structure and the __init__.py File

The base of a package is a directory with an __init__.py file.


Folder structure for package davehello:

davehello/                      # base package folder - name is discretionary
    davehello/                  # module folder -  usually same name
        __init__.py             # initial script -- this is run first
    setup.py                    # setup file -- discussed below

The initial code for our program: __init__.py

def greet():
    return 'hello, world!'

The names of the folders are up to you. The "outer" davehello/ is the name of the base package folder. The "inner" davehello/ is the name of your module. These can be the same or different. setup.py is discussed next.


setup.py: the installation script

This file describes the script and its authorship.


Inside setup.py put the following code, but replace the name, author, author_email and packages (this list should reflect the name in name):

from setuptools import setup

setup( name='davehello',
       version='0.1',
       description='This module greets the user.  ',
       url='',                # usually a github URL
       author='David Blaikie',
       author_email='david@davidbpython.com',
       license='MIT',
       packages=['davehello'],
       install_requires=[ ],
       zip_safe=False )

setuptools is a Python module for preparing modules. The setup() function establishes meta information for the package.


url can be left blank for now. Later on we will commit this package to github and put the github URL here. packages should be a list of packages that are part this package (as there can be sub-packages within a package); however we will just work with the one package.


Again, folder structure for package davehello with two files:

davehello/                      # base package folder - name is discretionary
    davehello/                  # module folder -  usually same name
        __init__.py             # initial script -- this is run first
    setup.py                    # setup file -- discussed below

Please doublecheck your folder structure and the placement of files -- this is vital to being able to run the files.


Installing Locally

pip install can install your module into your own local Python module directories.


First, make sure you're in the same directory as setup.py. Then from the Unix/Mac Terminal, or Windows Command Prompt :

$ pip install .                            # $ means Unix/Mac prompt
Processing /Users/david/Desktop/davehello
Installing collected packages: davehello
  Running setup.py install for davehello ... done
Successfully installed davehello-0.1

The install module copies your package files to a Python install directory that is part of your Python installation's sys.path. Remember that the sys.path holds a list of directories that will be searched when you import a module. If you get an error when you try to install, double check your folder structure and placement of files, and make sure you're in the same directory as setup.py.


If successful, you should now be able to open a new Terminal or Command Prompt window (on Windows use the Anaconda Prompt), cd into your home directory, launch a Python interactive session, and import your module:

$ cd /Users/david                # moving to my home directory, to make sure
                                 # we're running the installed version

$ python
Python 3.6.1 |Anaconda custom (64-bit)| (default, May 11 2017, 13:04:09)
[GCC 4.2.1 Compatible Apple LLVM 6.0 (clang-600.0.57)] on darwin
Type "help", "copyright", "credits" or "license" for more information.

>>> import davehello
>>> davehello.greet()
'hello, world!'
>>>

If you get a ModuleNotFound error when you try to import: 1. It is possible that the files are not in their proper places, for example if __init__.py is in the same directory as setup.py. 2. It is possible that the pip you used installed the module into a different distribution than the one you are running.


$ pip --version
pip 9.0.1 from /Users/david/anaconda/lib/python3.6/site-packages (python 3.6)

Davids-MacBook-Pro-2:~ david$ python -V
Python 3.6.1 :: Anaconda custom (64-bit)
Davids-MacBook-Pro-2:~ david$

Note that my pip --version path indicates that it's running under Anaconda, and my python -V also indicates Anaconda.


Package Development Directory vs. Installation Directory

Development Directory is where you created the files; Installation Directory is where Python copied them upon install.


Keep in mind that when you import a module, the current directory will be searched before any directories on the sys.path. So if your command line / Command Prompt / Terminal session is currently in the same directory as setup.py (as we had been before we did a cd to my home directory), you'll be reading from your local package, not the installed one. So you won't be testing the installation until you move away from the package directory.


To see which folder the module was installed into, make sure you're not in the package directory; then read the module's __file__ attribute:

$ cd /Users/david                # moving to my home directory, to make sure
                                 # we're running the installed version

$ python
Python 3.6.1 |Anaconda custom (64-bit)| (default, May 11 2017, 13:04:09)
[GCC 4.2.1 Compatible Apple LLVM 6.0 (clang-600.0.57)] on darwin
Type "help", "copyright", "credits" or "license" for more information.

>>> import davehello
>>> davehello.__file__

'/Users/david/anaconda/lib/python3.6/site-packages/davehello/__init__.py'

Note that this is not one of my directories or one that I write to; it is a common install directory for Anaconda Python.


Contrast this with the result if you're importing the module from the package directory (the same directory as setup.py):

$ cd /Users/david/Desktop/davehello/          # my package location

$ python

>>> import davehello
>>> davehello.__file__

'/Users/david/Desktop/davehello/davehello/__init__.py'

Note that this is one of my local directories.


Making Changes and Reinstalling

Changes to the package will not be reflected in the installed module unless we reinstall.


If you make a change to the package source files, it won't be reflected in your Python installation until you reinstall with pip install. (The exception to this is if you happen to be importing the module from within the package itself -- then the import will read from the local files.)


To reinstall a previously installed module, we must include the --upgrade flag:

$ pip install . --upgrade
Processing /Users/david/Desktop/davehello
Installing collected packages: davehello
  Found existing installation: davehello 0.1
    Uninstalling davehello-0.1:
      Successfully uninstalled davehello-0.1
  Running setup.py install for davehello ... done
Successfully installed davehello-0.1

Adding files

__init__.py is the "gateway" file; the bulk of code may be in other .py files in the package.


Many packages are made of up several .py files that work together. They may be files that are only called internally, or they may be intended to be called by the user. Your entire module could be contained within __init__.py, but I believe this is customarily used only as the gateway, with the bulk of module code in another .py file. In this step we'll move our function to another file.


hello.py (new file -- this can be any name)

def greet():
    return 'hello, new file!'

__init__.py

from .hello import greet       # .hello refers to hello.py in the base package directory

New folder structure for package davehello:

davehello/                      # base folder - name is discretionary
    davehello/                  # package folder -  usually same name
        __init__.py             # initial script -- this is run first
        hello.py                # new file
    setup.py                    # setup file -- discussed below

Don't forget to reinstall the module once you've finalized changes. However you can run the package locally (i.e., from the same directory as setup.py) without reinstalling. When the package is imported, Python reads and executes the ___init___.py program. This file is now importing greet from hello.py into the module's namespace, making it available to the user under the package name davehello.


The user can also reach the variable in hello.py directly, by using attribute syntax to reach the module -- so both of these calls to greet() should work:

>>> import davehello as dh
>>> dh.greet()                    # 'hello, new file!'
                                  # (because __init__.py imported greet from hello.py)

>>> dh.hello.greet()              # 'hello, new file!'
                                  # (calling it directly in hello.py)

>>> from davehello import hello
>>> hello.greet()                 # 'hello, new file!'

The mechanics of files and variables in packages

Packages provide for accessing variables within multiple files.



Specifying Dependencies

Dependencies are other modules that your module may need to import.


If your module imports a non-standard module like splain, it is known as a dependency. Dependencies must be mentioned in the setup() spec. The installer will make sure any dependent modules are installed so your module works correctly.


setup(name='davehello',
      version='0.1',
      description='This module greets the user.  ',
      url='',                # usually a github URL
      author='David Blaikie',
      author_email='david@davidbpython.com',
      license='MIT',
      packages=['davehello'],
      install_requires=[ 'splain' ],
      zip_safe=False)

This would not be necessary if the user already had splain installed. However, if they didn't, we would want the install of our module to result in the automatic installation of the splain module. (Please note that splain.py has not yet been uploaded to PyPI, so the above dependency will not work.)


Adding Tests

Tests belong in the package; thus anyone who downloads the source can run the tests.


In a package, tests should be added to a tests/ directory in the package root (i.e., in the same directory as setup.py).


We will use pytest for our testing -- the following configuration values need to be added to setup() in the setup.py file:

test_suite='pytest'
setup_requires=['pytest-runner']
tests_require=['pytest']

Here's our updated setup.py:

from setuptools import setup

setup(name='davehello',
      version='0.1',
      description='This module greets the user.  ',
      url='',                # usually a github URL
      author='David Blaikie',
      author_email='david@davidbpython.com',
      license='MIT',
      packages=['davehello'],
      install_requires=[ ],
      test_suite='pytest',
      setup_requires=['pytest-runner'],
      tests_require=['pytest'],
      zip_safe=False)

As is true for most testing suites, pytest requires that our test filenames should begin with test_, and test function names begin with test_.


Here is our test program test_hello.py, with test_greet(), which tests the greet() function.

import pytest
import davehello as dh

def test_greet():
    assert dh.greet() == 'hello, world!'

Here's a new folder structure for package davehello:

davehello/                      # base folder - name is discretionary
    davehello/                  # package folder -  usually same name
        __init__.py             # initial script -- this is run first
        hello.py                # new file
    tests/
        test_hello.py
    setup.py                    # setup file -- discussed below

Now when we'd like to run the package's tests, we run the following at the command line:

$ python setup.py pytest
running pytest
running egg_info
writing davehello.egg-info/PKG-INFO
writing dependency_links to davehello.egg-info/dependency_links.txt
writing top-level names to davehello.egg-info/top_level.txt
reading manifest file 'davehello.egg-info/SOURCES.txt'
writing manifest file 'davehello.egg-info/SOURCES.txt'
running build_ext
------------------------------- test session starts -------------------------------

platform darwin -- Python 3.6.1, pytest-3.0.7, py-1.4.33, pluggy-0.4.0
rootdir: /Users/david/Desktop/davehello, inifile:
collected 1 items

tests/test_hello.py F
----------------------------------- FAILURES -----------------------------------

    def test_greet():
>       assert dh.greet() == 'hello, world!'
E       AssertionError: assert 'hello, new file!' == 'hello, world!'
E         - hello, new file!
E         + hello, world!

tests/test_hello.py:7: AssertionError
--------------------------- 1 failed in 0.03 seconds ---------------------------

oops, our test failed. We're not supplying the right value to assert -- the function returns hello, new file! and our test is looking for hello, world!. We go into test_hello.py and modify the assert statement; or alternatively, we could change the output of the function.


After change has been made to test_hello.py to reflect the expected output:

$ python setup.py pytest
running pytest
running egg_info
writing dblaikie_hello.egg-info/PKG-INFO
writing dependency_links to dblaikie_hello.egg-info/dependency_links.txt
writing top-level names to dblaikie_hello.egg-info/top_level.txt
reading manifest file 'dblaikie_hello.egg-info/SOURCES.txt'
writing manifest file 'dblaikie_hello.egg-info/SOURCES.txt'
running build_ext
------------------------------- test session starts -------------------------------
platform darwin -- Python 3.6.1, pytest-3.0.7, py-1.4.33, pluggy-0.4.0
rootdir: /Users/david/Desktop/davehello, inifile:
collected 1 items

davehello/tests/test_hello.py .

----------------------------- 1 passed in 0.01 seconds ----------------------------

The output first shows us what setup.py is doing in the background, then shows collected 1 items to indicate that it's ready to run tests. The final statement indicates how many tests passed (or failed).


With these basic steps you can create a package, install it in your Python distribution, and prepare it for distribution to the world. May all beings be happy.




Distributing Modules and the PyPI Package Repo

PyPI: The Python Package Index

All publicly available modules can be found here.


https://testpypi.python.org/pypi


Registering your Application on PyPI

setuptools can do this for us automatically.


Davids-MacBook-Pro-2:dblaikie_hello david$ python setup.py register
running register
running egg_info
writing dblaikie_hello.egg-info/PKG-INFO
writing dependency_links to dblaikie_hello.egg-info/dependency_links.txt
writing top-level names to dblaikie_hello.egg-info/top_level.txt
reading manifest file 'dblaikie_hello.egg-info/SOURCES.txt'
writing manifest file 'dblaikie_hello.egg-info/SOURCES.txt'
running check
We need to know who you are, so please choose either:
 1. use your existing login,
 2. register as a new user,
 3. have the server generate a new password for you (and email it to you), or
 4. quit
Your selection [default 1]:

twine for uploading to PyPI

$ pip install twine



Containers: Lists, Sets and Tuples

Introduction: Collections of values can be used for various types of analysis.

With a collection of numeric values, we can perform many types of analysis that would not be possible with a simple count or sum.


We will summarize a year's worth of data in the Fama-French file as we did previously, but be able to say much more about it.

var = [1, 4.3, 6.9, 11, 15]                            # a list container

print(f'count is {len(var)}')              # count is 5
print(f'sum is {sum(var)}      ')              # sum is 38.2
print(f'average is {sum(var)}')                # average is
                                                         # 7.640000000000001

print(f'max val is {max(var)}')                # max val is 15
print(f'min val is {min(var)}')                # min val is 1

print(f'top two: {var[3]}, {var[4]}')      # top two: 11, 15

print(f'median is {var[int(len(var) / 2)]}')   # median is 6.9')

Introduction: Collections of values can be used to determine membership.

Checking one list against another is a core task in data analysis. We can validate arguments to a program, see if a user id is in a "whitelist" of valid users, see if a product is in inventory, etc.


We will apply membership testing to a spell checker, which simply checks every word in a file against a "whitelist" of correctly spelled words.

valid_actions = ['run', 'stop', 'search', 'reset']

uinput = input('please enter an action:  ')

if uinput in valid_actions:          # if string can be found in list
    print(f'great, I will "{uinput}"')
else:
    print('sorry, action not found')

Objectives for this Unit (Containers: Lists, Sets and Tuples)

Containers broaden our data analysis powers significantly over simple looping and summing. We can:


  • Use lists to build up sequences of non-unique values.
  • Use sets to build up collections of unique values.
  • Use summary functions to summarize numeric data in containers (sum, max, min, etc.)
  • Use sorting and slicing to do ordered analysis (top 5, median, etc.)
  • Use membership analysis (in) to check a value against a collection of values.


    Container Objects: List, Set, Tuple

    Compare and contrast the characteristics of each container.


  • list: ordered, mutable sequence of objects
  • tuple: ordered, immutable sequence of objects
  • set: unordered, mutable, unique collection of objects
  • dict: unordered, mutable collection of object key-value pairs, with unique keys (discussed upcoming)

  • Summary for object: "List" container object

    A list is an ordered sequence of values.


    Initialize a List

    var = []                     # initialize an empty list
    
    var2 = [1, 2, 3, 'a', 'b']   # initialize a list of values

    Append to a List

    var = []
    
    var.append(4)                # Note well! call is not assigned
    var.append(5.5)              # list is changed in-place
    
    print(var)                                # [4, 5.5]

    Add Two Lists Together with +

    var = ['a', 'b', 'c']
    var2 = ['d', 'e', 'f']
    
    var3 = var + var2                         # ['a', 'b', 'c', 'd', 'e', 'f']

    Slice a List (compare to string slicing)

    var2 = [1, 2, 3, 'a', 'b']                # initialize a list of values
    
    sublist = var2[2:4]                       # [3, 'a']

    Subscript a List

    mylist = [1, 2, 3, 'a', 'b']              # initialize a list of values
    
    xx = mylist[3]                            # 'a'

    Get Length of a List (compare to len() of a string)

    mylist = [1, 2, 3, 'a', 'b']
    
    yy = len(mylist)                          # 5 (# of elements in mylist)

    Test for membership in a List

    mylist = [1, 2, 3, 'a', 'b']
    
    if 'b' in mylist:                         # this is True for mylist
        print("'b' can be found in mylist")   # this will be printed
    
    print('b' in mylist)                      # "True":  the in operator
                                              # actually returns True or False

    Loop through a List (compare to looping through a file)

    mylist = [1, 2, 3, 'a', 'b']
    
    for var in mylist:
        print(var)                            # prints 1, then 2, then 3,
                                              # then a, then b

    Sort a List: sorted() returns a list of sorted values

    mylist = [4, 9, 1.2, -5, 200, 20]
    
    smyl = sorted(mylist)                     # [-5, 1.2, 4, 9, 20, 200]

    Summary for object: "Set" container object

    A set is an unordered, unique collection of values.


    Initialize a Set

    myset = set()                  # initialize an empty set (note empty curly
                                   # are reserved for dicts)
    
    myset = {'a', 9999, 4.3}       # initialize a set with elements
    
    myset = {'a', 9999, 4.3}  # legacy approach:  past a list
                                   # to set()

    Add to a Set

    myset = set()                  # initialize an empty set
    
    myset.add(4.3)                 # note well method call not assigned
    myset.add('a')
    
    print(myset)                   # {'a', 4.3}    (order is not
                                   #                necessarily maintained)

    Get Length of a Set

    mixed_set = {'a', 9999, 4.3}
    
    setlen = len(mixed_set)              # 3

    Test for membership in a Set

    myset = {'b', 'a', 'c'}
    if 'c' in myset:                    # test is True
        print("'c' is in myset")        # this will be printed

    Loop through a Set

    myset = {'b', 'a', 'c'}
    for el in myset:
        print(el)                       # may be printed in seeming 'random' order

    Sort a Set: sorted() returns a list of sorted object values

    myset = {'b', 'a', 'c'}
    
    zz = sorted(myset)                  # ['a', 'b', 'c']

    Summary for object: "Tuple" container object

    A tuple is an immutable ordered sequence of values. Immutable means it cannot be changed once initialized.


    Initialize a Tuple

    var = ('a', 'b', 'c', 'd')          # initialize a tuple

    Slice a Tuple

    var = ('a', 'b', 'c', 'd')
    varslice = var[1:3]                 # ('b', 'c')

    Subscript a Tuple

    mytup = ('a', 'b', 'c')
    last = mytup[2]                     # 'c'

    Get Length of a Tuple

    mytup = ('a', 'b', 'c')
    tuplen = len(mytup)
    
    print(tuplen)                       # 3

    Test for membership in a Tuple

    mytup = ('a', 'b', 'c')
    if 'c' in mytup:
        print("'c' is in mytup")

    Loop through a Tuple

    mytup = ('a', 'b', 'c')
    for el in mytup:
        print(el)

    Sort a Tuple

    xxxx = ('see', 'i', 'you', 'ah')
    
    yyyy = sorted(xxxx)                  # ['ah', 'i', 'see', 'you']

    Add Two Tuples Together with +

    var =  ('a', 'b', 'c')
    var2 = ('d', 'e', 'f')
    
    var3 = var + var2                    # ('a', 'b', 'c', 'd', 'e', 'f')

    Summary for functions: len(), sum(), max(), min()

    Summary functions offer a speedy answer to basic analysis questions: how many? How much? Highest value? Lowest value?


    mylist = [1, 3, 5, 7, 9]        # initialize a list
    mytup = (99, 98, 95.3)          # initialize a tuple
    myset = {2.8, 2.9, 1.7, 3.8}        # initialize a set
    
    print(len(mylist))                   # 5
    print(sum(mytup))                    # 292.3 sum of values in mytup
    print(min(mylist))                   # 1 smallest value in mylist
    print(max(myset))                    # 3.8 largest value in myset

    Summary for function: range()

    The range() function takes one, two or three arguments and produces an iterable that returns a sequence of integers.


    counter = range(10)
    for i in counter:
        print(i)                        # prints integers 0 through 9
    
    for i in range(3, 8):               # prints integers 3 through 7
        print(i)

    If we need an literal list of integers, we can simply pass the iterable to a list:

    intlist = list(range(5))
    print(intlist)                      # [0, 1, 2, 3, 4]

    Summary for function: sorted()

    The sorted() function takes any sequence as argument and returns a list of the elements sorted by numeric or string value.


    x = {1.8, 0.9, 15.2, 3.5, 2}
    
    y = sorted(x)                       # [0.9, 1.8, 2, 3.5, 15.2]

    Irregardless of the sequence passed to sorted(), a list is returned.


    Summary task: Adding to Containers

    We can add to a list with append() and to a set with add().


    Add to a list

    intlist1 = [1, 2, 55, 4, 9]          # list of integers
    
    intlist1.append('hello')
    
    print(intlist1)                      # [1, 2, 55, 4, 9, 'hello']

    Add to a set

    mixed_set = {'a', 9999, 4.3}         # initialize a set with a list or tuple
    
    mixed_set.add('a')                   # not added - duplicate
    mixed_set.add('cool')
    
    print(mixed_set)                     # {'a', 9999, 4.3, 'cool'}
    

    We cannot add to a tuple, of course, since they are immutable!


    Summary task: Looping through Containers

    We can loop through any container with for, just like a file.


    Loop through a List

    mylist = ['a', 'b', 'c']
    for el in mylist:
        print(el)

    Loop through a Set

    myset = {'b', 'a', 'c'}
    for el in myset:
        print(el)                         # will be printed in 'random' order

    Loop through a Tuple

    mytup = ('a', 'b', 'c')
    for el in mytup:
        print(el)

    Loop through a String(??)

    mystr = 'abcdefghi'
    
    for x in mystr:
        print(x)                          # what do you see?

    Strings can be seen as sequences of characters.


    Summary task: Subscripting and Slicing Containers

    We can slice any ordered container -- for us, list or tuple.


    letters = ['a', 'b', 'c', 'd', 'e', 'f', 'g', 'h']
    first_four = letters[0:4]
    print(first_four)                     # ['a', 'b', 'c', 'd']
    
    # no upper bound takes us to the end
    print(letters[5:])                    # ['f', 'g', 'h']

    Remember the rules with slices:


       1) the 1st index is 0
       2) the lower bound is the 1st element to be included
       3) the upper bound is one above the last element to be included
       4) no upper bound means "to the end"; no lower bound means "from 0"

    We cannot subscript or slice a set, of course, beacuse it is unordered!


    Summary task: Checking for Membership

    We can check for value membership of a value within any container with in.


    mylist = [1, 2, 3, 'a', 'b']
    
    if 'b' in mylist:                        # this is True for mylist
        print("'b' can be found in mylist")  # this will be printed
    

    Summary task: Sorting Containers

    Sorting allows us to rank values, find a median, and more.


    mylist = [9.3, 2.1, 0.8]
    xxx = sorted(mylist)                     # a list:  [0.8, 2.1, 9.3]
    
    names = {'David', 'George', 'Adam'}
    yyy = sorted(names)                      # a list:
                                             # ['Adam', 'David', 'George']
    
    ints = (5, 9, 0, 8)
    zzz = sorted(ints)                       # a list:  [0, 5, 8, 9]

    No matter what sequence is passed to sorted(), a list is returned. What about a string?!


    Summary Exception: AttributeError

    An AttributeError exception occurs when calling a method on an object type that doesn't support that method.


    mylines = ['line1\n', 'line2\n', 'line3\n']
    
    mylines = mylines.rstrip()         # AttributeError:
                                       # 'list' object has no attribute 'rstrip'

    However, an 'attribute' is any name that appears after an object and a dot (object.attribute). The attribute is often a method, though it doesn't need to be.

    x = 'hello'
    x.upper()                          # 'upper' attribute of string 'x'
    
    import sys
    sys.argv                           # 'argv' attribute of module 'sys'
    
    sys.copyright                      # 'copyright' attribute of module 'sys'

    This exception may sometimes result from a misuse of the append() method, which returns None (not a list):

    mylist = ['a', 'b', 'c']
    
    # oops:  returns None -- call to append() should not be assigned
    mylist = mylist.append('d')
    
    mylist = mylist.append('e')        # AttributeError:  'NoneType'
                                       # object has no attribute 'append'

    Since append() isn't designed to return an object, it returns None (this makes it different from string methods like upper(), which returns a string).


    The correct use of append() does not assign the call:

    mylist = ['a', 'b', 'c']
    
    mylist.append('d')                 # now mylist equals ['a', 'b', 'c', 'd']

    Summary Exception: IndexError

    An IndexError exception indicates use of an index for a list/tuple element that doesn't exist.


    mylist = ['a', 'b', 'c']
    
    print(mylist[5])                  # IndexError:  list index out of range

    Since mylist does not contain a sixth item (i.e., at index 5), Python tells us it cannot complete this operation.


    Practical: looping through a data source and building up containers

    The "summary algorithm" is very similar to building a float sum from a file source. We loop; select; add.


    list: build a list of states

    state_list = []                               # initialize an empty list
    for line in open('student_db.txt'):
    
        elements = line.split(':')
        state_list.append(elements[3])            # add the state for this row
                                                  # to state_list
    
    chosen_state = input('enter a state ID:  ')
    state_freq = state_list.count(chosen_state)   # count # of occurrences
                                                  # of this string in list
                                                  # of strings
    
    print(f'{chosen_state} occurs {state_freq} times')

    The list count() method counts the number of times an item value (in this case, a string "state" value) appears in the list of state string values.


    set: build a set of unique states

    state_set = set()                             # initialize an empty set
    for line in open('student_db.txt'):
    
        elements = line.split(':')
        state_set.add(elements[3])                # add the state for this row
                                                  # to state_list
    
    chosen_state = input('enter a state ID:  ')
    
    if chosen_state in state_set:
       print('that is a valid state')
    else:
        print('that is not a valid state')

    Practical: checking for membership

    We use in to compare two collections.


    In this example, we have a list of ids and a set of valid ids. With looping and in we can build a list of valid and invalid ids.

    student_states = ['CA', 'NJ', 'VT', 'ME', 'RI', 'CO', 'NY']
    ne_states = {'ME', 'VT', 'NH', 'MA', 'RI', 'CT'}
    
    ne_student_states = []
    for state in student_states:
        if state in ne_states:
            ne_student_states.append(state)
    
    print('students in our school are from these New England states: ')
    print(ne_student_states)

    This kind of analysis can also be done purely with sets and we'll discuss these methods later in the course.


    Practical: treating a file as a list

    Data files can be rendered as lists of lines, and slicing can manipulate them holistically rather than by using a counter.


    In this example, we want to skip the 'header' line of the student_db.txt file. Rather than count the lines and skip line 1, we simply treat the entire file as a list and slice the list as desired:

    fh = open('student_db.txt')
    file_lines_list = fh.readlines()          # a list of lines in the file
    print(file_lines_list)
          # [ "id:address:city:state:zip",
          #   "jk43:23 Marfield Lane:Plainview:NY:10023",
          #   "ZXE99:315 W. 115th Street, Apt. 11B:New York:NY:10027",
          #   "jab44:23 Rivington Street, Apt. 3R:New York:NY:10002"  ]
    
    wanted_lines = file_lines_list[1:]        # take all but 1st element
                                              # (i.e., 1st line)
    for line in wanted_lines:
        print(line.rstrip())                    # jk43:23 Marfield Lane:
                                              # Plainview:NY:10023
    
                                              # ZXE99:315 W. 115th Street,
                                              # Apt. 11B:New York:NY:10027
    
                                              # jab44:23 Rivington Street,
                                              # Apt. 3R:New York:NY:10002

    Sidebar: removing a container element

    We rarely need to remove elements from a container, but here is how we do it.


    mylist = ['a', 'hello', 5, 9]
    myset = {1, 3, 9, 11, 16}
    
    popped = mylist.pop(0)         # remove the first element from mylist
                                   # (argument specifies the index to remove)
    
    mylist.remove(5)               # remove an element by value
    
    
    myset.pop()                    # remove a random element
    myset.remove(3)                # remove an element by value
    



    Modules

    Introduction: Modules

    Modules are files that contain reusable Python code: we often refer to them as "libraries" because they contain code that can be used in other scripts. It is possible to import such library code directly into our programs through the import statement -- this simply means that the functions in the module are made available to our program. Modules consist principally of functions that do useful things and are grouped together by subject. Here are some examples:


    So when we import a module in our program, we're simply making other Python code (in the form of functions) available to our own programs. In a sense we're creating an assemblage of Python code -- some written by us, some by other people -- and putting it together into a single program. The imported code doesn't literally become part of our script, but it is part of our program in the sense that our script can call it and use it. We can also define our own modules -- collections of Python functions and/or other variables that we would like to make available to our other Python programs. We can even prepare modules designed for others to use, if we feel they might be useful. In this way we can collaborate with other members of our team, or even the world, by using code written by others and by providing code for others to use.


    Objectives for the Unit: Modules


    Summary Statement: import modulename

    Using import, we can import an entire Python module into our own code.


    messages.py: a Python module that prints messages

    import sys
    
    def print_warning(msg):
        """write a message to STDOUT"""
        sys.stdout.write(f'warning:  {msg}\n')
    
    def log_message(msg):
        """write a message to the log file"""
        try:
            fh = open('log.txt', 'a')
            fh.write(str(msg) + '\n')
        except FileNotFoundError:
            print_warning('log file not readable')

    test.py: a Python script that imports messages.py

    #!/usr/bin/env python
    
    import messages
    
    print("test program running...")
    
    messages.log_message('this is an important message')
    messages.print_warning("I think we're in trouble.")

    The global variables in the module become attributes of the module. The module's variables are accessible through the name of the module, as its attributes.


    Summary statement: import modulename as convenientname

    A module can be renamed at the point of import.


    import pandas as pd
    import datetime as dt
    
    users = pd.read_table('myfile.data', sep=',', header=None)
    
    print("yesterday's date:  {dt.date.today() - dt.timedelta(days=1)}")

    Summary statement: from modulename import variablename

    Individual variables can be imported by name from a module.


    #!/usr/bin/env python
    
    from messages import print_warning, log_message
    
    print("test program running...")
    
    log_message('this is an important message')
    print_warning("I think we're in trouble.")

    Summary: module search path

    Python must be told where to find our own custom modules.


    Python's standard module directories When it encounters an import, Python searches for the module in a selected list of standard module directories. It does not search through the entire filesystem for modules. Modules like sys and os are located in one of these standard directories. Our own custom module directories Modules that we create should be placed in one or more directories that we designate for this purpose. In order to let Python know about our own module directories, we have a couple of options: PYTHONPATH environment variable The standard approach to adding our own module directories to the list of those that Python searches is to create or modify the PYTHONPATH environment variable. This colon-separated list of paths indicates any paths to search in addition to the ones Python normally searches.


    In your Unix .bash_profile file in your home directory, you would place the following command:

    export PYTHONPATH=$PYTHONPATH:/path/to/my/pylib:/path/to/my/other/pylib

    ...where /path/to/my/pylib and /path/to/my/other/pylib are paths In Windows, you can set the PYTHONPATH environment variable through the Windows GUI.


    manipulating sys.path

    import sys
    
    print(sys.path)
    
        # ['', '/Users/dblaikie/lib', '//anaconda/lib/python2.7',
        #  '//anaconda/lib/python2.7/plat-darwin',
        #  '//anaconda/lib/python2.7/plat-mac',
        #  '//anaconda/lib/python2.7/plat-mac/lib-scriptpackages',
        #  '//anaconda/lib/python2.7/lib-tk',
        #  '//anaconda/lib/python2.7/lib-old',
        #  '//anaconda/lib/python2.7/lib-dynload',
        #  '//anaconda/lib/python2.7/site-packages',
        #  '//anaconda/lib/python2.7/site-packages/PIL',
        #  '//anaconda/lib/python2.7/site-packages/Sphinx-1.3.1-py2.7.egg',
        #  '//anaconda/lib/python2.7/site-packages/aeosa',
        #  '//anaconda/lib/python2.7/site-packages/setuptools-19.1.1-py2.7.egg']
    
    sys.path.append('/path/to/my/pylib')
    
    import mymod     # if mymod.py is in /path/to/my/pylib, it will be found

    Once a python script is running, Python makes the PYTHONPATH search path available in a list called sys.path. Since it is a list, it can be manipulated; you are free to add whatever paths you wish


    However, please note that this kind of manipulation is rare because the needed changes are customarily made to the PYTHONPATH environment variable.


    Summary: coding all Python scripts as modules

    All Python scripts should be coded as modules: able to be both imported and executed.


    #!/usr/bin/env python
    
    
    """ helloworld.py:  print the "Hello! message """
    
    def print_hello(arg):
        print(f"Hello, {arg}!")
    
    def main():
        print_hello('World')
    
    if __name__ == '__main__':              # value is '__main__'
                                            # if this script is executed
                                            # value is 'helloworld'
                                            # if this script is imported
        main()

    If we run the above script, we'll see "Hello, World!". But if we import the above script, we won't see anything. Why? Because the if statement above will be true only if the script is executed. This is important behavior, because there are some scripts that we may want to run directly, but also allow others to import (in order to use the script's functions). Whether we intend to import a script or not, it is considered a "best practice" to build all of our programs in this way -- with a "main body" of statements collected under function main() and the call to main() inside the if __name__ == '__main__' gate.


    Summary: raising exceptions

    Causing an exception to be raised is the principal way a module signals an error to the importing script.


    A file called mylib.py

    def get_yearsum(user_year):
    
        user_year = int(user_year)
        if user_year < 1929 or user_year > 2013:
          raise ValueError(f'year {user_year} out of range')
    
        # calculate value for the year
    
        return 5.9         # returning a sample value (for testing purposes only)

    An exception raised by us is indistinguishable from one raised by Python, and we can raise any exception type we wish. This allows the user of our function to handle the error if needed (rather than have the script fail):


    import mylib
    
    while True:
    
        year = input('please enter a year:  ')
    
        try:
            mysum = mylib.get_yearsum(year)
            break
        except ValueError:
            print('invalid year:  try again')
    
    print('mysum is', mysum)

    Summary: installing modules

    Third-party modules must be downloaded and installed into our Python distribution.


    Unix

    $ sudo pip search pandas         # searches for pandas in the PyPI repository
    $ sudo pip install pandas        # installs pandas

    Installation on Unix requires something called root permissions, which are permissions that the Unix system administrator uses to make changes to the system. The below commands include sudo, which is a way to temporarily be granted root permissions.


    Windows

    C:\\Windows > pip search pandas   # searches for pandas in the PyPI repo
    C:\\Windows > pip install pandas  # installs pandas

    PyPI: the Python Package Index The Python Package Index at https://pypi.python.org/pypi is a repository of software for the Python programming language. There are more than 70,000 projects uploaded there, from serious modules used by millions of developers to half-baked ideas that someone decided to share prematurely. Usually, we encounter modules in the field -- shared through blog posts and articles, word of mouth and even other Python code. But the PPI can be used directly to try to find modules that support a particular purpose.


    Summary: the Python standard distribution of modules

    Modules included with Python are installed when Python is installed -- they are always available.


    Python provides hundreds of supplementary modules to perform myriad tasks. The modules do not need to be installed because they come bundled in the Python distribution, that is they are installed at the time that Python itself is installed. The documentation for the standard library is part of the official Python docs.


    From the Standard Distribution: the time module

    time gives us simple access to the current time or any other time in terms of epoch seconds, or seconds since the epoch began, which was set by Unix developers at January 1, 1970 at midnight!


    The module has a simple interface - it generally works with seconds (which are often a float in order to specify milliseconds) and returns a float to indicate the time -- this float value can be manipulated and then passed back into time to see the time altered and can be formatted into any display.

    import time
    
    epochsecs = time.time()                     # 1450818009.925441
                                                # (current time in epoch seconds)
    
    print(time.ctime(epochsecs))                 # 'Tue Apr 12 13:29:19 2016'
    
    epochsecs = epochsecs + (60 * 60 *24)       # adding one day!
    
    print(time.ctime(epochsecs))                 # 'Wed Apr 13 13:29:19 2016'
    
    time.sleep(10)                              # pause execution 10 seconds

    From the standard distribution: datetime.date, datetime.datetime and datetime.timedelta

    Python uses the datetime library to allow us to work with dates. Using it, we can convert string representations of date and time (like "4/1/2001" or "9:30") to datetime objects, and then compare them (see how far apart they are) or change them (advance a date by a day or a year).


    import datetime as dt            # load the datetime library
    dt = dt.datetime.now()           # create a new date object set to now
    dt = dt.date.today()             # same
    
    dt = datetime(2011, 7, 14, 14, 22, 29)
                                     # create a new date object by setting
                                     # the year, month, day, hour, minute,
                                     # second (milliseconds if desired)
    
    dt = datetime.strptime("21/11/06 16:30", "%d/%m/%y %H:%M")
                                     # create a new date object
                                     # by giving formatted date and time,
                                     # then telling datetime what format we used

    Once a datetime object has been created, we can view the date in a number of ways

    print(dt)                        # print formatted (ISO) date
                                     # 2011-07-14 14:22:29.045814
    
    print(dt.strftime("%m/%d/%Y %H:%M:%S"))
                                    # print formatted date/time
                                    # using string format tokens
                                    # '07/14/2011 14:22:29'
    
    print(dt.year)                   # 2011
    print(dt.month)                  # 7
    print(dt.day)                    # 14
    print(dt.hour)                   # 14
    print(dt.minute)                 # 22
    print(dt.second)                 # 29
    print(dt.microsecond)
    print(dt.weekday())              # 'Tue'

    Comparing datetime objects

    Often we may want to compare two dates. It's easy to see whether one date comes before another by comparing two date objects as if they were numbers:

    d1 = datetime(2011, 7, 14, 9, 40, 15)
                                     # new date object:  July 14, 2011  9:40:15am
    
    d2 = datetime(2011, 6, 14, 9, 30, 00)
                                     # new date object:  June 14, 2011  9:30:00am
    
    print(d1 < d2)                   # False
    
    print(d1 > d2)                   # True

    "Delta" means change. If we want to measure the difference between two dates, we can subtract one from the other. The result is a timedelta object:

    td = d1 - d2                     # a new timedelta object
    print(td.days)                   # 30
    print(td.seconds)                # 615

    Between June 14, 2011 at 9:30am and July 14, 2011 at 9:45am, there is a difference of 30 days, 10 minutes and 15 seconds. timedelta doesn't show difference in minutes, however: instead, it shows the number of seconds -- seconds can easily be converted between seconds and minutes/hours with simple arithmetic.


    Changing a date using the timedelta object

    Timedelta can also be constructed and used to change a date. Here we create timedelta objects for 1 day, 1 hour, 1 minute, 1 second, and then change d1 to subtract one day, one hour, one minute and one second from the date.

    from datetime import timedelta
    add_day = timedelta(days=1)
    add_hour = timedelta(hours=1)
    add_minute = timedelta(minutes=1)
    add_second = timedelta(seconds=1)
    
    print(d1)                                  # 2011-07-14 09:40:15.678331
    
    d1 = d1 - add_day
    d1 = d1 - add_hour
    d1 = d1 - add_minute
    d1 = d1 - add_second
    
    print(d1.strftime("%m/%d/%Y %H:%M:%S"))    # 2011-07-13 08:39:14.678331



    pandas: Subscripting, Slicing, Joining, Appending

    Selecting a Series from a DataFrame

    Use a subscript (or attribute) to access columns by label; use the .loc[] or .iloc[] attributes to access rows by label or integer index.


    a DataFrame:

    df = pd.DataFrame( {'a': [1, 2, 3, 4],
                        'b': [1.0, 1.5, 2.0, 2.5],
                        'c': ['a', 'b', 'c', 'd'],
                        'd': [100, 200, 300, 400] },
                        index=['r1', 'r2', 'r3', 'r4'] )

    access column as Series:

    cola = df['a']       # Series with [1, 2, 3, 4] and index ['r1', 'r2', 'r3', 'r4']
    
    cola = df.a          # same -- can often use attribute labels for column name
    
    print(cola)
    
                # r1    1
                # r2    2
                # r3    3
                # r4    4
                # Name: a, dtype: int64

    access row as Series using index label 'r2':

    row2 = df.loc['r2']  # Series [2, 1.5, 'b', 200] and index ['a', 'b', 'c', 'd']

    access row as Series using integer index:

    row2 = df.iloc[1]    # Series [2, 1.5, 'b', 200] and index ['a', 'b', 'c', 'd'] (same as above)
    
    print(row2)
    
                # a      2
                # b    1.5
                # c      b
                # d    200
                # Name: r2, dtype: object

    (Note that the .ix DataFrame indexer is a legacy feature and is deprecated.)


    slicing

    DataFrames can be sliced along a column or row (Series) or both (DataFrame)


    Access a Series object through DataFrame column or index labels Again, we can apply any Series operation on any of the Series within a DataFrame - slice, access by Index, etc.


    df = pd.DataFrame( {'a': [1, 2, 3, 4],
                        'b': [1.0, 1.5, 2.0, 2.5],
                        'c': ['a', 'b', 'c', 'd'],
                        'd': [100, 200, 300, 400] },
                        index=['r1', 'r2', 'r3', 'r4'] )
    
    print(dfi['b'])
        # r1     1.0
        # r2     1.5
        # r3     2.0
        # r4     2.5
        # Name: b
    
        # print df['b'][0:3]
        # r1     1.0
        # r2     1.5
        # r3     2.0
    
        # dfi['b']['r2']
        # 1.5

    Create a DataFrame from columns of another DataFrame Oftentimes we want to eliminate one or more columns from our DataFrame. We do this by slicing Series out of the DataFrame, to produce a new DataFrame:


    >>> dfi[['a', 'c']]
         a   c
    r1   1   a
    r2   2   b
    r3   3   c
    r4   4   d

    Far less often we may want to isolate a row from a DataFrame - this is also returned to us as a Series. Note the column labels have become the Series index, and the row label becomes the Series Name. 2-dimensional slicing A double subscript can select a 2-dimensional slice (some rows and some columns).


    df[['a', 'b']]['alpha': 'gamma']

    Also note carefully the list inside the first square brackets.


    Mask: Conditional Slicing

    Oftentimes we want to select rows based on row criteria (i.e., conditionally). To do this, we establish a mask, which is a test placed within subscript-like square brackets.


    Selecting rows based on column criteria:


    import pandas as pd
    
    df = pd.DataFrame( { 'a': [1, 2, 3, 4],
                         'b': [-1.0, -1.5, 2.0, 2.5],
                         'c': ['a', 'b', 'c', 'd']  }, index=['r1', 'r2', 'r3', 'r4'] )
    
    print(df)
    
                        #     a    b  c
                        # r1  1 -1.0  a
                        # r2  2 -1.5  b
                        # r3  3  2.0  c
                        # r4  4  2.5  d
    
    print(df[ df['b'] < 0 ])              # select rows where 'b' value is < 0
    
                        #     a    b  c
                        # r1  1 -1.0  a
                        # r2  2 -1.5  b

    The mask by itself returns a boolean Series. Its values indicate whether the test return True for the value in the tested Series. This mask can of course be assigned to a name and used by name, which is common for complex criteria:

    mask = df['a'] > 2
    print(mask)          # we are printing this just for illustration
    
                        # r1    False
                        # r2    False
                        # r3    True
                        # r4    True
                        # Name: a, dtype: bool
    
    
    print(df[ mask ])
                        #     a    b  c
                        # r3  3  2.0  c
                        # r4  4  2.5  d

    negating a mask


    a tilde (~) in front of a mask creates its inverse:

    mask = df['a'] > 2
    print(df[ ~mask ])
    
                      #     a    b  c
                      # r1  1 -1.0  a
                      # r2  2 -1.5  b

    compound tests in a mask use & for 'and', | for 'or', and ( ) to separate tests


    The parentheses are needed to disambiguate the parts of the compound test.

    print(df[ (df.a > 3) & (df.a < 5) , 'b' ])
    
                      #     a    b  c
                      # r4  4  2.5  d

    Avoiding the 'copy of a slice' warning

    Use .loc[] rather than subscript slices for a guaranteed write to a slice


    We often begin work by reading a large dataset into a DataFrame, then slicing out a meaningful subset (eliminating columns and rows that are irrelevant to our analysis). Then we may wish to make some changes to the slice, or add columns to the slice. A recurrent problem in working with slices is that standard slicing may produce a link into the original data, or it may produce a temporary "copy". If a change is made to a temporary copy, our working data will not be changed.


    Here we are creating a slice by using a double subscript:

    dfi = pd.DataFrame({'c1': [0,   1,  2, 3,   4],
                       'c2': [5,   6,  7, 8,   9],
                       'c3': [10, 11, 12, 13, 14],
                       'c4': [15, 16, 17, 18, 19],
                       'c5': [20, 21, 22, 23, 24],
                       'c6': [25, 26, 27, 28, 29] },
               index = ['r1', 'r2', 'r3', 'r4', 'r5'])
    
    dfi_prime = dfi[ dfi['c1'] > 2 ]
    print(dfi_prime)
    
                      #     c1  c2  c3  c4  c5  c6
                      # r4   3   8  13  18  23  28
                      # r5   4   9  14  19  24  29
    
    dfi_prime['c3'] = dfi_prime['c1'] * dfi_prime['c2']
    
    
    A value is trying to be set on a copy of a slice from a DataFrame
    
    See the caveats in the documentation: http://pandas.pydata.org/pandas-docs/stable/indexing.html#indexing-view-versus-copy

    Note that in some cases this warning will not appear, and in others the warning will appear and yet the change will have taken effect.


    The same problem may occur with a simple slice selection:

    myslice = dfi[ ['c1', 'c2', 'c3'] ]
    print(myslice)
    
                      #     c1  c2  c3  c4  c5  c6
                      # r3   2   7  12  17  22  27
                      # r4   3   8  13  18  23  28
                      # r5   4   9  14  19  24  29
    
    myslice['c3'] = myslice['c1'] * myslice['c2']
    print(myslice)

    The problem here is that pandas cannot guarantee whether the slice is a view on the original data, or a temporary copy! If a temporary copy, a change will not take effect! What's particularly problematic about this warning is that we may not always see it in these situations. We may also see false positives and false negatives, as is acknowledged in the documentation. The takeaway is that whether or not you see a warning, whether or not you see the change when you run it, you should never trust that you can change a value on a slice, even if it seems to work when testing.


    The solution is to use .loc or .iloc:

    filtered = dfi.loc[ dfi['c3'] > 11, : ]     # filter by column, include all rows

    Keep in mind that you may get a warning even with this approach; you can consider it a false positive (i.e., disregard it).


    A more crude, yet unambiguous solution: make a copy!

    filtered = dfi[ dfi['c3'] > 11, : ].copy()

    Obviously this is less than optimal for large datasets.


    More details about .loc are in the next section.


    Using .loc[] to select data by column or row label

    If a slice is to changed, it should be derived using .loc[] rather than slicing.


    Again, starting with this DataFrame:

    dfi = pd.DataFrame({'c1': [0,   1,  2, 3,   4],
                       'c2': [5,   6,  7, 8,   9],
                       'c3': [10, 11, 12, 13, 14],
                       'c4': [15, 16, 17, 18, 19],
                       'c5': [20, 21, 22, 23, 24],
                       'c6': [25, 26, 27, 28, 29] },
               index = ['r1', 'r2', 'r3', 'r4', 'r5'])
    
      #     c1  c2  c3  c4  c5  c6
      # r1   0   5  10  15  20  25
      # r2   1   6  11  16  21  26
      # r3   2   7  12  17  22  27
      # r4   3   8  13  18  23  28
      # r5   4   9  14  19  24  29


    Slicing Columns: these examples select all rows and one or more columns.


    Slice a range of columns with a slice of column labels:

    dfi_slice = dfi.loc[:, 'c1': 'c3']
    
      #     c1  c2  c3
      # r1   0   5  10
      # r2   1   6  11
      # r3   2   7  12
      # r4   3   8  13
      # r5   4   9  14

    Note the slice upper bound is inclusive!


    Slice a single column Series with a string column label:

    dfi_slice = dfi.loc[:, 'c3']
    
      # r1    10
      # r2    11
      # r3    12
      # r4    13
      # r5    14
      # Name: c3, dtype: int64
    

    Slice a selection of columns with a tuple of column labels:

    dfi_slice = dfi.loc[:, ('c2', 'c3')]
    
      #    c2  c3
      # r1   5  10
      # r3   7  12
      # r4   8  13
      # r5   9  14

    A tuple is necessary here because it is "hashable" -- a list will not work here.



    Slicing Rows: these examples select one or more rows and all columns.


    Slice a range of rows with a slice of row labels:

    dfi_slice = dfi.loc['r1': 'r3':, :]
    
      #     c1  c2  c3  c4  c5  c6
      # r1   0   5  10  15  20  25
      # r2   1   6  11  16  21  26
      # r3   2   7  12  17  22  27

    Note the slice upper bound is inclusive!


    Slice a single row Series with a string row label:

    dfi_slice = dfi.loc['r2', :]
    
      # c1     1
      # c2     6
      # c3    11
      # c4    16
      # c5    21
      # c6    26
      # Name: r2, dtype: int64

    Slice a selection of rows with a tuple of row labels:

    dfi_slice = dfi.loc[('r1', 'r3', 'r5'), :]
    
      #     c1  c2  c3  c4  c5  c6
      # r1   0   5  10  15  20  25
      # r3   2   7  12  17  22  27
      # r5   4   9  14  19  24  29

    A tuple is necessary here because it is "hashable" -- a list will not work here.



    Slicing Rows and Columns


    We can of course specify both rows and columns:

    dfi.loc['r1': 'r3', 'c1': 'c3']
    
      #     c1  c2  c3
      # r1   0   5  10
      # r2   1   6  11
      # r3   2   7  12

    Using .loc[] to select data by 'mask' criteria

    A conditional can be used with .loc[] to select rows or columns


    Again, starting with this DataFrame:

    dfi = pd.DataFrame({'c1': [0,   1,  2, 3,   4],
                       'c2': [5,   6,  7, 8,   9],
                       'c3': [10, 11, 12, 13, 14],
                       'c4': [15, 16, 17, 18, 19],
                       'c5': [20, 21, 22, 23, 24],
                       'c6': [25, 26, 27, 28, 29] },
               index = ['r1', 'r2', 'r3', 'r4', 'r5'])
    
      #     c1  c2  c3  c4  c5  c6
      # r1   0   5  10  15  20  25
      # r2   1   6  11  16  21  26
      # r3   2   7  12  17  22  27
      # r4   3   8  13  18  23  28
      # r5   4   9  14  19  24  29

    .loc[] can also specify rows or columns based on criteria -- here are all the rows with 'c3' value greater than 11 (and all columns):

    dfislice = dfi.loc[ dfi['c3'] > 11, :]
    
      #     c1  c2  c3  c4  c5  c6
      # r3   2   7  12  17  22  27
      # r4   3   8  13  18  23  28
      # r5   4   9  14  19  24  29

    In order to add or change column values based on a row mask, we can specify which column should change and assign a value to it:


    dfi.loc[ dfi['c3'] > 11, 'c6'] = dfi['c6'] * 100  # 100 * 'c6' value if 'c3' > 11
    print(dfi)
    
       #     c1  c2  c3  c4  c5    c6
       # r1   0   5  10  15  20    25
       # r2   1   6  11  16  21    26
       # r3   2   7  12  17  22  5400
       # r4   3   8  13  18  23  5600
       # r5   4   9  14  19  24  5800

    DataFrame Concatenating / Appending

    pd.concat() is analogous to df.append()


    concat() can join dataframes either horizontally or vertically.

    df = pd.DataFrame( {'a': [1, 2, ],
                        'b': [1.0, 1.5 ] } )
    
    df2 = pd.DataFrame( {'b': [1, 2 ],
                         'c': [1.0, 1.5 ] } )
    df3 = pd.concat([df, df2])
    print(df3)
    
          #      a    b    c
          # 0  1.0  1.0  NaN
          # 1  2.0  1.5  NaN
          # 0  NaN  1.0  1.0
          # 1  NaN  2.0  1.5

    Note that the column labels have been aligned. As a result, some data is seen to be "missing", with the NaN value used (discussed shortly).


    In horizontal concatenation, the row labels are aligned but the column labels may be repeated:

    df4 = pd.concat([df, df2], axis=1)
    print(df4)
    
          #    a    b  b    c
          # 0  1  1.0  1  1.0
          # 1  2  1.5  2  1.5

    DataFrame append() is the method equivalent to pd.concat(), called on a DataFrame:

    df = df.append(df2)            # compare:  pd.concat([df, df2])
    
    df = df3.append(df4, axis=1)   # compare:  pd.concat([df, df2], axis=1)

    We can append a Series but must include the ignore_index=True parameter:

    df = pd.DataFrame( {'a': [1, 2, ],
                        'b': [1.0, 1.5 ] } )
    
    df = df.append(pd.Series(), ignore_index=True)
    
    print(df)
    
           #      a    b
           # 0  1.0  1.0
           # 1  2.0  1.5
           # 2  NaN  NaN

    Line and Bar Charts on a Slice

    Selected DataFrame columns (Series) can be plotted in a Bar or Line Chart.


    pandas .merge() (DataFrame .join())

    merge() provides database-like joins.


    Merge performs a relational database-like join on two dataframes. We can join on a particular field and the other fields will align accordingly.

    companies = pd.read_excel('company_states.xlsx', sheetname='Companies')
    states = pd.read_excel('company_states.xlsx', sheetname='States')
    
    print(companies)
         #      Company State
         # 0  Microsoft    WA
         # 1      Apple    CA
         # 2        IBM    NY
         # 3     PRTech    PR
    
    print(states)
    
         #   State Abbrev  State Long
         # 0           AZ     Arizona
         # 1           CA  California
         # 2           CO    Colorado
         # 3           NY    New York
         # 4           WA  Washington
    
    cs = pd.merge(companies, states,
                  left_on='State', right_on='State Abbrev')
    
    print(cs)
         #      Company State State Abbrev  State Long
         # 0  Microsoft    WA           WA  Washington
         # 1      Apple    CA           CA  California
         # 2        IBM    NY           NY    New York
         # 3     PRTech    PR          NaN  NaN

    When we merge, you can choose to join on the index (default), or one or more columns. The choices are similar to that in relationship databases:


    Merge method  SQL Join Name     Description
    left          LEFT OUTER JOIN   Use keys from left frame only
    right         RIGHT OUTER JOIN  Use keys from right frame only
    outer         FULL OUTER JOIN   Use union of keys from both frames
    inner         INNER JOIN        Use intersection of keys from both frames

    how= describes the type of join on= designates the column on which to join If the join columns are differently named, we can use left_on= and right_on=


    left join: include only keys from 'left' dataframe. Note that only states from the 'companies' dataframe are included.

    cs = pd.merge(companies, states, how='left',
                  left_on='State', right_on='State Abbrev')
    
    print(cs)
         #      Company State State Abbrev  State Long
         # 0  Microsoft    WA           WA  Washington
         # 1      Apple    CA           CA  California
         # 2        IBM    NY           NY    New York

    (Right join would be the same but with the dfs switched.)


    outer join: include keys from both dataframes. Note that all states are included, and the missing data from 'companies' is shown as NaN

    cs = pd.merge(companies, states, how='outer',
                  left_on='State', right_on='State Abbrev')
    
    print(cs)
         #      Company State State Abbrev  State Long
         # 0  Microsoft    WA           WA  Washington
         # 1      Apple    CA           CA  California
         # 2        IBM    NY           NY    New York
         # 3     PRTech    PR          NaN         NaN
         # 4        NaN   NaN           AZ     Arizona
         # 5        NaN   NaN           CO    Colorado

    inner join: include only keys common to both dataframes. Note taht

    cs = pd.merge(companies, states, how='inner',
                  left_on='State', right_on='State Abbrev')
    
    print(cs)
         #      Company State State Abbrev  State Long
         # 0  Microsoft    WA           WA  Washington
         # 1      Apple    CA           CA  California
         # 2        IBM    NY           NY    New York

    Manipulating the Index

    An Index can be set with a column or other sequence.


    Sometimes a pd.read_excel() includes index labels in the first column. We can easily set the index with .set_index():


    print(df)
        #     0  a    b  c           d
        # 0  r1  1  1.0  a  2016-11-01
        # 1  r2  2  1.5  b  2016-12-01
        # 2  r3  3  2.0  c  2017-01-01
        # 3  r4  4  2.5  d  2018-02-01
    
    df = df.set_index(df[0])
    df = df[['a', 'b', 'c', 'd']]
    print(df)
        #     a    b  c           d
        # 0
        # r1  1  1.0  a  2016-11-01
        # r2  2  1.5  b  2016-12-01
        # r3  3  2.0  c  2017-01-01
        # r4  4  2.5  d  2018-02-01

    We can reset the index with .reset_index(), although this makes the index into a new column.


    df2 = df.reset_index()
    print(df2)
        #     0  a    b  c           d
        #
        # 0  r1  1  1.0  a  2016-11-01
        # 1  r2  2  1.5  b  2016-12-01
        # 2  r3  3  2.0  c  2017-01-01
        # 3  r4  4  2.5  d  2018-02-01

    As mentioned we can move a column into the index with .set_index()

    df2 = df2.set_index(0)
    print(df2)
        #     a    b  c           d
        # 0
        # r1  1  1.0  a  2016-11-01
        # r2  2  1.5  b  2016-12-01
        # r3  3  2.0  c  2017-01-01
        # r4  4  2.5  d  2018-02-01

    To reset the index while dropping the original index, we can use drop=True:


    df3 = df2.reset_index(drop=True)
    print(df3)
        #    a    b  c           d
        #
        # 0  1  1.0  a  2016-11-01
        # 1  2  1.5  b  2016-12-01
        # 2  3  2.0  c  2017-01-01
        # 3  4  2.5  d  2018-02-01

    We can also sort the DataFrame by index using .sort_index():


    df4 = pd.DataFrame({'a': [1, 2, 3], 'b': [4, 5, 6]},
                       index=['Cello', 'Alpha', 'Bow'])
    df5 = df4.sort_index()
    print(df5)
        #        a  b
        # Alpha  2  5
        # Bow    3  6
        # Cello  1  4

    The default is to sort by the row index; axis=1 allows us to sort by columns. ascending=False reverses the sort:


    df6 = df5.sort_index(axis=1, ascending=False)
    print(df6)
        #        b  a
        # Alpha  5  2
        # Bow    6  3
        # Cello  4  1

    Note that .sort_values() offers the same options sorting a specified column or row.


    The columns and rows default to integers but can also be set with descriptive labels using the DataFrame's rename() method.

    df = pd.DataFrame([ [1, 10], [2, 20], [3, 30] ])
    
    print(df)
               #      0   1
               #   0  1  10
               #   1  2  20
               #   2  3  30
    
    df = df.rename(columns={0: 'A', 1: 'B'})            # reset column labels
    df = df.rename(index={0: 'R1', 1: 'R2', 2: 'R3'})   # reset row index labels
    
    print(df)
                #     A   B
                # R1  1  10
                # R2  2  20
                # R3  3  30

    Columns or index labels can be reset using the dataframe's rename() method.

    df = df.rename(columns={'a': 'A', 'b': 'B', 'c': 'C', 'd': 'D'},
                   index={'r1': 'R1', 'r2': 'R2', 'r3': 'R3', 'r4': 'R4'})

    The columns or index can also be set directly using the dataframe's attributes (although this is more prone to error).

    df.columns = ['A', 'B', 'C', 'D']
    
    # reset indices to integer starting with 0
    df.reset_index()
    
    
    # reindex ordering by index:
    df = df.reindex(reversed(df.index))
    
    df.reindex(columns=reversed(df.columns))
    
    print(df)
    
                          #  state  A    B  C    D
                          #  year
                          #  R1     1  1.0  a  100
                          #  R2     2  1.5  b  200
                          #  R3     3  2.0  c  300
                          #  R4     4  2.5  d  400

    Most Operations Produce a New DataFrame

    Some DataFrame operations provide the inplace=True option


    Keep in mind that many operations produce a new DataFrame copy. This means that if you are working with a large dataset, you can avoid allocating additional memory with this option.


    import pandas as pd
    
    df = pd.DataFrame({ 'a': [1, 2, 3, 4],
                        'b': [1.0, 1.5, 2.0, 2.5],
                        'c': ['a', 'b', 'c', 'd']   })
    
    print(df)
        #     a    b  c
        # r1  1  1.0  a
        # r2  2  1.5  b
        # r3  3  2.0  c
        # r4  4  2.5  d
    
    df2 = df.set_index('a')
    
    print(df2)             # new dataframe
    
        #      b  c
        # a
        # 1  1.0  a
        # 2  1.5  b
        # 3  2.0  c
        # 4  2.5  d
    
    print(df)              # unchanged
        #     a    b  c
        # r1  1  1.0  a
        # r2  2  1.5  b
        # r3  3  2.0  c
        # r4  4  2.5  d
    
    df.set_index('a', inplace=True)
    
    print(df)
    
        #      b  c
        # a
        # 1  1.0  a
        # 2  1.5  b
        # 3  2.0  c
        # 4  2.5  d

    .




    User-Defined Functions

    Introduction: User-Defined Functions

    A user-defined function is a named code block -- very simply, a block of Python code that we can call by name. These functions are used and behave very much like built-in functions, except that we define them in our own code.


    def addthese(val1, val2):  # function definition; argument signature
        valsum = val1 + val2
        return valsum          # return value
    
    x = addthese(5, 10)        # function call; 2 arguments passed;
                               # return value assigned to x
    print(x)                   # 15

    There are two primary reasons functions are useful: to reduce code duplication and to organize our code: Reduce code duplication: a named block of code can be called numerous times in a program, which means the same series of statements can be executed repeatedly, without having to type them out multiple times in the code. Organize code: large programs can be difficult to read, even with helpful comments. Dividing code into named blocks allows us to identify the major steps our code can take, and see at a glance what steps are being taken and the order in which they are taken. We have learned about using simple functions for sorting; in this unit we will learn about: 1) different ways to define function arguments 2) the "scoping" of variables within functions 3) the four "naming" scopes within Python


    Objectives for the Unit: User-Defined Functions

  • define functions that take arguments and return return values
  • define functions that take positional and keyword arguments
  • define functions that can take an arbitrary number of arguments
  • learn about the four variable scopes and how scopes interact


    Review: functions are named code blocks

    The block is executed every time the function is called.


    def print_hello():
        print("Hello, World!")
    
    print_hello()             # prints 'Hello, World!'
    print_hello()             # prints 'Hello, World!'
    print_hello()             # prints 'Hello, World!'

    When we run this program, we see the greeting printed three times.


    Review: function argument(s)

    Any argument(s) passed to a function are aliased to variable names inside the function definition.


    def print_hello(greeting, person):    # 2 strings aliased to objects
                                          # passed in the call
    
        full_greeting = f"{greeting}, {person}!"
        print(full_greeting)
    
    print_hello('Hello', 'World')     # pass 2 strings: prints "Hello, World!"
    print_hello('Bonjour', 'Python')  # pass 2 strings: prints "Bonjour, Python!"
    print_hello('squawk', 'parrot')   # pass 2 strings: prints "squawk, parrot!"

    Review: the return statement returns a value

    Object(s) are returned from a function using the return statement.


    def print_hello(greeting, person):
        full_greeting = greeting + ", " + person + "!"
        return full_greeting
    
    msg = print_hello('Bonjour', 'parrot')   # full_greeting
                                             # aliased to msg
    
    print(msg)                               # 'Bonjour, parrot!'

    Multi-target Assignment

    This convenience allows us to assign values in a list to individual variable names.


    line = 'Acme:23.9:NY'
    
    items = line.split(':')
    print(items)                  # ['Acme', '23.9', 'NY']
    
    name, revenue, state = items
    
    print(revenue)                # 23.9

    This kind of assignment is sometimes called "unpacking" because it assigns a container of items to individual variables.


    We can assign multiple values to multiple variables in one statement too:

    name, revenue, state = 'Acme', '23.9', 'NY'

    (The truth is that the 3 values on the right comprise a tuple even without the paretheses, so we are technically still unpacking.)


    If the number of items on the right does not match the number of variables on the left, an error occurs:

    mylist = ['a', 'b', 'c', 'd']
    
    x, y, z = mylist         # ValueError:  too many values to unpack
    
    v, w, x, y, z = mylist   # ValueError:  not enough values to unpack

    We also see multi-target assignment when returning multiple values from a function:

    def return_some_values():
    
        return 10, 20, 30, 40
    
    
    a, b, c, d = return_some_values()
    
    print(a)          # 10
    print(d)          # 40

    This means that like with standard unpacking, the number of variables to the right of a function call must match the number of values returned from the function call.


    Summary argument types: positional and keyword

    Your choice of type depends on whether they are required.


    positional: args are required and in particular order

    def sayname(firstname, lastname):
        print(f"Your name is {firstname} {lastname}")
    
    sayname('Joe', 'Wilson')         # passed two arguments:  correct
    
    sayname('Joe')    # TypeError: sayname() missing 1 required positional argument: 'lastname'

    keyword: args are not required, can be in any order, and the function specifies a default value

    def sayname(lastname, firstname="Citizen"):
        print(f"Your name is {firstname} {lastname}")
    
    
    sayname('Wilson', firstname='Joe')       # Your name is Joe Wilson
    
    sayname('Wilson')                        # Your name is Citizen Wilson

    Variable name scoping inside functions

    Variable names initialized inside a function are local to the function, and not available outside the function.


    def myfunc():
        a = 10
        return a
    
    var = myfunc()            # var is now 10
    print(a)                  # NameError ('a' does not exist here)

    Note that although the object associated with a is returned and assigned to var, the name a is not available outside the function. Scoping is based on names.


    global variables (i.e., ones defined outside a function) are available both inside and outside functions:

    var = 'hello global'
    
    def myfunc():
        print(var)
    
    myfunc()                  # hello global

    The four variable scopes: (L)ocal, (E)nclosing, (G)lobal and (B)uiltin

    Variable scopes "overlay" one another; a variable can be "hidden" by a same-named variable in a "higher" scope.


    From top to bottom:

  • Local: local to (defined in) a function
  • Enclosing: local to a function that may have other functions in it
  • Global: available anywhere in the script (also called file scope)
  • Built-in: a built-in name (usually a function like len() or str()) A variable in a given scope can be "hidden" by a same-named variable in a scope above it (see example below):


    def myfunc():
        len = 'inside myfunc'   # local scope:
                                # len is initialized in the function
        print(len)
    
    print(len)                  # built-in scope:
                                # prints '<built-in function len>'
    
    len = 'in global scope'     # assigned in global scope:  a global variable
    print(len)                  # global scope:  prints 'in global scope'
    
    myfunc()                    # prints 'inside myfunc'
                                # (i.e. the function executes)
    
    print(len)                  # prints 'in global scope' (the local
                                # len is gone, so we see the global
    
    del len                     # 'deletes' the global len
    print(len)                  # prints '<built-in function len>'

    Summary exception: UnboundLocalError

    An UnboundLocalError exception signifies a local variable that is "read" before it is defined.


    x = 99
    def selector():
        x = x + 1         # "read" the value of x; then assign to x
    selector()
    
    # Traceback (most recent call last):
    #   File "test.py", line 1, in <module>
    #   File "test.py", line 2, in selector
    # UnboundLocalError: local variable 'x' referenced before assignment

    Remember that a local variable is one that is initialized or assigned inside a function. In the above example, x is a local variable. So Python sees x not as the global variable (with value 99) but as a local variable. However, in the process of initializing x Python attempts to read x, and realizes that is hasn't been initialized yet -- the code has attempted to reference (i.e., read the value of) x before it has been assigned.


    Since we want Python to treat x as the global x, we need to tell it to do so. We can do this with the global keyword:

    x = 99
    def selector():
        global x
        x = x + 1
    selector()
    print(x)             # 100

    Proper Program Design

    Properly organized programs list imports, constants, functions and a main body.


    """ helloworld.py:  print the "Hello! message
    
        Author:  David Blaikie (dbb212@nyu.edu)
        Date:    2/19/2020
        Github:  https://github.com/dervid/testproj
    
    """
    
    import sys
    
    DEFAULT_NAME='Python'
    
    def greet(name=DEFAULT_NAME):
        """ greet the user by name
    
            Argument:  name   name to greet (optional)
    
            Return Value:  a string with the greeting
    
        """
    
        greeting = f'Hello, {name}!'
    
        return greeting
    
    
    def main():
        """ the main event:  calls greet() with an argument
            from sys.argv; if none is found,
    
            Argument:  N/A
    
            Return Values:  None
        """
    
        try:
            name = sys.argv[1]
            greet(name)
    
        except IndexError:
            name = None
            greet()
    
    
    if __name__ == '__main__':
    
        main()

    Professionally written Python programs have a basic structure that should be followed in all cases when writing code for a team or for review by others:


    If we run the above script, we'll see "Hello, World!". But if we import the above script, we won't see anything. Why? Because the if statement above will be true only if the script is executed. This is important behavior, because there are some scripts that we may want to run directly, but also allow others to import (in order to use the script's functions). Whether we intend to import a script or not, it is considered a "best practice" to build all of our programs in this way -- with a "main body" of statements collected under function main() and the call to main() inside the if __name__ == '__main__' gate.




    Dictionaries and Sorting

    Introduction: Dictionaries for Paired Data

    dicts pair unique keys with associated values.


    Much of the data we use tends to be paired:

  • companies paired with annual revenue for each
  • employees with contact information for each
  • students with grade point averages
  • dates with the high temperature for each
  • web pages with the number of times each was accessed


    To store paired data, we use a Python container called a dict, or dictionary. The dict contains keys paired with values.

    customer_balances = { 'jk125': 493.95,     # spacing for clarity
                          'xx3':   122.03,
                          'jp9':   23238.72 }
    
    print(customer_balances)                # { 'jk125': 493.95, 'xx3': 122.03,
                                            #   'jp9': 23238.72 }
    print(type(customer_balances))          # <class 'dict'>

    dicts have some of the same features as other containers. When standard container operations are applied, the keys are used:


    print(len(customer_balances))           # 3 keys in dict
    
    print('xx3' in customer_balances)       # True:  the key is there
    
    for yyy in customer_balances:
        print(yyy)                          # prints each key in customer_balances
    
    print(customer_balances['jp9'])         # 23238.72

    Objectives for the Unit: Dictionaries

    This key/value pairs container allows us to summarize data in powerful ways:



    Summary for Object: Dictionary (dict)

    A dictionary (or dict) is a collection of unique key/value pairs of objects.


    The keys in a dict are unique. In this way it is like a set. Traditionally, dicts are also like sets in that the keys are unordered as well. Hoever as of Python 3.6, dictionaries retain their key order. A dict key can be used to obtain the assocaiated value (that is, the dict is "addressible by key"). In this way it is also like a list or tuple, as it is addressible by ordered index.


    initialize a dict

    mydict = {}                        # empty dict
    
    mydict = {'a':1, 'b':2, 'c':3}     # dict with str keys and int values

    add a key/value pair to a dict

    mydict['d'] = 4                    # setting a new key and value
    
    print(mydict)                      # {'a': 1, 'c': 3, 'b': 2, 'd': 4}

    read a value based on a key

    dval = mydict['d']                 # value for 'd' is 4
    
    xxx = mydict['c']                  # value for 'c' is 3

    Note in the standard container features below, only the keys are used:


    loop through a dict and read keys and values

    mydict = {'a': 1, 'b': 2, 'c': 3, 'd': 4}
    
    for key in mydict:
        print(key, mydict[key])       # prints a 1, then c 3, then b 2, then d 4

    check for key membership

    mydict = {'a': 1, 'b': 2, 'c': 3}
    
    if 'a' in mydict:
        print("'a' is a key in mydict")

    sort a dict's keys

    mydict = {'xenophon': 10, 'abercrombie': 5, 'denusia': 3}
    
    mykeys = sorted(mydict)            # ['abercrombie', 'denusia', 'xenophon']

    retrieve an iterator of keys or of values

    mydict = {'a': 1, 'b': 2, 'c': 3}
    
    yyy = mydict.keys()                # dict_keys(['a', 'b', 'c'])
    
    zzz = mydict.values()              # dict_values([1, 2, 3])

    Summary dict Method: get()

    The dict get() method returns a value based on a key. An optional 2nd argument provides a default value if the key is missing.


    mydict = {'a': 1, 'b': 2, 'c': 3}
    
    xx = mydict.get('a', 0)          # 1 (key exists so paired value is returned)
    
    yy = mydict.get('zzz', 0)        # 0 (key does not exist so the
                                     #    default value is returned)

    The default value is your choice. This method is sometimes used as an alternative to testing for a key in a dict before reading it -- avoiding the KeyError exception that occurs when trying to read a nonexistent key.


    Summary dict Methods: keys() and values()

    keys() returns an iterator of keys; values() returns an iterator of values. An iterator is an object that can be looped over (or iterated over) with for. They can also be converted to a container of values using a container function (i.e., list(), tuple()).


    These methods are used for advanced manipulation of a dict.


    dict keys() method returns an iterator of keys in the dict

    mydict = {'a': 1, 'b': 2, 'c': 3}
    
    yyy = mydict.keys()                # dict_keys(['a', 'c', 'b'])
    
    for key in yyy:
        print(key)                     # prints each key in the dict

    dict values() method returns an iterator of values in the dict

    zzz = mydict.values()              # dict_values([1, 2, 3])
    
    for value in zzz:
        print(value)                   # prints each value in the dict

    values(), like, keys(), is also less often used -- to test for membership among values, for example.


    Summary Exception: KeyError

    The KeyError exception indicates that the requested key does not exist in the dictionary.


    mydict = {'a': 1, 'b': 2, 'c': 3}
    
    xx = mydict['a']                   # 1
    
    yy = mydict['NARNAR']              # KeyError:  'NARNAR'

    The above code results in this exception:

    Traceback (most recent call last):
      File "./keyerrortest.py", line 7, in <module>
        yy = mydict['NARNAR']
    KeyError: 'NARNAR'

    As you can see, the line of code and the key involved in the error are displayed. If you get this error the debugging procedure should be to check to see first whether the key seems to be in the dict or not; if it is, you may want to check the type of the object, since string '1' is not the same as int 1. String case also matters when Python is looking for a key -- the object value must match the dict's key exactly.


    One way to handle an error like this is to test the dict ahead of time using the in operator:

    mydict = {'a': 1, 'b': 2, 'c': 3}
    
    if 'NARNAR' not in mydict:
        yy = 0
    else:
        yy = mydict['NARNAR']          # else: block only executed
                                       # if key NARNAR exists in the dict

    Another approach is the use the dict get() method:

    mydict = {'a': 1, 'b': 2, 'c': 3}
    
    yy = mydict.get('a', 0)            # 1
    
    zz = mydict.get('zzz', 0)          # 0

    Summary Task: Read and Write to a dict

    Subscript syntax is used to add or read key/value pairs. The dict's key is the key!


    Note well: the syntax is the same for setting a key/value pair or getting a value based on a key.


    Setting a key/value pair in a dict

    mydict = {}
    mydict['a'] = 1                    # set a key and value in the dict
    mydict['b'] = 2                    # same
    
    print(mydict)                      # {'a': 1, 'b': 2}

    Getting a value from a dict based on a key

    val  = mydict['a']                 # get a value using a key:  1
    val2 = mydict['b']                 # get a value using a key:  2

    Summary Task: Looping through a Dict with for

    Looping through a dict means looping through its keys.


    mydict = {'a': 1, 'b': 2, 'c': 3, 'd': 4}
    
    for key in mydict:
        print(key, mydict[key])        # prints a 1, then c 3, then b 2, then d 4

    As with all standard container features (e.g. in, for, sorted()) the in operator test the keys of the dict.


    Here is another example of printing each key and value in the dict:

    mydict = {'a': 1, 'b': 2, 'c': 3, 'd': 4}
    
    for key in mydict:
        print(f"key: {key}; value: {mydict[key]}")
    
        ### key: a; value 1
        ### key: c; value 3
        ### key: b; value 2
        ### key: d; value 4

    Summary Task: Checking for Key Membership in a Dict with "in"

    Checking membership in a dict means checking for the presence of a key.


    mydict = {'a': 1, 'b': 2, 'c': 3}
    
    if 'a' in mydict:
        print("'a' is a key in mydict")

    As with all standard container features (e.g. in, for, sorted()) the in operator test the keys of the dict.


    Summary Task: Dictionary Size with len()

    len() counts the pairs in a dict.


    mydict = {'a': 1, 'b': 2, 'c': 3}
    
    print(len(mydict))                 # 3 (number of keys in dict)

    Summary Task: Obtaining Iterator of Keys, Iterator of Values, Iterator of key/value Items

    keys(), values() return iterators that can loop through a list of objects; items() returns an iterator of 2-element tuples. All three can be used with any of the summary container functions we have seen (len(), sum(), etc.). All three can also be passed to list() or set() to contain the items referred to by the iterator.


    keys(): return an iterator of keys in the dict

    mydict = {'a': 1, 'b': 2, 'c': 3}
    
    these_keys = mydict.keys()
    'c' in these_keys                  # True
    print(len(these_keys))             # 3
    print(list(these_keys))            # ['a', 'c', 'b']

    values(): return an iterator of values in the dict

    these_values = mydict.values()
    print(list(these_values))          # [1, 3, 2]
    print(sum(these_values))           # 6

    The values cannot be used to get the keys - it's a one-way lookup from the keys. However, we might want to check for membership in the values, or sort or sum the values, or some other less-used approach.


    The dict items() method: iterator of pairs as 2-element tuples

    these_items = mydict.items()
    print(list(these_items))           # [('a', 1), ('c', 3), ('b', 2)]

    There are three elements here: three tuples. Each tuple contains two elements: a key and value pair from the dict. There are a number of reasons we might wish to use a structure like this -- for example, to sort the dictionary and store it in sorted form. As you know, a dictionary's keys are unordered, but a lists are not. A list of tuples can also be manipulated in other ways pertaining to a list. It is a convenient structure that is preferred by some developers as an alternative to working with the keys.


    Review Summary Task: Sorting a Container with sorted()

    With a list, tuple or set, sorted() returns a list of sorted elements


    namelist = ['jo', 'pete', 'michael', 'zeb', 'avram']
    
    slist = sorted(namelist)          # ['avram', 'jo', 'michael', 'pete', 'zeb']

    Remember that no matter what container is passed to sorted(), the function returns list -- even a string!


    Summary Task: Reversing a Sort

    reverse=True, a special arg to sorted(), reverses the order of a sort.


    namelist = ['jo', 'pete', 'michael', 'zeb', 'avram']
    
    slist = sorted(namelist, reverse=True)    # ['zeb', 'pete',
                                              #  'michael', 'jo', 'avram']

    reverse is called a keyword argument -- it is no different from a regular positional arguments we have seen before; it is simply notated differently.


    Summary Task: Sorting a Dict (sorting its Keys)

    sorted() returns a sorted list of a dict's keys


    bowling_scores = {'jeb': 123, 'zeb': 98, 'mike': 202, 'janice': 184}
    keys = list(bowling_scores.keys())
    keys = sorted(keys)
    print(keys)                               # [ 'janice', 'jeb', 'mike', 'zeb' ]
    for key in keys:
        print(key + " = " + str(bowling_scores[key]))

    Or, we can even loop through the sorted dictionary keys directly:

    bowling_scores = {'jeb': 123, 'zeb': 98, 'mike': 202, 'janice': 184}
    for key in sorted(bowling_scores.keys()):
        print(key + " = " + str(bowling_scores[key]))

    Summary Task: Sorting a dictionary's keys by its values

    A special "sort criteria" argument can cause Python to sort a dict's keys by its values.


    bowling_scores = {'jeb': 123, 'zeb': 98, 'mike': 202, 'janice': 184}
    
    sorted_keys = sorted(bowling_scores, key=bowling_scores.get)
    
    print(sorted_keys)                 # ['zeb', 'jeb', 'janice', 'mike']
    
    
    for player in sorted_keys:
        print(f"{player} scored {bowling_scores[player]}")
    
            ##  zeb scored 98
            ##  jeb scored 123
            ##  janice scored 184
            ##  mike scored 202

    The key= argument allows us to specify an alternate criteria by which we might sort the keys. This will be discussed in depth shortly. Keep in mind that the key= argument is no so called in relation to a dictinoary's keys; it is coincidentally named key=.


    Practical: Build Up a Dict from Two Fields in a File

    As with all containers, we loop through a data source, select and add to a dict.


    ids_names = dict()                                  # initialize an
                                                        # empty dict
    for line in open('student_db.txt'):
        id, address, city, state, zip = line.split(':') # note "multi-target
                                                        # assignment" of
                                                        # 5 elements
    
        ids_names[id] = state                           # key id is paired to
                                                        # student's state
    
    print("here are ids and names from the students.txt file:  ")
    for id in ids_names:
        print("id " + id + " is from this state:  " + ids_names[id])
    
    print("here is the state for student 'jb29':  ")
    print(ids_names['jb29'])                            #  NJ

    Practical: Build a "Counting" or "Summing" Dictionary

    We can use a dict's keys and associated values to create an aggregation (correlating a sum or count to each of a collection of keys).


    state_count = dict()                               # initialize an empty dict
    for line in open('/Users/dblaikie/Desktop/python_data/student_db.txt'):
        items = line.split(':')
        state = items[3]
        if state not in state_count:
            state_count[state] = 0
        state_count[state] = state_count[state] + 1
    
    print("here is the count of states from the students.txt file:  ")
    for state in state_count:
        print(f"{state}:  {state_count[state]} occurrences")
    
    print("here is the count for 'NY':  ")
    print(state_count['NY'])                           # 4

    Practical: Build and Read a Dict of Lists

    A list can be added to a dict just like any other object.


    Here we're keying a dict to student id, and setting a list of the remaining elements as its value:

    ids_data = dict()                   # initialize an empty dict
    for line in open('student_db.txt'):
        item_list = line.split(':')     # note "multi-target assignment"
                                        # of 5 elements
        id = item_list[0]
        data = item_list[1:]
        ids_data[id] = data             # key id is paired to student's state
    
    print("here is the data for id 'jb29':  ", ids_data['jb29'])    #

    Practical: Working with dict items()

    dict items() produces an iterator of 2-item tuples. dict() can convert back to a dictionary.


    mydict = {'a': 1, 'b': 2, 'c': 3}
    these_items = list(mydict.items())   # list() produces a list from iterator
    print(these_items)                    # [('a', 1), ('c', 3), ('b', 2)]
    
    
    newdict = dict(these_items)
    
    print(newdict)                        # {'a': 1, 'b': 2, 'c': 3}

    2-item tuples can be sorted and sliced, so they are a handy alternate structure.


    zip() zips up parallel lists into tuples

    list1 = ['a', 'b', 'c', 'd']
    list2 = [ 1,   2,   3,   4 ]
    
    tupes = list(zip(list1, list2))
    
    print(tupes)          # [('a', 1), ('b', 2), ('c', 3), ('d', 4)]
    print(dict(tupes))    # {'a': 1,    'b': 2,   'c': 3,   'd': 4}

    Occasionally we are faced with lists that relate to each other one a 1-to-1 basis... or, we sometimes even shape our data into this form. Paralell lists like these can be zipped into multi-item tuples.




    CSV / Tabular File Parsing

    Table Data: Rows and Fields

    Tables consist of records (rows) and fields (column values).


    Tabular text files are organized into rows and columns.


    comma-separated values file (CSV)

        19260701,0.09,0.22,0.30,0.009
        19260702,0.44,0.35,0.08,0.009
        19270103,0.97,0.21,0.24,0.010
        19270104,0.30,0.15,0.73,0.010
        19280103,0.43,0.90,0.20,0.010
        19280104,0.14,0.47,0.01,0.010

    space-separated values file

        19260701    0.09    0.22    0.30   0.009
        19260702    0.44    0.35    0.08   0.009
        19270103    0.97    0.21    0.24   0.010
        19270104    0.30    0.15    0.73   0.010
        19280103    0.43    0.90    0.20   0.010
        19280104    0.14    0.47    0.01   0.010

    Our job for this lesson is to parse (separate) these values into usable data.


    Table Data in Text Files

    Text files are just sequences of characters. Newline characters separate text files into lines. Python reads text files line-by-line by separating lines at the newlines.


    If we print a CSV text file, we may see this:

        19260701,0.09,0.22,0.30,0.009
        19260702,0.44,0.35,0.08,0.009
        19270103,0.97,0.21,0.24,0.010
        19270104,0.30,0.15,0.73,0.010
        19280103,0.43,0.90,0.20,0.010
        19280104,0.14,0.47,0.01,0.010

    However, here's what a text file really looks like under the hood:

    19260701,0.09,0.22,0.30,0.009\n19260702,0.44,0.35,0.08,
    0.009\n19270103,0.97,0.21,0.24,0.010\n19270104,0.30,0.15,
    0.73,0.010\n19280103,0.43,0.90,0.20,0.010\n19280104,0.14,
    0.47,0.01,0.010

    The newline character separates the records in a CSV file. The delimeter (in this case, a comma) separates the fields. When displaying a file, your computer will translate the newlines into a line break, and drop down to the next line. This makes it seem as if each line is separate, but in fact they are only separated by newline characters.


    Goals for this Unit

    Summing, Counting, Averaging Values in Tabular Data


    Can we sum a column of values from a table? Our data analysis will resemble that of excel: it will read a column of value from a tabular file and sum up the values in that column. Here's how it works:

  • We set a summing float variable to 0.
  • We call function open() with a filename, returning a file object.
  • We loop through the file by using for with the file object.
  • Inside the for block, each line is a string. We split() the string on the delimeter (here, a comma), returning a list of strings.
  • We subscript the list to select the field value we want -- this is a string object with a numeric value.
  • We convert the string to a float object.
  • We add the float object to the summing float variable.
  • The loop continues until the file is exhausted.


    Tabular Data: Looping, Parsing and Summarizing

    Looping through file line strings, we can split and isolate fields on each line.


    The overall approach is to iterate over the file data with a for loop, in which each iteration produces a new line from the file. We then process the line by dividing it into fields, selecting one field, converting the field value to the needed type (if necessary). If our purpose is to produce a sum or list of values, we would establish a variable to hold the value(s) and add each field value to the variable.


    Here is "pseudocode" showing this process:

    ## establish a 'summary' variable (int, float, list, etc.) where data
    ## will be collected
    
    ## open() the file:  returns a file handle ('TextIOWrapper' object)
    
    ## loop through file handle with 'for':  each item in the iteration
    ## is a string file line
    
        ## use string rstrip() method to remove the newline from the line
        ## (if needed)
    
        ## split() the string into a list of items:  fields from that line
    
        ## subscript the list, returns an item/field from the line
    
        ## convert the item to the proper format (if it is a number,
        ## to convert from string to int or float)
    
        ## add the value to the summary variable
    
    ## when the loop is complete, report or otherwise use the value(s)
    ## scollected in the summary variable.

    Summary: File Object

    3 ways to read strings from a file.


    for: read (newline ('\n') marks the end of a line)

    fh = open('students.txt')                 # file object allows looping
                                              # through a series of strings
    for my_file_line in fh:                   # my_file_line is a string
        print(my_file_line)                   # prints each line of students.txt
    
    fh.close()                                # close the file

    read(): read entire file as a single string

    fh = open('students.txt')  # file object allows reading
    text = fh.read()                          # read() method called on file
                                              # object returns a string
    fh.close()                                # close the file
    
    print(text)

    The above prints:

    jw234,Joe,Wilson,Smithtown,NJ,2015585894
    ms15,Mary,Smith,Wilsontown,NY,5185853892
    pk669,Pete,Krank,Darkling,NJ,8044894893

    readlines(): read as a list of strings (each string a line)

    fh = open('students.txt')
    file_lines = fh.readlines()               # file.readlines() returns
                                              # a list of strings
    fh.close()                                # close the file
    print(file_lines)

    The above prints:

    ['jw234,Joe,Wilson,Smithtown,NJ,2015585894\n', 'ms15,Mary,Smith,Wilsontown,
    NY,5185853892\n', 'pk669,Pete,Krank,Darkling,NJ,8044894893\n']

    Summary: String Object

    Strings: 4 ways to manipulate strings from a file.


    split() a string into a list of strings

    mystr = 'jw234,Joe,Wilson,Smithtown,NJ,2015585894'
    elements = mystr.split(',')
    print(elements)                           # ['jw234', 'Joe', 'Wilson',
                                              #  'Smithtown', 'NJ', '2015585894']
    
    
    # alternative:  "multi-target" assignment
    # allows us to name each value on a row
    stuid, fname, lanme, city, state, stuzip = mystr.split(',')
    

    (included for completeness): join() a list of strings into a string

    mylist = ['jw234', 'Joe', 'Wilson', 'Smithtown', 'NJ', '2015585894']
    
    line = ','.join(mylist)          # 'jw234,Joe,Wilson,Smithtown,NJ,2015585894'

    slice a string

    mystr = '2014-03-13 15:33:00'
    year =  mystr[0:4]               # '2014'
    month = mystr[5:7]               # '03'
    day =   mystr[8:10]              # '13'

    rstrip() a string

    xx = 'this is a line with a newline at the end\n'
    
    yy = xx.rstrip()                 # return a new string without the newline
    
    print(yy)                        # 'this is a line with a newline at the end'

    Summary: List Object

    Lists: selecting individual elements of a list.


    A list is a sequence of objects of any type:


    initialize a list: lists are initalized with square brackets and comma-separated objects.

    aa = ['a', 'b', 'c', 3.5, 4.09, 2]

    subscript a list: using the list name, square brackets an an element index, starting at 0

    elements = ['jw234', 'Joe', 'Wilson', 'Smithtown', 'NJ', '2015585894']
    
    var = elements[0]               # 'jw234'
    var2 = elements[4]              # 'NJ'
    var3 = elements[-1]             # '2015585894' (-1 means last index)

    Summary: len() function for string and list length

    len() can be used to measure lists as well as strings.


    mystr = 'hello'
    mylist = [1.3, 1.9, 0.9, 0.3]
    
    lms = len(mystr)                # 5  (number of characters in mystr)
    lml = len(mylist)               # 4  (number of elements in mylist)

    Because it can measure lists or strings, len() can also measure files (when rendered as a list of strings or a whole string).


    Summary: repr() function for "true" representations of strings

    repr() takes any object and shows a more "true" representation of it. With a string, repr() will show us the newlines at the end of each line


    aa = open('small_db.txt')       # open a file, returns
                                    # a file object
    
    xx = aa.read()                  # read() on a file object,
                                    # returns a single string
    
    print(repr(xx))                 # the string with newlines visible:
                                    # '101:Acme:483982.90\n
                                    # 102:Boon:119001.94\n
                                    # 103:Cary:38322.15\n
                                    # 104:Daws:4838292.11\n105:Eaton:22000.00'

    Reading a file: options

    (Note that the remaining slides repeat some of the same material, but from a more practical perspective.)


    for loop: loop line-by-line The for loop repeats execution of its block until the file is completely read. Note that the for block is very similar to the while. The difference is that while relies on a test to continue executing, but for continues until it reaches the end of the file.


    fh = open('students.txt')              # file object allows
                                           # looping through a
                                           # series of strings
    
    for xx in fh:                          # xx is a string
        print(xx)                          # prints each line of students.txt
    fh.close()                             # close the file

    "xx" is called a control variable, and it is automatically reassigned each line in the file as a string. break and continue work with for as well as while loops.


    Table Records Are Read from a file as String Objects

    As Python reads files line-by-line, it handles each line as a string object.


    fh = open('students.txt')        # file object allows looping
                                     # through a series of strings
    for bb in fh:
         print(type(bb))            # <class 'str'>

    Again, the control variable bb is reassigned for each iteration of the loop. This means that if the file has 5 lines, the loop executes 5 times and bb is reassigned a new value 5 times.


    Stripping a file line with rstrip()

    When reading a file line-by-line, we should strip off the newline with the string method rstrip().


    fh = open('students.txt')   # file object allows looping through a
                                                # series of strings
    for xx in fh:               # xx is a string
    
        xx = xx.rstrip()        # remove "whitespace" from end of the line
                                # and return a new string with the string removed
    
        print(xx)               # prints each line of students.txt
    
    fh.close()                  # close the file

    String slicing

    A string can be sliced by position: we specify the start and end position of the slice.


    Indices start at 0; the "upper bound" is non-inclusive

    mystr = '19320805    3.62   -2.38    0.08   0.001'
    
    year =  mystr[0:4]             # '1932'
    month = mystr[4:6]             # '08'
    day = mystr[6:8]               # '05'

    To slice to the end, omit the upper bound

    mystr = '19320805    3.62   -2.38    0.08   0.001'
    
    rf_val = mystr[32:]            # '   0.001'

    Table Fields Are Parsed from File Line Strings into Lists of Strings

    The string split() method returns a list of strings, each string a field in a single record (row or line from the table).


    The delimeter tells Python how to split the string. Note that the delimeter does not appear in the list of strings.


    line_from_file = 'jw234:Joe:Wilson:Smithtown:NJ:2015585894\n'
    
    xx = line_from_file.split(':')
    
    print(xx)                         # ['jw234', 'Joe', 'Wilson',
                                      #  'Smithtown', 'NJ', '2015585894\n']

    If no delimeter is supplied, the string is split on whitespace:


    gg = 'this is a file    with    some     whitespace'
    
    hh = gg.split()                   # splits on any "whitespace character"
    
    print(hh)                         # ['this', 'is', 'a', 'file',
                                      #  'with', 'some', 'whitespace']

    Table Fields Are Selected from a List Using List Subscripts

    Each table record (row or line from the table) when rendered as a list of strings (from split()) is addressable by index.


    The index starts at 0. A negative index (-1, -2, etc.) will count from the end.


    gg = '2016:5.0:5.3:5.9:6.1'
    
    hh = gg.split(':')                     # splits on any "whitespace character"
    
    print(hh)                              # ['2016', '5.0', '5.3', '5.9', '6.1']
    
    kk = hh[0]                             # '2016'   (index starts at 0)
    
    mm = hh[1]                             # '5.0'
    
    zz = hh[-1]                            # '6.1'    (negative index selects
                                           #           from the end of the list)
    
    yy = hh[-2]                            # '5.9'

    Portions of a string can be "sliced" using string slicing

    Special slice syntax lets us specify a substring by position.


    split() separates a string based on a delimeter, but some strings have no delimeter but must be parsed by position:

    mystr = '20140313'
    year =  mystr[0:4]                     # '2014'  (the 0th through 3rd index)
    month = mystr[4:6]                     # '03'    (the 4 and 5 index values)
    day =   mystr[6:]                      # '13'    (note that no upper index
                                           #          means slice to the end)

    Note that the upper index is non-inclusive, which means that it specifies the index past the one desired. stride and negative stride


    A third value, the stride or step value, allows skipping over characters (every 2nd element every 3rd element, etc.)

    mystr = '20140303'
    
    skipper = mystr[0:7:2]                 # '2100'

    The negative stride actually reverses the string (when used with no other index):

    mystr = '20140303'
    
    reverser = mystr[::-1]                 # '30304102'



    pandas: Advanced groupby(), apply(), TimeSeries, Binning and Categorizing

    Series.apply(): apply a function call across a vector

    The function is called with each value in a row or column.


    Sometimes our computation is more complex than simple math, or we need to apply a function to each element. We can use apply():

    import pandas as pd
    
    df = pd.DataFrame( {'a': [1, 2, 3, 4],
                        'b': [1.0, 1.5, 2.0, 2.5],
                        'c': ['a', 'b', 'c', 'd'] }, index=['r1', 'r2', 'r3', 'r4'] )
    
    print(df)
    
                      #     a    b  c
                      # r1  1  1.0  a
                      # r2  2  1.5  b
                      # r3  3  2.0  c
                      # r4  4  2.5  d
    
    
    df['d'] = df['c'].apply(str.upper)
    
    print(df)
                      #     a    b  c  d
                      # r1  1  1.0  a  A
                      # r2  2  1.5  b  B
                      # r3  3  2.0  c  C
                      # r4  4  2.5  d  D

    apply() with custom function or lambda

    We use a custom named function or a lambda with apply():


    print(df)
                      #     a    b  c  d
                      # r1  1  1.0  a  A
                      # r2  2  1.5  b  B
                      # r3  3  2.0  c  C
                      # r4  4  2.5  d  D
    
    
    df['e'] = df['a'].apply(lambda x: '$' + str(x * 1000) )
    
    print(df)
                      #     a    b  c  d      e
                      # r1  1  1.0  a  A  $1000
                      # r2  2  1.5  b  B  $2000
                      # r3  3  2.0  c  C  $3000
                      # r4  4  2.5  d  D  $4000

    See below for an explanation of lambda.


    Review: lambda expressions

    A lambda describes a function in shorthand.


    Compare these two functions, both of which add/concatenate their arguments:

    def addthese(x, y):
      return x + y
    
    addthese2 = lambda x, y:  x + y
    
    print(addthese(5, 9))        # 14
    print(addthese2(5, 9))       # 14

    The function definition and the lambda statement are equivalent - they both produce a function with the same functionality.


    Advanced groupby(): multi-column aggregation

    Calculating a sum or count based on values in 2 or more columns.


    To aggregate by values in two combined columns, simply pass a list of columns by which to aggregate -- the result is called a "multi-column aggregation":


    print(df.groupby(['region', 'revtype']).sum())
    
                      #                   revenue
                      # region revtype
                      # NE     retail          12
                      #        wholesale       16
                      # NW     retail          18
                      #        wholesale       15
                      # SW     retail           3
                      #        wholesale       10

    Note that the index has 2 columns (you can tell in that the tops of the columns are 'recessed' beneath the column row). This is a MultiIndex or hierarchical index. In the above example the NE stands over both retail and wholesale in the first 2 rows -- we should read this as NE-retail and NE-wholesale.


    grouping functions: use a custom summary function

    Like passing a function to sorted(), we can pass a function to df.groupby()


    df = pd.DataFrame( { 'company': [ 'Alpha', 'Alpha', 'Alpha',
                                      'Beta', 'Beta', 'Beta',
                                      'Gamma', 'Gamma', 'Gamma'],
                         'region':  [ 'NE', 'NW', 'SW', 'NW', 'SW',
                                      'NE', 'NE', 'SW', 'NW'],
                         'revenue': [ 10, 9, 2, 15, 8, 2, 16, 3, 9],
                         'revtype': [ 'retail', 'retail', 'wholesale',
                                      'wholesale', 'wholesale', 'retail',
                                      'wholesale', 'retail', 'retail'     ] } )
    
    print(df)
    
                      #   company region  revenue    revtype
                      # 0   Alpha     NE       10     retail
                      # 1   Alpha     NW        9     retail
                      # 2   Alpha     SW        2  wholesale
                      # 3    Beta     NW       15  wholesale
                      # 4    Beta     SW        8  wholesale
                      # 5    Beta     NE        2     retail
                      # 6   Gamma     NE       16  wholesale
                      # 7   Gamma     SW        3     retail
                      # 8   Gamma     NW        9     retail

    groupby() functions using apply() We can design our own custom functions -- we simply use apply() and pass a function (you might remember similarly passing a function from the key= argument to sorted()). Here is the equivalent of the sum() function, written as a custom function:


    def get_sum(df_slice):
        return sum(df_slice['revenue'])
    
    print(df.groupby('region').apply(get_sum))  # custom function: same as groupby('region').sum()
    
                      # region
                      # NE    28
                      # NW    33
                      # SW    13
                      # dtype: int64

    As was done with sorted(), pandas calls our groupby function multiple times, once with each group. The argument that Python passes to our custom function is a dataframe slice containing just the rows from a single grouping -- in this case, a specific region (i.e., it will be called once with a silce of NE rows, once with NW rows, etc. The function should be made to return the desired value for that slice -- in this case, we want to see the sum of the revenue column (as mentioned, this is simply illustrating a function that does the same work as the built-in .sum() function). (For a better view on what is happening with the function, print df_slice inside the function -- you will see the values in each slice printed.) Here is a custom function that returns the median ("middle value") for each region:


    def get_median(df):
        listvals = sorted(list(df['revenue']))
        lenvals = len(listvals)
        midval = listvals[ int(lenvals / 2) ]
        return midval
    
    print(df.groupby('region').apply(get_median))
    
                      # region
                      # NE    10
                      # NW     9
                      # SW     3
                      # dtype: int64

    grouping functions: use a function to identify a group

    Standared aggregations group rows based on a column value ('NW', 'SW', etc.) or a combination of column values. If more work is needed to identify a group, we can supply a custom function for this operation as well. Perhaps we'd like to group our rows by whether or not they achieved a certain revenue target within a region. Basically we want to group each row by whether the value is 10 or greater (i.e., 10 or more for a company/region/revenue type). Our function will simply return the number of decimal places in the value. So, we can process this column value (or even include other column values) by referencing a function in the call to groupby():


    def bydecplace(idx):
        row = df.loc[idx]                 # a Series with the row values for this index
        return(len(str(row['revenue'])))  # '2' if 10; '1' if 9
    
    print((df.groupby(bydecplace).sum()))
                      #      revenue
                      #   1       33
                      #   2       41

    The value passed to the function is the index of a row. We can thus use the .loc attribute with the index value to access the row. This function isolates the revenue within the row and returns its string length. using lambdas as groupby() or grouping functions Of course any of these simple functions can be rewritten as a lambda (and in many cases, should be, as in the above case since the function references the dataframe directly, and we should prefer not to refer to outside variables in a standard function):


    def bydecplace(idx):
        row = df.loc[idx]                 # a Series with the row values for this index
        return(len(str(row['revenue'])))  # '2' if 10; '1' if 9
    
    print((df.groupby(lambda idx:  len(str(df.loc[idx]['revenue']))).sum()))
    
                      #        revenue
                      # alpha       21
                      # beta        25
                      # gamma       28

    Review: the Index -- DataFrame Column or Index Labels

    An Index object is used to specify a DataFrame's columns or index, or a Series' index.


    Columns and Indices


    A DataFrame makes use of two Index objects: one to represent the columns, and one to represent the rows.

    df = pd.DataFrame( {'a': [1, 2, 3, 4],
                        'b': [1.0, 1.5, 2.0, 2.5],
                        'c': ['a', 'b', 'c', 'd'],
                        'd': [100, 200, 300, 400] },
                        index=['r1', 'r2', 'r3', 'r4'] )
    print(df)
        #     a    b  c    d
        # r1  1  1.0  a  100
        # r2  2  1.5  b  200
        # r3  3  2.0  c  300
        # r4  4  2.5  d  400

    .rename() method: columns or index labels can be reset using this DataFrame method.

    df = df.rename(columns={'a': 'A', 'b': 'B', 'c': 'C', 'd': 'D'},
                   index={'r1': 'R1', 'r2': 'R2', 'r3': 'R3', 'r4': 'R4'})
    print(df)
        #     A    B  C    D
        # R1  1  1.0  a  100
        # R2  2  1.5  b  200
        # R3  3  2.0  c  300
        # R4  4  2.5  d  400

    .columns, .index: the columns or index can also be set directly using the DataFrame's attributes (this would have the same effect as above):

    df.columns = ['A', 'B', 'C', 'D']
    df.index = ['r1', 'r2', 'r3', 'r4']

    .set_index(): set any column to the index

    df2 = df.set_index('A')
    print(df2)
        #      B  C    D
        # A
        # 1  1.0  a  100
        # 2  1.5  b  200
        # 3  2.0  c  300
        # 4  2.5  d  400

    .reset_index(): we can reset the index to integers starting from 0; by default this converts the previous into a new column:

    df3 = df.reset_index()
    print(df3)
        #   index  A    B  C    D
        # 0    R1  1  1.0  a  100
        # 1    R2  2  1.5  b  200
        # 2    R3  3  2.0  c  300
        # 3    R4  4  2.5  d  400

    or to drop the index when resetting, include drop=True

    df4 = df.reset_index(drop=True)
    print(df4)
        #    A    B  C    D
        # 0  1  1.0  a  100
        # 1  2  1.5  b  200
        # 2  3  2.0  c  300
        # 3  4  2.5  d  400

    .reindex(): we can change the order of the indices and thus the rows:

    df5 = df.reindex(reversed(df.index))
    
    df5 = df5.reindex(columns=reversed(df.columns))
    
    print(df5)
    
                          #  state  A    B  C    D
                          #  year
                          #  R1     1  1.0  a  100
                          #  R2     2  1.5  b  200
                          #  R3     3  2.0  c  300
                          #  R4     4  2.5  d  400

    we can set names for index and column indices:

    df.index.name = 'year'
    df.columns.name = 'state'

    There are a number of "exotic" Index object types: Index (standard, default and most common Index type) RangeIndex (index built from an integer range) Int64Index, UInt64Index, Float64Index (index values of specific types) DatetimeIndex, TimedeltaIndex, PeriodIndex, IntervalIndex (datetime-related indices) CategoricalIndex (index related to the Categorical type)


    The MultiIndex: a sequence of tuples

    In a MultiIndex, we can think of a column or row label as having two items.


    A MultiIndex specifies an "index within an index" or "column within a column" for more sophisticated labeling of data.


    A DataFrame with multi-index columns this and that and multi-index index other and another

    this                  a                   b
    that                  1         2         1         2
    other another
    x     1       -1.618192  1.040778  0.191557 -0.698187
          2        0.924018  0.517304  0.518304 -0.441154
    y     1       -0.002322 -0.157659 -0.169507 -1.088491
          2        0.216550  1.428587  1.155101 -1.610666

    The MultiIndex can be generated by a multi-column aggregation, or it can be set directly, as below.


    The from_tuples() method creates a MultiIndex from tuple pairs that represent levels of the MultiIndex:

    arrays = [['bar', 'bar', 'baz', 'baz', 'foo', 'foo', 'qux', 'qux'],
              ['one', 'two', 'one', 'two', 'one', 'two', 'one', 'two']]
    
    tuples = list(zip(*arrays))    # zip two lists like a zipper
    
    #  [('bar', 'one'),
    #   ('bar', 'two'),
    #   ('baz', 'one'),
    #   ('baz', 'two'),
    #   ('foo', 'one'),
    #   ('foo', 'two'),
    #   ('qux', 'one'),
    #   ('qux', 'two')]
    
    
    index = pd.MultiIndex.from_tuples(tuples, names=['first', 'second'])
    
    #   MultiIndex(levels=[['bar', 'baz', 'foo', 'qux'], ['one', 'two']],
    #              labels=[[0, 0, 1, 1, 2, 2, 3, 3], [0, 1, 0, 1, 0, 1, 0, 1]],
    #              names=['first', 'second'])

    The notation above is somewhat hard to read; the labels= parameter specifies which of the two levels= lists values appears in each tuple pair.


    Here we're applying the above index to a Series object:

    s = pd.Series(np.random.randn(8), index=index)
    
    #   first  second
    #   bar    one       0.469112
    #          two      -0.282863
    #   baz    one      -1.509059
    #          two      -1.135632
    #   foo    one       1.212112
    #          two      -0.173215
    #   qux    one       0.119209
    #          two      -1.044236
    #   dtype: float64

    Slicing a MultiIndex

    Slicing works more or less as expected; tuples help us specify Multilevel indices.


    mindex = pd.MultiIndex(levels=[['bar', 'baz', 'foo', 'qux'], ['one', 'two']],
                           labels=[[0, 0, 1, 1, 2, 2, 3, 3], [0, 1, 0, 1, 0, 1, 0, 1]],
                           names=['first', 'second'])
    
    df = pd.DataFrame(np.random.randn(8, 4), columns=['A', 'B', 'C', 'D'], index=mindex)
    
    #                        A         B         C         D
    #   first second
    #   bar   one    -0.231171  0.340523  0.472207 -0.543819
    #         two     0.113923  0.367657  0.171424 -0.039921
    #   baz   one    -0.625282 -0.791371 -0.487958  0.568405
    #         two    -1.128698 -1.040629  2.536821 -0.844057
    #   foo   one    -1.319797 -1.277551 -0.614919  1.305367
    #         two     0.414166 -0.427726  0.929567 -0.524161
    #   qux   one     1.859414 -0.190417 -1.824712  0.454862
    #         two    -0.169519 -0.850846 -0.444302 -0.577360

    standard slicing

    df['A']
    #   first  second
    #   bar    one      -0.231171
    #          two       0.113923
    #   baz    one      -0.625282
    #          two      -1.128698
    #   foo    one      -1.319797
    #          two       0.414166
    #   qux    one       1.859414
    #          two      -0.169519
    #   Name: A, dtype: float64
    
    
    df.loc['bar']
    #                  A         B         C         D
    #   second
    #   one    -0.231171  0.340523  0.472207 -0.543819
    #   two     0.113923  0.367657  0.171424 -0.039921
    
    
    df.loc[('bar', 'one')]             # also:  df.loc['bar'].loc['one']
    #   A   -0.231171
    #   B    0.340523
    #   C    0.472207
    #   D   -0.543819
    #   Name: (bar, one), dtype: float64
    
    
    df.loc[('bar', 'two'), 'A']
    #   0.11392342023306047

    'cross-section' slicing with .xs

    The 'level' parameter allows slicing along a lower level


    mindex = pd.MultiIndex(levels=[['bar', 'baz', 'foo', 'qux'], ['one', 'two']],
                           labels=[[0, 0, 1, 1, 2, 2, 3, 3], [0, 1, 0, 1, 0, 1, 0, 1]],
                           names=['first', 'second'])
    
    df = pd.DataFrame(np.random.randn(8, 4), columns=['A', 'B', 'C', 'D'], index=mindex)
    
    #                        A         B         C         D
    #   first second
    #   bar   one    -0.231171  0.340523  0.472207 -0.543819
    #         two     0.113923  0.367657  0.171424 -0.039921
    #   baz   one    -0.625282 -0.791371 -0.487958  0.568405
    #         two    -1.128698 -1.040629  2.536821 -0.844057
    #   foo   one    -1.319797 -1.277551 -0.614919  1.305367
    #         two     0.414166 -0.427726  0.929567 -0.524161
    #   qux   one     1.859414 -0.190417 -1.824712  0.454862
    #         two    -0.169519 -0.850846 -0.444302 -0.577360
    
    
    # standard slicing
    df.xs('bar')
    #                  A         B         C         D
    #   second
    #   one    -0.231171  0.340523  0.472207 -0.543819
    #   two     0.113923  0.367657  0.171424 -0.039921
    
    df.xs(('baz', 'two'))
    #   A   -1.128698
    #   B   -1.040629
    #   C    2.536821
    #   D   -0.844057
    #   Name: (baz, two), dtype: float64
    
    
    # using the level= parameter
    df.xs('two', level='second')
    #                 A         B         C         D
    #   first
    #   bar    0.113923  0.367657  0.171424 -0.039921
    #   baz   -1.128698 -1.040629  2.536821 -0.844057
    #   foo    0.414166 -0.427726  0.929567 -0.524161
    #   qux   -0.169519 -0.850846 -0.444302 -0.577360

    TimeSeries: objects and methods

    These custom pandas objects provide powerful date calculation and generation.


    Timestamp: a single timestamp representing a date/time Timedelta: a date/time interval (like 1 months, 5 days or 2 hours) Period: a particular date span (like 4/1/16 - 4/3/16 or 4Q17) DatetimeIndex: DataFrame or Series Index of Timestamp PeriodIndex: DataFrame or Series Index of Period Timestamp: a single point in time


    Timestamp() constructor: creating a Timestamp object from string, ints or datetime():

    tstmp = pd.Timestamp('2012-05-01')
    tstmp = pd.Timestamp(2012, 5, 1)
    tstmp = pd.Timestamp(datetime.datetime(2012, 5, 1))
    
    year  = tstmp.year    # 2012
    month = tstmp.month   # 5
    day   = tstmp.day     # 1

    .to_datetime(): convert a string, list of strings or Series to dates

    tseries = pd.to_datetime(['2005/11/23', '2010.12.31'])
       # DatetimeIndex(['2005-11-23', '2010-12-31'], dtype='datetime64[ns]', freq=None)
    
    tseries = pd.to_datetime(pd.Series(['Jul 31, 2009', '2010-01-10', None]))
    
    # using European dates
    tstmp = pd.to_datetime('11/12/2010', dayfirst=True)   # 2010-11-12


    Timedelta: a time interval


    Timedelta() constructor: creating an interval

    # strings
    td = pd.Timedelta('1 days')                 # Timedelta('1 days 00:00:00')
    td =  pd.Timedelta('1 days 00:00:00')       # Timedelta('1 days 00:00:00')
    td = pd.Timedelta('1 days 2 hours')         # Timedelta('1 days 02:00:00')
    td = pd.Timedelta('-1 days 2 min 3us')      # Timedelta('-2 days +23:57:59.999997')
    
    # negative Timedeltas
    td = pd.Timedelta('-1us')                   # Timedelta('-1 days +23:59:59.999999')
    
    
    # with args similar to datetime.timedelta
    # note: these MUST be specified as keyword arguments
    td = pd.Timedelta(days=1, seconds=1)        # Timedelta('1 days 00:00:01')
    
    
    # integers with a unit
    td = pd.Timedelta(1, unit='d')              # Timedelta('1 days 00:00:00')


    Period: a specific datetime->datetime interval


    Period constructor: creating a date-to-date timespan

    perimon = pd.Period('2011-01')               # default interval is 'month' (end time is 2011-01-31 23:59:59.999)
    periday = pd.Period('2012-05-01', freq='D')  # specify 'daily' (end datetime is 2012-05-01 23:59:99.999)

    Filtering / Selecting Dates

    Let's start with data as it might come from a CSV file. We've designed the date column to be the DataFrame's index:

    import pandas as pd
    import numpy as np
    
    df = pd.DataFrame( {'impressions': [9,    10,   8,    3,    7,    12    ],
                        'sales':       [2.03, 2.38, 1.93, 0.63, 1.85, 2.53  ],
                        'clients':     [4,    6,    5,    1,    5,    7     ]  },
                        index=[ '2016-11-15', '2016-12-01', '2016-12-15',
                                '2017-01-01', '2017-01-15', '2017-02-01' ] )
    
    print(df)
                      #             clients  impressions  sales
                      # 2016-11-15        4            9   2.03
                      # 2016-12-01        6           10   2.38
                      # 2016-12-15        5            8   1.93
                      # 2017-01-01        1            3   0.63
                      # 2017-01-15        5            7   1.85
                      # 2017-02-01        7           12   2.53
    print(type(df.index[0]))                           # <class 'str'>

    Note that the index is listed as string. This would be standard in a read from a plaintext format like CSV (although not from a date-formatted column in Excel)


    We can convert the strings to Timestamp with astype():

    df.index = df.index.astype(np.datetime64)
    
    print(type(df.index))                      # <class 'pandas.tseries.index.DatetimeIndex'>
    print(type(df.index[0]))                   # <class 'pandas.tslib.Timestamp'>

    Now the index is a DatetimeIndex (no longer an Index), consisting of Timestamp objects, optimized for date calculation and selection.


    Filtering: a Series or DatetimeIndex of np.Timestamp objects, they can be selected or filtered quite easily:

    rng = pd.date_range('1/1/2016', periods=24, freq='M')
    
    
    # all entries from 2016
    print(df['2016'])
    
                      #             clients  impressions  sales
                      # 2016-11-15        4            9   2.03
                      # 2016-12-01        6           10   2.38
                      # 2016-12-15        5            8   1.93
    
    
    # all entries from Dec. 2016
    print(df['2016-12'])
    
                      #             clients  impressions  sales
                      # 2016-12-01        6           10   2.38
                      # 2016-12-15        5            8   1.93
    
    
    # all entries from Dec. 2016
    print(df['2016-12-10':])
    
                      #             clients  impressions  sales
                      # 2016-12-15        5            8   1.93
                      # 2017-01-01        1            3   0.63
                      # 2017-01-15        5            7   1.85
                      # 2017-02-01        7           12   2.53
    
    
    # all entries from 12/10/16 - 1/10/17
    print(df['2016-12-10': '2017-01-10'])
    
                      #             clients  impressions  sales
                      # 2016-12-15        5            8   1.93
                      # 2017-01-01        1            3   0.63

    Creating, comparing and calculating dates with pd.Timedelta

    We add or subtract a Timedelta interval from a Timestamp


    Comparing Timestamps

    ts1 = pd.Timestamp('2011-07-09 11:30')
    ts2 = pd.Timestamp('2011-07-10 11:35')
    
    print(ts1 > ts2)            # False
    print(ts1 < ts2)            # True

    Computing Timedeltas

    td1 = ts2 - ts1
    print(td1)                 # 1 days 00:05:00
    print((type(td1)))           # 
    
    # values in a Timedelta boil down to days and seconds
    print(td.days)              # 1
    print(td.seconds)           # 300
    
    ts3 = ts2 + td              # adding 1 day and 5 minutes
    print(ts3)                  # Timestamp('2011-07-11 11:40:00')

    Creating Timedeltas

    # strings
    pd.Timedelta('1 days')                 # Timedelta('1 days 00:00:00')
    
    pd.Timedelta('1 days 00:00:00')        # Timedelta('1 days 00:00:00')
    
    pd.Timedelta('1 days 2 hours')         # Timedelta('1 days 02:00:00')
    
    pd.Timedelta('-1 days 2 min 3us')      # Timedelta('-2 days +23:57:59.999997')
    
    # like datetime.timedelta
    # note: these MUST be specified as keyword arguments
    pd.Timedelta(days=1, seconds=1)        # Timedelta('1 days 00:00:01')
    
    # integers with a unit
    pd.Timedelta(1, unit='d')              # Timedelta('1 days 00:00:00')
    
    # from a datetime.timedelta/np.timedelta64
    pd.Timedelta(datetime.timedelta(days=1, seconds=1))
                                           # Timedelta('1 days 00:00:01')
    
    pd.Timedelta(np.timedelta64(1, 'ms'))  # Timedelta('0 days 00:00:00.001000')
    
    # negative Timedeltas
    pd.Timedelta('-1us')                   # Timedelta('-1 days +23:59:59.999999')

    Generating a date range with pd.date_range()

    date_range() provides evenly spaced Timestamp objects.


    date_range() with a start date, periods= and freq=:

    # By default date_range() returns a DatetimeIndex.
    # 5 hours starting with midnight Jan 1st, 2011
    rng = pd.date_range('1/1/2011', periods=5, freq='H')
    print(rng)
            # DatetimeIndex(['2011-01-01 00:00:00', '2011-01-01 01:00:00',
            #                '2011-01-01 02:00:00', '2011-01-01 03:00:00',
            #                '2011-01-01 04:00:00'],
            #                dtype='datetime64[ns]', freq='H')
    
    
    ts = pd.Series(list(range(0, len(rng))), index=rng)
    
    print(ts)
        # 2011-01-01 00:00:00    0
        # 2011-01-01 01:00:00    1
        # 2011-01-01 02:00:00    2
        # 2011-01-01 03:00:00    3
        # 2011-01-01 04:00:00    4
        # Freq: H, dtype: int64

    date_range() with a start date and end date

    start = pd.Timestamp('1/1/2011')
    end =  pd.Timestamp('1/5/2011')
    tindex = pd.date_range(start, end)
    
    print(tindex)
      # DatetimeIndex(['2011-01-01', '2011-01-02', '2011-01-03',
      #                '2011-01-04', '2011-01-05'])
      #               dtype='datetime64[ns]', length=5, freq='D')
    
      # note default frequency:  'D' (days)
    

    date_range() with a monthly period, dates are set to end of the month:

    tindex = pd.date_range(start='1/1/1980', end='11/1/1990', freq='M')

    date_range() with a monthly period, dates are set to start of the month:

    tindex = pd.date_range(start='1/1/1980', end='11/1/1990', freq='MS')

    date_range() with a start date, periods and freq

    tindex = pd.date_range('1/1/2011', periods=3, freq='W')
    
    print(tindex)
        # DatetimeIndex(['2011-01-02', '2011-01-09', '2011-01-16'],
        #               dtype='datetime64[ns]', freq='W-SUN')

    Note that freq= has defaulted to W-SUN which indicates weekly beginning on Sunday. pandas even adjusted our first day on this basis! We can specify the day of the week ourselves to start on a precise date.


    bdate_range() provides a date range that includes "business days" only:

    tbindex = pd.bdate_range(start, end)
    
    print(tbindex)
      # DatetimeIndex(['2011-01-03', '2011-01-04', '2011-01-05'],
      #               dtype='datetime64[ns]', freq='B')
    
      # (the 1st and 2nd of Jan. 2011 are Saturday and Sunday)

    See the offset aliases portion of the documentation.


    Comparing dates within intervals with pd.Period

    The Period represents an interval with a start date/time


    The .end_time attribute value is calulated as the start date/time + freq= value.

    # a 'day' period
    per = pd.Period('2016-05-03')    # Period('2016-05-03', 'D')
    
    print(per.start_time)            # Timestamp('2016-05-03 00:00:00')
    print(per.end_time)              # Timestamp('2016-05-03 23:59:59.999999999')
    
    # a 'month' period
    pdfm = pd.Period('2016-05-03', freq='M')
    
    print(pdfm.start_time)           # Timestamp('2016-05-01 00:00:00')
    
    print(pdfm.end_time)             # Timestamp('2016-05-31 23:59:59.999999999')

    "frequency" (or freq=) is a bat of a misnomer. It describes the size of the period -- that is, the amount of time it covers. Thus a freq='M' (month) period ends a month later than the start date.


    The Period object can be incremented to produce a new Period object. The freq interval determines the start date/time and size of the next Period.

    # a 'month' period
    pdfm = pd.Period('2016-05-03', freq='M')
    
    pdfm2 = pdfm + 1
    
    print(pdfm2.start_time)           # Timestamp('2016-06-01 00:00:00')
    print(pdfm2.end_time)             # Timestamp('2016-06-30 23:59:59.999999999')

    period_range(): produce a range of Period objects


    ps = pd.Series(list(range(12)), pd.period_range('1/2017', '12/2017', freq='M'))
    
    print(ps)
        # 2017-01     0
        # 2017-02     1
        # 2017-03     2
        # 2017-04     3
        # 2017-05     4
        # 2017-06     5
        # 2017-07     6
        # 2017-08     7
        # 2017-09     8
        # 2017-10     9
        # 2017-11    10
        # 2017-12    11
        # Freq: M, dtype: int64
    

    Above we have an index of Period objects; each period represents a monthly interval.


    This differs from TimeStamp in that a comparison or selection (such as a slice) will include any value that falls within the requested period, even if the date range is partial:

    print(ps['2017-03-15': '2017-06-15'])
    
        # 2017-03    2
        # 2017-04    3
        # 2017-05    4
        # 2017-06    5
        # Freq: M, dtype: int64

    Note that both 03 and 06 were included in the results, because the slice fell between their ranges.


    Quarterly Period Range

    prng = pd.period_range('1990Q1', '2000Q4', freq='Q-JAN')
    
    sq = pd.Series(list(range(0, len(prng))), prng)
    print(sq)
    
        # 1990Q1    0
        # 1990Q2    1
        # 1990Q3    2
        # 1990Q4    3
        # 1991Q1    4
        # 1991Q2    5
        # 1991Q3    6
        # 1991Q4    7
        # Freq: Q-JAN, dtype: int64
    
    sq[pd.Timestamp('1990-02-13')]    # 4

    Binning

    Dividing values into bins based on a category scheme


    Bins allow us to categorize values (often dates) into "bins" which are mapped to a value to be applied. Consider the table below, which might come from an Excel spreadsheet:


    dfbin = pd.DataFrame({'start_date': [1, 6, 11, 16],
                          'end_date': [5, 10, 15, 20],
                          'percent': [1, 2, 3, 10]})
    
    # order the columns
    dfbin = dfbin[['start_date', 'end_date', 'percent']]
    
    print(dfbin)
             #    start_date  end_date  percent
             # 0           1         5        1
             # 1           6        10        2
             # 2          11        15        3
             # 3          16        20       10

    Any date from 1-5 should key to 1%; any from 6-10, 2%, etc.


    We have data that needs to be categorized into the above bins:

    data = pd.DataFrame({'period': list(range(1, 21))})
    
    print(data)
             #         period
             #     0        1
             #     1        2
             #     2        3
             #     3        4
             #     4        5
             #     5        6
             #     6        7
             #     7        8
             #     8        9
             #     9       10
             #     10      11
             #     11      12
             #     12      13
             #     13      14
             #     14      15
             #     15      16
             #     16      17
             #     17      18
             #     18      19
             #     19      20
    
    
    print(dfbin)
             #    start_date  end_date  percent
             # 0           1         5        1
             # 1           6        10        2
             # 2          11        15        3
             # 3          16        20       10
    
    # converting the 'start_date' field into a list
    bins = list(dfbin['start_date'])
    
    # adding the last 'end_date' value to the end
    bins.append(dfbin.loc[len(dfbin)-1, 'end_date']+1)
    
    # category labels (which can be strings, but here are integers)
    cats = list(range(1, len(bins)))
    
    print(bins)
    print(cats)
             # [1, 6, 11, 16, 21]
             # [1, 2, 3, 4, 5]

    The cut function takes the data, bins and labels and sorts them by bin value:

    # 'right=False' keeps bins from overlapping (the bin does not include the rightmost edge)
    data['cat'] = pd.cut(data['period'], bins, labels=cats, right=False)
    print(data)
    
             #         period cat
             #     0        1   1
             #     1        2   1
             #     2        3   1
             #     3        4   1
             #     4        5   1
             #     5        6   2
             #     6        7   2
             #     7        8   2
             #     8        9   2
             #     9       10   2
             #     10      11   3
             #     11      12   3
             #     12      13   3
             #     13      14   3
             #     14      15   3
             #     15      16   4
             #     16      17   4
             #     17      18   4
             #     18      19   4
             #     19      20   4

    We are now free to use the bin mapping to apply the proper pct value to each row.




    pandas: Transforming, Sorting and Cleaning

    Vectorized Operations

    Operations to columns are vectorized, meaning they are propagated (broadcast) across all column Series in a DataFrame.


    import pandas as pd
    
    df = pd.DataFrame( {'a': [1, 2, 3, 4],
                        'b': [1.0, 1.5, 2.0, 2.5],
                        'c': ['a', 'b', 'c', 'd'] }, index=['r1', 'r2', 'r3', 'r4'] )
    
    print(df)
    
                      #     a    b  c
                      # r1  1  1.0  a
                      # r2  2  1.5  b
                      # r3  3  2.0  c
                      # r4  4  2.5  d
    
    
    # 'single value':  assign the same value to all cells in a column Series
    df['a'] = 0       # set all 'a' values to 0
    print(df)
    
                      #     a    b  c
                      # r1  0  1.0  a
                      # r2  0  1.5  b
                      # r3  0  2.0  c
                      # r4  0  2.5  d
    
    
    # 'calculation':  compute a new value for all cells in a column Series
    df['b'] = df['b'] * 2    # double all column 'b' values
    
    print(df)
    
                      #     a    b  c
                      # r1  0  2.0  a
                      # r2  0  3.0  b
                      # r3  0  4.0  c
                      # r4  0  5.0  d

    Adding New Columns with Vectorized Values

    We can also add a new column to the Dataframe based on values or computations:

    df = pd.DataFrame( {'a': [1, 2, 3, 4],
                        'b': [2.0, 3.0, 4.0, 5.0],
                        'c': ['a', 'b', 'c', 'd'] }, index=['r1', 'r2', 'r3', 'r4'] )
    
    
    df['d'] = 3.14           # new column, each field set to same value
    
    print(df)
    
                      #     a    b  c  d
                      # r1  1  2.0  a  3.14
                      # r2  2  3.0  b  3.14
                      # r3  3  4.0  c  3.14
                      # r4  4  5.0  d  3.14
    
    
    df['e'] = df['a'] + df['b']    # vectorized computation to new column
    
    print(df)
    
                      #     a    b  c     d  e
                      # r1  1  2.0  a  3.14  3.0
                      # r2  2  3.0  b  3.14  5.0
                      # r3  3  4.0  c  3.14  7.0
                      # r4  4  5.0  d  3.14  9.0

    Aggregate methods for DataFrame and Series

    Methods .sum(), .cumsum(), .count(), .min(), .max(), .mean(), .median(), et al. provide summary operations


    import numpy as np
    df = pd.DataFrame( {'a': [1, 2, 3, 4],
                        'b': [1.0, 1.5, np.nan, 2.5],
                        'c': ['a', 'b', 'b', 'a'] }, index=['r1', 'r2', 'r3', 'r4'] )
    
    print(df.sum())
    
         # a      10
         # b       5
         # c    abba
         # dtype: object
    
    
    print(df.cumsum())
    
         #      a    b     c
         # r1   1    1     a
         # r2   3  2.5    ab
         # r3   6  NaN   abb
         # r4  10    5  abba
    
    
    print(df.count())
    
         # a    4
         # b    3
         # c    4
         # dtype: int64

    Most of these methods work on a Series object as well:

    print(df['a'].median())
    
    2.5

    To see a list of attributes for any object, use dir() with a DataFrame or Series object. This is best done in jupyter notebook:

    dir(df['a'])     # attributes for Series

    The list of attributes is long, but this kind of exploration can provide some useful surprises.


    DataFrame groupby()

    A groupby operation performs the same type of operation as the database GROUP BY. Grouping rows of the table by the value in a particular column, you can perform aggregate sums, counts or custom aggregations.


    This simple hypothetical table shows client names, regions, revenue values and type of revenue.


    df = pd.DataFrame( { 'company': ['Alpha', 'ALPHA', 'ALPHA', 'BETA', 'Beta', 'Beta', 'Gamma', 'Gamma', 'Gamma'],
                         'region':  ['NE', 'NW', 'SW', 'NW', 'SW', 'NE', 'NE', 'SW', 'NW'],
                         'revenue': [10, 9, 2, 15, 8, 2, 16, 3, 9],
                         'revtype': ['retail', 'retail', 'wholesale', 'wholesale', 'wholesale',
                                     'retail', 'wholesale', 'retail', 'retail'] } )
    
    print(df)
    
                      #   company region  revenue    revtype
                      # 0   Alpha     NE       10     retail
                      # 1   Alpha     NW        9     retail
                      # 2   Alpha     SW        2  wholesale
                      # 3    Beta     NW       15  wholesale
                      # 4    Beta     SW        8  wholesale
                      # 5    Beta     NE        2     retail
                      # 6   Gamma     NE       16  wholesale
                      # 7   Gamma     SW        3     retail
                      # 8   Gamma     NW        9     retail

    groupby() built-in Aggregate Functions

    The "summary functions" like sum() count()


    Aggregations are provided by the DataFrame groupby() method, which returns a special groupby object. If we'd like to see revenue aggregated by region, we can simply select the column to aggregate and call an aggregation function on this object:


    # revenue sum by region
    rsbyr = df.groupby('region').sum()   # call sum() on the groupby object
    print(rsbyr)
    
                      #         revenue
                      # region
                      # NE           28
                      # NW           33
                      # SW           13
    
    
    # revenue average by region
    rabyr = df.groupby('region').mean()
    print(rabyr)
    
                      #           revenue
                      # region
                      # NE       9.333333
                      # NW      11.000000
                      # SW       4.333333

    The result is dataframe with the 'region' as the index and 'revenue' as the sole column. Note that although we didn't specify the revenue column, pandas noticed that the other columns were not numbers and therefore should not be included in a sum or mean. If we ask for a count, python counts each column (which will be the same for each). So if we'd like the analysis to be limited to one or more coluns, we can simply slice the dataframe first:


    # count of all columns by region
    print(df.groupby('region').count())
    
                      #         company  revenue  revtype
                      # region
                      # NE            3        3        3
                      # NW            3        3        3
                      # SW            3        3        3
    
    
    # count of companies by region
    dfcr = df[['company', 'region']]       # dataframe slice:  only 'company' and 'region'
    print(dfcr.groupby('region').count())
    
                      #         company
                      # region
                      # NE            3
                      # NW            3
                      # SW            3

    Multi-column aggregation To aggregate by values in two combined columns, simply pass a list of columns by which to aggregate -- the result is called a "multi-column aggregation":


    print(df.groupby(['region', 'revtype']).sum())
    
                      #                   revenue
                      # region revtype
                      # NE     retail          12
                      #        wholesale       16
                      # NW     retail          18
                      #        wholesale       15
                      # SW     retail           3
                      #        wholesale       10

    List of selected built-in groupby functions


                   count()
                   mean()
                   sum()
                   min()
                   max()
                   describe() (prints out several columns including sum, mean, min, max)

    DataFrame Sorting and Transposing

    Reorder and rotate a DataFrame


    import random
    rdf = pd.DataFrame({'a': [ random.randint(1,5) for i in range(5)],
                        'b': [ random.randint(1,5) for i in range(5)],
                        'c': [ random.randint(1,5) for i in range(5)]})
    print(rdf)
    
        #    a  b  c
        # 0  2  1  4
        # 1  5  3  3
        # 2  1  2  4
        # 3  5  2  4
        # 4  2  4  4
    
    # sorting by a column
    rdf = rdf.sort_values('a')
    print(rdf)
    
        #    a  b  c
        # 2  1  2  4
        # 0  2  1  4
        # 4  2  4  4
        # 1  5  3  3
        # 3  5  2  4
    
    # sorting by a row
    idf = rdf.sort_values(3, axis=1)
    print(idf)
    
        #    b  c  a
        # 0  1  4  2
        # 1  3  3  5
        # 2  2  4  1
        # 3  2  4  5
        # 4  4  4  2
    
    # sorting values by two columns (first by 'c', then by 'b')
    rdf = rdf.sort_values(['c', 'b'])
    print(rdf)
    
        #    a  b  c
        # 1  5  3  3
        # 0  2  1  4
        # 2  1  2  4
        # 3  5  2  4
        # 4  2  4  4
    
    # sorting by index
    rdf = rdf.sort_index()
    print(rdf)
    
        #    a  b  c
        # 0  2  1  4
        # 1  5  3  3
        # 2  1  2  4
        # 3  5  2  4
        # 4  2  4  4
    
    # sorting options:  ascending=False, axis=1
    

    Transposing


    Transposing simply means inverting the x and y axes -- in a sense, flipping the values diagonally:

    rdft = rdf.T
    print(rdft)
    
        #    0  1  2  3  4
        # a  2  5  1  5  2
        # b  1  3  2  2  4
        # c  4  3  4  4  4

    Working with Missing Data (NaN)

    "Not a Number" is numpy's None value.


    If pandas can't insert a value (because indexes are misaligned or for other reasons), it inserts a special value noted as NaN (not a number) in its place. This value belongs to the numpy module, accessed through np.nan


    import numpy as np
    
    df = pd.DataFrame({ 'c1': [6, 6, np.nan],
                        'c2': [np.nan, 1, 3],
                        'c3': [2, 2, 2]  })
    
    print(df)
                      #     c1   c2  c3
                      # 0  6.0  NaN   2
                      # 1  6.0  1.0   2
                      # 2  NaN  3.0   2

    Note that we are specifying the NaN value with np.nan, athough in most cases the value is generated by "holes" in mismatched data.


    We can fill missing data with fillna():

    df2 = df.fillna(0)
    print(df2)
                      #     c1   c2  c3
                      # 0  6.0  0.0   2
                      # 1  6.0  1.0   2
                      # 2  0.0  3.0   2

    Or we can choose to drop rows or columns that have any NaN values with dropna():

    df3 = df.dropna()
    
    print(df3)
                      #     c1   c2  c3
                      # 1  6.0  1.0   2
    
    # axis=1:  drop columns
    df4 = df.dropna(axis=1)
    
    print(df4)
    
                      #    c3
                      # 0   2
                      # 1   2
                      # 2   2

    Testing for NaN We may well be interested in whether a column or row has missing data. .isnull() provides a True/False mapping.


    print(df)
                      #     c1   c2  c3
                      # 0  6.0  NaN   2
                      # 1  6.0  1.0   2
                      # 2  NaN  3.0   2
    
    df['c1'].isnull().any()  # True
    df['c3'].isnull().any()  # False
    
    
    df['c1'].isnull().all()  # False



    Higher-Order Functions and Decorators

    Object References

    Everything is an Object; Objects Assigned by Reference


    object: a data value of a particular type variable: a name bound to an object


    When we create a new object and assign it to a name, we call it a variable. This simply means that the object can now be referred to by that name.

    a = 5                 # bind an int object to name a
    
    b = 'hello'           # bind a str object to name b

    Here are three classic examples demonstrating that objects are bound by reference and that assignment from one variable to another is simply binding a 2nd name to the same object (i.e. it simply points to the same object -- no copy is made).


    Aliasing (not copying) one object to another name:

    a = ['a', 'b', 'c']   # bind a list object to a (by reference)
    
    b = a                 # 'alias' the object to b as well -- 2 references
    
    b.append('d')
    
    print(a)               # ['a', 'b', 'c', 'd']

    a was not manipulated directly, but it changed. This underscores that a and b are pointing to the same object.


    Passing an object by reference to a function:

    def modifylist(this_list):   # The object bound to a
        this_list.append('d')    # Modify the object bound to a
    
    a = ['a', 'b', 'c']
    
    modifylist(a)                # Pass object bound to a by reference
    
    print(a)                       # ['a', 'b', 'c', 'd']

    The same dynamics at work: a is pointing at the same list object as this_list, so a change through one name is a change to the one object -- and the other name pointing to the same object will see the same changed object.


    Alias an object as a container item:

    a = ['a', 'b', 'c']          # list bound to a
    
    xx = [1, 2, a]               # 3rd item is reference to list bound to a
    
    xx[2].append('d')            # appending to list object referred to in list item
    
    print(a)                       # ['a', 'b', 'c', 'd']

    The same dynamic applied to a container item: the only difference here is that a name and a container item are pointing to the same object.


    Function References: Renaming Functions

    Functions are variables (objects bound to names) like any other; thus they can be renamed.

    def doubleit(val):
        val2 = val * 2
        return val2
    
    print(doubleit)         # <function doubleit at 0x105554d90>
    
    newfunc = doubleit
    
    xx = newfunc(5)        # 10

    The output <function doubleit at 0x105554d90> is Python's way of visualizing a function (i.e. showing its printed value). The hex code refers to the function object's location in memory (this can be used for debugging to identify the individual function).


    Function References: Functions in Containers

    Functions are "first-class" objects, and as such can be stored in containers, or passed to other functions.


    Functions can be stored in containers the same way any other object (int, float, list, dict) can be:

    def doubleit(val):
        val2 = val * 2
        return val2
    
    def tripleit(val):
        return val * 3
    
    funclist = [ doubleit, tripleit ]
    
    print(funclist[0](10))   # 20
    print(funclist[1](10))   # 30

    Higher-Order Built-in Functions: map(), grep() and sorted()

    These functions allow us to pass a function as an argument to enable its behavior.


    We can pass a function name (or a lambda, which is also a reference to a function) to any of these built-in functions. The function controls the built-in function's behavior.


    One example is the function passed to sorted():

    def by_last(name):
        first, last = name.split()
        return last
    
    names = ['Joe Wilson', 'Zeb Applebee', 'Abe Zimmer']
    
    snames = sorted(names, key=by_last)   # ['Zeb Applebee, 'Joe Wilson', 'Abe Zimmer']

    In this example, we are passing the function by_last to sorted(). sorted() calls the function once with each item in the list as argument, and sorts that item by the return value from the function.


    In the case of map() (apply a function to each item and return the result) and filter() (apply a function to each item and include it in the returned list if the function returns True), the function is required:

    def make_intval(val):
        return int(val)
    
    def over9(val):
        if int(val) > 99:
            return True
        else:
            return False
    
    x = ['1', '100', '11', '10', '110']
    
    # apply make_intval() to each item and sort by the return value
    sx = sorted(x, key=make_intval)     # ['1', '10', '11', '100', '110']
    
    # apply make_intval() to each item and store the return value in the resulting list
    mx = map(make_intval, x)
    print(list(mx))                      # [ 1, 100, 11, 10, 110 ]
    
    # apply over9() to each item and if the return value is True, store in resulting list
    fx = filter(over9, x)
    print(list(fx))                      # [ '100', '110' ]

    Higher-Order Functions: Functions as Arguments and as Return Values

    A "higher order" function is one that can be passed to another function, or returned from another function.


    The charge() function takes a function as an argument, and uses it to calculate its return value:

    def charge(func, val):
        newval = func(val) + val
        return '${}'.format(newval)
    
    def tax_ny(val):
        val2 = val * 0.085
        return val2
    
    def tax_ca(val):
        val2 = val * 0.065
        return val2
    
    nyval = charge(tax_ny, 10)          # pass tax_ny to charge():  $10.85
    caval = charge(tax_ca, 10)          # pass tax_ca to charge():  $10.65

    Any function that takes a function as an argument or returns one as a return value is called a "higher order" function.


    Using a function to build another function

    A function can be kind of a "factory" of functions.


    This function returns a function as return value, after "seeding" it with a value:

    def makemul(mul):
        def times(startval):
            return mul * startval
        return times
    
    doubler = makemul(2)
    tripler = makemul(3)
    
    print(doubler(5))      # 10
    print(tripler(5))      # 15

    In the two examples above, the values 2 and 3 are made part of the returning function -- "seeded" as built-in values .


    Decorators

    A decorator accepts a function as an argument and returns a replacement function as a return value.


    Python decorators return a function to replace the function being decorated -- when the original function is called in the program, Python calls the replacement. A decorator can be added to any function through the use of the @ sign and the decorator name on the line above the function.


    Here's a simple example of a function that returns another function (from RealPython blog):

    def my_decorator(some_function):
    
        def wrapper():
            print("Something is happening before some_function() is called.")
    
            some_function()
    
            print("Something is happening after some_function() is called.")
        return wrapper
    
    
    def this_function():
        print("Wheee!")
    
    
    # now the same name points to a replacement function
    this_function = my_decorator(this_function)
    
    # calling the replacement function
    this_function()

    This is not a decorator yet, but it shows the concept and mechanics


    If we wished to use this as a Python decorator, we can simply use the @ decorator notation:

    def my_decorator(some_function):
    
        def wrapper():
            print("Something is happening before some_function() is called.")
    
            some_function()
    
            print("Something is happening after some_function() is called.")
        return wrapper
    
    @my_decorator
    def this_function():
        print('Wheee!')
    
    this_function()
                           # Something is happening before...
                           # Whee!
                           # Something is happening after...

    The benefit here is that, rather than requiring the user to explicitly pass a function to a processing function, we can simply decorate each function to be processed and it will behave as advertised.


    *args and **kwargs to capture all arguments

    To allow a decorated function to accept arguments, we must accept them and pass them to the decorated function.


    Here's a decorator function that adds integer 1 to the return value of a decorated function:

    def addone(oldfunc):
        def newfunc(*args, **kwargs):
            return oldfunc(*args, **kwargs) + 1
        return newfunc
    
    @addone
    def sumtwo(val1, val2):
        return val1 + val2
    
    y = sumtwo(5, 10)
    print(y)                  # 16

    Now look closely at def newfunc(*args, **kwargs): *args in a function definition means that all positional arguments passed to it will be collected into a tuple called args. **kwargs in a function definition means that all keyword arguments passed to it will be collected into a dictionary called kwargs. (The * and ** are not part of the variable names; they are notations that allow the arguments to be passed into the tuple and dict.) Then on the next line, look at return oldfunc(*args **kwargs) + 1 *args in a function call means that the tuple args will be passed as positional arguments (i.e., the reverse of what happened above) **kwargs in a function call means that the dict kwargs will be passed as keyword arguments (i.e., the reverse of what happened above) This means that if we wanted to decorate a function that takes different arguments, *args and **kwargs would faithfully pass on those arguments as well.


    Here's another example, adapted from the Jeff Knupp blog:

    def currency(f):                              # decorator function
        def wrapper(*args, **kwargs):
            return '$' + str(f(*args, **kwargs))
        return wrapper
    
    @currency
    def price_with_tax(price, tax_rate_percentage):
        """Return the price with *tax_rate_percentage* applied.
        *tax_rate_percentage* is the tax rate expressed as a float, like
        "7.0" for a 7% tax rate."""
    
        return price * (1 + (tax_rate_percentage * .01))
    
    print(price_with_tax(50, .10))           # $50.05
    

    In this example, *args and **kwargs represent "unlimited positional arguments" and "unlimited keyword arguments". This is done to allow the flexibility to decorate any function (as it will match any function argument signature).




    "Whole File" Parsing

    Goals for this Unit

    The Line, Word, Character Count


    Can we parse and count the lines, words and characters in a file? We will emulate the work of the Unix wc (word count) utility, which does this work. Here's how it works:

  • We call function open() with a filename, returning a file object.
  • We call read() on the file object, returning a string object containing the entire file text.
  • We call splitlines() on the string, returning a list of strings. len() will then tell us the number of lines.
  • We call split() on the same string, returning a list of strings. len() will then tell us the number of words.
  • We call len() on the string to count the number of characters.


    Tabular Data: Slicing and Dicing

    A file can be rendered as a single string or a list of strings. Strings can be split into fields.


    file (TextIOWrapper) object

    # read(): file text as a single strings
    fh = open('students.txt')  # file object allows reading
    text = fh.read()                          # read() method called on
                                              # file object returns a string
    
    fh.close()                                # close the file
    
    print(text)
    
        # single string, entire text
        # 'id,fname,lname,city,state,tel\njw234,Joe,Wilson,Smithtown,NJ,
        #  2015585894\nms15,Mary,Smith,Wilsontown,NY,5185853892\npk669,
        #  Pete,Krank,Darkling,VA,8044894893'
    
    
    # readlines():  file text as a list of strings
    fh = open('students.txt')
    file_lines = fh.readlines()
    
        # list of strings, each line an item in the list (note newlines)
        # [ 'id,fname,lname,city,state,tel\n',
        #   'jw234,Joe,Wilson,Smithtown,NJ,2015585894\n',
        #   'ms15,Mary,Smith,Wilsontown,NY,5185853892\n',
        #   'pk669,Pete,Krank,Darkling,VA,8044894893'   ]
    
    fh.close()                        # close the file
    
    print(file_lines)                  # list of strings,
                                      # each string a line

    string object

    # split():  separate a string into a list of strings
    
    file_text = 'There was a russling of dresses, and the standing
    congregation sat down.\nThe boy whose history this book relates
    did not enjoy the prayer, \nhe only endured it, if he even did
    that much.  He was restive all through it; he \nkept tally of
    the details of the prayer, unconshiously, for he was not...'
    
    elements = mystr.split()      # split entire file on whitespace (spaces or newlines)
    print elements
        # ['There', 'was', 'a', 'russling', 'of', 'dresses,', 'and', 'the', 'standing',
        #  'congregation', 'sat', 'down.', 'The', 'boy', 'whose', 'history', 'this', 'book',
        #  'relates', 'did', 'not', 'enjoy', 'the', 'prayer,', 'he', 'only', 'endured', 'it,',
        #  'if', 'he', 'even', 'did', 'that', 'much.', 'He', 'was', 'restive', 'all',
        #  'through', 'it;', 'he', 'kept', 'tally', 'of', 'the', 'details', 'of', 'the',
        #  'prayer,', 'unconshiously,', 'for', 'he', 'was', 'not...']
    
    
    
    # splitlines():  separate a multiline string
    fh = open('students.txt')                # open the file, return
                                             # a file object
    text = fh.read()                         # read the entire file into
                                             # a string
                                             # (of course this includes newlines)
    
    fh.close()
    
    lines = text.splitlines()                # returns a list of strings
                                             # (similar to fh.readlines(),
                                             # except without newlines)
    
    print(lines)
    
        # list of strings, each line an item in the list (note no newlines)
        # [ 'id,fname,lname,city,state,tel',
        #   'jw234,Joe,Wilson,Smithtown,NJ,2015585894',
        #   'ms15,Mary,Smith,Wilsontown,NY,5185853892',
        #   'pk669,Pete,Krank,Darkling,VA,8044894893'   ]

    list object

    # subscript a list of lines
    lines = [ 'id,fname,lname,city,state,tel',
              'jw234,Joe,Wilson,Smithtown,NJ,2015585894',
              'ms15,Mary,Smith,Wilsontown,NY,5185853892',
              'pk669,Pete,Krank,Darkling,VA,8044894893'   ]
    
    header = lines[0]           # 'id,fname,lname,city,state,tel'
    last_line = lines[-1]       # 'pk669,Pete,Krank,Darkling,VA,8044894893'
    
    
    # slice a list
    data = lines[1:]            # from first line to end of file
    
    print(data)
    
       # [ 'jw234,Joe,Wilson,Smithtown,NJ,2015585894',
       #   'ms15,Mary,Smith,Wilsontown,NY,5185853892',
       #   'pk669,Pete,Krank,Darkling,VA,8044894893'   ]
    
    
    # get the length of a list of lines (# of lines in a file)
    x = len(lines)               # 4
    

    Summary: File Object

    3 ways to read strings from a file.


    for: read (newline ('\n') marks the end of a line)

    fh = open('students.txt')                 # file object allows looping
                                              # through a series of strings
    for my_file_line in fh:                   # my_file_line is a string
        print(my_file_line)                   # prints each line of students.txt
    
    fh.close()                                # close the file

    read(): read entire file as a single string

    fh = open('students.txt')  # file object allows reading
    text = fh.read()                          # read() method called on file
                                              # object returns a string
    fh.close()                                # close the file
    
    print(text)

    The above prints:

    jw234,Joe,Wilson,Smithtown,NJ,2015585894
    ms15,Mary,Smith,Wilsontown,NY,5185853892
    pk669,Pete,Krank,Darkling,NJ,8044894893

    readlines(): read as a list of strings (each string a line)

    fh = open('students.txt')
    file_lines = fh.readlines()               # file.readlines() returns
                                              # a list of strings
    fh.close()                                # close the file
    print(file_lines)

    The above prints:

    ['jw234,Joe,Wilson,Smithtown,NJ,2015585894\n', 'ms15,Mary,Smith,Wilsontown,
    NY,5185853892\n', 'pk669,Pete,Krank,Darkling,NJ,8044894893\n']

    Summary: String Object

    Strings: 4 ways to manipulate strings from a file.


    split() a string into a list of strings

    mystr = 'jw234,Joe,Wilson,Smithtown,NJ,2015585894'
    elements = mystr.split(',')
    print(elements)                           # ['jw234', 'Joe', 'Wilson',
                                              #  'Smithtown', 'NJ', '2015585894']

    (included for completeness): join() a list of strings into a string

    mylist = ['jw234', 'Joe', 'Wilson', 'Smithtown', 'NJ', '2015585894']
    
    line = ','.join(mylist)          # 'jw234,Joe,Wilson,Smithtown,NJ,2015585894'

    slice a string

    mystr = '2014-03-13 15:33:00'
    year =  mystr[0:4]               # '2014'
    month = mystr[5:7]               # '03'
    day =   mystr[8:10]              # '13'

    rstrip() a string

    xx = 'this is a line with a newline at the end\n'
    
    yy = xx.rstrip()                 # return a new string without the newline
    
    print(yy)                        # 'this is a line with a newline at the end'

    splitlines() a multiline string

    fh = open('students.txt') # open the file, return
                                             # a file object
    text = fh.read()                         # read the entire file into a string
                                             # (of course this includes newlines)
    
    lines = text.splitlines()                # returns a list of strings
                                             # (similar to fh.readlines(),
                                             # except without newlines)

    Reading a file: options

    (Note that the remaining slides repeat some of the same material, but from a more practical perspective.)


    for loop: loop line-by-line The for loop repeats execution of its block until the file is completely read. Note that the for block is very similar to the while. The difference is that while relies on a test to continue executing, but for continues until it reaches the end of the file.


    fh = open('students.txt')              # file object allows
                                           # looping through a
                                           # series of strings
    
    for xx in fh:                          # xx is a string
        print(xx)                          # prints each line of students.txt
    fh.close()                             # close the file

    "xx" is called a control variable, and it is automatically reassigned each line in the file as a string. break and continue work with for as well as while loops. readlines(): work with the file as a list of string lines


    To capture the entire file into a list of lines, use the file readlines() method:

    fh = open('students.txt')
    
    lines = fh.readlines()
    
    for line in lines:
        line = line.rstrip()
        print(line)
    
    print(lines[0])                        # the first line from the file
    
    print(len(lines))                      # the number of lines in the file

    We can then loop through the list, or perform other operations (select a single line or slice, get the number of lines with len() of the list, etc.) read() with splitlines(): an easy way to drop the newlines


    A handy trick is to read() the file into a string, then call splitlines() on the string to split on newlines.

    fh = open('students.txt')
    
    text = fh.read()
    lines = text.splitlines()
    
    for line in lines:
        print(line)

    This has the effect of delivering the entire file as a list of lines, but with the newlines removed (because the string was split on them with splitlines()).




    pandas: DataFrame, Series, Index; column dtypes

    pandas Object Types: DataFrame, Series, Index

    DataFrame: rows and columns; Series: a single column or single row; Index: column or row labels.


    DataFrame: * is the core pandas structure -- a 2-dimensional array or list of lists * is like an Excel spreadsheet - rows, columns, and row and column labels * is also like a "dict of dicts" in that it holds column- and row-indexed Series * offers "vectorized" operations (sum rows or columns, modify values across rows, etc.) * offers database-like and excel-like manipulations (merge, groupby, pivot table etc.)


    import pandas as pd
    df = pd.DataFrame({'a': [1, 2, 3],
                       'b': [10, 20, 30],
                       'c': [100, 200, 300]},
                       index=['r1', 'r2', 'r3'])
    
    print(df)
                #      a   b    c
                # r1   1  10  100
                # r2   2  20  200
                # r3   3  30  300
    
    print(df['c']['r2'])    # 200

    Series * a "dictionary-like list" -- ordered values by associating them with an index * has a dtype attribute that holds its objects' common type


    # read a column as a Series (use DataFrame subscript)
    bcol = df['b']
    print(bcol)
                # r1    10
                # r2    20
                # r3    30
                # Name: b, dtype: int64
    
    
    # read a row as a Series (use subscript of df.loc[])
    oneidx = df.loc['r2']
    print(oneidx)
                # a      2
                # b     20
                # c    200
                # Name: r2, dtype: int64

    Index * an object that provides indexing for both the Series (its item index) and the DataFrame (its column or row index).


    columns = df.columns   # Index(['a',  'b',  'c'],  dtype='object')
    idx = df.index         # Index(['r1', 'r2', 'r3'], dtype='object')
    

    The DataFrame: Initializing and Subscripting

    A dataframe can be indexed like a list and subscripted like a dictionary.


    import pandas as pd
    import numpy as np
    
    # initialize a new, empty DataFrame
    df = pd.DataFrame()
    
    # initialize a DataFrame with sample data
    df = pd.DataFrame( {'a': [1, 2, 3, 4],
                        'b': [1.0, 1.5, 2.0, 2.5],
                        'c': ['a', 'b', 'c', 'd'] }, index=['r1', 'r2', 'r3', 'r4'] )
    
    print(df)
    
    # a DataFrame, printed
    #     a    b  c
    # r1  1  1.0  a
    # r2  2  1.5  b
    # r3  3  2.0  c
    # f4  4  2.5  d
    
    

    DataFrame subscript: column Series


    s = df['a']
    
    print(s)        # r1    1
                    # r2    2
                    # r3    3
                    # r4    4
                    # Name: a, dtype: int64
    
    print(type(s))   # Series

    The Series: Initializing and Subscripting

    A Series is pandas object representing a column or row in a DataFrame.


    Every DataFrame column or row is a Series:

    df = pd.DataFrame( {'a': [1, 2],
                        'b': [8, 9] },
                        index=['r1', 'r2'] )
    
    print(df)
                               #     a  b
                               # r1  1  8
                               # r2  2  9
    
    # DataFrame string subscript accesses a column
    print(df['a'])             # 0    1
                               # 1    2
                               # Name: a, dtype: int64
    
    print(type(df['a']))       # <class 'pandas.core.series.Series'>
    
    
    # DataFrame .loc[] indexer accesses the rows
    print(df.loc['r1'])        # a    1
                               # b    8
                               # Name: r1, dtype: int64
    
    print(type(df.loc['r1']))  # <class 'pandas.core.series.Series'>

    A Series can be also be initialized on its own:

    s1 = pd.Series([1, 2, 3, 4])
    s2 = pd.Series([5, 6, 7, 8])

    We can combine Series into DataFrames:

    df = pd.DataFrame([s1, s2])       # add Series as rows
    
    df = pd.concat([s1, s2], axis=1)  # add Series as columns

    The Index: DataFrame Column or Index Labels

    An Index object is used to specify a DataFrame's columns or index, or a Series' index.


    Columns and Indices


    A DataFrame makes use of two Index objects: one to represent the columns, and one to represent the rows.

    df = pd.DataFrame( {'a': [1, 2, 3, 4],
                        'b': [1.0, 1.5, 2.0, 2.5],
                        'c': ['a', 'b', 'c', 'd'],
                        'd': [100, 200, 300, 400] },
                        index=['r1', 'r2', 'r3', 'r4'] )
    print(df)
        #     a    b  c    d
        # r1  1  1.0  a  100
        # r2  2  1.5  b  200
        # r3  3  2.0  c  300
        # r4  4  2.5  d  400

    .rename() method: columns or index labels can be reset using this DataFrame method.

    df = df.rename(columns={'a': 'A', 'b': 'B', 'c': 'C', 'd': 'D'},
                   index={'r1': 'R1', 'r2': 'R2', 'r3': 'R3', 'r4': 'R4'})
    print(df)
        #     A    B  C    D
        # R1  1  1.0  a  100
        # R2  2  1.5  b  200
        # R3  3  2.0  c  300
        # R4  4  2.5  d  400

    .columns, .index: the columns or index can also be set directly using the DataFrame's attributes (this would have the same effect as above):

    df.columns = ['A', 'B', 'C', 'D']
    df.index = ['r1', 'r2', 'r3', 'r4']

    .set_index(): set any column to the index

    df2 = df.set_index('A')
    print(df2)
        #      B  C    D
        # A
        # 1  1.0  a  100
        # 2  1.5  b  200
        # 3  2.0  c  300
        # 4  2.5  d  400

    .reset_index(): we can reset the index to integers starting from 0; by default this converts the previous into a new column:

    df3 = df.reset_index()
    print(df3)
        #   index  A    B  C    D
        # 0    R1  1  1.0  a  100
        # 1    R2  2  1.5  b  200
        # 2    R3  3  2.0  c  300
        # 3    R4  4  2.5  d  400

    or to drop the index when resetting, include drop=True

    df4 = df.reset_index(drop=True)
    print(df4)
        #    A    B  C    D
        # 0  1  1.0  a  100
        # 1  2  1.5  b  200
        # 2  3  2.0  c  300
        # 3  4  2.5  d  400

    .reindex(): we can change the order of the indices and thus the rows:

    df5 = df.reindex(reversed(df.index))
    
    df5 = df5.reindex(columns=reversed(df.columns))
    
    print(df5)
    
                          #  state  A    B  C    D
                          #  year
                          #  R1     1  1.0  a  100
                          #  R2     2  1.5  b  200
                          #  R3     3  2.0  c  300
                          #  R4     4  2.5  d  400

    we can set names for index and column indices:

    df.index.name = 'year'
    df.columns.name = 'state'

    There are a number of "exotic" Index object types: Index (standard, default and most common Index type) RangeIndex (index built from an integer range) Int64Index, UInt64Index, Float64Index (index values of specific types) DatetimeIndex, TimedeltaIndex, PeriodIndex, IntervalIndex (datetime-related indices) CategoricalIndex (index related to the Categorical type)


    DataFrame: Initializing from Data Source

    DataFrame can be read from CSV, JSON, Excel and XML formats.


    Note: df is a common variable name for pandas DataFrame objects; you will see this name used frequently in these examples.


    CSV

    # read from file
    df = pd.read_csv('quarterly_revenue_2017Q4.csv')
    
    
    # write to file
    wfh = open('output.csv', 'w')
    df.to_csv(wfh, na_rep='NULL')
    
    
    # reading from Fama-French file (the abbreviated file, no header)
    # sep= indicates the delimiter on which to split() the fields
    # names= indicates the column heads
    df = pd.read_csv('FF_abbreviated.txt', sep='\s+',
                                           names=['date', 'MktRF', 'SMB', 'HML', 'RF'])
    
    
    # reading from Fama-French non-abbreviated (the main file including headers and footers)
    # skiprows=5:  start reading 5 rows down
    df = pd.read_csv('F-F_Research_Data_Factors_daily.txt', skiprows=5, sep='\s+',
                                                            names=['date', 'MktRF', 'SMB', 'HML', 'RF'])
    
    df.to_csv('newfile.csv')

    Excel

    # reading from excel file to DataFrame
    df = pd.read_excel('revenue.xlsx', sheet_name='Sheet1')
    
    # optional:  produce a 'reader' object used to obtain sheet names, etc.
    xls_file = pd.ExcelFile('data.xls')    # produce a file 'reader' object
    df = xls_file.parse('Sheet1')          # parse a selected sheet
    
    
    # write to excel
    df.to_excel('data2.xls', sheet_name='Sheet1')

    JSON

    # sample df for demo purposes
    df = pd.DataFrame( {'a': [1, 2, 3, 4],
                        'b': [1.0, 1.5, 2.0, 2.5],
                        'c': ['a', 'b', 'c', 'd'] }, index=['r1', 'r2', 'r3', 'r4'] )
    
    
    
    # write dataframe to JSON
    pd.json.dump(df, open('df.json', 'w'))
    
    mydict = pd.json.load(open('df.json'))
    new_df = pd.DataFrame(mydict)

    Relational Database

    import sqlite3                        # file-based database format
    conn = sqlite3.connect('example.db')  # a db connection object
    
    df = pd.read_sql('SELECT this FROM that', conn)

    The above can be used with any database connection (MySQL, Oracle, etc.)


    From Clipboard: this option is excellent for cutting and pasting data from websites

    df = pd.read_clipboard(skiprows=5, sep='\s+',
                           names=['date', 'MktRF', 'SMB', 'HML', 'RF'])

    Dataframe and Series dtypes

    pandas deduces a column type based on values and applies it to the column automatically.


    Pandas is built on top of numpy, a numeric processing module, compiled in C for efficiency. Unlike core Python containers (but similar to a database table), numpy cares about object type. Wherever possible, numpy will assign a type to a column of values and attempt to maintain the type's integrity. This is done for the same reason it is done with database tables: speed and space efficiency. In the below DataFrame, numpy/pandas "sniffs out" the type of a column Series. It will set the type most appropriate to the values.


    import pandas as pd
    
    df = pd.DataFrame( {'a': [1, 2, 3, 4],
                        'b': [1.0, 1.5, 2.0, 2.5],
                        'c': ['a', 'b', 'c', 'd'],
                        'd': ['2016-11-01', '2016-12-01', '2017-01-01', '2018-02-01'] },
                        index=['r1', 'r2', 'r3', 'r4'] )
    
    print(df)
                      #     a    b  c          d
                      # r1  1  1.0  a 2016-11-01
                      # r2  2  1.5  b 2016-12-01
                      # r3  3  2.0  c 2017-01-01
                      # r4  4  2.5  d 2017-02-01
    
    print(df.dtypes)
    
                      # a      int64       # note special pandas types int64 and float64
                      # b    float64
                      # c     object       # 'object' is general-purpose type,
                      # d     object       #     covers strings or mixed-type columns
                      # dtype: object

    You can use the regular integer index to set element values in an existing Series. However, the new element value must be the same type as that defined in the Series; if not, pandas may refuse, or it may upconvert or cast the Series column to a more general type (usually object, because numpy seems to care only about numeric and datetime types).


    print(df.b.dtype)       # float64
    df.loc['b'] = 'hello'
    print(df.b.dtype)       # object

    Note that we never told pandas to store these values as floats. But since they are all floats, pandas decided to set the type.


    We can change a dtype for a Series ourselves with .astype():

    df.a = df.a.astype('object')     # or df['a'] = df['a'].astype('object')
    
             #  df.loc[0, 'a'] = 'hello'

    The numpy dtypes you are most likely to see are:

    int64
    float64
    datetime64
    object

    Checking the memory usage of a DataFrame


    .info() provides approximate memory size of a DataFrame

    df.info()   # on the original example at the top
    
             #  
             #  Index: 4 entries, r1 to r4
             #  Data columns (total 4 columns):
             #  a    4 non-null int64
             #  b    4 non-null float64
             #  c    4 non-null object
             #  d    4 non-null object
             #  dtypes: float64(1), int64(1), object(2)
             #  memory usage: 160.0+ bytes

    '+' means "probably larger" -- info() only sizes numeric types, not 'object'


    With memory_usage='deep', size includes type'object'

    df.info(memory_usage='deep')
    
             # memory usage: 832 bytes

    Selecting a Series from a DataFrame

    Use a subscript (or attribute) to access columns by label; use the .loc[] or .iloc[] attributes to access rows by label or integer index.


    a DataFrame:

    df = pd.DataFrame( {'a': [1, 2, 3, 4],
                        'b': [1.0, 1.5, 2.0, 2.5],
                        'c': ['a', 'b', 'c', 'd'],
                        'd': [100, 200, 300, 400] },
                        index=['r1', 'r2', 'r3', 'r4'] )

    access column as Series:

    cola = df['a']       # Series with [1, 2, 3, 4] and index ['r1', 'r2', 'r3', 'r4']
    
    cola = df.a          # same -- can often use attribute labels for column name
    
    print(cola)
    
                # r1    1
                # r2    2
                # r3    3
                # r4    4
                # Name: a, dtype: int64

    access row as Series using index label 'r2':

    row2 = df.loc['r2']  # Series [2, 1.5, 'b', 200] and index ['a', 'b', 'c', 'd']

    access row as Series using integer index:

    row2 = df.iloc[1]    # Series [2, 1.5, 'b', 200] and index ['a', 'b', 'c', 'd'] (same as above)
    
    print(row2)
    
                # a      2
                # b    1.5
                # c      b
                # d    200
                # Name: r2, dtype: object

    (Note that the .ix DataFrame indexer is a legacy feature and is deprecated.)


    Manipulating the Index

    An Index can be set with a column or other sequence.


    Sometimes a pd.read_excel() includes index labels in the first column. We can easily set the index with .set_index():


    print(df)
        #     0  a    b  c           d
        # 0  r1  1  1.0  a  2016-11-01
        # 1  r2  2  1.5  b  2016-12-01
        # 2  r3  3  2.0  c  2017-01-01
        # 3  r4  4  2.5  d  2018-02-01
    
    df = df.set_index(df[0])
    df = df[['a', 'b', 'c', 'd']]
    print(df)
        #     a    b  c           d
        # 0
        # r1  1  1.0  a  2016-11-01
        # r2  2  1.5  b  2016-12-01
        # r3  3  2.0  c  2017-01-01
        # r4  4  2.5  d  2018-02-01

    We can reset the index with .reset_index(), although this makes the index into a new column.


    df2 = df.reset_index()
    print(df2)
        #     0  a    b  c           d
        #
        # 0  r1  1  1.0  a  2016-11-01
        # 1  r2  2  1.5  b  2016-12-01
        # 2  r3  3  2.0  c  2017-01-01
        # 3  r4  4  2.5  d  2018-02-01

    As mentioned we can move a column into the index with .set_index()

    df2 = df2.set_index(0)
    print(df2)
        #     a    b  c           d
        # 0
        # r1  1  1.0  a  2016-11-01
        # r2  2  1.5  b  2016-12-01
        # r3  3  2.0  c  2017-01-01
        # r4  4  2.5  d  2018-02-01

    To reset the index while dropping the original index, we can use drop=True:


    df3 = df2.reset_index(drop=True)
    print(df3)
        #    a    b  c           d
        #
        # 0  1  1.0  a  2016-11-01
        # 1  2  1.5  b  2016-12-01
        # 2  3  2.0  c  2017-01-01
        # 3  4  2.5  d  2018-02-01

    We can also sort the DataFrame by index using .sort_index():


    df4 = pd.DataFrame({'a': [1, 2, 3], 'b': [4, 5, 6]},
                       index=['Cello', 'Alpha', 'Bow'])
    df5 = df4.sort_index()
    print(df5)
        #        a  b
        # Alpha  2  5
        # Bow    3  6
        # Cello  1  4

    The default is to sort by the row index; axis=1 allows us to sort by columns. ascending=False reverses the sort:


    df6 = df5.sort_index(axis=1, ascending=False)
    print(df6)
        #        b  a
        # Alpha  5  2
        # Bow    6  3
        # Cello  4  1

    Note that .sort_values() offers the same options sorting a specified column or row.


    Most Operations Produce a New DataFrame

    Some DataFrame operations provide the inplace=True option


    Keep in mind that many operations produce a new DataFrame copy. This means that if you are working with a large dataset, you can avoid allocating additional memory with this option.


    import pandas as pd
    
    df = pd.DataFrame({ 'a': [1, 2, 3, 4],
                        'b': [1.0, 1.5, 2.0, 2.5],
                        'c': ['a', 'b', 'c', 'd']   })
    
    print(df)
        #     a    b  c
        # r1  1  1.0  a
        # r2  2  1.5  b
        # r3  3  2.0  c
        # r4  4  2.5  d
    
    df2 = df.set_index('a')
    
    print(df2)             # new dataframe
    
        #      b  c
        # a
        # 1  1.0  a
        # 2  1.5  b
        # 3  2.0  c
        # 4  2.5  d
    
    print(df)              # unchanged
        #     a    b  c
        # r1  1  1.0  a
        # r2  2  1.5  b
        # r3  3  2.0  c
        # r4  4  2.5  d
    
    df.set_index('a', inplace=True)
    
    print(df)
    
        #      b  c
        # a
        # 1  1.0  a
        # 2  1.5  b
        # 3  2.0  c
        # 4  2.5  d

    DataFrame and Series as list, set, etc.

    DataFrames behave as you might expect when converted to any Python container


    df = pd.DataFrame( {'a': [1, 2, 3, 4],
                        'b': [1.0, 1.5, 2.0, 2.5],
                        'c': ['a', 'b', 'b', 'a'] }, index=['r1', 'r2', 'r3', 'r4'] )
    
    
    print(len(df))             # 4
    
    print(len(df.columns))     # 3
    
    print(max(df['a']))        # 4
    
    print(list(df['a']))       # [1, 2, 3, 4]     (column for 'a')
    
    print(list(df.loc['r2']))  # [2, 1.5, 'b']   (row for 'r2')
    
    print(set(df['c']))        # {'b', 'a'}       (a set of unique values)

    DataFrame .values -- convert to a list of numpy arrays


    An numpy array is a list-like object. simple list comprehension could convert these to a list of lists:

    print((df.values))
    
        # array([[1, 1.0, 'a'],
        #        [2, 1.5, 'b'],
        #        [3, 2.0, 'b'],
        #        [4, 2.5, 'a']], dtype=object)
    
    
    lol = list( [ list(item) for item in df.values ])
    
    print(lol)
    
                              # [ [1, 1.0, 'a'],
                              #   [2, 1.5, 'b'],
                              #   [3, 2.0, 'b'],
                              #   [4, 2.5, 'a'] ]

    looping - loops through columns


    for colname in df:
        print('{}:  {}'.format(colname, type(df[colname])))
    
                              # a:  <pandas.core.series.Series>
                              # b:  <pandas.core.series.Series>
                              # c:  <pandas.core.series.Series>
    
    
    # looping with iterrows -- loops through rows
    for row_index, row_series in df.iterrows():
        print('{}:  {}'.format(row_index, type(row_series)))
    
                              # r1:  <pandas.core.series.Series>
                              # r2:  <pandas.core.series.Series>
                              # r3:  <pandas.core.series.Series>
                              # r4:  <pandas.core.series.Series>

    Although keep in mind that we generally prefer vectorized operations across columns or rows to looping.




    Python Data Model

    Python's Data Model: Overview

    The Data Model specifies how objects, attributes, methods, etc. function and interact in the processing of data.


    The Python Language Reference provides a clear introduction to Python's lexical analyzer, data model, execution model, and various statement types. This session covers the basics of Python's data model. Mastery of these concepts allows you to create objects that behave (i.e., using the same interface -- operators, looping, subscripting, etc.) like standard Python objects, as well as in becoming conversant on StackOverflow and other discussion sites.


    Special / "Private" / "Magic" Attributes

    All objects contain "private" attributes that may be methods that are indirectly called, or internal "meta" information for the object.


    The __dict__ attribute shows any attributes stored in the object.

    >>> list.__dict__.keys()
    ['__getslice__', '__getattribute__', 'pop', 'remove', '__rmul__', '__lt__', '__sizeof__',
     '__init__', 'count', 'index', '__delslice__', '__new__', '__contains__', 'append',
     '__doc__', '__len__', '__mul__', 'sort', '__ne__', '__getitem__', 'insert',
     '__setitem__', '__add__', '__gt__', '__eq__', 'reverse', 'extend', '__delitem__',
     '__reversed__', '__imul__', '__setslice__', '__iter__', '__iadd__', '__le__', '__repr__',
     '__hash__', '__ge__']

    The dir() function will show the object's available attributes, including those available through inheritance.

    >>> dir(list)
    ['__add__', '__class__', '__contains__', '__delattr__', '__delitem__', '__delslice__',
     '__doc__', '__eq__', '__format__', '__ge__', '__getattribute__', '__getitem__',
     '__getslice__', '__gt__', '__hash__', '__iadd__', '__imul__', '__init__', '__iter__',
     '__le__', '__len__', '__lt__', '__mul__', '__ne__', '__new__', '__reduce__',
     '__reduce_ex__', '__repr__', '__reversed__', '__rmul__', '__setattr__', '__setitem__',
     '__setslice__', '__sizeof__', '__str__', '__subclasshook__', 'append', 'count',
     'extend', 'index', 'insert', 'pop', 'remove', 'reverse', 'sort']

    In this case, dir(list) includes attributes not found in list.__dict__. What class(es) does list inherit from? We can use __bases__ to see:

    >>> list.__bases__
    (object,)

    This is a tuple of classes from which list inherits - in this case, just the super object object.


    >>> object.__dict__.keys()
    ['__setattr__', '__reduce_ex__', '__new__', '__reduce__', '__str__', '__format__',
     '__getattribute__', '__class__', '__delattr__', '__subclasshook__', '__repr__',
     '__hash__', '__sizeof__', '__doc__', '__init__']

    Of course this means that any object that inherits from object will have the above attributes. Most if not all built-in objects inherit from object. (In Python 3, all classes inherit from object.) (Note that of course the term "private" in this context does not refer to unreachable data as would be used in C++ or Java.)


    Object Inspection And Modification Built-in Functions

    Object Inspection

    isinstance()Checks to see if this object is an instance of a class (or parent class)
    issubclass()Checks to see if this class is a subclass of another
    callable()Checks to see if this object is callable
    hasattr()Checks to see if this object has an attribute of this name

    Object Attribute Modification

    setattr()sets an attribute in an object (using a string name)
    getattr()retrieves an attribute from an object (using a string name)
    delattr()deletes an attribute from an object (using a string name)


    Special Attributes: "operator overloading"

    Some special attributes are methods, usually called implictly as the result of function calls, the use of operators, subscripting or slicing, etc.


    We can replace any operator and many functions with the corresponding "magic" methods to achieve the same result:

    var = 'hello'
    var2 = 'world'
    
    print(var + var2)         # helloworld
    print(var.__add__(var2))  # helloworld
    
    print(len(var))           # 5
    print(var.__len__())      # 5
    
    if 'll' in var:
        print('yes')
    
    if var.__contains__('ll'):
        print('yes')

    Here is an example of a new class, Number, that reproduces the behavior of a number in that you can add, subtract, multiply, divide them with other numbers.

    class Number(object):
      def __init__(self, start):
        self.data = start
      def __sub__(self, other):
        return Number(self.data - other)
      def __add__(self, other):
        return Number(self.data + other)
      def __mul__(self, other):
        return Number(self.data * other)
      def __div__(self, other):
        return Number(self.data / float(other))
      def __repr__(self):
        print("Number value: ", end=' ')
        return str(self.data)
    
    X = Number(5)
    X = X - 2
    print(X)               # Number value: 3

    Of course this means that existing built-in objects make use of these methods -- you can find them listed from the object's dir() listing.


    Special Attributes: Reimplementing __repr__ and __str__

    __str__ is invoked when we print an object or convert it with str(); __repr__ is used when __str__ is not available, or when we view an object at the Python interpreter prompt.


    class Number(object):
      def __init__(self, start):
        self.data = start
      def __str__(self):
        return str(self.data)
      def __repr__(self):
        return 'Number(%s)' % self.data
    
    X = Number(5)
    print(X)          # 5  (uses __str__ -- without repr or str, would be <__main__.Y object at 0x105d61190>

    __str__ is intended to display a human-readable version of the object; __repr__ is supposed to show a more "machine-faithful" representation.


    Special attributes available in class design

    Here is a short listing of attributes available in many of our standard objects.


    You view see many of these methods as part of the attribute dictionary through dir(). There is also a more exhaustive exhaustive list with explanations provided by Rafe Kettler.

    object construction and destruction:

    __init__object constructor
    __del__del x (invoked when reference count goes to 0)
    __new__special 'metaclass' constructor

    object rendering:

    __repr__"under the hood" representation of object (in Python interpreter)
    __str__string representation (i.e., when printed or with str()

    object comparisons:

    __lt__<
    __le__<=
    __eq__==
    __ne__!=
    __gt__>
    __ge__>=
    __nonzero__(bool(), i.e. when used in a boolean test)

    calling object as a function:

    __call__when object is "called" (i.e., with ())

    container operations:

    __len__handles len() function
    __getitem__subscript access (i.e. mylist[0] or mydict['mykey'])
    __missing__handles missing keys
    __setitem__handles dict[key] = value
    __delitem__handles del dict[key]
    __iter__handles looping
    __reversed__handles reverse() function
    __contains__handles 'in' operator
    __getslice__handles slice access
    __setslice__handles slice assignment
    __delslice__handles slice deletion

    attribute access (discussed in upcoming session):

    __getattr__object.attr read: attribute may not exist
    __getattribute__object.attr read: attribute that already exists
    __setattr__object.attr write
    __delattr__object.attr deletion (i.e., del this.that)

    'descriptor' class methods (discussed in upcoming session)

    __get__when an attribute w/descriptor is read
    __set__when an attribute w/descriptor is written
    __delete__when an attribute w/descriptor is deleted with del

    numeric types:

    __add__addition with +
    __sub__subtraction with -
    __mul__multiplication with *
    __div__division with \/
    __floordiv__"floor division", i.e. with //
    __mod__modulus


    "Introspection" Special Attributes

    The name, module, file, arguments, documentation, and other "meta" information for an object can be found in special attributes.


    Below is a partial listing of special attributes; available attributes are discussed in more detail on the data model documentation page.

    user-defined functions

    __doc__doc string
    __name__this function's name
    __module__module in which this func is defined
    __defaults__default arguments
    __code__the "compiled function body" of bytecode of this function. Code objects can be inspected with the inspect module and "disassembled" with the dis module.
    __globals__global variables available from this function
    __dict__attributes set in this function object by the user

    user-defined methods

    im_classclass for this method
    __self__instance object
    __module__name of the module

    modules

    __dict__globals in this module
    __name__name of this module
    __doc__docstring
    __file__file this module is defined in

    classes

    __name__class name
    __module__module defined in
    __bases__classes this class inherits from
    __doc__docstring

    class instances (objects)

    im_classclass
    im_selfthis instance


    Variable Naming Conventions

    Underscores are used to designate variables as "private" or "special".


    lower-case separated by underscoresmy_nice_var"public", intended to be exposed to users of the module and/or class
    underscore before the name_my_private_var"non-public", *not* intended for importers to access (additionally, "from modulename import *" doesn't import these names)
    double-underscore before the name__dont_inherit"private"; its name is "mangled", available only as _classname__dont_inherit
    double-underscores before and after the name __magic_me__"magic" attribute or method, specific to Python's internal workings
    single underscore after the name file_used to avoid overwriting built-in names (such as the file() function)


    class GetSet(object):
    
        instance_count = 0
    
        __mangled_name = 'no privacy!'
    
        def __init__(self,value):
            self._attrval = value
            instance_count += 1
    
        def getvar(self):
            print('getting the "var" attribute')
            return self._attrval
    
        def setvar(self, value):
            print('setting the "var" attribute')
            self._attrval = value
    
    cc = GetSet(5)
    cc.var = 10
    print(cc.var)
    print(cc.instance_count)
    
    print(cc._attrval)                 # "private", but available:  10
    print(cc.__mangled_name)           # "private", apparently not available...
    print(cc._GetSet__mangled_name)    # ...and yet, accessible through "mangled" name
    
    cc.__newmagic__ = 10              # MAGICS ARE RESERVED BY PYTHON -- DON'T DO THIS

    Subclassing Builtin Objects

    Inheriting from a class (the base or parent class) makes all methods and attributes available to the inheriting class (the child class).


    class NewList(list):     # an empty class - does nothing but inherit from list
        pass
    
    x = NewList([1, 2, 3, 'a', 'b'])
    x.append('HEEYY')
    
    print(x[0])   # 1
    print(x[-1])  # 'HEEYY'

    Overriding Base Class Methods


    This class automatically returns a default value if a key can't be found -- it traps and works around the KeyError that would normally result.

    class DefaultDict(dict):
    
        def __init__(self, default=None):
            dict.__init__(self)
            self.default = default
    
        def __getitem__(self, key):
            try:
                return dict.__getitem__(self, key)
            except KeyError:
                return self.default
        def get(self, key, userdefault):
            if not userdefault:
                userdefault = self.default
            return dict.get(self, key, userdefault)
    
    xx = DefaultDict()
    
    xx['c'] = 5
    
    print(xx['c'])          # 5
    print(xx['a'])          # None

    Since the other dict methods related to dict operations (__setitem__, extend(), keys(), etc.) are present in the dict class, any calls to them also work because of inheritance.


    WARNING! Avoiding method recursion Note the bolded statements in DefaultDict above (as well as MyList below) -- are calling methods in the parent in order to avoid infinite recursion. If we were to call DefaultDict.get() from inside DefaultDict.__getitem__(), Python would again call DefaultDict.__getitem__() in response, and an infinite loop of calls would result. We call this infinite recursion


    The same is true for MyList.__getitem__() and MyList.__setitem__() below.

        # from DefaultDict.__getitem__()
        dict.get(self, key, userdefault)       # why not self.get(key, userdefault)?
    
        # from MyList.__getitem__()
        return list.__getitem__(self, index)   # why not self[index]?
    
        # from MyList.__setitem__()                   # (from example below)
        list.__setitem__(self, index, value)   # why not self[index] = value?

    Another example -- a custom list that indexes items starting at 0:

    class MyList(list):         # inherit from list
      def __getitem__(self, index):
        if index == 0:  raise IndexError
        if index > 0: index = index - 1
        return list.__getitem__(self, index)  # this method is called when we access
                                                     # a value with subscript (x[1], etc.)
      def __setitem__(self, index, value):
        if index == 0:  raise IndexError
        if index > 0: index = index - 1
        list.__setitem__(self, index, value)
    
    x = MyList(['a', 'b', 'c'])  # __init__() inherited from builtin list
    
    print(x)                      # __repr__() inherited from builtin list
    
    x.append('spam');            # append() inherited from builtin list
    
    print(x[1])                   # 'a' (MyList.__getitem__
                                 #      customizes list superclass method)
                                 # index should be 0 but it is 1!
    
    print(x[4])                   # 'spam' (index should be 3 but it is 4!)

    So MyList acts like a list in most respects, but its index starts at 0 instead of 1 (at least where subscripting is concerned -- other list methods would have to be overridden to complete this 1-indexing behavior).


    Iterator Protocol

    The protocol specifies methods to be implemented to make our objects iterable.


    "Iterable" simply means able to be looped over or otherwise treated as a sequence or collection. The for loop is the most obvious feature that iterates, however a great number of functions and other features perform iteration, including list comprehensions, max(), min(), sorted(), map(), filter(), etc., because each of these must consider every item in the collection.


    We can make our own objects iterable by implementing __iter__ and next, and by raising the StopIteration exception

    class Counter:
        def __init__(self, low, high):
            self.current = low
            self.high = high
    
        def __iter__(self):
            return self
    
        def __next__(self):                   # Python 3: def __next__(self)
            if self.current > self.high:
                raise StopIteration
            else:
                self.current += 1
                return self.current - 1
    
    
    for c in Counter(3, 8):
        print(c)

    Reading a file with 'with'

    A file is automatically closed upon exiting the 'with' block


    A 'best practice' is to open files using a 'with' block. When execution leaves the block, the file is automatically closed.

    with open('myfile.txt') as fh:
        for line in fh:
            print(line)
    
    ## at this point (outside the with block), filehandle fh has been closed.

    The conventional approach:

    fh = open('myfile.txt')
    for line in fh:
        print(line)
    
    fh.close()        # explicit close() of the file

    Although open files do not block other processes from opening the same file, they do leave a small additional temporary file on the filesystem (called a file descriptor); if many files are left open (especially for long-running processes) these small additional files could accumulate. Therefore, files should be closed as soon as possible.


    Implementing a 'with' context

    Any object definition can include a 'with' context; what the object does when leaving the block is determined in its design.


    A 'with' context is implemented using the magic methods __enter__() and __exit()__.

    class CustomWith:
        def __init__(self):
            """ when object is created """
            print('new object')
    
        def __enter__(self):
            """ when 'with' block begins (normally same time as __init__()) """
            print('entering "with"')
            return self
    
        def __exit__(self, exc_type, exc_value, exc_traceback):
            """ when 'with' block is left """
            print('leaving "with"')
    
            # if an exception should occur inside the with block:
            if exc_type:
                print('oops, an exception')
                raise exc_type(exc_value)     # raising same exception (optional)
    
    with CustomWith() as fh:
        print('ok')
    
    print('done')

    __enter__() is called automatically when Python enters the with block. This is usually also when the object is created with __init__() although it is possible to create __exit__() is called automatically when Python exits the with block. If an exception occurs inside the with block, Python passes the exception object, any value passed to the exception (usually a string error message) and a traceback string ("Traceback (most recent call last):...") In our above program, if an exception occurred (if type has a value) we are choosing to re-raise the same exception. Your program can choose any action at that point.


    Internal Types

    Some implicit objects can provide information on code execution.


    Traceback objects


    Traceback objects become available during an exception. Here's an example of inspection of the exception type using sys.exc_info()

    import sys, traceback
    try:
        some_code_i_wrote()
    except BaseException as e:
        error_type, error_string, error_tb =  sys.exc_info()
        if not error_type == SystemExit:
            print('error type:    {}'.format(error_type))
            print('error string:  {}'.format(error_string))
            print('traceback:     {}'.format(''.join(traceback.format_exception(error_type, e, error_tb))))

    Code objects In CPython (the most common distribution), a code object is a piece of compiled bytecode. It is possible to query this object / examine its attributes in order to learn about bytecode execution. A detailed exploration of code objects can be found here. Frame objects A frame object represents an execution frame (a new frame is entered each time a function is called). They can be found in traceback objects (which trace frames during execution).

    f_backprevious stack frame
    f_codecode object executed in this frame
    f_localslocal variable dictionary
    f_globalsglobal variable dictionary
    f_builtinsbuilt-in variable dictionary


    For example, this line placed within a function prints the function name, which can be useful for debugging -- here we're pulling a frame, grabbing the code object of that frame, and reading the attribute co_name to read it.

    import sys
    
    def myfunc():
        print('entering {}()'.format(sys._getframe().f_code.co_name ))

    Calling this function, the frame object's function name is printed:

    myfunc()         # entering myfunc()



    Generators and Recursion

    Generators and generator comprehensions

    A generator is like an iterator, but may generate an indefinite number of items.


    A generator is a special kind of object that returns a succession of items, one at a time. Unlike functions that create a list of results in memory and then return the entire list (like the range() function in Python 2), generators perform lazy fetching, using up only enough memory to produce one item, returning it, and then proceeding to the next item retrieval. For example, in Python 2 range() produced a list of integers:


    import sys; print sys.version      # 2.7.10
    
    x = range(10)
    print(x)                           # [0, 1, 2, 3, 4, 5, 6, 7, 8, 9]

    But in Python 3, range() produces a special range() object that can be iterated over to obtain the list:


    import sys; print sys.version      # 3.7.0
    x = range(10)
    print(x)                           # range(0,10)
    
    for el in x:
        print(el)        # 0
                         # 1
                         # 2 etc...

    It makes sense that range() should use lazy fetching, since most of the time using it we're only interested in iterating over it, one item at a time. (Strictly speaking range() is not a generator, but we can consider its behavior in that context when discussing lazy fetching.) If we do want a list of integers, we can simply pass the object to list():


    print(list(range(10)))       # [0, 1, 2, 3, 4, 5, 6, 7, 8, 9]

    Since list() is an explicit call, this draws the reader's attention to the memory being allocated, in line with Python's philosophy of "explicit is better than implicit". Without this explicit call, the memory allocation might not be so clear. A generator comprehension is a list comprehension that uses lazy fetching to produce a generator object, rather than producing an entirely new list:


    convert_list = ['THIS', 'IS', 'QUITE', 'UPPER', 'CASE']
    
    lclist = [ x.lower() for x in convert_list ]   # list comprehension (square brackets)
    
    gclist = ( x.lower() for x in convert_list )   # generator comprehension (parentheses)
    
    
    print(lclist)         # ['this', 'is', 'quite', 'upper', 'case']
    
    print(gclist)         # <generator object <genexpr> at 0x10285e7d0>

    We can then iterate over the generator object to retrieve each item in turn. In Python 3, a list comprehension is just "syntactic sugar" for a generator comprehension wrapped in list():


    lclist = list(( x.lower() for x in convert_list ))

    Generator Functions

    We can create our own generator functions, which might be necessary if we don't want our list-returning function to produce the entire list in memory.


    Writing your own generator function would be useful, or even needed, if: 1) we are designing a list-producing function 2) the items from the list were coming from a generator-like source (for example, calculating prime numbers, or looping through a file and modifying each line of the file) 3) the list coming back from the function was too big to be conveniently held in memory (or too big for memory altogether). The generator function contains a new statement, yield, that returns an item produced by the function but remembers its place in the list-generating process. Here is a simplest version of a generator, containing 3 yield statements:


    def return_val():
        yield 'hello'
        yield 'world'
        yield 'wassup'
    
    for msg in return_val():
        print(msg, end=' ')      # hello world wassup
    
    x = return_val()
    print(x)                      # <generator object return_val at 0x10285e7d0>

    As with range() or a generator comprehension, a generator function produces an object that performs lazy fetching. Consider this simulation of the range() function, which generates a sequence of integers starting at 0:


    def my_range(max):
        x = 0
        while x < max:
            yield x
            x += 1
    
    xr = my_range(5)
    
    print(xr)                    # <generator object my_range at 0x10285e870>
    
    for val in my_range(5):
        print(val)               # 0 1 2 3 4
    
    print list(my_range(5))      # [0, 1, 2, 3, 4]

    Generators are particularly useful in producing a sequence of n values, i.e. not a fixed sequence, but an unlimited sequence. In this example we have prepared a generator that generates primes up to the specified limit.


    def get_primes(num_max):
        """ prime number generator """
        candidate = 2
        found = []
        while True:
            if all(candidate % prime != 0 for prime in found):
                yield candidate
                found.append(candidate)
            candidate += 1
            if candidate >= num_max:
                raise StopIteration
    
    my_iter = get_primes(100)
    print(next(my_iter))        # 2
    print(next(my_iter))        # 3
    print(next(my_iter))        # 5
    
    for i in get_primes(100):
        print(i)

    Recursive functions

    A recursive function calls itself until a condition has been reached.


    Recursive functions are appropriate for processes that iterate over a structure of an unknown "depth" of items or events. A factorial is the product of a range of numbers (1 * 2 * 3 * 4 ...).


    factorial: "linear" approach

    def factorial_linear(n):
        prod = 1
        for i in range(1, n+1):
            prod = prod * i
        return prod

    factorial: "recursive" approach

    def factorial(n):
       if n < 1:                                     # base case (reached 0):  returns
           return 1
       else:
           return_num = n * factorial(n - 1)  # recursive call
           return return_num
    
    print(factorial(5))       # 120

    Recursive functions are appropriate for processes that iterate over a structure of an unknown "depth" of items or events. Such situations could include files within a directory tree, where listing the directory is performed over and over until all directories within a tree are exhausted; or similarly, visiting links to pages within a website, where listing the links in a page is performed repeatedly. Recursion features three items: a recursive call, which is a call by the function to itself; the function process itself; and a base condition, which is the point at which the chain of recursions finally returns. A directory tree is a recursive structure in that requires the same operation (listing files in a directory) to be applied to "nodes" of unknown depth:


    Recurse through a directory tree

    import os
    
    def list_dir(this_dir):
        print(('* entering list_dir {} *'.format(this_dir)))
        for name in os.listdir(this_dir):
            pathname = os.path.join(this_dir, name)
            if os.path.isdir(pathname):
                list_dir(pathname)
            else:
                print(('  ' + name))
        print('* leaving list_dir *')   # base condition:  looping is complete
    list_dir('/Users/david/test')

    * entering list_dir /Users/david/test *
      recurse.py
    * entering list_dir /Users/david/test *
      file1
      file2
    * entering list_dir /Users/david/test/test2 *
      file3
      file4
    * leaving list_dir *
    * entering list_dir /Users/david/test/test3 *
      file5
      file6
    * leaving list_dir *
    * leaving list_dir *
    * entering list_dir /Users/david/test4 *
      file7
      file8
    * leaving list_dir *
    * leaving list_dir *

    The function process is the listing of the items in a directory and printing the files. The recursive call is the call to walk(path) at the bottom of the loop -- this is called when the directory looping encounters a directory. The base condition occurs when the file listing is completed. There are no more directories to loop through, so the function call returns.




    Sequence Manipulation

    Identifying Sequences and Iterables

    A sequence is an iterable container of items; an iterable is anything that can be looped through


    So far we have explored the reading and parsing of data; the loading of data into built-in structures; and the aggregation and sorting of these structures. This session explores advanced tools for container processing.


    set operations

    a = set(['a', 'b', 'c'])
    b = set(['b', 'c', 'd'])
    print(a.difference(b))      # set(['a'])
    print(a.union(b))           # set(['a', 'b', 'c', 'd'])
    print(a.intersection(b))    # set(['b', 'c'])

    list comprehensions

    a = ['hello', 'there', 'harry']
    print([ var.upper() for var in a if var.startswith('h') ])
                               # ['HELLO', 'HARRY']

    lambda functions

    names = ['Joe Wilson', 'Pete Johnson', 'Mary Rowe']
    sorted_names = sorted(names, key=lambda x: x.split()[1])
    print(sorted_names)             # ['Pete Johnson', 'Mary Rowe', 'Joe Wilson']

    ternary assignment

    rev_sort = True if user_input == 'highest' else False
    
    pos_val = x if x >= 0 else x * -1

    conditional assignment

    val = this or that       # 'this' if this is True else 'that'
    val = this and that      # 'this' if this is False else 'that'

    Container processing: Set Comparisons

    We have used the set to create a unique collection of objects. The set also allows comparisons of sets of objects. Methods like set.union (complete member list of two or more sets), set.difference (elements found in this set not found in another set) and set.intersection (elements common to both sets) are fast and simple to use.


    set_a = set([1, 2, 3, 4])
    set_b = set([3, 4, 5, 6])
    
    print(set_a.union(set_b))           # set([1, 2, 3, 4, 5, 6])  (set_a + set_b)
    print(set_a.difference(set_b))      # set([1, 2])              (set_a - set_b)
    print(set_a.intersection(set_b))    # set([3, 4])  (what is common between them?)

    List comprehensions: filtering a container's elements

    List comprehensions abbreviate simple loops into one line.


    Consider this loop, which filters a list so that it contains only positive integer values:

    myints = [0, -1, -5, 7, -33, 18, 19, 55, -100]
    myposints = []
    for el in myints:
      if el > 0:
        myposints.append(el)
    
    print(myposints)                   # [7, 18, 19, 55]

    This loop can be replaced with the following one-liner:

    myposints = [ el for el in myints if el > 0 ]

    See how the looping and test in the first loop are distilled into the one line? The first el is the element that will be added to myposints - list comprehensions automatically build new lists and return them when the looping is done.


    The operation is the same, but the order of operations in the syntax is different:

    # this is pseudo code
    # target list = item for item in source list if test

    Hmm, this makes a list comprehension less intuitive than a loop. However, once you learn how to read them, list comprehensions can actually be easier and quicker to read - primarily because they are on one line. This is an example of a filtering list comprehension - it allows some, but not all, elements through to the new list.


    List comprehensions: transforming a container's elements

    Consider this loop, which doubles the value of each value in it:


    nums = [1, 2, 3, 4, 5]
    dblnums = []
    for val in nums:
      dblnums.append(val*2)
    
    print(dblnums)                          # [2, 4, 6, 8, 10]

    This loop can be distilled into a list comprehension thusly:

    dblnums = [ val * 2 for val in nums ]

    This transforming list comprehension transforms each value in the source list before sending it to the target list:

    # this is pseudo code
    # target list = item transform for item in source list

    We can of course combine filtering and transforming:

    vals = [0, -1, -5, 7, -33, 18, 19, 55, -100]
    doubled_pos_vals = [ i*2 for i in vals if i > 0 ]
    print(doubled_pos_vals)                # [14, 36, 38, 110]

    List comprehensions: examples

    If they only replace simple loops that we already know how to do, why do we need list comprehensions? As mentioned, once you are comfortable with them, list comprehensions are much easier to read and comprehend than traditional loops. They say in one statement what loops need several statements to say - and reading multiple lines certainly takes more time and focus to understand.


    Some common operations can also be accomplished in a single line. In this example, we produce a list of lines from a file, stripped of whitespace:

    stripped_lines = [ i.rstrip() for i in open('FF_daily.txt').readlines() ]

    Here, we're only interested in lines of a file that begin with the desired year (1972):

    totals = [ i for i in open('FF_daily.txt').readlines() if i.startswith('1972') ]

    If we want the MktRF values for our desired year, we could gather the bare amounts this way:

    mktrf_vals = [ float(i.split()[1]) for i in open('FF_daily.txt').readlines() if i.startswith('1972') ]

    And in fact we can do part of an earlier assignment in one line -- the sum of MktRF values for a year:

    mktrf_sum = sum([ float(i.split()[1]) for i in open('FF_daily.txt').readlines() if i.startswith('1972') ])

    From experience I can tell you that familiarity with these forms make it very easy to construct and also to decode them very quickly - much more quickly than a 4-6 line loop.


    List Comprehensions with Dictionaries

    Remember that dictionaries can be expressed as a list of 2-element tuples, converted using items(). Such a list of 2-element tuples can be converted back to a dictionary with dict():


    mydict =  {'a': 5, 'b': 0, 'c': -3, 'd': 2, 'e': 1, 'f': 4}
    
    my_items = list(mydict.items())      # my_items is now [('a',5), ('b',0), ('c',-3), ('d',2), ('e',1), ('f',4)]
    mydict2 = dict(my_items)       # mydict2 is now   {'a':5,   'b':0,   'c':-3,   'd':2,   'e':1,   'f':4}

    It becomes very easy to filter or transform a dictionary using this structure. Here, we're filtering a dictionary by value - accepting only those pairs whose value is larger than 0:

    mydict = {'a': 5, 'b': 0, 'c': -3, 'd': 2, 'e': -22, 'f': 4}
    filtered_dict = dict([ (i, j) for (i, j) in list(mydict.items()) if j > 0 ])

    Here we're switching the keys and values in a dictionary, and assigning the resulting dict back to mydict, thus seeming to change it in-place:

    mydict = dict([ (j, i) for (i, j) in list(mydict.items()) ])
    

    The Python database module returns database results as tuples. Here we're pulling two of three values returned from each row and folding them into a dictionary.

    # 'tuple_db_results' simulates what a database returns
    tuple_db_results = [
      ('joe', 22, 'clerk'),
      ('pete', 34, 'salesman'),
      ('mary', 25, 'manager'),
    ]
    
    names_jobs = dict([ (name, role) for name, age, role in tuple_db_results ])

    Sorting dictionary items by value: operator.itemgetter

    Recall that we can render a dictionary as a list of tuples with the dict.items() method:


    mydict = { 'a': 5, 'b': 2, 'c': 1, 'z': 0 }
    mydict_items = list(mydict.items())                        # [(a, 5), (c, 1), (b, 2), (z, 0)]

    To sort dictionary items by value, we need to sort each two-element tuple by its second element. The built-in module operator.itemgetter will return whatever element of a sequence we wish - in this way it is like a subscript, but in function format (so it can be called by the Python sorting algorithm).

    import operator
    mydict = { 'a': 5, 'b': 2, 'c': 1, 'z': 0 }
    mydict_items = list(mydict.items())                        # [(a, 5), (c, 1), (b, 2), (z, 0)]
    mydict_items.sort(key=operator.itemgetter(1))
    print(mydict_items)                                   # [(z, 0), (c, 1), (b, 2), (a, 5)]
    for key, val in mydict_items:
        print("{0} = {1}".format(key, val))
    
                        ## z = 0
                        ## c = 1
                        ## b = 2
                        ## a = 5

    The above can be conveniently combined with looping, effectively allowing us to loop through a "sorted" dict:

    for key, val in sorted(list(mydict.items()), key=operator.itemgetter(1)):
        print("{0} = {1}".format(key, val))

    Database results come as a list of tuples. Perhaps we want our results sorted in different ways, so we can store as a list of tuples and sort using operator.itemgetter. This example sorts by the third field, then by the second field (last name, then first name):

    import operator
    items =[ (123, 'Joe', 'Wilson', 35, 'mechanic'),
             (124, 'Sam', 'Jones', 22, 'mechanic'),
             (125, 'Pete', 'Jones', 40, 'mechanic'),
             (126, 'Irina', 'Bibi', 31, 'mechanic'),
           ]
    items.sort(key=operator.itemgetter(2,1)) # sorts by last, first name
    for this_pair in items:
      print("{0} {1}".format(this_pair[1], this_pair[2]))
    
           ## Irina Bibi
           ## Pete Jones
           ## Sam Jones
           ## Joe Wilson



    Flask

    Web Frameworks

    Introduction


    A "web framework" is an application or package that facilitates web programming. Server-side apps (for example: a catalog, content search and display, reservation site or most other interactive websites) use a framework to handle the details of the web network request, page display and database input/output -- while freeing the programmer to supply just the logic of how the app will work. Full Stack Web Frameworks A web application consists of layers of components that are configured to work together (i.e., built into a "stack"). Such components may include: * authenticating and identifying users through cookies * handling data input from forms and URLs * reading and writing data to and from persistant storage (e.g. databases) * displaying templates with dynamic data inserted * providing styling to templates (e.g., with css) * providing dynamic web page functionality, as with AJAX The term "full stack developer" used by schools and recruiters refers to a developer who is proficient in all of these areas. Django is possibly the most popular web framework. This session would probably focus on Django, but its configuration and setup requirements are too lengthy for the available time. "Lightweight" Web Frameworks A "lightweight" framework provides the base functionality needed for a server-side application, but allows the user to add other stack components as desired. Such apps are typically easier to get started with because they require less configuration and setup. Flask is a popular lightweight framework with many convenient defaults allows us to get our web application started quickly.


    The Flask app object, the app.run() method and @app.route dispatch decorators

    "@app.route()" functions describe what happens when the user visits a particular "page" or URL shown in the decorator.


    hello_flask.py


    Here is a basic template for a Flask app.

    #!/usr/bin/env python
    
    import flask
    app = flask.Flask(__name__)           # a Flask object
    
    @app.route('/hello')                 # called when visiting web URL 127.0.0.1:5000/hello/
    def hello_world():
        print('*** DEBUG:  inside hello_world() ***')
        return '<PRE>Hello, World!</PRE>'            # expected to return a string (usu. the HTML to display)
    
    if __name__ == '__main__':
        app.run(debug=True, port=5000)    # app starts serving in debug mode on port 5000

    The first two lines and last two lines will always be present (production apps will omit the debug= and port= arguments). app is an object returned by Flask; we will use it for almost everything our app does. We call it a "god object" because it's always available and contains most of what we need. app.run() starts the Flask application server and causes Flask to wait for new web requests (i.e., when a browser visit the server). @app.route() functions are called when a particular URL is requested. The decorator specifies the string to be found at the end of the URL. For example, the above decorator @app.route('/hello') specifies that the URL to reach the function should be http://localhost:5000/hello/ The string returned from the function is passed on to Flask and then to the browser on the other end of the web request.


    Running a Flask app locally

    Flask comes complete with its own self-contained app server. We can simply run the app and it begins serving locally. No internet connection is required.


    $ python hello_flask.py
     * Running on http://127.0.0.1:5000/ (Press CTRL+C to quit)
     * Restarting with stat
    127.0.0.1 - - [13/Nov/2016 15:58:16] "GET / HTTP/1.1" 200 -
    *** DEBUG:  inside hello_world() ***

    The Flask app prints out web server log messages showing the URL requested by each visitor. You can also print error messages directly from the application, and they will appear in the log (these were printed with *** strings, for visibility.) Changes to Flask code are detected and cause Flask to restart the server; errors cause it to exit


    Whenever you make a change and save your script, the Flask server will restart -- you can see it issue messages to this effect:

     * Detected change in '/Users/dblaikie/Dropbox/tech/nyu/advanced_python/solutions/flask/guestbook/guestbook_simple.py', reloading
     * Restarting with stat
     * Debugger is active!
     * Debugger pin code: 161-369-356

    If there is an error in your code that prevents it from running, the code will raise an exception and exit.

     * Detected change in '/Users/dblaikie/Dropbox/tech/nyu/advanced_python/solutions/flask/guestbook/guestbook_simple.py', reloading
     * Restarting with stat
      File "./guestbook_simple.py", line 17
        return hello, guestbook_id ' + guestbook_id
                                                  ^
    SyntaxError: EOL while scanning string literal

    At that point, the browser will simply report "This site can't be reached" with no other information.


    Therefore we must keep an eye on the window to see if the latest change broke the script -- fix the error in the script, and then re-run the script in the Terminal.


    Redirection and Event Flow

    Once an 'event' function is called, it may return a string, call another function, or redirect to another page.


    Return a plain string


    A plain string will simply be displayed in the browser. This is the simplest way to display text in a browser.

    @app.route('/hello')
    def hello_world():
        return '<PRE>Hello, World!</PRE>'            # expected to return a string (usu. the HTML to display)


    Return HTML (also a string)


    HTML is simply tagged text, so it is also returned as a string.

    @app.route('/hello_template')
    def hello_html():
        return """
    <HTML>
      <HEAD>
        <TITLE>My Greeting Page</TITLE>
      </HEAD>
      <BODY>
        <H1>Hello, world!</H1>
      </BODY>
    </HTML>"""


    Return an HTML Template (to come)


    This method also returns a string, but it is returned from the render_template() function.

    @app.route('/hello_html')
    def hello_html():
        return flask.render_template('response.html')    # found in templates/response.html


    Return another function call


    Functions that aren't intended to return strings but to perform other actions (such as making database changes) can simply call other functions that represent the desired destination:

    def hello_html():
        return """
    <HTML>
      <HEAD>
        <TITLE>My Greeting Page</TITLE>
      </HEAD>
      <BODY>
        <H1>Hello, world (from another function)  !</H1>
      </BODY>
    </HTML>"""
    
    @app.route('/hello_template')
    def hello():
        return hello_html()

    Because hello() is calling hello_html() in its return statement, whatever is returned from there will be returned from hello().


    Redirecting to another program URL with flask.redirect() and flask.url_for()


    At the end of a function we can call the flask app again through a page redirect -- that is, to have the app call itself with new parameters.

    from datetime import date
    
    @app.route('/sunday_hello')
    def sunday_hello():
        return "It's Sunday!  Take a rest!"
    
    @app.route('/shello')
    def hello():
        if date.today().strftime('%a') == 'Sun':
            return flask.redirect(flask.url_for('sunday_hello'))
        else:
            return 'Hello, workday (or Saturday)!'

    redirect() issues a redirection to a specified URL; this can be http://www.google.com or any desired URL.


    url_for() simply produces the URL that will call the flask app with /login.


    Building a Multi-Page Application

    Use flask.url_for() to build links to other apps


    This app has three pages; we can build a "closed system" of pages by having each page link to another within the site.


    happy_image = 'http://davidbpython.com/advanced_python/python_data/happup.jpg'
    sad_image = 'http://davidbpython.com/advanced_python/python_data/sadpup.jpg'
    
    @app.route('/question')
    def ask_question():
        return """
    <HTML>
      <HEAD><TITLE>Do you like puppies?</TITLE></HEAD>
      <BODY>
          <H3>Do you like puppies?</H3>
          <A HREF="{}">arf!</A><BR>
          <A HREF="{}">I prefer cats...</A>
      </BODY>
    </HTML>""".format(flask.url_for('yes'), flask.url_for('no'))
    
    
    @app.route('/yes')
    def yes():
        return """
    <HTML>
      <HEAD><TITLE>C'mere Boy!</TITLE></HEAD>
      <BODY>
          <H3>C'mere, Boy!</H3>
          <IMG SRC="{}"><BR>
          <BR>
          Change your mind?  <A HREF="{}">Let's try again.</A>
      </BODY>
    </HTML>""".format(happy_image, flask.url_for('ask_question'))
    
    
    @app.route('/no')
    def no():
        return """
    <HTML>
      <HEAD><TITLE>Aww...</TITLE></HEAD>
      <BODY>
          <H3>Aww...really?</H3>
          <IMG SRC="{}"><BR>
          <BR>
          Change your mind?  <A HREF="{}">Let's try again.</A>
      </BODY>
    </HTML>""".format(sad_image, flask.url_for('ask_question'))

    Simple Templating

    Use {{ varname }} to create template tokens and flask.render_template() to insert to them.


    HTML pages are rarely written into Flask apps; instead, we use standalone template files. The template files are located in a templates directory placed in the same directory as your Flask script.


    question.html

    <HTML>
      <HEAD><TITLE>Do you like puppies?</TITLE></HEAD>
      <BODY>
          <H3>Do you like puppies?</H3>
          <A HREF="{{ yes_link }}">arf!</A><BR>
          <A HREF="{{ no_link }}">I prefer cats...</A>
      </BODY>
    </HTML>

    puppy.html

    <HTML>
      <HEAD><TITLE><{{ title_message }}</TITLE></HEAD>
      <BODY>
          <H3>{{ title_message }}</H3>
          <IMG SRC="{{ puppy_image }}"><BR>
          <BR>
          Change your mind?  <A HREF="{{ question_link }}">Let's try again.</A>
      </BODY>
    </HTML>

    puppy_question.py

    happy_image = 'http://davidbpython.com/advanced_python/python_data/happup.jpg'
    sad_image =   'http://davidbpython.com/advanced_python/python_data/sadpup.jpg'
    
    
    @app.route('/question')
    def ask_question():
        return flask.render_template('question.html',
                                      yes_link=flask.url_for('yes'),
                                      no_link=flask.url_for('no'))
    
    @app.route('/yes')
    def yes():
        return flask.render_template('puppy.html',
                                      puppy_image=happy_image,
                                      question_link=flask.url_for('ask_question'),
                                      title_message='Cmere, boy!')
    
    @app.route('/no')
    def no():
        return flask.render_template('puppy.html',
                                      puppy_image=sad_image,
                                      question_link=flask.url_for('ask_question'),
                                      title_message='Aww... really?')

    Embedding Python Code in Templates

    Use {% %} to embed Python code for looping, conditionals and some functions/methods from within the template.


    Template document: "template_test.html"


    <!DOCTYPE html>
    <html lang="en">
      <head>
        <title>Important Stuff</title>
      </head>
      <body>
    
      <h1>Important Stuff</h1>
    
      Today's magic number is {{ number }}<br><br>
    
      Today's strident word is {{ word.upper() }}<br><br>
    
      Today's important topics are:<br>
      {% for item in mylist %}
          {{ item }}<br>
      {% endfor %}
      <br><br>
    
      {% if reliability_warning %}
          WARNING:  this information is not reliable
      {% endif %}
    
      </body>
    </html>

    Flask code


    @app.route('template_test')
    def template_test():
        return flask.render_template('template_test.html', number=1035,
                                                           word='somnolent',
                                                           mylist=['children', 'animals', 'bacteria'],
                                                           reliability_warning=True)

    As before, {{ variable }} can used for variable insertions, as well as instance attributes, method and function calls
    {% for this in that %} can be used for 'if' tests, looping with 'for' and other basic control flow


    Reading args from URL or Form Input

    Input from a page can come from a link URL, or from a form submission.


    name_question.html

    <HTML>
      <HEAD>
      </HEAD>
      <BODY>
        What is your name?<BR>
        <FORM ACTION="{{ url_for('greet_name') }}" METHOD="post">
          <INPUT NAME="name" SIZE="20">
          <A HREF="{{ url_for('greet_name') }}?no_name=1">I don't have a name</A>
          <INPUT TYPE="submit" VALUE="tell me!">
        </FORM>
      </BODY>
    </HTML>

    flaskapp.py

    @app.route('/name_question')
    def ask_name():
        return flask.render_template('name_question.html')
    
    @app.route('/greet', methods=['POST', 'GET'])
    def greet_name():
        name =    flask.request.form.get('name')     # from a POST (form with 'method="POST"')
        no_name = flask.request.args.get('no_name')  # from a GET (URL)
    
        if name:
            msg = 'Hello, {}!'.format(name)
        elif no_name:
            msg = 'You are anonymous.  I respect that.'
        else:
            raise ValueError('\nraised error:  no "name" or "no_name" params passed in request')
    
        return '<PRE>{}</PRE>'.format(msg)
    
    

    Base templates

    Many times we want to apply the same HTML formatting to a group of templates -- for example the <head> tag, which my include css formatting, javascript, etc.


    We can do this with base templates:

    {% extends "base.html" %}         # 'base.html' can contain HTML from another template
        <h1>Special Stuff</h1>
        Here is some special stuff from the world of news.

    The base template "surrounds" any template that imports it, inserting the importing template at the {% block body%} tag:

    <html>
      <head>
      </head>
      <body>
        <div class="container">
          {% block body %}
             <H1>This is the base template default body.</H1>
          {% endblock %}
        </div>
      </body>
    </html>

    There are many other features of Jinja2 as well as ways to control the API, although I have found the above features to be adequate for my purposes.


    Sessions

    Sessions (usually supported by cookies) allow Flask to identify a user between requests (which are by nature "anonymous").


    When a session is set, a cookie with a specific ID is passed from the server to the browser, which then returns the cookie on the next visit to the server. In this way the browser is constantly re-identifying itself through the ID on the cookie. This is how most websites keep track of a user's visits.


    flask_session.py

    import flask
    app = flask.Flask(__name__)
    
    app.secret_key = 'A0Zr98j/3yX R~XHH!jmN]LWX/,?RT'  # secret key
    
    @app.route('/index')
    def hello_world():
    
        # see if the 'login' link was clicked:  set a session ID
        user_id = flask.request.args.get('login')
        if user_id:
            flask.session['user_id'] = user_id
            is_session = True
    
        # else see if the 'logout' link was clicked:  clear the session
        elif flask.request.args.get('logout'):
            flask.session.clear()
    
        # else see if there is already a session cookie being passed:  retrieve the ID
        else:
            # see if a session cookie is already active between requests
            user_id = flask.session.get('user_id')
    
        # tell the template whether we're logged in (user_id is a numeric ID, or None)
        return flask.render_template('session_test.html', is_session=user_id)
    
    
    if __name__ == '__main__':
        app.run(debug=True, port=5001)    # app starts serving in debug mode on port 5001

    session_test.html

    <!DOCTYPE html>
    <html lang="en">
      <head>
        <title>Session Test</title>
      </head>
      <body>
    
      <h1>Session Test</h1>
    
      {% if is_session %}
      <font color="green">Logged In</font>
      {% else %}
      <font color="red">Logged Out</font>
      {% endif %}
    
      <br><br>
    
      <a href="index?login=True">Log In</a><br>
      <a href="index?logout=True">Log Out</a><br>
    
      </body>
    </html>

    Config Values

    Configuration values are set to control how Flask works as well as to be set and referenced by an individual application.


    Flask sets a number of variables for its own behavior, among them DEBUG=True to display errors to the browser, and SECRET_KEY='!jmNZ3yX R~XWX/r]LA098j/,?RTHH' to set a session cookie's secret key. A list of Flask default configuration values is here. Retrieving config values


    value = app.config['SERVER_NAME']

    Setting config values individually


    app.config['DEBUG'] = True

    Setting config values from a file


    app.config.from_pyfile('flaskapp.cfg')

    Such a file need only contain python code that sets uppercased constants -- these will be added to the config. Setting config values from a configuration Object Similarly, the class variables defined within a custom class can be read and applied to the config with app.config.from_object(). Note in the example below that we can use inheritance to distribute configs among several classes, which can aid in organization and/or selection:


    In a file called configmodule.py:

    class Config(object):
        DEBUG = False
        TESTING = False
        DATABASE_URI = 'sqlite://:memory:'
    
    class ProductionConfig(Config):
        DATABASE_URI = 'mysql://user@localhost/foo'
    
    class DevelopmentConfig(Config):
        DEBUG = True
    
    class TestingConfig(Config):
        TESTING = True

    In the flask script:

    app.config.from_object('configmodule.ProductionConfig')

    Environment Variables

    Environment Variables are system-wide values that are set by the operating system and apply to all applications. They can also be set by individual applications.


    The OpenShift web container sets a number of environment variables, among them OPENSHIFT_LOG_DIR for log files and OPENSHIFT_DATA_DIR for data files. Flask employs jinja2 templates. A list of Openshift environment variables can be found here.


    Flask and security

    An important caveat regarding web security: Flask is not considered to be a secure approach to handling sensitive data.


    ...at least, that was the opinion of a former student, a web programmer who worked for Bank of America, about a year ago -- they evaluated Flask and decided that it was not reliable and could have security vulnerabilities. His team decided to use CGI -- the baseline protocol for handling web requests. Any framework is likely to have vulnerabilities -- only careful research and/or advice of a professional can ensure reliable privacy. However for most applications security is not a concern -- you will simply want to avoid storing sensitive data on a server without considering security.


    Flask-Specific Errors

    Keep these Flask-specific errors in mind.


    Page not found


    URLs specified in @app.route() functions should not end in a trailing slash, and URLs entered into browsers must match

    @app.route(/'hello')

    If you omit a trailing slash from this path but include one in the URL, the browser will respond that it can't find the page.


    What's worse, some browsers sometimes try to 'correct' your URL entries, so if you type a URL with a trailing slash and get "Page not found", the next time you type it differently (even if correctly) the browser may attempt to "correct" it to the way you typed it the first time (i.e., incorrectly). This can be extremely frustrating; the only remedy I have found is to clear the browser's browsing data.


    Functions must return strings (or redirect to another function or URL); they do not print page responses.

    @app.route(/'hello')
    def hello():
        return 'Hello, world!'        # not print 'Hello, world!'

    Each routing function expects a string to be returned -- so the function must do one of these:


       1) return a string (this string will be displayed in the browser)    2) call another @app.route() function that will return a string    3) issue a URL redirect (described later) Method not allowed usually means that a form was submitting that specifies method="POST" but the @app.route decorator doesn't specify methods=['POST']. See "Reading args from URL or Form Input", above.


    name_question.html

    <HTML>
      <HEAD>
      </HEAD>
      <BODY>
        What is your name?<BR>
        <FORM ACTION="{{ url_for('greet_name') }}" METHOD="post">
          <INPUT NAME="name" SIZE="20">
          <A HREF="{{ url_for('greet_name') }}?no_name=1">I don't have a name</A>
          <INPUT TYPE="submit" VALUE="tell me!">
        </FORM>
      </BODY>
    </HTML>

    If the form above submits data as a "post", the app.route() function would need to specify this as well:

    @app.route('/greet', methods=['POST', 'GET'])
    def greet_name():
       ...



    Regular Expressions: Text Matching and Extraction

    Text Matching: Why is it so Useful?

    Short answer: validation and extraction of formatted text. Either we want to see whether a string contains the pattern we seek, or we want to pull selected information from the string.


    In the case of fixed-width text, we have been able to use a slice.

    line = '19340903  3.4 0.9'
    year = line[0:4]                 # year == 1934

    In the case of delimited text, we have been able to use split()

    line = '19340903,3.4,0.9'
    els = line.split(',')
    yearmonthday = els[0]            # 193400903
    MktRF = els[1]                   # 3.4

    In the case of formatted text, there is no obvious way to do it.

    # how would we extract 'Jun' from this string?
    log_line = '66.108.19.165 - - [09/Jun/2003:19:56:33 -0400] "GET /~jjk265/cd.jpg HTTP/1.1" 200 175449'

    We may be able to use split() and slicing in some combination to get what we want, but it would be awkward and time consuming. So we're going to learn how to use regular expressions.


    Preview: a complex example

    Just as an example to show you what we're doing, the following regex pattern could be used to pull 'Jun' from the log line:


    import re
    
    log_line = '66.108.19.165 - - [09/Jun/2003:19:56:33 -0400] "GET /~jjk265/cd.jpg HTTP/1.1" 200 175449'
    
    reg = re.search(r'\d{2,3}\.\d{2,3}\.\d{2,3}\.\d{2,3} - - \[\d\d\/(\w{3})\/\d{4}', log_line)
    
    print(reg.group(1))   # Jun

    Reading from left to right, the pattern (shown in the r'' string) says this: "2-3 digits, followed by a period, followed by 2-3 digits, followed by a period, followed by 2-3 digits, followed by a period, followed by 2-3 digits, followed by a space, dash, space, dash, followed by a square bracket, followed by 2 digits, followed by a forward slash, followed by 3 word characters (and this text grouped for extraction), followed by a slash, followed by 4 digit characters." Now, that may seem complex. Many of our regexes will be much simpler, although this one isn't terribly unusual. But the power of this tool is in allowing us to describe a complex string in terms of the pattern of the text -- not the specific text -- and either pull parts of it out or make sure it matches the pattern we're looking for. This is the purpose of regular expressions.


    Python's re module and re.search()

    import re makes regular expressions available to us. Everything we do with regular expressions is done through re.


    re.search() takes two arguments: the pattern string which is the regex pattern, and the string to be searched. It can be used in an if expression, and will evaluate to True if the pattern matched.

    # weblog contains string lines like this:
      '66.108.19.165 - - [09/Jun/2003:19:56:33 -0400] "GET /~jjk265/cd.jpg HTTP/1.1" 200 175449'
      '66.108.19.165 - - [09/Jun/2003:19:56:44 -0400] "GET /~dbb212/mysong.mp3 HTTP/1.1" 200 175449'
      '66.108.19.165 - - [09/Jun/2003:19:56:45 -0400] "GET /~jjk265/cd2.jpg HTTP/1.1" 200 175449'
    
    # script snippet:
    for line in weblog.readlines():
      if re.search(r'~jjk265', line):
        print line                      # prints 2 of the above lines

    not for negating a search

    As with any if test, the test can be negated with not. Now we're saying "if this pattern does not match".


    # again, a weblog:
      '66.108.19.165 - - [09/Jun/2003:19:56:33 -0400] "GET /~jjk265/cd.jpg HTTP/1.1" 200 175449'
      '66.108.19.165 - - [09/Jun/2003:19:56:44 -0400] "GET /~dbb212/mysong.mp3 HTTP/1.1" 200 175449'
      '66.108.19.165 - - [09/Jun/2003:19:56:45 -0400] "GET /~jjk265/cd2.jpg HTTP/1.1" 200 175449'
    
    # script snippet:
    for line in weblog.readlines():
      if not re.search(r'~jjk265', line):
        print line                      # prints 1 of the above lines -- the one without jjk265

    The raw string (r'')

    The raw string is like a normal string, but it does not process escapes. An escaped character is one preceded by a backslash, that turns the combination into a special character. \n is the one we're famliar with - the escaped n converts to a newline character, which marks the end of a line in a multi-line string.


    A raw string wouldn't process the escape, so r'\n' is literally a backslash followed by an n.

    var = "\n"            # one character, a newline
    var2 = r'\n'          # two characters, a backslash followed by an n

    The re Bestiary

    We call it a "bestiary" because it contains many strange animals. Each of these animals takes the shape of a special character (like $, ^, |); or a regular text character that has been escaped (like \w, \d, \s); or a combination of characters in a group (like {2,3}, [aeiou]). Our bestiary can be summarized thusly:


    Anchor Characters and the Boundary Character: $, ^, \b Character Classes: \w, \d, \s, \W, \S, \D Custom Character Classes: [aeiou], [a-zA-Z] The Wildcard: . Quantifiers: +, *, ? Custom Quantifiers: {2,3}, {2,}, {2} Groupings: (parentheses groups)


    Patterns can match anywhere, but must match on consecutive characters

    Patterns can match any string of consecutive characters. A match can occur anywhere in the string.


    import re
    
    str1 = 'hello there'
    str2 = 'why hello there'
    str3 = 'hel lo'
    
    if re.search(r'hello', str1):  print('matched')   # matched
    if re.search(r'hello', str2):  print('matched')   # matched
    if re.search(r'hello', str3):  print('matched')   # does not match

    Note that 'hello' matches at the start of the first string and the middle of the second string. But it doesn't match in the third string, even though all the characters we are looking for are there. This is because the space in str3 is unaccounted for - always remember - matches take place on consecutive characters.


    Anchors and Boundary

    Our match may require that the search text appear at the beginning or end of a string. The anchor characters can require this.


    This program lists only those files in the directory that end in .txt:

    import os, re
    for filename in os.listdir(r'/path/to/directory'):
      if re.search(r'\.txt$', filename):     # look for '.txt' at end of filename
        print(filename)

    This program prints all the lines in the file that don't begin with a hash mark:

    for text_line in open(r'/path/to/file.py'):
      if not re.search(r'^#', text_line):        # look for '#' at start of filename
        print(text_line)

    When they are used as anchors, we will always expect ^ to appear at the start of our pattern, and $ to appear at the end.


    Character Classes

    A character class is a special regex entity that will match on any of a set of characters. The three built-in character classes are these:


    \d: [0-9] (Digits) \w: [a-zA-Z0-9_] (Word characters -- letters, numbers or underscores) \s: [ \n\t] ('Whitespace' characters -- spaces, newlines, or tabs) So a \d will match on a 5, 9, 3, etc.; a \w will match on any of those, or on a, Z, _ (underscore). Keep in mind that although they match on any of several characters, a single instance of a character class matches on only one character. For example, a \d will match on a single number like '5', but it won't match on both characters in '55'. To match on 55, you could say \d\d.


    Built-in Character Classes: digits

    The \d character class matches on any digit. This example lists only those files with names formatted with a particular syntax -- YYYY-MM-DD.txt:


    import re
    dirlist = ('.', '..', '2010-12-15.txt', '2010-12-16.txt', 'testfile.txt')
    for filename in dirlist:
      if re.search(r'^\d\d\d\d-\d\d-\d\d\.txt$', filename):
        print(filename)

    Here's another example, validation: this regex uses the pattern ^\d\d\d\d$ to check to see that the user entered a four-digit year:

    import re
    answer = input("Enter your birth year in the form YYYY\n")
    if re.search(r'^\d\d\d\d$', answer):
      print("Your birth year is ", answer)
    else:
      print("Sorry, that was not YYYY")
    

    Built-in Character Classes: "word" characters

    A word character casts a wider net: it will match on any number, letter or underscore.


    In this example, we require the user to enter a username with any word characters:

    username = input()
    if not re.search(r'^\w\w\w\w\w$', username):
      print("use five numbers, letters, or underscores\n")

    As you can see, the anchors prevent the input from exceeding 5 characters.


    Built-in Character Classes: spaces

    A space character class matches on any of three characters: a space (' '), a newline ('\n') or a tab ('\t'). This program searches for a space anywhere in the string and if it finds it, the match is successful - which means the input isn't successful:


    new_password = input()
    if re.search(r'\s', new_password):
      print("password must not contain spaces")

    Note in particular that the regex pattern \s is not anchored anywhere. So the regex will match if a space occurs anywhere in the string. You may also reflect that we treat spaces pretty roughly - always stripping them off. They always get in the way! And they're invisible, too, and still we feel the need to persecute them. What a nuisance.


    Inverse Character Classes

    These are more aptly named inverse character classes - they match on anything that is not in the usual character set.


    Not a digit: \D


    So \D matches on letters, underscores, special characters - anything that is not a digit. This program checks for a non-digit in the user's account number:

    account_number = input()
    if re.search(r'\D', account_number):
      print("account number must be all digits!")

    Not a word character: \W


    Here's a username checker, which simply looks for a non-word:

    account_number = input()
    if re.search(r'\W', account_number):
      print("account number must be only letters, numbers, and underscores")

    Not a space character: \S


    These two regexes check for a non-space at the start and end of the string:

    sentence = input()
    if re.search(r'^\S', sentence) and re.search(r'\S$', sentence):
      print("the sentence does not begin or end with a space, tab or newline.")

    Custom Character Classes

    Consider this table of character classes and the list of characters they match on:


    \d: [0123456789] \w: [abcdefghijklmnopqrstuvwxyzABCDEFGHIJKLMNOOPQRSTUVWXYZ0123456789_] or
    [a-zA-Z0-9_]
    \s: [ \t\n] In fact, the bracketed ranges can be used to create our own character classes. We simply place members of the class within the brackets and use it in the same way we might use \d or the others.


    A custom class can contain a range of characters. This example looks for letters only (there is no built-in class for letters):

    import re
    input = input("please enter a username, starting with a letter:  ")
    if not re.search(r'^[a-zA-Z]', input):
      exit("invalid user name entered")

    This custom class [.,;:?!] matches on any one of these punctuation characters, and this example identifies single punctuation characters and removes them:

    import re
    text_line = 'Will I?  I will.  Today, tomorrow; yesterday and before that.'
    for word in text_line.split():
      while re.search(r'[.,;:?! -]$', word):
        word = word[:-1]
      print(word)

    Inverse Custom Character Classes

    Like \S for \s, the inverse character class matches on anything not in the list. It is designated with a carrot just inside the open bracket:


    import re
    for text_line in open('unknown_text.txt'):
      for word in text_line.split():
        while re.search(r'[^a-zA-Z]$', word):
          word = word[:-1]
        print(word)

    It would be easy to confuse the carrot at the start of a string with the carrot at the start of a custom character class -- just keep in mind that one appears at the very start of the string, and the other at the start of the bracketed list.


    The Wildcard (.)

    The ultimate character class, it matches on every character except for a newline. (We might surmise this is because we are often working with line-oriented input, with pesky newlines at the end of every line. Not matching on them means we never have to worry about stripping or watching out for newlines.)

    import re
    username = input()
    if not re.search(r'^.....$', username):   # five dots here
      print("you can use any characters except newline, but there must \
      be five of them.\n")
    

    Quantifiers: specifies how many to match on

    A quantifier appears immediately after a character, character class, or grouping (coming up). It describes how many of the preceding characters there may be in our matched text.


    We can say three digits (\d{3}), between 1 and 3 word characters (\w{1,3}), one or more letters [a-zA-Z]+, zero or more spaces (\s*), one or more x's (x+). Anything that matches on a character can be quantified.

    +      : 1 or more
    *      : 0 or more
    ?      : 0 or 1
    {3,10} : between 3 and 10

    In this example directory listing, we are interested only in files with the pattern config_ followed by an integer of any size. We know that there could be a config_1.txt, a config_12.txt, or a config_120.txt. So, we simply specify "one or more digits":

    import re
    filenames = ['config_1.txt', 'config_10.txt', 'notthis.txt', '.', '..']
    wanted_files = []
    for file in filenames:
      if re.search(r'^config_\d+\.txt$', file):
        wanted_files.append(file)

    Here, we validate user input to make sure it matches the pattern for valid NYU ID. The pattern for an NYU Net ID is: two or three letters followed by one or more numbers:

    import re
    input = input("please enter your net id:  ")
    if not re.search(r'^[A-Za-z]{2,3}\d+$', input):
      print("that is not valid NYU Net ID!")

    A simple email address is one or more word characters followed by an @ sign, followed by a period, followed by 2-4 letters:

    import re
    email_address = input()
    if re.search(r'^\w+@\w+\.[A-Za-z]{2,}$', email_address):
      print("email address validated")

    Of course email addresses can be more complicated than this - but for this exercise it works well.


    flags: re.IGNORECASE

    We can modify our matches with qualifiers called flags. The re.IGNORECASE flag will match any letters, whether upper or lowercase. In this example, extensions may be upper or lowercase - this file matcher doesn't care!


    import re
    dirlist = ('thisfile.jpg', 'thatfile.txt', 'otherfile.mpg', 'myfile.TXT')
    for file in dirlist:
      if re.search(r'\.txt$', file, re.IGNORECASE):   #'.txt' or '.TXT'
        print(file)

    The flag is passed as the third argument to search, and can also be passed to other re search methods.


    re.search(), re.compile() and the compile object

    re.search() is the one-step method we've been using to test matching. Actually, regex matching is done in two steps: compiling and searching. re.search() conveniently puts the two together.


    In some cases, a pattern should be compiled first before matching begins. This would be useful if the pattern is to be matched on a great number of strings, as in this weblog example:

    import re
    access_log = '/home1/d/dbb212/public_html/python/examples/access_log'
    weblog = open(access_log)
    patternobj = re.compile(r'edg205')
    for line in weblog.readlines():
      if patternobj.search(line):
        print(line, end=' ')
    weblog.close()

    The pattern object is returned from re.compile, and can then be called with search. Here we're calling search repeatedly, so it is likely more efficient to compile once and then search with the compiled object.


    Grouping for Alternates: Vertical Bar

    We can group several characters together with parentheses. The parentheses do not affect the match, but they do designate a part of the matched string to be handled later. We do this to allow for alternate matches, for quantifying a portion of the pattern, or to extract text.


    Inside a group, the vertical bar can indicate allowable matches. In this example, a string will match on any of these words, and because of the anchors will not allow any other characters:

    import re
    import sys
    
    program_arg = sys.argv[1]
    if not re.search(r'^Q(1|2|3|4)\-\d{4}$', program_arg):
        exit("quarter argument must match the pattern 'Q[num]-YYYY' "
             "where [num] is 1-4 and YYYY is a 4-digit year")
    

    Grouping for Quantifying

    Let's expand our email address pattern and make it possible to match on any of these examples:

    good_emails = [
        'joe@apex.com',
        'joe.wilson@apex.com',
        'joe.wilson@eng.apex.com',
        'joe.g.zebulon.wilson@my.subdomain.eng.apex.com'
    ]

    And let's make sure our regex fails on any of these:

    bad_emails = [
        '.joe@apex.com',          # leading period
        'joe.wilson@apex.com.',   # trailing period
        'joe..wilson@apex.com'    # two periods together
    ]

    How can we include the period while making sure it doesn't appear at the start or end, or repeated, as it does in the bad_emails list?


    Look for a repeating pattern of groups of characters in the good_emails. In these combinations, we are attempting to account for subdomains, which could conceivably be chained togtehter. In this case, there is a pattern joe., that we can match with \w+\. (a period, the wildcard, must be escaped). If we see that this may repeat, we can group the pattern and apply a quantifier to it:

    import re
    for address in good_emails + bad_emails:                # concatenates two lists
        if re.search(r'^(\w+\.)*\w+@(\w+\.)+[A-Za-z]{2,}$', address):
            print("{0}:  good".format(address))
        else:
            print("{0}:  bad".format(address))

    Grouping for Extraction: the matchobject group() method.

    We use the group() method of the match object to extract the text that matched the group.


    Here's an example, using our log file. What if we wanted to capture the last two numbers (the status code and the number of bytes served), and place the values into structures?

    log_lines = [
    '66.108.19.165 - - [09/Jun/2003:19:56:33 -0400] "GET /~jjk265/cd.jpg HTTP/1.1" 200 175449',
    '216.39.48.10 - - [09/Jun/2003:19:57:00 -0400] "GET /~rba203/about.html HTTP/1.1" 200 1566',
    '216.39.48.10 - - [09/Jun/2003:19:57:16 -0400] "GET /~dd595/frame.htm HTTP/1.1" 400 1144'
    ]
    
    import re
    bytes_sum = 0
    for line in log_lines:
      matchobj = re.search(r'(\d+) (\d+)$', line) # last two numbers in line
      status_code = matchobj.group(1)
      bytes = matchobj.group(2)
      bytes_sum += int(bytes)                     # sum the bytes

    groups()

    groups() returns all grouped matches.


    If you wish to grab all the matches into a tuple rather than call them by number, use groups(). You can then read variables from the tuple, or assign groups() to named variables.


    In this example, the Salesforce Customer Relationship Management system has a field in one of its objects that discounts certain revenue and explains the reason. Our job is to extract the code and the reason from the string:

    import re
    my_GL_codes = [
        '12520 - Allowance for Customer Concessions',
        '12510 - Allowance for Unmet Mins',
        '40000 - Platform Revenue',
        '21130 - Pre-paid Revenue',
        '12500 - Allowance for Doubtful Accounts'
    ]
    for field in my_GL_codes:
        codeobj = re.search(r'^(\d+)\s*\-\s*(.+)$', field)      # GL code
        name_tuple = codeobj.groups()
        print('tuple from groups:  ', name_tuple)
        code, reason = name_tuple
        print("extracted:  '{0}':  '{1}'".format(code, reason))
        print()

    findall() for multiple matches

    findall() matches the same pattern repeatedly, returning all matched text within a string.


    findall() with a groupless pattern


    Usually re tries to a match a pattern once -- and after it finds the first match, it quits searching. But we may want to find as many matches as we can -- and return the entire set of matches in a list. findall() lets us do that:

    text = "There are seven words in this sentence";
    words = re.findall(r'\w+', text)
    print(words)  # ['There', 'are', 'seven', 'words', 'in', 'this', 'sentence']

    This program prints each of the words on a separate line. The pattern \b\w+\b is applied again and again, each time to the text remaining after the last match. This pattern could be used as a word counting algorithm (we would count the elements in words), except for words with punctuation. findall() with groups


    When a match pattern contains more than one grouping, findall returns multiple tuples:

    text = "High: 33, low: 17"
    temp_tuples = re.findall(r'(\w+):\s+(\d+)', text)
    print(temp_tuples)                       # [('High', '33'), ('low', '17')]

    re.sub() for substitutions

    re.sub() replaces matched text with replacement text.


    Regular expressions are used for matching so that we may inspect text. But they can also be used for substitutions, meaning that they have the power to modify text as well.


    This example replaces Microsoft '\r\n' line ending codes with Unix '\n'.

    text = re.sub(r'\r\n', '\n', text)

    Here's another simple example:

    string = "My name is David"
    string = re.sub('David', 'John', string)
    
    print(string)                            # 'My name is John'

    Practical: matching on an entire file string

    An entire file as a single string opens up additional matching possibilities.


    This example opens and reads a web page (which we might have retrieved with a module like urlopen), then looks to see if the word "advisory" appears in the text. If it does, it prints the page:


    file = open('weather-ny.html')
    text = file.read()
    if re.search(r'advisory', text, re.I):
      print("weather advisory:  ", text)
    

    re.MULTILINE: ^ and $ can match at start or end of line

    Within a file of many lines, we can specify start or end of a single line.


    We have been working with text files primarily in a line-oriented (or, in database terminology, record-oriented way, and regexes are no exception - most file data is oriented in this way. However, it can be useful to dispense with looping and use regexes to match within an entire file - read into a string variable with read().


    In this example, we surely can use a loop and split() to get the info we want. But with a regex we can grab it straight from the file in one line:

    # passwd file:
    nobody:*:-2:-2:Unprivileged User:/var/empty:/usr/bin/false
    root:*:0:0:System Administrator:/var/root:/bin/sh
    daemon:*:1:1:System Services:/var/root:/usr/bin/false

    # python script:
    import re
    passwd_text = open('/etc/passwd').read()
    mobj =  re.search(r'^root:[^:]+:[^:]+:[^:]:([^:]+):([^:]+)', passwd_text, re.MULTILINE)
    if mobj:
        info = mobj.groups()
        print("root:  Name %s, Home Dir %s" % (info[0], info[1]))

    We can even use findall to extract all the information rfrom a file - keep in mind, this is still being done in two lines:

    import re
    passwd_text = open('/etc/passwd').read()
    lot = re.findall(r'^(\w+):[^:]+:[^:]+:[^:]+:[^:]+:([^:]+)', passwd_text, re.MULTILINE)
    
    mydict = dict(lot)
    
    print(mydict)

    re.DOTALL -- allow the wildcard (.) to match on newlines

    Matching the wildcard on newlines may be needed for a multi-line file string.


    Normally, the wildcard doesn't match on newlines. When working with whole files, we may want to grab text that spans multiple lines, using a wildcard.


    # search file sample.txt
    some text we don't want
    ==start text==
    this is some text that we do want.
    the extracted text should continue,
    including just about any character,
    until we get to
    ==end text==
    other text we don't want

    # python script:
    import re
    text = open('sample.txt').read()
    matchobj = re.search(r'==start text==(.+)==end text==', text, re.DOTALL)
    print(matchobj.group(1))

    flags: re.IGNORECASE

    re.I provides a case-insensitive match.


    We can modify our matches with qualifiers called flags. The re.IGNORECASE flag will match any letters, whether upper or lowercase. In this example, extensions may be upper or lowercase - this file matcher doesn't care!


    import re
    dirlist = ('thisfile.jpg', 'thatfile.txt', 'otherfile.mpg', 'myfile.TXT')
    for file in dirlist:
      if re.search(r'\.txt$', file, re.IGNORECASE):   #'.txt' or '.TXT'
        print(file)

    The flag is passed as the third argument to search, and can also be passed to other re search methods.




    SQL Part 1: SQLite3, SELECT statements

    Starting the SQLite3 Client and Opening a Database Archive

    Most databases come with a tool for issuing SQL queries from a command prompt.


    Special Note for Windows Users: the Sqlite3 client must be installed. Surprisingly, the sqlite3 CLI (command-line interface) does not come with the Windows distribution of Anaconda Python, as it does with the Mac distribution.

    1. Download the client for windows with this link: https://www.sqlite.org/2018/sqlite-tools-win32-x86-3220000.zip
    2. Find and double-click the file; a "zip" window (looks like a regular Explorer file listing window) should open with a folder inside. Double-click this folder to open, which should reveal 3 files.
    3. Open a new Explorer window and navigate to C:\Users\[your username]\Anaconda3.
    4. Drag the 3 files from the other zip window opened above, into the Anaconda3 window. They should be copied into this window.
    5. Open an Anaconda prompt. In the search window, type cmd. One of the choices should be "Anaconda Prompt". Choose this.
    6. If you can't find the Anaconda prompt, you would need to change your PATH variable in order to conveniently work with sqlite3. The path to add will be: C:\Users\[your username]\Anaconda3 (i.e., the same path where you copied the sqlite3 files). The PATH value is a list of directories separated by semicolons. Follow the directions appropriate to your system: https://www.java.com/en/download/help/path.xml (Disregard the Java references -- just supply the path to Anaconda when changing the PATH variable) Note that this path may already be included in your PATH variable, depending on whether you added Anaconda to your PATH when you installed it. If it is there already, you do not need to add it again. Once you have added the Anaconda path to your PATH variable, open a regular command prompt by using "cmd" (no quotes) in the search or Run... open blank.


    Please let me know if you find anything missing from these instructions! Help others - be a good citizen - let me know! You can start the SQLite3 client in one of two ways:

    1. Open a new database archive: sqlite3 new.db
    2. Open an existing database archive: sqlite3 session_1.db
    3. Open an "in-memory" database (which can be saved later): sqlite3


    Note carefully that the syntax for opening a new file and opening an existing file are the same! This means that if you intend to open a new file but misspell the name, SQLite3 will simply create a new file; you'll then be confused to see that the file you thought you opened has nothing in it! Special Note: sqlite3 client column formatting


    At the start of your session, issue the following two commands -- these will format your sqlite3 output so it is clearer, and add columns headers.

    sqlite> .mode column
    sqlite> .headers on

    sqlite> SELECT * FROM revenue;
    
        # company     state       price
        # ----------  ----------  ----------
        # Haddad's    PA          239.5
        # Westfield   NJ          53.9
        # The Store   NJ          211.5
        # Hipster's   NY          11.98
        # Dothraki F  NY          5.98
        # Awful's     PA          23.95
        # The Clothi  NY          115.2

    RDBMS (Relational Database Management System) Tables

    A relational database stores data in tabular form, similar to CSV, Excel, etc.


    A Database Table is a tabular structure (like CSV) stored in binary form which can only be displayed through a database client or database driver (for a language like Python) The database client is provided by the database and allows command-prompt access to the database. Every database provides its own client. A database driver is a module that provides programmatic access to the database. Python has a full suite of drivers for most major databases (mariadb, oracle, postrgres, etc.) Database Tables


    A mysql database table description looks like this:

    +-------+--------+------+-----+---------+-------+
    | Field | Type   | Null | Key | Default | Extra |
    +-------+--------+------+-----+---------+-------+
    | date  | int(8) | YES  |     | NULL    |       |
    | mktrf | float  | YES  |     | NULL    |       |
    | hml   | float  | YES  |     | NULL    |       |
    | smb   | float  | YES  |     | NULL    |       |
    | rf    | float  | YES  |     | NULL    |       |
    +-------+--------+------+-----+---------+-------+

    "Field" is the column name. "Type" specifies the required data type for that column.


    In sqlite3 the database table description looks like this:

    CREATE TABLE ff_table (date INTEGER, mktrf FLOAT, hml FLOAT, smb FLOAT, rf FLOAT)

    The available database column types are defined by the specific database. Most are very similar, with small variations between databases.


    SQLite3 (file-based) and mariadb/mysql (traditional db) RDBMSs

    We might use SQLite3 to prototype our tables, and mariadb/mysql for production.


    mariadb (formerly known as mysql, can be colloquially referred to as either although the <) is a production-quality RDBMS that is free, and fully embraced by the industry. It supports most SQL statements and is very similar to Oracle, postgres, SQL Server, etc. SQLite is a file-based RDBMS that is extremely lightweight and requires no installation. It also supports many SQL statements although its data types are more limited. It is claimed to be the most-used database on the planet because it can work in very small environments (for example "internet of things" devices.) It is also ideal for learning. If you are new to databses, you are recommended to use SQLite for your coursework here. If you wish to try mysql/mariadb, you must install it and you'll need to do some research to complete some steps toward completing the homework. Basic Commands for SQLite3 and mysql/MariaDB

    SQLiteMariaDB/mysql


    start the command-prompt utility

    $ sqlite3


     

    $ mysql


    show existing databases

    (in SQLite, each file is a database,
    so opening a file (below) provides
    this connection)


     

    Maria DB []> show databases;


    connect to a database

    sqlite> .open sqlite3_trades.db


     

    Maria DB []> use trades;


    show tables in the database

    sqlite> .tables


     

    Maria DB [trades]> show tables;


    describe a table

    sqlite> .schema stocks


     

    Maria DB [trades]> desc stocks;


    select specified columns from the table

    sqlite> SELECT date, trans_type,
            symbol, qty FROM stocks;


     

    Maria DB [trades]> SELECT date, trans_type,
                       symbol, qty FROM stocks;


    select ALL columns from the table

    sqlite> SELECT * FROM stocks;


     

    Maria DB [trades]> SELECT * FROM stocks;


    select only rows WHERE a value is found

    sqlite> SELECT date, symbol, qty
            FROM stocks
            WHERE trans_type = 'BUY';


     

    Maria DB [trades]> SELECT date, symbol, qty
                       FROM stocks
                       WHERE trans_type = 'BUY';


    INSERT INTO: insert a row

    sqlite> INSERT INTO stocks
            (date, trans_type, symbol, qty)
            VALUES ('2015-01-01', 'BUY', 'MSFT', 10);


     

    Maria DB [trades]> INSERT INTO stocks
                       (date, trans_type, symbol, qty)
                       VALUES ('2015-01-01', 'BUY', 'MSFT', 10);


    connect to database from Python

    import sqlite3
    conn = sqlite3.connect('sqlite3_trades.db')
    
    c = conn.cursor()


     

    import pymysql
    
    host = 'localhost'
    database = 'test'
    port = 3306
    username = 'ta'
    password = 'pepper'
    
    conn = pymysql.connect(host=host,
                           port=port,
                           user=username,
                           passwd=password,
                           db=database)
    
    cur = conn.cursor()


    execute a SELECT query from Python

    c.execute('SELECT date, symbol, qty
               FROM stocks
               WHERE trans_type = 'BUY')


     

    cur.execute('SELECT date, symbol, qty
                 FROM stocks
                 WHERE trans_type = 'BUY')


    retrieve one row or many rows from a result set

    c.execute('SELECT * FROM stocks
                    ORDER BY price')
    
    tuple_row = c.fetchone()      # tuple row
    
    tuple_row = c.fetchmany(3)    # list of tuple rows - specify # of rows
    
    tuple_rows = c.fetchall()     # list of tuple rows - entire result set


     

    c.execute('SELECT * FROM stocks
                    ORDER BY price')
    
    tuple_row = c.fetchone()      # tuple row
    
    tuple_rows = c.fetchmany()    # list of tuple rows


    iterate over database result set

    c.execute('SELECT * FROM stocks
                    ORDER BY price')
    
    for tuple_row in c:
        print(tuple_row)


     

    for tuple_row in c:
        print(tuple_row)


    insert a row

    c.execute("INSERT INTO stocks
                    (date, trans_type, symbol, qty)
                    VALUES ('2015-01-01', 'BUY', 'MSFT', 10)")


     

    c.execute("INSERT INTO stocks
                    (date, trans_type, symbol, qty)
                    VALUES ('2015-01-01', 'BUY', 'MSFT', 10)")


    list common commands

    .help


     

    here are some mysql "important commands"




    sqlite3 from Python

    The sqlite3 module provides programmatic access to sqlite3 databases.


    Keep in mind that the interface you use for SQLite3 will be very similar to one that you use for other databases such as mysql/MariaDB.


    import sqlite3
    conn = sqlite3.connect('example.db')  # a db connection object
    
    # generate a cursor object
    c = conn.cursor()                     # a cursor object for issuing queries
    
    
    
    # execute change to database
    c.execute("INSERT INTO test.dbo.revenue VALUES ('Acme, Inc.', 'CA', 23.9)")
    
    # commit change
    conn.commit()
    
    
    
    # select data from database
    c.execute('SELECT * FROM test.dbo.revenue')
    
    
    # cursor.description:  database columns
    desc = c.description
    
    for field in desc:
        fname = field[0]
        ftype = field[1]
        print(f'{fname}:  {ftype}')
    
          ### id:  
          ### first_name:  
          ### last_name:  
          ### birthday:  
    
    
    # fetching options (customarily only one of these will be used)
    
    # loop through results
    for row in c:
        print(row)
    
    
    # fetch one row
    row = c.fetchone()
    print(row)
    
    
    # fetch several rows
    rows = c.fetchmany(3)
    print(rows)
    
    
    # fetch all rows
    rows = c.fetchall()
    print(rows)

    Connect to database; retrieve cursor object

    The connect string validates the user and specifies a host and database


    Connecting to Database


    The 'connect string' passed to connect() is the name of a database file. In other databases it would be a longer "connect string" that includes user and database information.

    import sqlite3
    conn = sqlite3.connect('example.db')  # a db connection object
    
    # generate a cursor object
    c = conn.cursor()                     # a cursor object for issuing queries
    


    Generating a cursor object


    A cursor object represents a request to the database. We will use the cursor object to execute queries and retrieve query results.

    c = conn.cursor()
    

    Note that sqlite3 datatypes are nonstandard and don't exactly match types found in databases such as mysql/MariaDB:

    INTEGERall int types (TINYINT, BIGINT, INT, etc.)
    REALFLOAT, DOUBLE, REAL, etc.
    NUMERICDECIMAL, BOOLEAN, DATE, DATETIME, NUMERIC
    TEXTCHAR, VARCHAR, etc.
    BLOBBLOB (non-typed (binary) data, usually large)


    Execute changes to database; commit changes

    Any change to database must be committed to be made permanent


    c.execute("INSERT INTO revenue VALUES ('Acme, Inc.', 'CA', 23.9)")
    
    conn.commit()

    Inserting values into query dynamically SQL requires quotes around text values and no quotes around numeric values. This can cause confusion or issues, especially when our text values have quotes in them.


    sqlite3 can be made to add the quotes for us, thus escaping any values as needed:

    name = "Joe's Fashions"   # the single quote could be a problem when inserted into an SQL statement
    state = 'NY'
    value = 1.09
    
    c.execute("INSERT INTO revenue VALUES (?, ?, ?)", (name, state, value))
    
    conn.commit()

    we can also insert multiple rows in one statement using this method:

    insert_rows = [ ("Joe's Fashions", 'NY', 1.09), ('Beta Corp', 'CA', 1.39) ]
    
    c.executemany("INSERT INTO test.dbo.revenue VALUES (?, ?, ?)", insert_rows)
    
    conn.commit()

    Select data from database; retrieve results

    Four options for retrieving result rows: fetchone(), fetchmany(), fetchall() and 'for' looping


    executing a select query


    c.execute('SELECT * FROM revenue')

    fetching options: depending on the size of the result set and how we'd like process the results, we have four options for retrieving results from the cursor object once a query has been executed.


    loop through result set: similar to file looping

    for row in c:
        print(row)       # returns a tuple row with each iteration

    retrieve one row: most appropriate when only one row is expected

    row = c.fetchone()  # returns a single tuple row

    fetch several rows: we may wish to process results in batches

    rows = c.fetchmany(3)  # returns a list of 3 tuple rows

    fetch all rows: appropriate only if result set is not very large

    rows = c.fetchall()   # returns a list of tuple rows

    cursor.description: database columns information This attribute of the cursor object contains a tuple structure that describes each column: its name and type, as well as a number of internal attributes


    desc = c.description
    
    for field in desc:
        fname = field[0]
        ftype = field[1]
        print(f'{fname}:  {ftype}')
    
          ### id:  
          ### first_name:  
          ### last_name:  
          ### birthday:  



    Reading Multidimensional Containers

    Introduction: Reading Multidimensional Containers

    Data can be expressed in complex ways using nested containers.


    Real-world data is often more complex in structure than a simple sequence (i.e., a list) or a collection of pairs (i.e. a dictionary).


    Complex data can be structured in Python through the use of multidimensional containers, which are simply containers that contain other containers (lists of lists, lists of dicts, dict of dicts, etc.) in structures of arbitrary complexity. Most of the time we are not called upon to handle structures of greater than 2 dimensions (lists of lists, etc.) although some config and data transmitted between systems (such as API responses) can go deeper. In this unit we'll look at the standard 2-dimensional containers we are more likely to encounter or want to build in our programs.


    Objectives for the Unit: Reading Multidimensional Containers


    Summary Structure: List of Lists

    A list of lists provides a "matrix" structure similar to an Excel spreadsheet.


    value_table =       [
                           [ '19260701', 0.09, -0.22, -0.30, 0.009 ],
                           [ '19260702', 0.44, -0.35, -0.08, 0.009 ],
                           [ '19260703', 0.17, 0.26,  -0.37, 0.009 ]
    
                        ]

    Probably used more infrequently, a list of lists allows us to access values through list methods (looping and indexed subscripts). The "outer" list has 3 items -- each items is a list, and each list represents a row of data. Each row list has 4 items, which represent the row data from the Fama-French file: the date, the Mkt-RF, SMB, HML and RF values. Looping through this structure would be very similar to looping through a delimited file, which after all is an iteration of lines that can be split into fields.


    for rowlist in value_table:
        print("the MktRF for {rowlist[0]} is {rowlist[1]}")

    Summary Structure: List of Dicts

    A list of dicts structures tabular rows into field-keyed dictionaries.


    value_table = [
                    { 'date': '19260701', 'MktRF': 0.09, 'SMB': -0.22,
                                          'HML': -0.30, 'RF': 0.009 },
    
                    { 'date': '19260702', 'MktRF': 0.44, 'SMB': -0.35,
                                          'HML': -0.08, 'RF': 0.009 },
    
                    { 'date': '19260706', 'MktRF': 0.17, 'SMB': 0.26,
                                          'HML': -0.37, 'RF': 0.009 }
                  ]

    The "outer" list contains 3 items, each being a dictionary with identical keys. The keys in each dict correspond to field / column labels from the table, so it's easy to identify and access a given value within a row dict.


    A structure like this might look elaborate, but is very easy to build from a data source. The convenience of named subscripts (as contrasted with the numbered subscripts of a list of lists) lets us loop through each row and name the fields we wish to access:

    for rowdict in value_table:
        print("the MktRF for {rowdict['date']} is {rowdict['MktRF']}")

    Summary Structure: Dict of Lists

    A dict of lists allows association of a sequence of values with unique keys.


    yr_vals = { '1926': [ 0.09,  0.44,  0.17, -0.15, -0.06,
                          -0.55,  0.61,  0.05, 0.51 ],
    
                '1927': [ -0.97,  0.30,  0.13, -0.18,  0.31,
                          0.39,  0.14, -0.27, 0.05 ],
    
                '1928': [ 0.43, -0.14, -0.71,  0.61,  0.13,
                          -0.88, -0.85,  0.12, 0.48 ]  }

    The "outer" dict contains 3 string keys, each associated with a list of float values -- in this case, the MktRF values from each of the trading days for each year (only the first 9 are included here for clarity). With a structure like this, we can perform calculations like those we have done on this data for a given year, namely to identify the max(), min(), sum(), average, etc. for a given year


    for year in yr_vals:
        print(f'for year {year}: ')
        print(f'  len: {len(yr_vals[year])}')
        print(f'  sum: {sum(yr_vals[year])}')
        print(f'  avg: {sum(yr_vals[year]) / len(yr_vals[year])}')

    Summary Structure: Dict of Dicts

    In a dict of dicts, each unique key points to another dict with keys and values.


    date_values = {
      '19260701':   { 'MktRF':  0.09,
                      'SMB':   -0.22,
                      'HML':   -0.30,
                      'RF':    0.009 },
      '19260702':   { 'MktRF':  0.44,
                      'SMB':   -0.35,
                      'HML':   -0.08,
                      'RF':    0.009 },
    }

    The "outer" dict contains string keys, each of which is associated with a dictionary -- each "inner" dictionary is a convenient key/value access to the fields of the table, as we had with a list of dicts.


    Again, this structure may seem complex (perhaps even needlessly so?). However, a structure like this is extremely easy to build and is then very convenient to query. For example, the 'HML' value for July 2, 1926 is accessed in a very visual way:

    print(date_values['19260702']['HML'])        # -0.08

    Looping through a dict of dicts is probably the most challenging part of working with multidimensional structures:


    x = {
        'a':  { 'zz': 1,
                  'yy': 2  },
          'b':  { 'zz': 5,
                  'yy': 10 }
        }
    
    x['a']['yy']  # 2
    
    
    for i in x:
      print(i)
      for j in x[i]:
        print(x[i][j], end=' ')                          # 1  2  5  10

    Summary Structure: arbitrary dimensions

    Containers can nest in "irregular" configurations, to accomodate more complex orderings of data.


    See if you can identify the object type and elements of each of the containers represented below:

    conf = [
      {
        "domain": "www.example1.com",
        "database": {
          "host": "localhost1",
          "port": 27017
        },
        "plugins": [
           "plugin1",
           "eslint-plugin-plugin1",
           "plugin2",
           "plugin3"
        ]
      },   # (additional dicts would follow this one in the list)
    ]

    Above we have a list with one item! The item is a dictionary with 3 keys. The "domain" key is associated with a string value. The "database" key is associated with another dictionary of string keys and values. The "plugins" key is associated with a list of strings. Presumably this "outer" list of dicts would have more than one item, and would be followed by additional dictionaries with the same keys and structure as this one.


    Summary Task: retrieving an "inner" element value

    Nested subscripts are the usual way to travel "into" a nested structure to obtain a value.


    A list of lists

    value_table =       [
                           [ '19260701', 0.09, -0.22, -0.30, 0.009 ],
                           [ '19260702', 0.44, -0.35, -0.08, 0.009 ],
                           [ '19260703', 0.17, 0.26,  -0.37, 0.009 ]
    
                        ]
    
    print(f"SMB for 7/3/26 is {value_table[2][2]}")

    A dict of dicts

    date_values = {
      '19260701':   { 'MktRF':  0.09,
                      'SMB':   -0.22,
                      'HML':   -0.30,
                      'RF':    0.009 },
      '19260702':   { 'MktRF':  0.44,
                      'SMB':   -0.35,
                      'HML':   -0.08,
                      'RF':    0.009 },
    }
    
    MktRF_thisday = date_values['19260701']['MktRF']   # value is 0.09
    
    print(date_values['19260701']['SMB'])               # -0.22
    print(date_values['19260701']['HML'])               # -0.3

    Summary Task: looping through a complex structure

    Looping through a nested structure often requires an "inner" loop within an "outer" loop.


    looping through a list of lists

    value_table =       [
                           [ '19260701', 0.09, -0.22, -0.30, 0.009 ],
                           [ '19260702', 0.44, -0.35, -0.08, 0.009 ],
                           [ '19260703', 0.17, 0.26,  -0.37, 0.009 ]
    
                        ]
    
    for row in value_table:
        print(f"MktRF for {row[0]} is {row[1]}")

    looping through a dict of dicts

    date_values = {
      '19260701':   { 'MktRF':  0.09,
                      'SMB':   -0.22,
                      'HML':   -0.30,
                      'RF':    0.009 },
      '19260702':   { 'MktRF':  0.44,
                      'SMB':   -0.35,
                      'HML':   -0.08,
                      'RF':    0.009 },
    }
    
    for this_date in date_values:
        print(f"MktRF for {this_date} is {date_values[this_date]['MktRF']}")

    Summary Function: pprint

    pprint() prints a complex structure in readable format.


    import pprint
    
    dvs = {'19260701': {'HML': -0.3, 'RF': 0.009, 'MktRF': 0.09, 'SMB': -0.22},
    '19260702': {'HML': -0.08, 'RF': 0.009, 'MktRF': 0.44, 'SMB': -0.35}}
    
    pprint.pprint(dvs)
    
       ### {'19260701': {'HML': -0.3, 'MktRF': 0.09, 'RF': 0.009, 'SMB': -0.22},
       ###  '19260702': {'HML': -0.08, 'MktRF': 0.44, 'RF': 0.009, 'SMB': -0.35}}

    Lightweight data storage and transfer with JSON

    JavaScript Object Notation is a simple "data interchange" format for sending structured data through text.


    Structured simply means that the data is organized into standard programmatic containers (lists and dictionaries). In fact, JSON uses the same notation as Python (and vice versa) so it is immediately recognizable to us. Once data is loaded from JSON, it takes the form of a standard Python multidimensional structure.


    Here is some simple JSON with an arbitrary structure, saved into a file called mystruct.json:

    {
       "key1":  ["a", "b", "c"],
       "key2":  {
                  "innerkey1": 5,
                  "innerkey2": "woah"
                },
       "key3":  55.09,
       "key4":  "hello"
    }

    Initializing a Python structure read from JSON


    We can load this structure from a file or read it from a string:

    import json
    
    fh = open('mystruct.json')          # open file in 'binary' mode
    mys = json.load(fh)                 # load from a file
    fh.close()
    
    fh = open('mystruct.json')
    file_text = fh.read()
    
    mys = json.loads(file_text)         # load from a string
    
    fh.close()
    
    print(mys['key2']['innerkey2'])     # woah

    Note: although it resembles Python structures, JSON notation is slightly less forgiving than Python -- for example, double quotes are required around strings, and no trailing comma is allowed after the last element in a dict or list (Python allows this).


    For example, I added a comma to the end of the outer dict in the example above:

      "key4":  "hello",

    When I then tried to load it, the json module complained with a helpfully correct location:

    ValueError: Expecting property name: line 9 column 1 (char 164)

    Dumping a Python structure to JSON


    json.dumps(['foo', {'bar': ('baz', None, 1.0, 2)}])
    
    json.dump(['streaming API'], io)



    csv module

    Importing Python Modules

    A module is Python code (a code library) that we can import and use in our own code -- to do specific types of tasks.


    As an example, the datetime module provides convenient handling of dates.

    import datetime           # make datetime (a library module) part of our code
    
    dt = datetime.date.today()           # generate a new date object (dt)
    print(dt)                            # prints today date in YYYY-MM-DD format
    dt = dt + datetime.timedelta(days=1)
    print(dt)                            # prints tomorrow's date

    Once a module is imported, its Python code is made available to our code. We can then call specialized functions and use objects to accomplish specialized tasks. Python's module support is profound and extensive. Modules can do powerful things, like manipulate image or sound files, munge and process huge blocks of data, do statistical modeling and visualization (charts) and much, much, much more. The Python 3 Standard Library documentation can be found at https://docs.python.org/3/library/index.html


    CSV

    The CSV module parses CSV files, splitting the lines for us. We read the CSV object in the same way we would a file object.


    import csv
    
    
    # open a file for reading
    fh = open('students.txt')
    
    # pass the 'read' file object to the csv reader constructor
    reader = csv.reader(fh)
    
    header_line = next(fh)    # return one row (useful to capture or skip header lines)
    print(header_line)        # 'id,first_name,last_name'
    
    
    for record in reader:     # loop through each row
    
        print(record)         # ['123', 'Joe', 'Wilson']
    
        print(f'id:{record[0]}')
        print(f'first name:  {record[1]}')
        print(f'last name:  {record[2]}')

    This module takes into account more advanced CSV formatting, such as quotation marks (which are used to allow commas within data.)


    Writing is similarly easy:

    import csv
    
    
    # open a file for writing (note 'w')
    wfh = open('some.csv', 'w', newline='')       # "newline='' fixes a Windows issue"
    
    # pass the 'write' file object to the csv writer constructor
    writer = csv.writer(wfh)
    
    
    # write one row
    writer.writerow(['some', 'values', "boy, don't you like long field values?"])
    
    
    # write multiple rows
    writer.writerows([['a', 'b', 'c'], ['d', 'e', 'f'], ['g', 'h', 'i']])
    
    # always remember to close a write file handle or
    # you may not see the change in the file
    wfh.close()
    
    # Again, please be advised that you will not see writes to a file
    # until you close the file with fh.close() or until the program
    # ends execution.  This is particularly critical with Jupyter as
    # it is basically a running program.



    Jinja2 Templating

    Jinja2: Getting Started

    Jinja2 is a module for inserting data into templates.


    Last week we used the simple .format() method to insert "dynamic" data (that is, variables that may be any value) into "static" templates (text that does not change). Jinja2 offers a full-featured templating system, including the following features:


    Here is a basic example showing these features:


    test.html, stored the test_templates/ directory

    Hello, {{ name }}.
    
    Please say it loudly:  {{ compliment.upper() }}!
    
    Must I tell you:
    {% for item in pronouncements %}
       {{ item }}
    {% endfor %}
    
    Or {{ pronouncements[0] }}?
    
    {% if not reconciled: %}
      We have work left to do.
    {% else %}
      I'm glad we worked that out.
    {% endif %}

    test.py, in the same dir as test_templates

    import jinja2
    
    env = jinja2.Environment()
    env.loader = jinja2.FileSystemLoader('test_templates')
    
    template = env.get_template('test.html')
    
    print(template.render(name='Joe', compliment="you're great",
                          pronouncements=['over', 'over', 'over again'],
                          reconciled=False))

    The rendered template!

    Hello, Joe.
    
    Please say it loudly:  YOU'RE GREAT!
    
    Must I tell you:
    
       over
       over
       and over again
    
    Or over?
    
    
      We have work left to do.



    Building Multidimensional Containers

    Multidimensional structures - building

    Usually, we don't initialize multi-dimensional structures within our code. Sometimes one will come to us, as with dict.items(), which returns a list of tuples. Database results also come as a list of tuples.


    Most commonly, we will build a multi-dimensional structure of our own design based on the data we are trying to store. For example, we may use the Fama-French file to build a dictionary of lists - the key of the dictionary being the date, and the value being a 4-element list of the values for that date.

    outer_dict = {}                            # new dict
    for line in open('FF_abbreviated.txt').read().splitlines():
        columns = line.split()                 # split each line into
                                               # a list of string values
    
        date = columns[0]                      # the first value is the date
        values = columns[1:]                   # slice this list into a
                                               # list of floating-point
                                               # values
    
        outer_dict[date] = values              # so values is a
                                               # list, assigned as value
                                               # to key date

    Summary Structure: List of Lists

    A list of lists provides a "matrix" structure similar to an Excel spreadsheet.


    value_table =       [
                           [ '19260701', 0.09, -0.22, -0.30, 0.009 ],
                           [ '19260702', 0.44, -0.35, -0.08, 0.009 ],
                           [ '19260703', 0.17, 0.26,  -0.37, 0.009 ]
    
                        ]

    Probably used more infrequently, a list of lists allows us to access values through list methods (looping and indexed subscripts). The "outer" list has 3 items -- each items is a list, and each list represents a row of data. Each row list has 4 items, which represent the row data from the Fama-French file: the date, the Mkt-RF, SMB, HML and RF values. Building this structure is simplified in that the 'inner' list is produced from the split():


    outer_list = []                            # new list
    
    for line in open('FF_abbreviated.txt').read().splitlines():
    
        columns = line.split()                 # split row into list of strings
        outer_list.append(columns)             # append row list to outer list

    Summary Structure: List of Dicts

    A list of dicts structures tabular rows into field-keyed dictionaries.


    value_table = [
                    { 'date': '19260701', 'MktRF': 0.09, 'SMB': -0.22,
                                          'HML': -0.30, 'RF': 0.009 },
    
                    { 'date': '19260702', 'MktRF': 0.44, 'SMB': -0.35,
                                          'HML': -0.08, 'RF': 0.009 },
    
                    { 'date': '19260706', 'MktRF': 0.17, 'SMB': 0.26,
                                          'HML': -0.37, 'RF': 0.009 }
                  ]

    The "outer" list contains 3 items, each being a dictionary with identical keys. The keys in each dict correspond to field / column labels from the table, so it's easy to identify and access a given value within a row dict.


    Building this structure simply means constructing a new dict for each row. This does not have to be done piecemeal, but as a single initialized dictionary:

    outer_list = []                            # new list
    
    for line in open('FF_abbreviated.txt').read().splitlines():
    
        columns = line.split()                 # split row into list of strings
    
        inner_dict = { 'date':  columns[0],
                       'MktRF': columns[1],
                       'SMB':   columns[2],
                       'HML':   columns[3],
                       'RF':    columns[4]  }
    
        outer_list.append(inner_dict)

    Summary Structure: Dict of Dicts

    In a dict of dicts, each unique key points to another dict with keys and values.


    date_values = {
      '19260701':   { 'MktRF':  0.09,
                      'SMB':   -0.22,
                      'HML':   -0.30,
                      'RF':    0.009 },
      '19260702':   { 'MktRF':  0.44,
                      'SMB':   -0.35,
                      'HML':   -0.08,
                      'RF':    0.009 },
    }

    The "outer" dict contains string keys, each of which is associated with a dictionary -- each "inner" dictionary is a convenient key/value access to the fields of the table, as we had with a list of dicts.


    Similar to the list of dicts, we can very simply build an 'inner' dict inside the loop and associate it with a given key:

    outer_dict = {}                            # new dict
    
    for line in open('FF_abbreviated.txt').read().splitlines():
    
        columns = line.split()                 # split row into list of strings
    
        inner_dict = { 'MktRF': columns[1],
                       'SMB':   columns[2],
                       'HML':   columns[3],
                       'RF':    columns[4]  }
    
        outer_dict[columns[0]] = inner_dict

    Summary Structure: Dict of Lists

    A dict of lists allows association of a sequence of values with unique keys.


    value_table = {  '1926': [  0.09,  0.44,  0.17, -0.15, -0.06, -0.55,
                                0.61,  0.05, 0.51 ],
    
                     '1927': [  -0.97,  0.30,  0.13, -0.18,  0.31,  0.39,
                                0.14, -0.27, 0.05 ],
    
                     '1928': [  0.43, -0.14, -0.71,  0.61,  0.13, -0.88,
                                -0.85,  0.12, 0.48 ]                       }

    The "outer" dict contains 3 string keys, each associated with a list of float values -- in this case, the MktRF values from each of the trading days for each year (only the first 9 are included here for clarity). To produce a structure like this we need only consider the counting or summing dictionary. This structure associates a list with each key, so instead of summing a new value to the current value associated with each key, we can append the new value to the list associated with the key.


    Writing to JSON

    To write to JSON we simply open a file for writing and dump the structure to the file.


    Here is some simple JSON with an arbitrary structure:

    struct = {
               "key1":  ["a", "b", "c"],
               "key2":  {
                          "innerkey1": 5,
                          "innerkey2": "woah"
                        },
               "key3":  55.09,
               "key4":  "hello"
            }

    We can 'dump' this structure to a file directly, or to a string:

    fh = open('mystruct.json' ,'w')
    mys = json.dump(struct, fh)                # write struct to a file
    fh.close()
    
    
    struct_str = json.dumps(struct)            # write struct to a string
    
    fh = open('mystruct.json', 'w')
    fh.write(struct_str)                       # write string to the file
    fh.close()



    Functions and Proper Code Organization

    Summary Statement: Functions (user-defined)

    A user-defined function is simply a named code block that can be called and executed any number of times.


    def print_hello():
        print("Hello, World!")
    
    print_hello()             # prints 'Hello, World!'
    print_hello()             # prints 'Hello, World!'
    print_hello()             # prints 'Hello, World!'

    Function argument(s)


    A function's arguments are renamed in the function definition, and the function refers to them by these names.

    def print_hello(greeting, person):
      full_greeting = greeting + ", " + person + "!"
      print(full_greeting)
    
    print_hello('Hello', 'World')         # prints 'Hello, World!'
    print_hello('Bonjour', 'Python')      # prints 'Bonjour, Python!'
    print_hello('squawk', 'parrot')       # prints 'squawk, parrot!'

    (The argument objects are copied to the argument names -- they are the same objects.) Function return value


    A function's return value is passed back from the function with the return statement.

    def print_hello(greeting, person):
      full_greeting = greeting + ", " + person + "!"
      return full_greeting
    
    msg = print_hello('Bonjour', 'parrot')
    print(msg)                                       # 'Bonjour, parrot!'

    Coding all Python scripts as modules

    All Python scripts should be coded as modules: able to be both imported and executed.


    #!/usr/bin/env python
    
    """ helloworld.py:  print the "Hello! message """
    
    PI = 3.14
    LOGFILE = 'mylog.txt'
    
    def print_hello(arg):
        """ greets the user """
        print(f"Hello, {arg}!")
    
    def logmsg(msg):
        wfh = open(LOGFILE, 'w')
        wfh.write(f'{msg}\n')
        wfh.close()
    
    def main():
        """ the main event """
        print_hello('World')
    
    if __name__ == '__main__':
        logmsg('starting...')
        main()
        logmsg('ending...')
    

    If we run the above script, we'll see "Hello, World!". But if we import the above script, we won't see anything. Why? Because the if statement above will be true only if the script is executed. This is important behavior, because there are some scripts that we may want to run directly, but also allow others to import (in order to use the script's functions). Whether we intend to import a script or not, it is considered a "best practice" to build all of our programs in this way -- with a "main body" of statements collected under function main() and the call to main() inside the if __name__ == '__main__' gate.


    Raising exceptions

    Causing an exception to be raised is the principal way to signal errors from a function.


    A file called mylib.py

    def get_yearsum(user_year):
    
        user_year = int(user_year)
        if user_year < 1929 or user_year > 2013:
          raise ValueError(f'year {user_year} out of range')
    
        # calculate value for the year
    
        return 5.9         # returning a sample value (for testing purposes only)

    An exception raised by us is indistinguishable from one raised by Python, and we can raise any exception type we wish. This allows the user of our function to handle the error if needed (rather than have the script fail):


    import mylib
    
    while True:
    
        year = input('please enter a year:  ')
    
        try:
            mysum = mylib.get_yearsum(year)
            break
        except ValueError:
            print('invalid year:  try again')
    
    print('mysum is', mysum)



    Attribute Access in Classes and Instances

    getattr() and setattr()

    These built-in functions allow attribute access through string arguments.


    class This(object):       # a simple class with one class variable
        a = 5
    
    x = This()
    
    print(getattr(x, 'a'))     # 5:  finds the 'a' attribute in the class
    
    setattr(x, 'b', 10)       #     set x.b = 10 in the instance
    
    print(x.b)                 # 10: retrieve x.b from the instance

    Similar to the dict method get(), a default argument can be passed to getattr() to return a value if the attribute doesn't exist (otherwise, a missing attribute will raise an AttributeError exception).

    class InstMake(object):        # create a featureless class so we can
                                   # play with its instance
        pass
    
    x = InstMake()
    
    curr_val = getattr(x, 'intval', 0)   # no 'intval' attribute, so default 0
    setattr(x, 'intval', 10)             # set 'intval' to 10
    
    print(x.__dict__)                     # {'intval': 10}

    We might want to use these functions as a dispatch utility: if our program is working with a string value that is the name of an attribute, we can use the string directly.

    var = 'hElLo'
    for methodname in ('upper', 'lower', 'title'):
        print(getattr(var, methodname)())            # call each method through the attribute
                                                    # HELLO
                                                    # hello
                                                    # Hello

    Special methods __getattr__, __getattribute__ and __setattr__

    These special methods are called when we access an attribute (setting or getting). We can implement them for broad control over our custom class' attributes.


    class MyClass(object):
    
        classval = 5
    
        # when read attribute does not exist
        def __getattr__(self, name):
            default = 0
            print('getattr: "{}" not found; setting default'.format(name))
            setattr(self, name, default)
            return default
    
        # when attribute is read
        def __getattribute__(self, name):
            print('getattribute:  attempting to access "{}"'.format(name))
            return object.__getattribute__(self, name)
    
        # when attribute is assigned
        def __setattr__(self, name, value):
            print('setattr:  setting "{}" to value "{}"'.format(name, value))
            self.__dict__[name] = value
    
    x = MyClass()
    
    x = MyClass()
    
    x.a = 5             # setattr:  setting "a" to value "5"
    
    print(x.a)           # getattribute:  attempting to access "__dict__"
                        # getattribute:  attempting to access "a"
                        # 5
    
    print(x.ccc)         # getattribute:  attempting to access "ccc"
                        # getattr: "ccc" not found; setting default
                        # setattr:  setting "ccc" to value "0"
                        # getattribute:  attempting to access "__dict__"
                        # 0

    __getattribute__: implicit call upon attribute read. Anytime we attempt to access an attribute, Python calls this method if it is implemented in the class. __getattr__: implicit call for non-existent attribute. If an attribute does not exist, Python calls this method -- regardless of whether it called __getattribute__. recursion alert: we must use alternate means of getting or setting attributes lest Python call these methods repeatedly: __getattribute__(): use object.__getattribute__(self, name) __setattr__(): use self.__dict__[name] = value e.g., use of self.attr = val in __setattr__ would cause the method to call itself.


    @property: attribute control

    This decorator allows behavior control when an individual attribute is accessed, through separate @property, @setter and @deleter methods.


    class GetSet(object):
    
        def __init__(self,value):
            self.attrval = value
    
        @property
        def var(self):
            print('thanks for calling me -- returning {}'.format(self.attrval))
            return self.attrval
    
        @var.setter
        def var(self, value):
            print("thanks for setting me -- setting 'var' to {}".format(value))
            self.attrval = value
    
        @var.deleter
        def var(self):
            print('should I thank you for deleting me?')
            self.attrval = None
    
    me = GetSet(5)
    
    me.var = 1000    # setting the "var" attribute
    
    print(me.var)     # thanks for calling me -- returning 1000
                     # 1000
    
    del me.var       # should I thank you for deleting me?
    print(me.var)     # thanks for calling me -- returning None
                     # None
    

    Note that each decorated method is called def var. This would cause conflicts if it weren't for the decorators.


    One caveat: since the interface for attribute access appears very simple, it can be misleading to attach computationally expensive operations to an attribute decorated with @property.


    Attribute access: descriptors

    A descriptor is an attribute that is linked to a separate class that defines __get__(), __set__() or __delete__().


    class RevealAccess(object):
        """ A data descriptor that sets and returns values and prints a message declaring access. """
    
        def __init__(self, initval=None):
            self.val = initval
    
        def __get__(self, obj, objtype):
            print('Getting attribute from object', obj)
            print('...and doing some related operation that should take place at this time')
            return self.val
    
        def __set__(self, obj, val):
            print('Setting attribute from object', obj)
            print('...and doing some related operation that should take place at this time')
            self.val = val
    
    
    # the class we will work with directly
    class MyClass(object):
        """ A simple class with a class variable as descriptor """
        def __init__(self):
            print('initializing object ', self)
    
        x = RevealAccess(initval=0)  # attach a descriptor to class attribute 'x'
    
    
    mm = MyClass()                   # initializing object  <__main__.MyClass object at 0x10066f7d0>
    
    mm.x = 5                         # Setting attribute from object <__main__.MyClass object at 0x1004de910>
                                     # ...and doing some related operation that should take place at this time
    
    val = mm.x                       # Getting attribute from object <__main__.MyClass object at 0x1004de910>
                                     # ...and doing some related operation that should take place at this time
    
    print('retrieved value: ', val)   # retrieved value:  5

    You may observe that descriptors behave very much like the @property decorator. And it's no coincidence: @property is implemented using descriptors.


    __slots__

    This class variable causes object attributes to be stored in a specially designated space rather than in a dictionary (as is customary).


    class MyClass(object):
        __slots__ = ['var', 'var2', 'var3']
    
    a = MyClass()
    
    a.var = 5
    a.var2 = 10
    a.var3 = 20
    a.var4 = 40   # AttributeError:  'MyClass' object has no attribute 'var4'

    All objects store attributes in a designated dictionary under the attribute __dict__. This takes up a fairly large amount of memory space for each object created. __slots__, initialized as a list and as a class variable, causes Python not to create an object dictionary; instead, just enough memory needed for the attributes is allocated. When many instances are being created a marked improvement in performance is possible. The hotel rating website Oyster.com reported a problem solution in which they reduced their memory consumption by a third by using __slots__. Please note however that slots should not be used to limit the creation of attributes. This kind of control is considered "un-pythonic" in the sense that privacy and control are mostly cooperative schemes -- a user of your code should understand the interface and not attempt to subvert it by establishing unexpected attributes in an object.




    Classes

    Introduction: Classes

    Classes allow us to create a custom type of object -- that is, an object with its own behaviors and its own ways of storing data. Consider that each of the objects we've worked with previously has its own behavior, and stores data in its own way: dicts store pairs, sets store unique values, lists store sequential values, etc. An object's behaviors can be seen in its methods, as well as how it responds to operations like subscript, operators, etc. An object's data is simply the data contained in the object or that the object represents: a string's characters, a list's object sequence, etc.


    Objectives for this Unit: Classes

  • Understand what classes, objects and attributes are and why they are useful
  • Create our own classes -- our own object types
  • Set attributes in objects and read attributes from objects
  • Define methods in classes that can be used by objects
  • Define object initializers with __init__()
  • Use getter and setter methods to enforce encapsulation
  • Understand class inheritance
  • Understand polymorphism


    Class Example: the date and timedelta object types

    First let's look at object types that demonstrate the convenience and range of behaviors of objects.


    A date object can be set to any date and knows how to calculate dates into the future or past. To change the date, we use a timedelta object, which can be set to an "interval" of days to be added to or subtracted from a date object.


    from datetime import date, timedelta
    
    dt = date(1926, 12, 30)         # create a new date object set to 12/30/1926
    td = timedelta(days=3)          # create a new timedelta object:  3 day interval
    
    dt = dt + timedelta(days=3)     # add the interval to the date object:  produces a new date object
    
    print(dt)                        # '1927-01-02' (3 days after the original date)
    
    
    dt2 = date.today()              # as of this writing:  set to 2016-08-01
    dt2 = dt2 + timedelta(days=1)   # add 1 day to today's date
    
    print(dt2)                       # '2016-08-02'
    
    print(type(dt))                  # <type 'datetime.datetime'>
    print(type(td))                  # <type 'datetime.timedelta'>

    Class Example: the proposed server object type

    Now let's imagine a useful object -- this proposed class will allow you to interact with a server programmatically. Each server object represents a server that you can ping, restart, copy files to and from, etc.


    import time
    from sysadmin import Server
    
    
    s1 = Server('blaikieserv')
    
    if s1.ping():
        print('{} is alive '.format(s1.hostname))
    
    s1.restart()                       # restarts the server
    
    s1.copyfile_up('myfile.txt')       # copies a file to the server
    s1.copyfile_down('yourfile.txt')   # copies a file from the server
    
    print(s1.uptime())                  # blaikieserv has been alive for 2 seconds

    A class block defines an object "factory" which produces objects (instances) of the class.

    Method calls on the object refer to functions defined in the class.


    class Greeting(object):
        """ greets the user """
    
        def greet(self):
            print('hello, user!')
    
    
    c = Greeting()
    
    c.greet()                    # hello, user!
    
    print(type(c))                # <class '__main__.Greeting'>

    Each class object or instance is of a type named after the class. In this way, class and type are almost synonymous.


    Each object holds an attribute dictionary

    Data is stored in each object through its attributes, which can be written and read just like dictionary keys and values.


    class Something(object):
        """ just makes 'Something' objects """
    
    obj1 = Something()
    obj2 = Something()
    
    obj1.var = 5             # set attribute 'var' to int 5
    obj1.var2 = 'hello'      # set attribute 'var2' to str 'hello'
    
    obj2.var = 1000          # set attribute 'var' to int 1000
    obj2.var2 = [1, 2, 3, 4] # set attribute 'var2' to list [1, 2, 3, 4]
    
    
    print(obj1.var)           # 5
    print(obj1.var2)          # hello
    
    print(obj2.var)           # 1000
    print(obj2.var2)          # [1, 2, 3, 4]
    
    obj2.var2.append(5)      # appending to the list stored to attribute var2
    
    print(obj2.var2)          # [1, 2, 3, 4, 5]

    In fact the attribute dictionary is a real dict, stored within a "magic" attribute of the object:

    print(obj1.__dict__)      # {'var': 5, 'var2': 'hello'}
    
    print(obj2.__dict__)      # {'var': 1000, 'var2': [1, 2, 3, 4, 5]}

    The class also holds an attribute dictionary

    Data can also be stored in a class through class attributes or through variables defined in the class.


    class MyClass():
        """ The MyClass class holds some data """
    
        var = 10              # set a variable in the class (a class variable)
    
    
    MyClass.var2 = 'hello'    # set an attribute directly in the class object
    
    print(MyClass.var)         # 10      (attribute was set as variable in class block)
    print(MyClass.var2)        # 'hello' (attribute was set as attribute in class object)
    
    print(MyClass.__dict__)    # {'var': 10,
                              #  '__module__': '__main__',
                              #  '__doc__': ' The MyClass class holds some data ',
                              #  'var2': 'hello'}

    The additional __module__ and __doc__ attributes are automatically added -- __module__ indicates the active module (here, that the class is defined in the script being run); __doc__ is a special string reserved for documentation on the class).


    object.attribute lookup tries to read from object, then from class

    If an attribute can't be found in an object, it is searched for in the class.


    class MyClass(object):
      classval = 10         # class attribute
    
    a = MyClass()
    b = MyClass()
    
    b.classval = 99         # instance attribute of same name
    
    print(a.classval)        # 10 - still class attribute
    print(b.classval)        # 99 - instance attribute
    
    del b.classval          # delete instance attribute
    
    print(b.classval)        # 10 -- now back to class attribute

    Method calls pass the object as first (implicit) argument, called self

    Object methods or instance methods allow us to work with the object's data.


    class Do(object):
        def printme(self):
            print(self)      # <__main__.Do object at 0x1006de910>
    
    x = Do()
    
    print(x)                 # <__main__.Do object at 0x1006de910>
    x.printme()

    Note that x and self have the same hex code. This indicates that they are the very same object.


    Instance methods / object methods and object attributes: changing object "state"

    Since instance methods pass the object, and we can store values in object attributes, we can combine these to have a method modify an object's values.


    class Sum(object):
        def add(self, val):
            if not hasattr(self, 'x'):
                self.x = 0
            self.x = self.x + val
    
    myobj = Sum()
    myobj.add(5)
    myobj.add(10)
    
    print(myobj.x)      # 15

    Objects are often modified using getter and setter methods

    These methods are used to read and write object attributes in a controlled way.


    class Counter(object):
        def setval(self, val):     # arguments are:  the instance, and the value to be set
            if not isinstance(val, int):
                raise TypeError('arg must be a string')
    
            self.value = val        # set the value in the instance's attribute
    
        def getval(self):          # only one argument:  the instance
            return self.value       # return the instance attribute value
    
        def increment(self):
            self.value = self.value + 1
    
    a = Counter()
    b = Counter()
    
    a.setval(10)       # although we pass one argument, the implied first argument is a itself
    
    a.increment()
    a.increment()
    
    print(a.getval())   # 12
    
    
    b.setval('hello')  # TypeError

    __init__() is automagically called when a new instance is created

    The initializer of an object allows us to set the initial attribute values of the object.


    class MyCounter(object):
      def __init__(self, initval):   # self is implied 1st argument (the instance)
        try:
          initval = int(initval)     # test initval to be an int,
        except ValueError:           # set to 0 if incorrect
          initval = 0
        self.value = initval         # initval was passed to the constructor
    
      def increment_val(self):
        self.value = self.value + 1
    
      def get_val(self):
        return self.value
    
    a = MyCounter(0)
    b = MyCounter(100)
    
    a.increment_val()
    a.increment_val()
    a.increment_val()
    
    b.increment_val()
    b.increment_val()
    
    print(a.get_val())    # 3
    print(b.get_val())    # 102

    Classes can be organized into an an inheritance tree

    When a class inherits from another class, attribute lookups can pass to the parent class when accessed from the child.


    class Animal(object):
      def __init__(self, name):
        self.name = name
      def eat(self, food):
        print('{} eats {}'.format(self.name, food))
    
    class Dog(Animal):
      def fetch(self, thing):
        print('{} goes after the {}!'.format(self.name, thing))
    
    class Cat(Animal):
      def swatstring(self):
        print('{} shreds the string!'.format(self.name))
      def eat(self, food):
        if food in ['cat food', 'fish', 'chicken']:
          print('{} eats the {}'.format(self.name, food))
        else:
          print('{}:  snif - snif - snif - nah...'.format(self.name))
    
    d = Dog('Rover')
    c = Cat('Atilla')
    
    d.eat('wood')                 # Rover eats wood.
    c.eat('dog food')             # Atilla:  snif - snif - snif - nah...

    Conceptually similar methods can be unified through polymorphism

    Same-named methods in two different classes can share a conceptual similarity.


    class Animal(object):
      def __init__(self, name):
        self.name = name
      def eat(self, food):
        print('{} eats {}'.format(self.name, food))
    
    class Dog(Animal):
      def fetch(self, thing):
        print('{} goes after the {}!'.format(self.name, thing))
      def speak(self):
        print('{}:  Bark!  Bark!'.format(self.name))
    
    class Cat(Animal):
      def swatstring(self):
        print('{} shreds the string!'.format(self.name))
      def eat(self, food):
        if food in ['cat food', 'fish', 'chicken']:
          print('{} eats the {}'.format(self.name, food))
        else:
          print('{}:  snif - snif - snif - nah...'.format(self.name))
      def speak(self):
        print('{}:  Meow!'.format(self.name))
    
    for a in (Dog('Rover'), Dog('Fido'), Cat('Fluffy'), Cat('Precious'), Dog('Rex'), Cat('Kittypie')):
      a.speak()
    
                       # Rover:  Bark!  Bark!
                       # Fido:  Bark!  Bark!
                       # Fluffy:  Meow!
                       # Precious:  Meow!
                       # Rex:  Bark!  Bark!
                       # Kittypie:  Meow!

    Static Methods and Class Methods

    A class method can be called through the instance or the class, and passes the class as the first argument. We use these methods to do class-wide work, such as counting instances or maintaining a table of variables available to all instances. A static method can be called through the instance or the class, but knows nothing about either. In this way it is like a regular function -- it takes no implicit argument. We can think of these as 'helper' functions that just do some utility work and don't need to involve either class or instance.


    class MyClass(object):
    
      def myfunc(self):
        print("myfunc:  arg is {}".format(self))
    
      @classmethod
      def myclassfunc(klass):      # we spell it differently because 'class' will confuse the interpreter
        print("myclassfunc:  arg is {}".format(klass))
    
      @staticmethod
      def mystaticfunc():
        print("mystaticfunc: (no arg)")
    
    a = MyClass()
    
    a.myfunc()             # myfunc:  arg is <__main__.MyClass instance at 0x6c210>
    
    MyClass.myclassfunc()  # myclassfunc:  arg is __main__.MyClass
    a.myclassfunc()        # [ same ]
    
    a.mystaticfunc()       # mystaticfunc: (no arg)

    Here is an example from Learning Python, which counts instances that are constructed:


    class Spam:
    
      numInstances = 0
    
      def __init__(self):
        Spam.numInstances += 1
    
      @staticmethod
      def printNumInstances():
        print "instances created:  ", Spam.numInstances
    
    s1 = Spam()
    s2 = Spam()
    s3 = Spam()
    
    Spam.printNumInstances()        # instances created:  3
    s3.printNumInstances()          # instances created:  3



    File Parsing Algorithms

    File Looping and Counting

    An integer value set to 0 before a 'for' loop over a file can be used to count lines in the file.


    This algorithm uses a basic 'for' loop counter, which you can apply to any 'for' loop if you want to count the iterations. For a file, counting iterations means counting lines from the file. We might use this pattern to count lines that have a particular value, or only those lines that have data.


    the basic pattern is as follows:

    set counter to integer zero
    
    loop over lines in the file
        increment counter
    
    print or use the count integer

    This example counts the data lines - first we advance past the header line to the 2nd line, then begin counting:

    fh = open('revenue.csv')
    
    header_line = next(fh)   # advance the file pointer past
                             # the first line (only if needed)
    
    line_counter = 0
    
    for line in fh:
        line_counter = line_counter + 1     # 1 (first value)
    
    fh.close()
    
    print(f'{line_counter} lines')          # '7 lines'

    Of course if we're only interested in knowing how many lines are in the file, we can also read the file as a list of lines (e.j. with .readlines()) and measure the length of that list). A counter would be more appropriate if we weren't counting every line.


    File Looping and Summing

    An integer or float value that is set to 0 before a 'for' loop over a file, can be used to sum up values from the file.


    "Column summing" is a common and useful algorithm. It's also meaningful as an exercise in splitting out or selecting out values to summarize some of the data in a file.


    set summing variable to 0
    loop over lines in the file
        split or select out data from the line (e.g. a column value)
        add value to summing variable
    
    print or use the sum

    This example splits out the 3rd column value from each row, converts to float and adds the value to a float variable initialized before the loop begins:

    fh = open('revenue.csv')
    
    header_line = next(fh)   # advance the file pointer past
                             # the first line (only if needed)
    
    value_summer = 0.0       # set to a float because a
                             # float value is expected
    
                                            # first values:
    for line in fh:                         # "Haddad's,PA,239.5\n"
        line = line.rstrip()                # "Haddad's,PA,239.5"
        items = line.split(',')             # ["Haddad's", 'PA', '239.5']
        rev = items[2]                      # '239.5'
        frev = float(rev)                   # 239.5
    
        value_summer = value_summer + frev  # 239.5
    
    fh.close()
    
    print(f'sum of values in 3rd column:  {value_summer}')
    
                                            # 662.0100000000001
                                            # (note fractional value may differ)

    File Looping and Collecting

    An empty list or set initialized before a 'for' loop over a file can be used to collect values from the file.


    Collecting values as a loop progresses is also a very common idiom. We may be looping over lines from a file, a database result set or a more complex structure such as that read from a JSON file.


    The central concept behind this algorithm is a collector - such as a list or set - that we initialize as empty before the loop and add to as we loop through the data:

    initialize a list or set as empty
    
    loop over lines in file
        split or select out value to add
        add value to list or set
    
    use the list of collected values

    This example splits out the 3rd column value from each row, converts to float and append the value to a list that was initialized before the loop began:

    fh = open('revenue.csv')
    
    header_line = next(fh)   # advance the file pointer past
                             # the first line (only if needed)
    
    value_collector = []     # empty list or set for collecting
    
                                       # first values:
    for line in fh:                    # "Haddad's,PA,239.5\n"
        line = line.rstrip()           # "Haddad's,PA,239.5"
        items = line.split(',')        # [ "Haddad's", 'PA', '239.5' ]
        rev = items[2]                 # '239.5'
        frev = float(rev)              # 239.5
    
        value_collector.append(frev)   # [ 239.5 ]
    
    fh.close()
    
    print(value_collector)             # [ 239.5, 53.9, 211.5, 11.98,
                                       #   5.98, 239.5, 115.2 ]

    Building a dict Lookup from File

    A dict lookup taken from a file usually stores one pair per row of a file.


    Dicts (or "mappings") pair up keys with values, and in the case of a lookup dict, we're interested in being able to look up a value associated with a key. For example, we might look up a full name based on an abbreviation; a city based on a zip code; an employee name based on an id, etc.


    Similar to a list or set collection, we add pairs to a dict from each row or selected rows in a file:

    initialize a dict as empty
    loop through file
        split or select out the key and value to be added
        add the key/value as a pair
    
    use the dict for lookup or other purpose

    This example splits out the 1st and 2nd column value from each line, and adds that values as a pair to the dict.

    fh = open('revenue.csv')
    
    header_line = next(fh)   # advance the file pointer past
                             # the first line (only if needed)
    
    company_states = {}      # empty dict for collecting
    
                                      # first values:
    for line in fh:                   # "Haddad's,PA,239.5\n"
        items = line.split(',')       # [ "Haddad's", 'PA', '239.5\n' ]
        co = items[0]                 # "Haddad's"
        st = items[1]                 # 'PA'
        company_states[co] = st       # {"Haddad's", 'PA'}
    
    fh.close()
    
    print(company_states)      # { "Haddad's": 'PA',
                               #   'Westfield': 'NJ',
                               #   'The Store': 'NJ',
                               #   'Hipster's': 'NY',
                               #   'Dothraki Fashions': 'NY',
                               #   "Awful's': 'PA',
                               #   'The Clothiers': 'NY',
                               #  }

    Building a dict Aggregation (Summing or Counting) from File

    A dict aggregation from a file relies on the presence of a key to determine whether to set a pair or change the value associated with a key.


    An aggregation, or grouping, allows us to compile a sum or a count based on a "primary key". For example: counting large cities in each state or students under each major; summing up revenue by sales associate or total population of cities in each country.


    A dictionary can accomplish this by setting the "primary key" (the key under which values are collected) as the key in the dict and the value to an int or float that is then updated inside the file loop:

    initialize a dict as empty
    
    loop through the file
        split and select out the "key" and associated "value"
        if the key is not in the dict
            set the key to 0
        add the "value" to the current value for the key
    
    use the dict for lookup or other purpose

    fh = open('revenue.csv')
    
    header_line = next(fh)   # advance the file pointer past
                             # the first line (only if needed)
    
    company_states = {}      # empty dict for collecting
    
                                         # first values:
    for line in fh:                      # "Haddad's,PA,239.5\n"
        items = line.split(',')          # [ "Haddad's", 'PA', '239.5\n' ]
        state = items[1]                 # 'PA'
    
        if state not in company_states:
            company_states[state] = 0    # {'PA': 0}
    
        company_states[state] = company_states[state] + 1
                                         # {'PA': 1}
    
    print(company_states)                # {'PA': 2, 'NJ': 2, 'NY': 3}
    
    fh.close()

    "Whole File" Parsing: Accessing File Text as a String

    Files can be read as a single string and processed as a string


    Treating an entire file as a string means we can process the entire file with one whole operation, using string transforming methods like .upper() or .replace(), "inspection" methods like .count(), in or len(), or methods that convert the string into another useful form like .split() (split into "words") or .splitlines() (split into lines).


    fh = open('pyku.txt')
    
    text = fh.read()       # returns single string, full text of file
    
    fh.close()

    "Whole File" Parsing: Accessing Lines as a List

    Files can be read as lists of string lines.


    Dividing a file into a list of lines means we can access any line by position in the file (first line, last line, 5th line, etc.) by subscripting the list. We can also slice a portion of the file by position (first 5 lines, last 3 lines, 2nd through last line).


    Calling readlines() on a file returns the list of lines (strings):

    fh = open('revenue.csv')
    
    lines = fh.readlines()     # list of strings, each line in file
    
    print(lines)
    
    fh.close()

    Calling the .read() method on a file to get a string, and then the .splitlines() method on the string returns the list of lines (strings)

    fh = open('revenue.csv')
    
    text = fh.read()                 # single string, full text of file
    lines = text.splitlines()        # list of strings, lines from file
    
    print(lines)
    
    fh.close()

    We may even want to combine the read and splitlines into one statement -- this is just a convenient way to say it quickly:

    text = fh.read().splitlines()

    (As with any file read, be sure to read the file only once - if you try to read it a second time, python will return an empty list or string.)


    "Whole File" Parsing: Accessing "words" as a List

    Files can be read as a string, then split on whitespace, creating a list of words from the file.


    fh = open('pyku.txt')
    
    text = fh.read()                  # single string, full text of file
    words = text.split()              # entire file as a list of words
    
    print(words)
    
    fh.close()

    CSV <-> CSV Transforms: Selecting Columns

    We may "sculpt" a CSV file by reading the file and building a list of rows for writing to another file.


    initialize an empty list
    open a file for reading, pass to a CSV reader
    read the file row-by-row
        select columns from or add column values to the row
        append the row to the list
    
    open a new file for writing, pass to a CSV writer
    write the list of rows to the file
    always remember to close a write file

    Selecting column values for each row is as easy as creating a new row list, then appending the new list to a row collector list:

    import csv
    
    fh = open('revenue.csv')
    reader = csv.reader(fh)
    
                                           # advance the file pointer past
                                           # the first line (only if needed)
    header_row = next(reader)              # ['company', 'state', 'price']
    
    new_lines = []                         # new collector list
    new_lines.append(['company', 'price']) # add header line with selected columns
    
                                           # first values:
    for row in reader:                     # ["Haddad's", 'PA', '239.5']
        name = row[0]                      # "Haddad's"
        state = row[1]                     # 'PA'
        price = row[2]                     # '239.5'
    
        new_row = [name, price]            # ["Haddad's", '239.5']
    
        new_lines.append(new_row)          # [ ['company', 'price'],
                                           #   ["Haddad's", '239.5'] ]
                                           # (list of lists)
    
    fh.close()
    
    
    wfh = open('revenue_new.csv', 'w', newline='')
    
    writer = csv.writer(wfh)
    writer.writerows(new_lines)         # writing list of lists to new file
                                        # as all lines at once
    
    wfh.close()
                                        # company,price
                                        # Haddad's,239.5
                                        # Westfield,53.9
                                        # The Store,211.5
                                        # Hipster's,11.98
                                        # Dothraki Fashions,5.98
                                        # Awful's,23.95
                                        # The Clothiers,115.2

    Adding a column can be done just as easily - building a collector list of rows with an extra value on each row list:

    import csv
    
    fh = open('revenue.csv')
    reader = csv.reader(fh)
    
                                        # advance the file pointer past
                                        # the first line (only if needed)
    header_row = next(reader)           # ['company', 'state', 'price']
    
    header_row.append('tax')            # ['company', 'state', 'price', 'tax']
    
    new_lines = [ ]                     # new collector list
    new_lines.append(header_row)        # [ ['company', 'state', 'price', 'tax'] ]
    
                                        # first values:
    for row in reader:                  # ["Haddad's", 'PA', '239.5']
        name = row[0]                   # "Haddad's"
        state = row[1]                  # 'PA'
        revenue = float(row[2])         # 239.5
    
        tax = round(revenue * .08, 2)   # 19.16
    
        new_row = [ name, state, revenue, tax ]   # [ "Haddad's", 'PA', 239.5, 19.16 ]
    
                                        # adding above row to list - list of lists
        new_lines.append(new_row)       # [ ['company', 'state', 'price', 'tax'],
                                        #   ["Haddad's", 'PA', 239.5, 19.16] ]
    
    
    wfh = open('revenue_new.csv', 'w', newline='')
    writer = csv.writer(wfh)
    
    writer.writerows(new_lines)         # writing list of lists to new file
                                        # as all lines at once
    
    fh.close()
                                        # company,state,price,tax
                                        # Haddad's,PA,239.5,19.16
                                        # Westfield,NJ,53.9,4.31
                                        # The Store,NJ,211.5,16.92
                                        # Hipster's,NY,11.98,0.96
                                        # Dothraki Fashions,NY,5.98,0.48
                                        # Awful's,PA,23.95,1.92
                                        # The Clothiers,NY,115.2,9.22

    CSV <-> CSV Transforms: Selecting Rows by Column Value

    We may selectively choose rows from a file for addition to a new file.


    Here we are building a new collector list of rows that have a column value above a threshold value:

    import csv
    
    fh = open('revenue.csv')
    reader = csv.reader(fh)
                                        # advance the file pointer past
                                        # the first line (only if needed)
    header_row = next(reader)           # ['company', 'state', 'price']
    
    new_lines = [ ]                     # new collector list
    new_lines.append(header_row)        # [ ['company', 'state', 'price'] ]
    
                                        # first values:
    for row in reader:                  # ["Haddad's", 'PA', '239.5']
        name = row[0]                   # "Haddad's"
        state = row[1]                  # 'PA'
        revenue = float(row[2])         # 239.5
    
        if revenue >= 100:
            new_lines.append(row)       # [ ['company', 'state', 'price'],
                                        #   ["Haddad's", 'PA', '239.5']    ]
    
    
    wfh = open('revenue_new.csv', 'w', newline='')
    writer = csv.writer(wfh)
    writer.writerows(new_lines)         # writes entire list of lists to file
    
    fh.close()
                                        # company,state,price
                                        # Haddad's,PA,239.5
                                        # The Store,NJ,211.5
                                        # The Clothiers,NY,115.2

    CSV <-> CSV Transforms: Selecting Rows by Position

    We may choose rows from a file based on position, build them into a list of rows, then write those rows to a new file.


    Similar to "whole file" parsing of lines, we can read the file into a list of rows, then select rows from the list for writing to a new file. The cvs .writerows() method makes this particularly easy.

    import csv
    
    fh = open('revenue.csv')
    reader = csv.reader(fh)
                                        # advance the file pointer past
                                        # the first line (only if needed)
    header_row = next(reader)           # ['company', 'state', 'price']
    
    new_lines = [ ]                     # new collector list
    new_lines.append(header_row)        # [ ['company', 'state', 'price'] ]
    
    data_lines = list(reader)           # list of lists - entire file
    
    wanted_lines = data_lines[3:]       # just the 4th through last rows
                                        # (omitting 1st 3 rows)
    
    new_lines.extend(wanted_lines)      # add selected rows to list of rows
    
    
    wfh = open('revenue_new.csv', 'w', newline='')
    writer = csv.writer(wfh)
    writer.writerows(new_lines)         # writes entire list of lists to file
    
    fh.close()
    
                                        # company,state,price
                                        # Hipster's,NY,11.98
                                        # Dothraki Fashions,NY,5.98
                                        # Awful's,PA,23.95
                                        # The Clothiers,NY,115.2
    



    Matplotlib

    Matplotlib documentation

    (my slides are the clearest though)


    Matplotlib documentation can be found here: http://matplotlib.org/ A very good rundown of features is in the Python for Data Analysis 2nd Edition PDF, Chapter 9 A clear tutorial on the central plotting function pyplot (part of which was used for this presentation) can be found here: https://matplotlib.org/users/pyplot_tutorial.html


    Plotting in a Python Script

    Use plt.plot() to plot; plt.savefig() to save as an image file.


    Python script using pyplot object:

    import matplotlib.pyplot as plt
    import numpy as np
    
    line_1_data = [1, 2, 3, 2, 4, 3, 5, 4, 6]
    line_2_data = [6, 4, 5, 3, 4, 2, 3, 2, 1]
    
    plt.plot(line_1_data)          # plot 1st line
    plt.plot(line_2_data)          # plot 2nd line
    
    plt.savefig('linechart.png')   # use any image extension
                                   # for an image of that type

    Plotting in a Jupyter Notebook

    Jupyter notebook session using pyplot object

    # load matplotlib visualization functionality
    %matplotlib notebook
    import matplotlib.pyplot as plt
    
    linedata = np.random.randn(1000).cumsum()
    
    plt.plot(linedata)

    Any calls to .plot() will display the figure in Jupyter.


    The Figure and Subplot Objects

    The figure represents the overall image; a figure may contain multiple subplots.


    Here we are establishing a figure with one subplot. The subplot object ax can be used to plot


    import numpy as np
    import matplotlib.pyplot as plt
    
    fig = plt.figure()
    ax = fig.add_subplot(1, 1, 1)            # this figure will have one subplot
                                             # 1 row, 1 column, position 1 within that
    ax.plot(np.random.randn(1000).cumsum())
    ax.plot(np.random.randn(1000).cumsum())

    Multiple Subplots Within a Figure

    We may create a column of 3 plots, a 2x2 grid of 4 plots, etc.


    import matplotlib.pyplot as plt
    import numpy as np
    
    fig = plt.figure()
    ax1 = fig.add_subplot(2, 2, 1)
    plt.plot(np.random.randn(50).cumsum(), 'k--')
    ax2 = fig.add_subplot(2, 2, 2)
    plt.plot(np.random.randn(50).cumsum(), 'b-')
    ax3 = fig.add_subplot(2, 2, 3)
    plt.plot(np.random.randn(50).cumsum(), 'r.')

    Establishing a grid of subplots with pyplot.subplots()


    import matplotlib.pyplot as plt
    fig, axes = plt.subplots(2, 3)
    
    print(axes)
       # array([ [ <matplotlib.axes._subplots.AxesSubplot object at 0x7fb626374048>,
       #           <matplotlib.axes._subplots.AxesSubplot object at 0x7fb62625db00>,
       #           <matplotlib.axes._subplots.AxesSubplot object at 0x7fb6262f6c88> ],
       #         [ <matplotlib.axes._subplots.AxesSubplot object at 0x7fb6261a36a0>,
       #           <matplotlib.axes._subplots.AxesSubplot object at 0x7fb626181860>,
       #           <matplotlib.axes._subplots.AxesSubplot object at 0x7fb6260fd4e0> ] ],
       #           dtype=object)

    A fine discussion can be found at http://www.labri.fr/perso/nrougier/teaching/matplotlib/matplotlib.html#figures-subplots-axes-and-ticks


    Line Plotting along 1 or 2 axes

    plot() with a list plots the values along the y axis, indexed by list indices along the x axis (0-4):

    fig = plt.figure()
    ax1 = fig.add_subplot(1, 1, 1)
    ax1.plot([1, 2, 3, 4])         # indexed against 0, 1, 2, 3 on x axis

    With two lists, plots the values in the first list along the y axis, indexed by the second list along the x axis:

    fig = plt.figure()
    ax1 = fig.add_subplot(1, 1, 1)
    ax1.plot([10, 20, 30, 40], [1, 4, 9, 16])

    Adding Seaborn to any Plot

    Simply import seaborn makes plots look better.


    import seaborn as sns
    import matplotlib.pyplot as plt
    
    barvals = [10, 30, 20, 40, 30, 50]
    barpos = [0, 1, 2, 3, 4, 5]
    
    fig = plt.figure()
    ax = fig.add_subplot(1, 1, 1)
    ax.bar(barpos, barvals)

    seaborn is an add-on library that can make any matplotlib plot more attractive, through the use of muted colors and additional styles. The library can be used for detailed control of style, but simply importing it provides a distinct improvement over the default primary colors.


    Line Color, Style, Markers

    import matplotlib.pyplot as plt
    
    line_1_data = [1, 2, 3, 2, 4, 3, 5, 4, 6]
    line_2_data = [6, 4, 5, 3, 4, 2, 3, 2, 1]
    
    fig = plt.figure()
    ax = fig.add_subplot(1, 1, 1)
    ax.plot(line_1_data, linestyle='dotted', color='red')
    ax.plot(line_2_data, linestyle='dashed', color='green', marker='o')


    Line Styles

    '-'solid line style
    '--'dashed line style
    '-.'dash-dot line style
    ':'dotted line style
    '.'point marker
    ','pixel marker
    'o'circle marker
    'v'triangle_down marker
    '^'triangle_up marker
    '<'triangle_left marker
    '>'triangle_right marker
    '1'tri_down marker
    '2'tri_up marker
    '3'tri_left marker
    '4'tri_right marker
    's'square marker
    'p'pentagon marker
    '*'star marker
    'h'hexagon1 marker
    'H'hexagon2 marker
    '+'plus marker
    'x'x marker
    'D'diamond marker
    'd'thin_diamond marker
    '|'vline marker
    '_'hline marker
    Line Styles
    'b'blue
    'g'green
    'r'red
    'c'cyan
    'm'magenta
    'y'yellow
    'k'black
    'w'white


    Setting Axis Ticks and Tick Range

    import imp
    plt = imp.reload(plt)
    
    ydata = [1, 2, 3, 2, 4, 3, 5, 4, 6]
    xdata = [0, 10, 20, 30, 40, 50, 60, 70, 80]    # (this is the default if no list is passed for x)
    
    fig = plt.figure()
    ax = fig.add_subplot(1, 1, 1)
    
    ax.set_yticks([2, 4, 6, 8])
    ax.set_xticks([0, 25, 50, 75, 100])
    
    ax.set_ylim(0, 10)
    ax.set_xlim(0, 100)
    
    ax.set_xticklabels(['zero', 'twenty-five', 'fifty', 'seventy-five', 'one hundred'],
                       rotation=30, fontsize='small')
    
    line1, = ax.plot(xdata, ydata)
    line2, = ax.plot([i+10 for i in xdata], ydata)
    
    ax.legend([line1, line2], ['this line', 'that line'])

    The ticks on the y axis (vertical) are set based on the data values of the first list passed. The ticks on the x axis (horizontal) are set based on the data values of the second list passed. setting the tick range limit


    ax.set_ylim(0, 10)
    ax.set_xlim(0, 100)

    setting the ticks specifically


    ax.set_yticks([2, 4, 6, 8])
    ax.set_xticks([0, 25, 50, 75, 100])

    setting tick labels


    ax.set_xticklabels(['zero', 'twenty-five', 'fifty', 'seventy-five', 'one hundred'],
                        rotation=30, fontsize='small')

    plt.grid(True) to add a grid to the figure


    plt.grid(True)

    setting a legend


    line1, = ax.plot(xdata, ydata)
    line2, = ax.plot([i+10 for i in xdata], ydata)
    
    ax.legend([line1, line2], ['this line', 'that line'])

    Saving to File

    fig.savefig() saves the figure to a file.


    fig.savefig('myfile.png')

    The filename extension of a saved figure determines the filetype.

    print(fig.canvas.get_supported_filetypes())
    
        # {'eps': 'Encapsulated Postscript',
        #  'pdf': 'Portable Document Format',
        #  'pgf': 'PGF code for LaTeX',
        #  'png': 'Portable Network Graphics',
        #  'ps': 'Postscript',
        #  'raw': 'Raw RGBA bitmap',
        #  'rgba': 'Raw RGBA bitmap',
        #  'svg': 'Scalable Vector Graphics',
        #  'svgz': 'Scalable Vector Graphics'}

    Visualizing pandas Series and DataFrame

    pandas has fully incorporated matplotlib into its API.


    pandas Series objects have a plot() method that works

    import pandas as pd
    import numpy as np
    
    ts = pd.Series(np.random.randn(1000), index=pd.date_range('1/1/2000', periods=1000))
    ts = ts.cumsum()
    ts.plot(kind="line")   # "line" is default

    pandas DataFrames also have a .plot() method that plots multiple lines


    import pandas as pd
    df = pd.DataFrame({'a': [1, 2, 3], 'b': [2, 3, 4]})
    df.plot()

    Pandas DataFrames also have a set of methods that create the type of chart desired.


    df.plot.area     df.plot.barh     df.plot.density  df.plot.hist     df.plot.line     df.plot.scatter
    df.plot.bar      df.plot.box      df.plot.hexbin   df.plot.kde      df.plot.pie

    The pandas visualization documentation can be found here: http://pandas.pydata.org/pandas-docs/stable/visualization.html


    Bar Charts

    .


    import matplotlib.pyplot as plt
    
    langs =    ('Python', 'C++', 'Java', 'Perl', 'Scala', 'Lisp')
    langperf = [     10,     8,      6,      4,       2,      1]
    
    y_pos = np.arange(len(langs))
    
    plt.bar(y_pos, langperf, align='center', alpha=0.5)
    plt.xticks(y_pos, langs)
    plt.ylabel('Usage')
    plt.title('Programming language usage')

    Pie Charts

    Pie charts set slice values as portions of a summed whole


    import numpy as np
    import matplotlib.pyplot as plt
    plt.pie([2, 3, 10, 20])

    Scatterplot

    Scatterplots Set points at x,y coordinates, at varying sizes and colors


    import matplotlib.pyplot as plt
    import numpy as np
    
    N = 50
    x = np.random.rand(N)
    y = np.random.rand(N)
    colors = np.random.rand(N)
    area = np.pi * (15 * np.random.rand(N))**2  # 0 to 15 point radii
    
    plt.scatter(x, y, s=area, c=colors, alpha=0.5)



    SQL Part 2: Primary Key, JOIN, GROUP BY, ORDER BY

    Reminder: sqlite3 client column formatting

    Use these sqlite3 commands to format your output readably.


    At the start of your session, issue the following two commands -- these will format your sqlite3 output so it is clearer, and add columns headers.

    sqlite> .mode column
    sqlite> .headers on

    Now output is clearly lined up with column heads displayed:

    sqlite> SELECT * FROM revenue;
    
        # company     state       price
        # ----------  ----------  ----------
        # Haddad's    PA          239.5
        # Westfield   NJ          53.9
        # The Store   NJ          211.5
        # Hipster's   NY          11.98
        # Dothraki F  NY          5.98
        # Awful's     PA          23.95
        # The Clothi  NY          115.2

    However, note that each column is only 10 characters wide. It is possible to change these widths although not usually necessary.


    Primary Key (PK) in a Table

    The PK is defined in the CREATE TABLE definition.


    Here's a table description in SQLite for a table that has a "instructor_id" primary key:

    CREATE TABLE instructors ( instructor_id INT PRIMARY KEY,
                               password TEXT,
                               first_name TEXT,
                               last_name TEXT   );

    The primary key in a database cannot be duplicated -- an error will occur if this is attempted.


    JOINing Tables on a Primary Key

    Two tables may have info keyed to the same primary key -- these can be joined into one table.


    Relational database designs attempt to separate data into individual tables, in order to avoid repetition. For example, consider one table that holds data for instructors at a school (in which one instructor appears per row) and another that holds records of a instructor's teaching a class (in which the same instructor may appear in multiple rows).


    Here is a CREATE TABLE description for tables instructors and instructor_classes. instructors contains:

    sqlite3> .schema instructors
    CREATE TABLE instructors ( instructor_id INT PRIMARY KEY,
                               password TEXT,
                               first_name TEXT,
                               last_name TEXT   );
    
    sqlite3> .schema instructor_classes
    CREATE TABLE instructor_classes ( instructor_id INT,
                                      class_name TEXT,
                                      day TEXT );

    Select all rows from both tables:

    sqlite3> SELECT * from instructors
    instructor_id  password    first_name  last_name
    -------------  ----------  ----------  ----------
    1              pass1       David       Blaikie
    2              pass2       Joe         Wilson
    3              xxyx        Jenny       Warner
    4              yyyy        Xavier      Yellen
    
    sqlite> SELECT * from instructor_classes
    instructor_id  class_name    day
    -------------  ------------  ----------
    1              Intro Python  Thursday
    1              Advanced Pyt  Monday
    2              php           Monday
    2              js            Tuesday
    3              sql           Wednesday
    3              mongodb       Thursday
    99             Golang        Saturday

    Why is instructor_classes data separated from instructors data? If we combined all of this data into one table, there would be repetition -- we'd see the instructor's name repeated on all the rows that indicate the instructor's class assignments. So it makes sense to separate the data that has a "one-to-one" relationship of instructors to the data for each instructor (as in the instructors table) from the data that has a "many-to-one" relationship of the instructor to the data for each instructor (as in the instructor_classes table). But there are times where we will want to see all of this data shown together in a single result set -- we may see repetition, but we won't be storing repetition. We can create these combined result sets using database joins.


    LEFT JOIN

    all rows from "left" table, and matching rows in right table


    A left join includes primary keys from the "left" table (this means the table mentioned in the FROM statement) and will include only those rows in right table that share those same keys.

    sqlite3> SELECT * FROM instructors LEFT JOIN instructor_classes
             on instructors.instructor_id = instructor_classes.instructor_id;
    
    instructor_id  password    first_name  last_name   instructor_id  class_name       day
    -------------  ----------  ----------  ----------  -------------  ---------------  ----------
    1              pass1       David       Blaikie     1              Advanced Python  Monday
    1              pass1       David       Blaikie     1              Intro Python     Thursday
    2              pass2       Joe         Wilson      2              js               Tuesday
    2              pass2       Joe         Wilson      2              php              Monday
    3              xxyx        Jenny       Warner      3              mongodb          Thursday
    3              xxyx        Jenny       Warner      3              sql              Wednesday
    4              yyyy        Xavier      Yellen

    Note the missing data on the right half of the last line. The right table instructor_classes had no data for instructor id 4.


    RIGHT JOIN

    all rows from the "right" table, and matching rows in the left table


    A right join includes primary keys from the "right" table (this means the table mentioned in the JOIN clause) and will include only those rose in the left table that share the same keys as those in the right.


    Unfortunately, SQLite does not support RIGHT JOIN (although many other databases do). The workaround is to use a LEFT JOIN and reverse the table names

    sqlite3> SELECT * FROM instructor_classes LEFT JOIN instructors ON instructors.instructor_id = instructor_classes.instructor_id;
    
    instructor_id  class_name    day         instructor_id  password    first_name  last_name
    -------------  ------------  ----------  -------------  ----------  ----------  ----------
    1              Intro Python  Thursday    1              pass1       David       Blaikie
    1              Advanced Pyt  Monday      1              pass1       David       Blaikie
    2              php           Monday      2              pass2       Joe         Wilson
    2              js            Tuesday     2              pass2       Joe         Wilson
    3              sql           Wednesday   3              xxyx        Jenny       Warner
    3              mongodb       Thursday    3              xxyx        Jenny       Warner
    99             Golang        Saturday

    Now only rows that appear in instructor_classes appear in this table, and data not found in instructors is missing (In this case, Golang has no instructor and it is given the default id 99).


    <=H>INNER JOIN and OUTER JOIN <=SH>Select only PKs common to both tables, or all PKs for all tables INNER JOIN: rows common to both tables


    An inner join includes only those rows that have primary key values that are common to both tables:

    sqlite3> SELECT * from instructor_classes INNER JOIN instructors ON instructors.instructor_id = instructor_classes.instructor_id;
    instructor_id  class_name    day         instructor_id  password    first_name  last_name
    -------------  ------------  ----------  -------------  ----------  ----------  ----------
    1              Intro Python  Thursday    1              pass1       David       Blaikie
    1              Advanced Pyt  Monday      1              pass1       David       Blaikie
    2              php           Monday      2              pass2       Joe         Wilson
    2              js            Tuesday     2              pass2       Joe         Wilson
    3              sql           Wednesday   3              xxyx        Jenny       Warner
    3              mongodb       Thursday    3              xxyx        Jenny       Warner

    Rows are joined where both instructors and instructor_classes have data.


    OUTER JOIN: all rows from both tables


    An outer join includes all rows from both tables, regardless of whether a PK id appears in the other table. Here's what the query would be if sqlite3 supported outer joins:

    SELECT * from instructor_classes OUTER JOIN instructors ON instructors.instructor_id = instructor_classes.instructor_id;

    unfortunately, OUTER JOIN is not currently supported in sqlite3. In these cases it's probably best to use another approach, i.e. built-in Python or pandas merge() (to come).


    Aggregating data with GROUP BY

    "Aggregation" means counting, summing or otherwise summarizing multiple values based on a common key.


    Consider summing up a count of voters by their political affiliation (2m Democrats, 2m Republicans, .3m Independents), a sum of revenue of companies by their sector (manufacturing, services, etc.), or an average GPA by household income. All of these require taking into account the individual values of multiple rows and compiling some sort of summary value based on those values.


    Here is a sample that we'll play with:

    sqlite3> SELECT date, name, rev FROM companyrev;
    
    date        name         rev
    ----------  -----------  ----------
    2019-01-03  Alpha Corp.  10
    2019-01-05  Alpha Corp.  20
    2019-01-03  Beta Corp.   5
    2019-01-07  Beta Corp.   7
    2019-01-09  Beta Corp.   3

    If we wish to sum up values by company, we can say it easily:

    sqlite3> SELECT name, sum(rev) FROM companyrev GROUP BY name;
    
    name         sum(rev)
    -----------  ----------
    Alpha Corp.  30
    Beta Corp.   15

    If we wish to count the number entries for each company, we can say it just as easily:

    sqlite3> SELECT name, count(name) FROM companyrev GROUP BY name;
    
    name         count(name)
    -----------  -----------
    Alpha Corp.  2
    Beta Corp.   3

    Sorting a Result Set with ORDER BY

    This is SQL's way of sorting results.


    The ORDER BY clause indicates a single column, or multiple columns, by which we should order our results:

    sqlite3> SELECT name, rev FROM companyrev ORDER BY rev;
    
    name        rev
    ----------  ----------
    Beta Corp.  3
    Beta Corp.  5
    Beta Corp.  7
    Alpha Corp  10
    Alpha Corp  20



    HTTP and Web Clients

    Basic HTTP

    Headers, Cookies and Response Codes


    HTTP (HyperText Transport Protocol) is the protocol for sending and receiving messages from a browser (the client) to a web server (the server) and back again. We call the browser's message the request and the server's response the response. HTTP request A request may consist of four parts: the URL, headers, key/value parameters, and a payload or body of the request. Each of these may be used to define what is being requested. Only the URL and headers will be present in every request. parameters


    parameters are key/value pairs that you see in some URLs after a question mark:

    http://mysite.com/getinfo?what=news&when=yesterday

    Parameters are used to specify what data is requested. If you look at URLs you are clicking on, many times you will see parameters in this format (including my homework system -- check it out). HTTP headers are meta information sent along with a request. This may include session (cookie) information. payload This is the "body" of a POST request (GET requests have no payload). This may contain information to be posted to the server. HTTP response The content of a response is the HTML, text or other data (can be binary data, or anything else a browser can send or receive). The response may also include headers. These include the response code, the size of the response, and may also contain cookies. The response code of a response indicates whether the request was handled without error (200), whether there was a server error (500), etc. GET vs. POST A request is generally one of two methods: GET (for retrieving data) or POST (for sending data, like form input). (Note in this context method is unrelated to a Python method.) An HTTP GET request is customarily used to retrieve information; a POST request is usually for sending information. a get request is the default made by hyperlinks a post request is usually made by HTML forms


    A form submission is usually sent as a POST request and includes parameter data. Here is a sample form:

    <FORM ACTION="http://www.mywebsite.com/user" METHOD="POST">
      <INPUT NAME="firstname"><BR>
      <INPUT NAME="lastname"><BR>
      <INPUT NAME="password" TYPE="password">
      <INPUT TYPE="submit">
    </FORM>

    This form produces key/value data in the body of the request.


    Python on the Client side: the requests module

    requests is highly popular for web client functionality


    A web client is any program that can send HTTP requests to a web server. Your browser is a web client. The requests module lets us issue HTTP requests in the same way a browser would. We can therefore have our Python program behave like a browser, i.e. visit web pages and download data. Requesting a web page through requests.get()


    A page is usually requested as a GET request

    import requests
    
    response = requests.get('http://www.nytimes.com')
    
    page_text =   response.text
    status_code = response.status_code
    
    page_text = page_text.encode('utf-8')      # if necessary
    
    print('status code:  {}'.format(status_code))
    print('======================= page text =======================')
    print(page_text)

    Decoding a JSON response


    API calls are also made over HTTP, and if the call returns JSON, this can easily be decoded through requests.

    import requests
    
    response = requests.get('http://api.wunderground.com/api/d2e101aa48faa661/conditions/q/CA/San_Francisco.json')
    conditions_json = response.json()
    
    print(conditions_json["current_observation"]["temp_f"])        # 84.5 (accessing data within the JSON)

    requests: URL, parameters, headers, payload

    The URL plus one or more of the below


    URL


    The URL may contain just the address

    
          
          
            

    parameters


    Many web requests include parameters specifying what content or action is desired. Here's a link to my homework application:

    http://homework-davidbpython.rhcloud.com/route_view?assignment_id=1.1&student_id=bill_hanson

    The parameters here are assignment_id (value 1.1) and student_id (value bill_hanson)


    To pass parameters, we simply include a dict keyed to params.

    param_dict = {'assignment_id': '1.1', 'student_id': 'bill_hanson'}
    response = requests.get('http://homework-davidbpython.rhcloud.com/route_view', params=param_dict)

    headers These are key/value pairs that are sent along with the request.


    
          
          
            

    payload


    
          
          
            

    We can replicate this kind of request by using request.post() and passing a dict of param keys and values:


    userdata = {"firstname": "John", "lastname": "Doe", "password": "jdoe123"}
    
    resp = requests.post('http://www.mywebsite.com/user', params=userdata)

    Some other features of requests: * International Domains and URLs * Keep-Alive & Connection Pooling * Sessions with Cookie Persistence * Browser-style SSL Verification * Basic/Digest Authentication * Elegant Key/Value Cookies * Automatic Decompression * Unicode Response Bodies * Multipart File Uploads * Connection Timeouts


    Python as a web client: the urllib module

    urllib is another module for making web requests.


    Although the requests module is strongly favored by some for its simplicity, it has not yet been added to the Python builtin distribution.


    The urlopen method takes a url and returns a file-like object that can be read() as a file:

    import urllib.request
    my_url = 'http://www.google.com'
    readobj = urllib.request.urlopen(my_url)
    text = readobj.read()
    print(text)
    readobj.close()

    Alternatively, you can call readlines() on the object (keep in mind that many objects that can deliver file-like string output can be read with this same-named method:

    for line in readobj.readlines():
      print(line)
    readobj.close()

    The text that is downloaded is CSV, HTML, Javascript, and possibly other kinds of data. TypeError: can't use a string pattern on a bytes-like object This error may occur with some websites. It indicates that an undecoded unicode response was received.


    The response usually comes to us as a special object called a byte string. In order to work with the response as a string, we may need to use the decode() method:

    text = text.decode('utf-8')

    UnicodeEncodeError This error may occur if the downloaded page contains characters that Python doesn't know how to handle. It is in most cases fixed by using the text.decode line above on the text immediately after it is retrieves from urlopen(). SSL Certificate Error Many websites enable SSL security and require a web request to accept and validate an SSL certificate (certifying the identity of the server). urllib by default requires SSL certificate security, but it can be bypassed (keep in mind that this may be a security risk).


    import ssl
    ctx = ssl.create_default_context()
    ctx.check_hostname = False
    ctx.verify_mode = ssl.CERT_NONE
    
    my_url = 'http://www.nytimes.com'
    readobj = urllib.request.urlopen(my_url, context=ctx)

    Download binary files: images and other files can be saved locally using urllib.request.urlretrieve().


    import urllib.request
    
    urllib.request.urlretrieve('http://www.azquotes.com/picture-quotes/quote-python-is-an-experiment-in-how-much-freedom-programmers-need-too-much-freedom-and-nobody-guido-van-rossum-133-51-31.jpg', 'guido.jpg')

    Note the two arguments to urlretrieve(): the first is a URL to an image, and the second is a filename -- this file will be saved locally under that name.


    Encoding Parameters: urllib.request.urlencode()

    When including parameters in our requests, we must encode them into our request URL. The urlencode() method does this nicely:


    import urllib.request, urllib.parse
    
    params = urllib.parse.urlencode({'choice1': 'spam and eggs', 'choice2': 'spam, spam, bacon and spam'})
    print("encoded query string: ", params)
    f = urllib.request.urlopen("http://www.google.com?{}".format(params))
    print(f.read())

    this prints:

    encoded query string: choice1=spam+and+eggs&choice2=spam%2C+spam%2C+bacon+and+spam
    
    choice1:  spam and eggs<BR>
    choice2:  spam, spam, bacon and spam<BR>



    Advanced Container Structures

    array

    The array is a type-specific list.


    The array container provides a list of a uniform type. An array's type must be specified at initialization. A uniform type makes an array more efficient than a list, which can contain any type.


    from array import array
    
    myarray = array('i', [1, 2])
    
    myarray.append(3)
    
    print(myarray)           # array('i', [1, 2, 3])
    
    print(myarray[-1])       # acts like a list
    for val in myarray:
        print(val)
    
    myarray.append(1.3)     # error

    Available array types:

    Type code C Type Python Type Minimum size in bytes
    'c' char character 1
    'b' signed char int 1
    'B' unsigned char int 1
    'u' Py_UNICODE Unicode character 2
    'h' signed short int 2
    'H' unsigned short int 2
    'i' signed int int 2
    'I' unsigned int long 2
    'l' signed long int 4
    'L' unsigned long long 4
    'f' float float 4
    'd' double float 8


    Collections: deque

    A "double-ended queue" provides fast adds/removals.


    The collections module provides a variety of specialized container types. These containers behave in a manner similer to the builtin ones with which we are familiar, but with additional functionality based around enhancing convenience and efficiency. lists are optimized for fixed-length operations, i.e., things like sorting, checking for membership, index access, etc. They are not optimized for appends, although this is of course a common use for them. A deque is designed specifically for fast adds -- to the beginning or end of the sequence:


    from collections import deque
    
    x = deque([1, 2, 3])
    
    x.append(4)               # x now [1, 2, 3, 4]
    x.appendleft(0)           # x now [0, 1, 2, 3, 4]
    
    popped = x.pop()          # removes '4' from the end
    
    popped2 = x.popleft()     # removes '1' from the start

    A deque can also be sized, in which case appends will push existing elements off of the ends:


    x = deque(['a', 'b', 'c'], 3)      # maximum size:  3
    x.append(99)                       # now: deque(['b', 'c', 99])  ('a' was pushed off of the start)
    x.appendleft(0)                    # now: deque([0, 'b', 'c'])   (99 was pushed off of the end)

    Collections: Counter

    Counter provides a counting dictionary.


    This structure inherits from dict and is designed to allow an integer count as well as a default 0 value for new keys. So instead of doing this:


    c = {}
    if 'a' in c:
        c['a'] = 0
    else:
        c['a'] = c['a'] + 1

    We can do this:


    from collections import Counter
    
    c = Counter()
    c['a'] = c['a'] + 1

    Counter also has related functions that return a list of its keys repeated that many times, as well as a list of tuples ordered by frequency:


    from collections import Counter
    
    c = Counter({'a': 2, 'b': 1, 'c': 3, 'd': 1})
    
    for key in c.elements():
        print(key, end=' ')            # c c c a a b b
    
    print(','.join(c.elements()))   # c,c,c,a,a,b,b
    
    
    print(c.most_common(2))   # [('c', 3), ('a', 2)]
                             # 2 arg says "give me the 2 most common"
    
    c.clear()                # set all counts to 0 (but not remove the keys)

    And, you can use Counter's implementation of the math operators to work with multiple counters and have them sum their values:


    c = Counter({'a': 1, 'b': 2})
    d = Counter({'a': 10, 'b': 20})
    
    print(c + d)                     # Counter({'b': 22, 'a': 11})

    Collections: defaultdict

    defaultdict is a dict that provides a default object for new keys.


    Similar to Counter, defaultdict allows for a default value if a key doesn't exist; but it will accept a function that provides a default value.


    A defaultdict with a default list value for each key

    from collections import defaultdict
    
    ddict = defaultdict(list)
    
    ddict['a'].append(1)
    ddict['b']
    
    print(ddict)                    # defaultdict(, {'a': [1], 'b': []})

    A defaultdict with a default dict value for each key

    ddict = defaultdict(dict)
    
    print(ddict['a'])         # {}    (key/value is created, assigned to 'a')
    
    print(list(ddict.keys()))       # dict_keys(['a'])
    
    ddict['a']['Z'] = 5
    ddict['b']['Z'] = 5
    ddict['b']['Y'] = 10
    
          # defaultdict(<class 'dict'>, {'a': {'Z': 5}, 'b': {'Z': 5, 'Y': 10}})



    Testing

    Testing: Introduction

    Code without tests is like driving without seatbelts.


    All code is subject to errors -- not just ValueErrors and TypeErrors encountered during development, but errors related to unexpected data anomalies or user input, or the unforeseen effects of functions run in untested combinations. Unit testing is the front line of the effort to ensure code quality. Many developers say they won't take a software package seriously unless it comes with tests. testing: a brief rundown Unit testing is the most basic form and the one we will focus on here. Other styles of testing:


    Unit Testing

    "Unit" refers to a function. Unit testing calls individual functions and validates the output or result of each.


    The most easily tested scripts are made up of small functions that can be called and validated in isolation. Therefore "pure functions" (functions that do not refer or change "external state" -- i.e., global variables) are best for testing. Testing for success, testing for failure A unit test script performs test by importing the script to be tested and calling its functions with varying arguments, including ones intended to cause an error. Basically, we are hammering the code as many ways as we can to make sure it succeeds properly and fails properly. Test-driven development As we develop our code, we can write tests simultaneously and run them periodically as we develop. This way we can know that further changes and additions are not interfering with anything we have done previously. Any time in the process we can run the testing program and it will run all tests. In fact commonly accepted wisdom supports writing tests before writing code! The test is written with the function in mind: after seeing that the tests fail, we write a function to satisfy the tests. This called test-driven development.


    The assert statement

    assert raises an AssertionError exception if the test returns False


    assert 5 == 5        # no output
    
    assert 5 == 10       # AssertionError raised

    We can incorporate this facility in a simple testing program: program to be tested: "myprogram.py"


    import sys
    
    def doubleit(x):
        var = x * 2
        return var
    
    if __name__ == '__main__':
        input_val = sys.argv[1]
        doubled_val = doubleit(input_val)
    
        print("the value of {0} is {1}".format(input_val, doubled_val))

    testing program: "test_myprogram.py"


    import myprogram
    
    def test_doubleit_value():
        assert myprogram.doubleit(10) == 20

    If doubleit() didn't correctly return 20 with an argument of 10, the assert would raise an AssertionError. So even with this basic approach (without a testing module like pyttest or unittest), we can do testing with assert.


    pytest Basics

    All programs named test_something.py that have functions named test_something() will be noticed by pytest and run automatically when we run the pytest script py.test.


    instructions for running tests 1. Download the testing program test_[name].py (where name is any name) and place in the same directory as your script. 2. Open up a command prompt. In Mac/Unix, open the Terminal program (you can also use the Terminal window in PyCharm). In Windows, you must launch the Anaconda Command Prompt, which should be accessible by searching for cmd on Windows 10 -- let me know if you have trouble finding the Anaconda prompt. 3. Make sure your homework script and the test script are in the same directory, and that your Command Prompt or Terminal window is also in that directory (let me know if you have any trouble using cd to travel to this directory in your Command Prompt or Terminal window.) 4. Execute the command py.test at the command line (keep in mind this is not the Python prompt, but your Mac/Unix Terminal program or your Anaconda cmd/Command Prompt window). py.test is a special command that should work from your command line if you have Anaconda installed. (py.test is not a separate file.) 5. If your program and functions are named as directed, the testing program will import your script and test each function to see that it is providing correct output. If there are test failures, look closely at the failure output -- look for the assert test showing what values were involved in the failure. You can also look at the testing program to see what it is requiring (look for the assert statements). 6. If you see collected 0 items it means there was no test_[something].py file (where [something] is a name of your choice) or there was no test_[something]() function inside the test program. These names are required by py.test. 7. If your run of py.test hangs (i.e., prints something out but then just waits), or if you see a lot of colorful error output saying not found here and there, it may be for the above reason.


    running py.test from the command line or Anaconda command prompt:

    $ py.test
     =================================== test session starts ====================================
    platform darwin -- Python 2.7.10 -- py-1.4.27 -- pytest-2.7.1
    rootdir: /Users/dblaikie/testpytest, inifile:
    collected 1 items
    
    test_myprogram.py .
    
     ================================= 1 passed in 0.01 seconds =================================

    noticing failures


    def doubleit(x):
        var = x * 2
        return x      # oops, returned the original value rather than the doubled value

    Having incorporated an error, run py.test again:


    $ py.test
     =================================== test session starts ====================================
    platform darwin -- Python 2.7.10 -- py-1.4.27 -- pytest-2.7.1
    rootdir: /Users/dblaikie/testpytest, inifile:
    collected 1 items
    
    test_myprogram.py F
    
     ========================================= FAILURES =========================================
    ___________________________________ test_doubleit_value ____________________________________
    
        def test_doubleit_value():
    >       assert myprogram.doubleit(10) == 20
    E       assert 10 == 20
    E        +  where 10 = (10)
    E        +    where  = myprogram.doubleit
    
    test_myprogram.py:7: AssertionError
     ================================= 1 failed in 0.01 seconds =================================

    Writing tests

    Use assert to test values returned from a function against expected values


    assert raises an AssertionError exception if the test returns False

    assert 5 == 5        # no output
    
    assert 5 == 10       # AssertionError raised

    We can incorporate this facility in a simple testing program: program to be tested: "myprogram.py"


    import sys
    
    def doubleit(x):
        var = x * 2
        return var
    
    if __name__ == '__main__':
        input_val = sys.argv[1]
        doubled_val = doubleit(input_val)
    
        print("the value of {0} is {1}".format(input_val, doubled_val))

    testing program: "test_myprogram.py"


    import myprogram
    
    def test_doubleit_value():
        assert myprogram.doubleit(10) == 20

    If doubleit() didn't correctly return 20 with an argument of 10, the assert would raise an AssertionError. So even with this basic approach (without a testing module like pyttest or unittest), we can do testing with assert.


    Writing tests with pytest

    All programs named test_something.py that have functions named test_something() will be noticed by pytest and run automatically when we run the pytest script py.test.


    instructions for writing and running tests using pytest 1. Make sure your program, myprogram.py and your testing program test_myprogram.py are in the same directory. 2. Open up a command prompt. In Mac/Unix, open the Terminal program (you can also use the Terminal window in PyCharm). In Windows, you must launch the Anaconda Command Prompt, which should be accessible by searching for cmd on Windows 10 -- let me know if you have trouble finding the Anaconda prompt. 3. Use cd to change the present working directory for your Command Prompt or Terminal window session to the same directory (let me know if you have any trouble with this step.) 4. Execute the command py.test at the command line (keep in mind this is not the Python prompt, but your Mac/Unix Terminal program or your Anaconda cmd/Command Prompt window). py.test is a special command that should work from your command line if you have Anaconda installed. (py.test is not a separate file.) 5. If your program and functions are named as directed, the testing program will import your script and test each function to see that it is providing correct output. If there are test failures, look closely at the failure output -- look for the assert test showing what values were involved in the failure. You can also look at the testing program to see what it is requiring (look for the assert statements). 6. If you see collected 0 items it means there was no test_[something].py file (where [something] is a name of your choice) or there was no test_[something]() function inside the test program. These names are required by py.test. 7. If your run of py.test hangs (i.e., prints something out but then just waits), or if you see a lot of colorful error output saying not found here and there, it may be for the above reason.


    running py.test from the command line or Anaconda command prompt:

    $ py.test
     =================================== test session starts ====================================
    platform darwin -- Python 2.7.10 -- py-1.4.27 -- pytest-2.7.1
    rootdir: /Users/dblaikie/testpytest, inifile:
    collected 1 items
    
    test_myprogram.py .
    
     ================================= 1 passed in 0.01 seconds =================================

    noticing failures


    def doubleit(x):
        var = x * 2
        return x      # oops, returned the original value rather than the doubled value

    Having incorporated an error, run py.test again:


    $ py.test
     =================================== test session starts ====================================
    platform darwin -- Python 2.7.10 -- py-1.4.27 -- pytest-2.7.1
    rootdir: /Users/dblaikie/testpytest, inifile:
    collected 1 items
    
    test_myprogram.py F
    
     ========================================= FAILURES =========================================
    ___________________________________ test_doubleit_value ____________________________________
    
        def test_doubleit_value():
    >       assert myprogram.doubleit(10) == 20
    E       assert 10 == 20
    E        +  where 10 = (10)
    E        +    where  = myprogram.doubleit
    
    test_myprogram.py:7: AssertionError
     ================================= 1 failed in 0.01 seconds =================================

    Testing the expected raising of an exception

    Many of our tests will deliberately pass bad input and test to see that an appropriate exception is raised.


    import sys
    
    def doubleit(x):
        if not isinstance(x, (int, float)):           # make sure the arg is the right type
            raise TypeError('must be int or float')   # if not, raise a TypeError
        var = x * 2
        return var
    
    if __name__ == '__main__':
        input_val = sys.argv[1]
        doubled_val = doubleit(input_val)
    
        print("the value of {0} is {1}".format(input_val, doubled_val))

    Note that without type testing, the function could work, but incorrectly (for example if a string or list were passed instead of an integer). To verify that this error condition is correctly raised, we can use with pytest.raises(TypeError).


    import myprogram
    import pytest
    
    def test_doubleit_value():
        assert myprogram.doubleit(10) == 20
    
    def test_doubleit_type():
        with pytest.raises(TypeError):
            myprogram.doubleit('hello')

    with is the same context manager we have used with open(): it can also be used to detect when an exception occured inside the with block.


    Grouping tests into a class

    We can organize related tests into a class, which can also include setup and teardown routines that are run automatically (discussed next).


    """ test_myprogram.py -- test functions in a testing class """
    
    import myprogram
    import pytest
    
    class TestDoubleit(object):
    
        def test_doubleit_value(self):
            assert myprogram.doubleit(10) == 20
    
        def test_doubleit_type(self):
            with pytest.raises(TypeError):
                myprogram.doubleit('hello')

    So now the same rule applies for how py.test looks for tests -- if the class begins with the word Test, pytest will treat it as a testing class.


    Mock data: setup and teardown

    Tests should not be run on "live" data; instead, it should be simulated, or "mocked" to provide the data the test needs.


    """ myprogram.py -- makework functions for the purposes of demonstrating testing """
    
    import sys
    
    def doubleit(x):
        """ double a number argument, return doubled value """
        if not isinstance(x, (int, float)):
            raise TypeError('arg to doublit() must be int or float')
        var = x * 2
        return var
    
    def doublelines(filename):
        """open a file of numbers, double each line, write each line to a new file"""
        with open(filename) as fh:
            newlist = []
            for line in fh:                   # file is assumed to have one number on each line
                floatval = float(line)
                doubleval = doubleit(floatval)
                newlist.append(str(doubleval))
        with open(filename, 'w') as fh:
            fh.write('\n'.join(newlist))
    
    if __name__ == '__main__':
        input_val = sys.argv[1]
        doubled_val = doubleit(input_val)
    
        print("the value of {0} is {1}".format(input_val, doubled_val))

    For this demo I've invented a rather arbitrary example to combine an external file with the doubleit() routine: doublelines() opens and reads a file, and for each line in the file, doubles the value, writing each value as a separate line to a new file (supplied to doublelines()).


    """ test_myprogram.py -- test the doubleit.py script """
    
    import myprogram
    import os
    import pytest
    import shutil
    
    class TestDoubleit(object):
    
        numbers_file_template = 'testnums_template.txt'  # template for test file (stays the same)
        numbers_file_testor = 'testnums.txt'             # filename used for testing
                                                         # (changed during testing)
    
        def setup_class(self):
            shutil.copy(TestDoubleit.numbers_file_template, TestDoubleit.numbers_file_testor)
    
        def teardown_class(self):
            os.remove(TestDoubleit.numbers_file_testor)
    
        def test_doublelines(self):
            myprogram.doublelines(TestDoubleit.numbers_file_testor)
            old_vals = [ float(line) for line in open(TestDoubleit.numbers_file_template) ]
            new_vals = [ float(line) for line in open(TestDoubleit.numbers_file_testor) ]
            for old_val, new_val in zip(old_vals, new_vals):
                assert float(new_val) == float(old_val) * 2
    
        def test_doubleit_value(self):
            assert myprogram.doubleit(10) == 20
    
        def test_doubleit_type(self):
            with pytest.raises(TypeError):
                myprogram.doubleit('hello')

    setup_class and teardown_class run automatically. As you can see, they prepare a dummy file and when the testing is over, delete it. In between, tests are run in order based on the function names.


    Writing tests

    Use assert to test values returned from a function against expected values


    assert raises an AssertionError exception if the test returns False

    assert 5 == 5        # no output
    
    assert 5 == 10       # AssertionError raised

    We can incorporate this facility in a simple testing program: program to be tested: "myprogram.py"


    import sys
    
    def doubleit(x):
        var = x * 2
        return var
    
    if __name__ == '__main__':
        input_val = sys.argv[1]
        doubled_val = doubleit(input_val)
    
        print("the value of {0} is {1}".format(input_val, doubled_val))

    testing program: "test_myprogram.py"


    import myprogram
    
    def test_doubleit_value():
        assert myprogram.doubleit(10) == 20

    If doubleit() didn't correctly return 20 with an argument of 10, the assert would raise an AssertionError. So even with this basic approach (without a testing module like pyttest or unittest), we can do testing with assert.


    Writing tests with pytest

    All programs named test_something.py that have functions named test_something() will be noticed by pytest and run automatically when we run the pytest script py.test.


    instructions for writing and running tests using pytest 1. Make sure your program, myprogram.py and your testing program test_myprogram.py are in the same directory. 2. Open up a command prompt. In Mac/Unix, open the Terminal program (you can also use the Terminal window in PyCharm). In Windows, you must launch the Anaconda Command Prompt, which should be accessible by searching for cmd on Windows 10 -- let me know if you have trouble finding the Anaconda prompt. 3. Use cd to change the present working directory for your Command Prompt or Terminal window session to the same directory (let me know if you have any trouble with this step.) 4. Execute the command py.test at the command line (keep in mind this is not the Python prompt, but your Mac/Unix Terminal program or your Anaconda cmd/Command Prompt window). py.test is a special command that should work from your command line if you have Anaconda installed. (py.test is not a separate file.) 5. If your program and functions are named as directed, the testing program will import your script and test each function to see that it is providing correct output. If there are test failures, look closely at the failure output -- look for the assert test showing what values were involved in the failure. You can also look at the testing program to see what it is requiring (look for the assert statements). 6. If you see collected 0 items it means there was no test_[something].py file (where [something] is a name of your choice) or there was no test_[something]() function inside the test program. These names are required by py.test. 7. If your run of py.test hangs (i.e., prints something out but then just waits), or if you see a lot of colorful error output saying not found here and there, it may be for the above reason.


    running py.test from the command line or Anaconda command prompt:

    $ py.test
     =================================== test session starts ====================================
    platform darwin -- Python 2.7.10 -- py-1.4.27 -- pytest-2.7.1
    rootdir: /Users/dblaikie/testpytest, inifile:
    collected 1 items
    
    test_myprogram.py .
    
     ================================= 1 passed in 0.01 seconds =================================

    noticing failures


    def doubleit(x):
        var = x * 2
        return x      # oops, returned the original value rather than the doubled value

    Having incorporated an error, run py.test again:


    $ py.test
     =================================== test session starts ====================================
    platform darwin -- Python 2.7.10 -- py-1.4.27 -- pytest-2.7.1
    rootdir: /Users/dblaikie/testpytest, inifile:
    collected 1 items
    
    test_myprogram.py F
    
     ========================================= FAILURES =========================================
    ___________________________________ test_doubleit_value ____________________________________
    
        def test_doubleit_value():
    >       assert myprogram.doubleit(10) == 20
    E       assert 10 == 20
    E        +  where 10 = (10)
    E        +    where  = myprogram.doubleit
    
    test_myprogram.py:7: AssertionError
     ================================= 1 failed in 0.01 seconds =================================

    Testing the expected raising of an exception

    Many of our tests will deliberately pass bad input and test to see that an appropriate exception is raised.


    import sys
    
    def doubleit(x):
        if not isinstance(x, (int, float)):           # make sure the arg is the right type
            raise TypeError('must be int or float')   # if not, raise a TypeError
        var = x * 2
        return var
    
    if __name__ == '__main__':
        input_val = sys.argv[1]
        doubled_val = doubleit(input_val)
    
        print("the value of {0} is {1}".format(input_val, doubled_val))

    Note that without type testing, the function could work, but incorrectly (for example if a string or list were passed instead of an integer). To verify that this error condition is correctly raised, we can use with pytest.raises(TypeError).


    import myprogram
    import pytest
    
    def test_doubleit_value():
        assert myprogram.doubleit(10) == 20
    
    def test_doubleit_type():
        with pytest.raises(TypeError):
            myprogram.doubleit('hello')

    with is the same context manager we have used with open(): it can also be used to detect when an exception occured inside the with block.


    Grouping tests into a class

    We can organize related tests into a class, which can also include setup and teardown routines that are run automatically (discussed next).


    """ test_myprogram.py -- test functions in a testing class """
    
    import myprogram
    import pytest
    
    class TestDoubleit(object):
    
        def test_doubleit_value(self):
            assert myprogram.doubleit(10) == 20
    
        def test_doubleit_type(self):
            with pytest.raises(TypeError):
                myprogram.doubleit('hello')

    So now the same rule applies for how py.test looks for tests -- if the class begins with the word Test, pytest will treat it as a testing class.


    Mock data: setup and teardown

    Tests should not be run on "live" data; instead, it should be simulated, or "mocked" to provide the data the test needs.


    """ myprogram.py -- makework functions for the purposes of demonstrating testing """
    
    import sys
    
    def doubleit(x):
        """ double a number argument, return doubled value """
        if not isinstance(x, (int, float)):
            raise TypeError('arg to doublit() must be int or float')
        var = x * 2
        return var
    
    def doublelines(filename):
        """open a file of numbers, double each line, write each line to a new file"""
        with open(filename) as fh:
            newlist = []
            for line in fh:                   # file is assumed to have one number on each line
                floatval = float(line)
                doubleval = doubleit(floatval)
                newlist.append(str(doubleval))
        with open(filename, 'w') as fh:
            fh.write('\n'.join(newlist))
    
    if __name__ == '__main__':
        input_val = sys.argv[1]
        doubled_val = doubleit(input_val)
    
        print("the value of {0} is {1}".format(input_val, doubled_val))

    For this demo I've invented a rather arbitrary example to combine an external file with the doubleit() routine: doublelines() opens and reads a file, and for each line in the file, doubles the value, writing each value as a separate line to a new file (supplied to doublelines()).


    """ test_myprogram.py -- test the doubleit.py script """
    
    import myprogram
    import os
    import pytest
    import shutil
    
    class TestDoubleit(object):
    
        numbers_file_template = 'testnums_template.txt'  # template for test file (stays the same)
        numbers_file_testor = 'testnums.txt'             # filename used for testing
                                                         # (changed during testing)
    
        def setup_class(self):
            shutil.copy(TestDoubleit.numbers_file_template, TestDoubleit.numbers_file_testor)
    
        def teardown_class(self):
            os.remove(TestDoubleit.numbers_file_testor)
    
        def test_doublelines(self):
            myprogram.doublelines(TestDoubleit.numbers_file_testor)
            old_vals = [ float(line) for line in open(TestDoubleit.numbers_file_template) ]
            new_vals = [ float(line) for line in open(TestDoubleit.numbers_file_testor) ]
            for old_val, new_val in zip(old_vals, new_vals):
                assert float(new_val) == float(old_val) * 2
    
        def test_doubleit_value(self):
            assert myprogram.doubleit(10) == 20
    
        def test_doubleit_type(self):
            with pytest.raises(TypeError):
                myprogram.doubleit('hello')

    setup_class and teardown_class run automatically. As you can see, they prepare a dummy file and when the testing is over, delete it. In between, tests are run in order based on the function names.




    pandas: Introduction

    pandas: Introduction

    pandas is a Python module used for manipulation and analysis of tabular data. * Excel-like numeric calculations, particularly column-wise and row-wise calculations (vectorization) * SQL-like merging, grouping and aggregating * visualizing (line chart, bar chart, etc.) * emphasis on: - aligning data from multiple sources - "slicing and dicing" by rows and columns - concatenating and joining - cleaning and normalizing missing or incorrect data - working with time series - categorizing * ability to read and write to CSV, XML, Excel, database queries, etc. numpy is a data analysis library upon which pandas is built. We sometimes make direct calls to numpy - some of its variables (such as np.nan), variable-generating functions (such as np.arange or np.linlist) and some processing functions.


    pandas Reference

    Use the docs for an ongoing study of pandas' rich feature set.


    pandas official documentation


    full docs (HTML, pdf)

    http://pandas.pydata.org/pandas-docs/stable
    http://pandas.pydata.org/pandas-docs/version/0.19.0/pandas.pdf

    "10 minutes to pandas"

    https://pandas.pydata.org/pandas-docs/stable/10min.html

    pandas cookbook

    http://pandas.pydata.org/pandas-docs/stable/cookbook.html



    matplotlib official documentation

    http://matplotlib.org/api/pyplot_api.html

    Tutorials and Learning Aids

    Online resources are many; when you find one you like, stick with it


    pandas textbook "Python for Data Analysis" by Wes McKinney


    http://astronomi.erciyes.edu.tr/wp-content/uploads/astronom/pdf/OReilly%20Python%20for%20Data%20Analysis.pdf

    (If the above link goes stale, simply search Python for Data Analysis pdf.)


    The Second Edition is available from O'Reilly on Safari Bookshelf:

    http://shop.oreilly.com/product/0636920050896.do

    Please keep in mind that pandas is in active development, which means that features may be added, removed and changed (latest version: 0.25.2)

    blog tutorials

    These often provide the kind of "insider view" that is most helpful when getting oriented.


    Tom Augspurger blog (6-part series)

    http://tomaugspurger.github.io/modern-1.html

    Greg Reda blog (3-part series)

    http://gregreda.com/2013/10/26/intro-to-pandas-data-structures/

    cheat sheet (Treehouse)

    https://s3.amazonaws.com/assets.datacamp.com/blog_assets/PandasPythonForDataScience.pdf



    Inspecting attributes and docs on pandas and pandas objects

    An object type that is new to us can be explored through attribute inspection -- we can list the object's attributes with dir() and see brief documentation on an attribute with help().


    import pandas as pd
    
    # list of pandas attributes (global vars)
    print(dir(pd))
    
    # list of pandas functions (filtering for <class 'function'> only)
    import types
    print([ attr for attr in dir(pd) if isinstance(getattr(pd, attr), types.FunctionType) ])
    
    # short doc on read_csv() function
    help(pd.read_csv)       # help on the read_csv function of pandas
    
    
    # list of Series attributes
    s1 = pd.Series()
    print((dir(s1)))
    
    # list of DataFrame attributes
    df = pd.DataFrame()
    print(dir(df))
    
    # list of DataFrame methods (filtering for <class 'method'> only)
    print([ attr for attr in dir(df)
            if isinstance(getattr(df, attr), types.MethodType) ])
    
    # short doc on DataFrame join() method
    help(df.join)           # help on the join() method of a DataFrame



    pandas Object Types

    DataFrame: rows and columns; Series: a single column or single row; Index: column or row labels.


    Series * an ordered sequence of values indexed by labels * items addressable by index label (dict-like) * items addressable by sequence ineger (list-like) * has a dtype attribute that holds its objects' common type DataFrame: * is the core pandas structure -- a 2-dimensional array * is like a "dict of dicts" in that columns and rows can be indexed by label * is like a "list of lists" in that columns and rows can be indexed by integer index * is also like an Excel spreadsheet - rows, columns, and row and column labels Index * an object that provides indexing for both the Series (its item index) and the DataFrame (its column and row indices).


    The Series: Ordered Sequence of Values Indexed by Label

    Like a list, but with item labels... so like a dict too.



    Initialize a single series

    import pandas as pd
    
    s1 = pd.Series([10, 20, 30], index=['r1', 'r2', 'r3'], name='a')
    
    print(s1)
                # r1    10
                # r2    20
                # r3    30
                # Name: b, dtype: int64
    
    s2 = pd.Series(['x', 'y', 'z'], index=['r1', 'r2', 'r3'], name='b')
    
    
    # combine Series to make a DataFrame
    df = pd.DataFrame(s1, s2)

    The DataFrame

    The DataFrame is the pandas workhorse structure. It is a 2-dimensional structure with columns and rows (i.e., a lot like a spreadsheet).


    Initializing

    import pandas as pd
    
    # initialize a new, empty DataFrame
    df = pd.DataFrame()
    
    
    # init with dict of lists (keyed to columns) and index
    df = pd.DataFrame( {'a': [1, 2, 3],
                        'b': [1.0, 1.5, 2.0],
                        'c': ['a', 'b', 'c']  },
                        index=['r1', 'r2', 'r3'] )
    
    print(df)
                #     a    b  c
                # r1  1  1.0  a
                # r2  2  1.5  b
                # r3  3  2.0  c

    previewing the DataFrame


    print(len(df))     # 3 (# of rows)
    
    print(df.head(2))  # 1st 2 rows
    
    print(df.tail(2))  # last 2 rows

    column attribute / subscripting: delivers a Series


    sa = df.a         # or df['a']
    
    print(sa)          # pandas.Series([1, 2, 3], index=['r1', 'r2', 'r3'], name='a')

    label and positional access


    print(df.loc['r2', 'b'])    # 1.5
    print(df.iloc[1, 1])        # 1.5

    .columns and .index attributes


    Columns and rows can be accessed through the DataFrame's attributes:

    print(df.columns)        # Index(['a', 'b', 'c'], dtype='object')
    print(df.index)          # Index(['r1', 'r2', 'r3'], dtype='object')

    DataFrame as a list of Series objects Again, any DataFrame's columns or rows can be sliced out as a Series:


    # read a column as a Series (use DataFrame subscript)
    bcol = df['b']
    print(bcol)
                # r1    1.0
                # r2    1.5
                # r3    2.0
                # Name: b, dtype: float64
    
    
    
    # read a row as a Series (use subscript of df.loc[])
    oneidx = df.loc['r2']
    print(oneidx)
                # a      2
                # b    1.5
                # c      b
                # Name: r2, dtype: object

    Note: df is a common variable name for pandas DataFrame objects; you will see this name used frequently in these examples.


    The Index: DataFrame Column or Index Labels

    An Index object is used to specify a DataFrame's columns or index, or a Series index.


    Columns and Indices


    A DataFrame makes use of two Index objects: one to represent the columns, and one to represent the rows.

    df = pd.DataFrame( {'a': [1, 2, 3, 4],
                        'b': [1.0, 1.5, 2.0, 2.5],
                        'c': ['a', 'b', 'c', 'd'],
                        'd': [100, 200, 300, 400] },
                        index=['r1', 'r2', 'r3', 'r4'] )
    
    print((df.index))    # Index(['r1', 'r2', 'r3', 'r4'], dtype=object)
    
    print((df.columns))  # Index(['a', 'b', 'c', 'd'], dtype=object)
    
    # set name for index and columns
    df.index.name = 'year'
    df.columns.name = 'state'
    
    s_index = s1.index     # Index(['r1', 'r2', 'r3'])
    columns = df.columns   # Index(['a',  'b',  'c'],  dtype='object')
    idx = df.index         # Index(['r1', 'r2', 'r3'], dtype='object')
    

    There are a number of "exotic" Index object types as well - all come from Index and behave in the same ways: Index (standard, default and most common Index type) RangeIndex (index built from an integer range) Int64Index, UInt64Index, Float64Index (index values of specific types) DatetimeIndex, TimedeltaIndex, PeriodIndex, IntervalIndex (datetime-related indices) CategoricalIndex (index related to the Categorical type)


    pandas Objects are Like Python Objects

    DataFrames behave as you might expect when converted to any Python container


    df = pd.DataFrame( {'a': [1, 2, 3, 4],
                        'b': [1.0, 1.5, 2.0, 2.5],
                        'c': ['a', 'b', 'b', 'a'] }, index=['r1', 'r2', 'r3', 'r4'] )
    
    
    print(len(df))             # 4
    
    print(len(df.columns))     # 3
    
    print(max(df['a']))        # 4
    
    print(list(df['a']))       # [1, 2, 3, 4]     (column for 'a')
    
    print(list(df.loc['r2']))  # [2, 1.5, 'b']   (row for 'r2')
    
    print(set(df['c']))        # {'b', 'a'}       (a set of unique values)

    DataFrame .values -- convert to a list of numpy arrays


    An numpy array is a list-like object. simple list comprehension could convert these to a list of lists:

    print((df.values))
    
        # array([[1, 1.0, 'a'],
        #        [2, 1.5, 'b'],
        #        [3, 2.0, 'b'],
        #        [4, 2.5, 'a']], dtype=object)
    
    
    lol = list( [ list(item) for item in df.values ])
    
    print(lol)
    
                              # [ [1, 1.0, 'a'],
                              #   [2, 1.5, 'b'],
                              #   [3, 2.0, 'b'],
                              #   [4, 2.5, 'a'] ]

    looping - loops through columns


    for colname in df:
        print('{}:  {}'.format(colname, type(df[colname])))
    
                              # a:  <pandas.core.series.Series>
                              # b:  <pandas.core.series.Series>
                              # c:  <pandas.core.series.Series>
    
    
    # looping with iterrows -- loops through rows
    for row_index, row_series in df.iterrows():
        print('{}:  {}'.format(row_index, type(row_series)))
    
                              # r1:  <pandas.core.series.Series>
                              # r2:  <pandas.core.series.Series>
                              # r3:  <pandas.core.series.Series>
                              # r4:  <pandas.core.series.Series>

    Although keep in mind that we generally prefer vectorized operations across columns or rows to looping (discussed later).


    DataFrame: Initializing from Data Source

    DataFrame can be read from CSV, JSON, Excel and XML formats.


    CSV

    # read from file
    df = pd.read_csv('quarterly_revenue_2017Q4.csv')
    
    
    # write to file
    wfh = open('output.csv', 'w')
    df.to_csv(wfh, na_rep='NULL')
    
    
    # reading from Fama-French file (the abbreviated file, no header)
    # sep= indicates the delimiter on which to split() the fields
    # names= indicates the column heads
    df = pd.read_csv('FF_abbreviated.txt', sep='\s+',
                                           names=['date', 'MktRF', 'SMB', 'HML', 'RF'])
    
    
    # reading from Fama-French non-abbreviated (the main file including headers and footers)
    # skiprows=5:  start reading 5 rows down
    df = pd.read_csv('F-F_Research_Data_Factors_daily.txt', skiprows=5, sep='\s+',
                                                            names=['date', 'MktRF', 'SMB', 'HML', 'RF'])
    
    df.to_csv('newfile.csv')

    Excel

    # reading from excel file to DataFrame
    df = pd.read_excel('revenue.xlsx', sheet_name='Sheet1')
    
    # optional:  produce a 'reader' object used to obtain sheet names, etc.
    xls_file = pd.ExcelFile('data.xls')    # produce a file 'reader' object
    df = xls_file.parse('Sheet1')          # parse a selected sheet
    
    
    # write to excel
    df.to_excel('data2.xls', sheet_name='Sheet1')

    JSON

    # sample df for demo purposes
    df = pd.DataFrame( {'a': [1, 2, 3, 4],
                        'b': [1.0, 1.5, 2.0, 2.5],
                        'c': ['a', 'b', 'c', 'd'] }, index=['r1', 'r2', 'r3', 'r4'] )
    
    
    
    # write dataframe to JSON
    pd.json.dump(df, open('df.json', 'w'))
    
    mydict = pd.json.load(open('df.json'))
    new_df = pd.DataFrame(mydict)

    Relational Database

    import sqlite3                        # file-based database format
    conn = sqlite3.connect('example.db')  # a db connection object
    
    df = pd.read_sql('SELECT this FROM that', conn)

    The above can be used with any database connection (MySQL, Oracle, etc.)


    From Clipboard: this option is excellent for cutting and pasting data from websites

    df = pd.read_clipboard(skiprows=5, sep='\s+',
                           names=['date', 'MktRF', 'SMB', 'HML', 'RF'])

    Dataframe and Series dtypes

    pandas infers a column type based on values and applies it to the column automatically.


    Pandas is built on top of numpy, a numeric processing module, compiled in C for efficiency. Unlike core Python containers (but similar to a database table), numpy cares about object type. Wherever possible, numpy will assign a type to a column of values and attempt to maintain the type's integrity. This is done for the same reason it is done with database tables: speed and space efficiency. In the below DataFrame, numpy/pandas "sniffs out" the type of a column Series. It will set the type most appropriate to the values.


    import pandas as pd
    
    df = pd.DataFrame( {'a': [1, 2, 3, 4],
                        'b': [1.0, 1.5, 2.0, 2.5],
                        'c': ['a', 'b', 'c', 'd'],
                        'd': ['2016-11-01', '2016-12-01', '2017-01-01', '2018-02-01'] },
                        index=['r1', 'r2', 'r3', 'r4'] )
    
    print(df)
                      #     a    b  c          d
                      # r1  1  1.0  a 2016-11-01
                      # r2  2  1.5  b 2016-12-01
                      # r3  3  2.0  c 2017-01-01
                      # r4  4  2.5  d 2017-02-01
    
    print(df.dtypes)
    
                      # a      int64       # note special pandas types int64 and float64
                      # b    float64
                      # c     object       # 'object' is general-purpose type,
                      # d     object       #     covers strings or mixed-type columns
                      # dtype: object

    You can use the regular integer index to set element values in an existing Series. However, the new element value must be the same type as that defined in the Series; if not, pandas may refuse, or it may upconvert or cast the Series column to a more general type (usually object, because numpy is focused on the efficiency of numeric and datetime types).


    print(df.b.dtype)       # float64
    df.loc['b'] = 'hello'
    print(df.b.dtype)       # object

    Note that we never told pandas to store these values as floats. But since they are all floats, pandas decided to set the type.


    We can change a dtype for a Series ourselves with .astype():

    df.a = df.a.astype('object')     # or df['a'] = df['a'].astype('object')
    
             #  df.loc[0, 'a'] = 'hello'

    The numpy dtypes you are most likely to see are:

    int64
    float64
    datetime64
    object

    Checking the memory usage of a DataFrame


    .info() provides approximate memory size of a DataFrame

    df.info()   # on the original example at the top
    
             #  
             #  Index: 4 entries, r1 to r4
             #  Data columns (total 4 columns):
             #  a    4 non-null int64
             #  b    4 non-null float64
             #  c    4 non-null object
             #  d    4 non-null object
             #  dtypes: float64(1), int64(1), object(2)
             #  memory usage: 160.0+ bytes

    '+' means "probably larger" -- info() only sizes numeric types, not 'object'


    With memory_usage='deep', size includes type'object'

    df.info(memory_usage='deep')
    
             # memory usage: 832 bytes

    Line Chart on a Series

    A Series object (usually a column from a DataFrame) can be easily plotted as a line.


    import pandas as pd
    
    df = pd.read_csv('weather_newyork.csv')
    
    print(len(df))   # 365   (weather information for each day)
    
    df.mean_temp.plot()



    Complex Sorting

    Summary Function: sorted()

    sorted() takes a sequence argument and returns a sorted list. The sequence items are sorted according to their respective types.


    sorted() with numbers

    mylist = [4, 3, 9, 1, 2, 5, 8, 6, 7]
    
    sorted_list = sorted(mylist)
    print(sorted_list)                # [1, 2, 3, 4, 5, 6, 7, 8, 9]

    sorted() with strings

    namelist = ['jo', 'pete', 'michael', 'zeb', 'avram']
    
    print(sorted(namelist))           # ['avram', 'jo', 'michael', 'pete', 'zeb']

    Summary Task (Review): sorting a dictionary's keys

    Sorting a dict means sorting the keys -- sorted() returns a list of sorted keys.


    bowling_scores = {'jeb': 123, 'zeb': 98, 'mike': 202, 'janice': 184}
    
    sorted_keys = sorted(bowling_scores)
    
    print(sorted_keys)                     # ['janice', 'jeb', 'mike', 'zeb']

    Indeed, any "listy" sort of operation on a dict assumes the keys: for looping, subscripting, sorted(); even sum(), max() and min().


    Summary Task (Review): sorting a dictionary's keys by its values

    The dict get() method returns a value based on a key -- perfect for sorting keys by values.


    bowling_scores = {'jeb': 123, 'zeb': 98, 'mike': 202, 'janice': 184}
    
    sorted_keys = sorted(bowling_scores, key=bowling_scores.get)
    
    print(sorted_keys)                     # ['zeb', 'jeb', 'janice', 'mike']
    
    
    for player in sorted_keys:
          print(f"{player} scored {bowling_scores[player]}")
    
            ##  zeb scored 98
            ##  jeb scored 123
            ##  janice scored 184
            ##  mike scored 202

    Summary Feature: Custom sort using an sorting helper function

    A sorting helper function returns to python the value by which a given element should be sorted.


    Here is the same dict sorted by value in the same way as previously, through a custom sorting helper function.

    def by_value(dict_key):                       # a key to be sorted
                                                  # (for example, 'mike'
    
        dict_value = bowling_scores[dict_key]     # retrieving the value based on
                                                  #  'mike':  202
    
        return dict_value                         # returning the value 202
    
    bowling_scores = {'jeb': 123, 'zeb': 98, 'mike': 202, 'janice': 184}
    sorted_keys = sorted(bowling_scores, key=by_value)
    
    print(sorted_keys)          # ['zeb', 'jeb', 'janice', 'mike']

    The dict's keys are sorted by value because of the by_value() sorting helper function: 1. sorted() sees by_value referenced in the function call. 2. sorted() calls the by_value() four times: once with each key in the dict. 3. by_value() is called with 'jeb' (which returns 123), 'zeb' (which returns 98), 'mike' (which returns 202), and 'janice' (which returns 184). 4. The return value of the function is the value by which the key will be sorted Therefore because of the return value of the sorting helper function, jeb will be sorted by 123, zeb by 98, etc.


    Summary Task: sort a numeric string by its numeric value

    Numeric strings (as we might receive from a file) sort alphabetically:


    numbers_from_file = ['1', '10', '3', '20', '110', '1000' ]
    sorted_numbers = sorted(numbers_from_file)
    
    print(sorted_numbers)    # ['1', '1000', '110', '20', '3'] (alphabetic sort)

    To sort numerically, the sorting helper function can convert to int or float.

    def by_numeric_value(this_string):
        return int(this_string)
    
    numbers_from_file = ['1', '10', '3', '20', '110', '1000' ]
    sorted_numbers = sorted(numbers_from_file, key=by_numeric_value)
    
    print(sorted_numbers)    # ['1', '3', '10', '20', '110', '1000']

    Note that the values returned do not change; they are simply sorted by their integer equivalent.


    Summary Task: sort a string by its case-insensitive value

    Python string sorting sorts uppercase before lowercase:


    namelist = ['Jo', 'pete', 'Michael', 'Zeb', 'avram']
    print(sorted(namelist))            # ['Jo', 'Michael', 'Zeb', 'avram', 'pete']

    To sort "insensitively", the sorting helper function can lowercase each string.

    def by_lowercase(my_string):
        return my_string.lower()
    
    namelist = ['Jo', 'pete', 'michael', 'Zeb', 'avram']
    
    print(sorted(namelist, key=by_lowercase))
    
                                      # ['avram', 'Jo', 'michael', 'pete', 'Zeb']

    Summary Task: sort a string by a portion of the string

    To sort a string by a portion of the string (for example, the last name in these 2-word names), we can split or slice the string and return the portion.


    full_names = ['Jeff Wilson', 'Abe Zimmerman', 'Zoe Apple', 'Will Jefferson']
    
    def by_last_name(fullname):
        fname, lname = fullname.split()
        return lname
    
    sfn = sorted(full_names, key=by_last_name)
    
    print(sfn)                                     #  ['Zoe Apple',
                                                   #   'Will Jefferson',
                                                   #   'Jeff Wilson',
                                                   #   'Abe Zimmerman']

    Summary Task: sort a file line by a field within the line

    To sort a string of fields (for example, a CSV line) by a field within the line, we can split() and return a field from the split.


    def by_third_field(this_line):
        els = this_line.split(',')
        return els[2]
    
    lines = open('students.txt')
    sorted_lines = sorted(lines, key=by_third_field)
    print(sorted_lines)
    
    
            # [ 'pk669,Pete,Krank,Darkling,NJ,8044894893\n',
            #   'ms15,Mary,Smith,Wilsontown,NY,5185853892\n',
            #   'jw234,Joe,Wilson,Smithtown,NJ,2015585894\n'    ]

    Summary Task: custom sort using a built-in function as the sorting helper function

    Built-in functions can be used to help sorted() decide how to sort in the same way as custom functions -- by telling Python to pass an element and sort by the return value.


    len() returns string length - so it can be used to sort strings by length

    mystrs = ['angie', 'zachary', 'zeb', 'annabelle']
    
    print(sorted(mystrs, key=len))      # ['zeb', 'angie', 'zachary', 'annabelle']

    Using a builtin function


    os.path.getsize() returns the byte size of any file based on its name (in this example, in the present working directory):

    import os
    
    print(os.path.getsize('test.txt'))  # return 53, the byte size of test.txt

    To sort files by their sizes, we can simply pass this function to sorted()

    import os
    
    files = ['test.txt', 'myfile.txt', 'data.csv', 'bigfile.xlsx']
    
                                        # some files in my current dir
    
    size_files = sorted(files, key=os.path.getsize)
                                        # pass each file to getsize()
    
    for this_file in size_files:
          print("{this_file}:  {os.path.getsize(this_file)} bytes")

    (Please note that this will only work if your terminal's present working directory is the same as the files being sorted. Otherwise, you would have to prepend the path -- see File I/O, later in this course.)


    Using methods

    namelist = ['Jo', 'pete', 'michael', 'Zeb', 'avram']
    print(sorted(namelist, key=str.lower))
    
                                      # ['avram', 'Jo', 'michael', 'pete', 'Zeb']

    Using methods called on existing objects

    companydict = {'IBM': 18.68, 'Apple': 50.56, 'Google': 21.3}
    
    revc = sorted(companydict, key=companydict.get)   # [ 'IBM',
                                                             #   'Google',
                                                             #   'Apple'   ]

    You can use a method here in the same way you would use a function, except that you won't be specifying the specific object as you would normally with a method. To refer to a method "in the abstract", you can say str.upper or str.lower. However, make sure not to actually call the method (which is done with the parentheses). Instead, you simply refer to the method, i.e., mention the method without using the parentheses.)


    Summary task: sorting dict of dicts with custom function

    Having built multidimensional structures in various configurations, we should now learn how to sort them -- for example, to sort the keys in a dictionary of dictionaries by one of the values in the inner dictionary (in this instance, the last name):


    (The "thing to sort" is a dict key. The "value by which it should be sorted" is a value within the dict associated with that key, in this case the 'lname' value.)


    def by_last_name(key):
        return dod[key]['lname']
    
    dod = {
             'db13':  {
                         'fname': 'Joe',
                         'lname': 'Wilson',
                         'tel':   '9172399895'
                      },
             'mm23':  {
                         'fname': 'Mary',
                         'lname': 'Doodle',
                         'tel':   '2122382923'
                      }
           }
    
    sorted_keys = sorted(dod, key=by_last_name)
    print(sorted_keys)                             # ['mm23', 'db13']

    The trick here will be to put together what we know about obtaining the value from an inner structure with what we have learned about custom sorting.


    Summary task: sorting list of dicts with custom function

    Similar to itemgetter, we may want to sort a complex structure by some inner value - If we have a list of dicts to sort, we can use the custom sub to specify the sort value from inside each dict.


    (The "thing to sort" is a dict. The "value by which it should be sorted" is the 'lname' in the dict.)


    def by_dict_lname(this_dict):
        return this_dict['lname'].lower()
    
    list_of_dicts = [
      { 'id': 123,
        'fname': 'Joe',
        'lname': 'Wilson',
      },
      { 'id': 124,
        'fname': 'Sam',
        'lname': 'Jones',
      },
      { 'id': 125,
        'fname': 'Pete',
        'lname': 'abbott',
      },
    ]
    list_of_dicts.sort(key=by_dict_lname)      # custom sort function (above)
    for this_dict in list_of_dicts:
        print(f"{this_dict['fname']} {this_dict['lname']}")
    
    # Pete abbot
    # Sam Jones
    # Joe Wilson

    So, although we are sorting dicts, our sub says "take this dictionary and sort by this inner element of the dictionary".


    Sidebar: cascading sort

    Sort a list by multiple criteria by having your sorting helper function return a 2-element tuple.


    def by_last_first(name):
      fname, lname = name.split()
      return (lname, fname)
    
    names = ['Zeb Will', 'Deb Will', 'Joe Max', 'Ada Max']
    
    lnamesorted = sorted(names, key=by_last_first)
    
                                 # ['Ada Max', 'Joe Max', 'Deb Will', 'Zeb Will']

    Sorting review

    A quick review of sorting: recall how Python will perform a default sort (numeric or ASCII-betical) depending on the objects sorted. If we wish to modify this behavior, we can pass each element to a function named by the key= parameter:


    mylist = ['Alpha', 'Gamma', 'episilon', 'beta', 'Delta']
    
    print(sorted(mylist))                      # ASCIIbetical sort
                                              # ['Alpha', 'Gamma', 'Delta', 'beta', 'epsilon']
    
    mylist.sort()                             # sort mylist in-place
    
    print(sorted(mylist, key=str.lower))       # alphabetical sort
                                              # (lowercasing each item by telling Python to pass it
                                              # to str.lower)
                                              # ['Alpha', 'beta', 'Delta', 'epsilon', 'Gamma']
    
    print(sorted(mylist, key=len))             # sort by length
                                              # ['beta', 'Alpha', 'Gamma', 'Delta', 'epsilon']

    Sorting review: sorting dictionary keys by value: dict.get

    When we loop through a dict, we can loop through a list of keys (and use the keys to get values) or loop through items, a list of (key, value) tuple pairs. When sorting a dictionary by the values in it, we can also choose to sort keys or items.


    To sort keys, mydict.get is called with each key - and get returns the associated value. So the keys of the dictionary are sorted by their values.

    mydict = { 'a': 5, 'b': 2, 'c': 1, 'z': 0 }
    mydict_sorted_keys = sorted(mydict, key=mydict.get)
    for i in mydict_sorted_keys:
        print(f"{i} = {mydict[i]}")
    
                     ## z = 0
                     ## c = 1
                     ## b = 2
                     ## a = 5

    Sorting dictionary items by value: operator.itemgetter

    Recall that we can render a dictionary as a list of tuples with the dict.items() method:


    mydict = { 'a': 5, 'b': 2, 'c': 1, 'z': 0 }
    mydict_items = list(mydict.items())                        # [(a, 5), (c, 1), (b, 2), (z, 0)]

    To sort dictionary items by value, we need to sort each two-element tuple by its second element. The built-in module operator.itemgetter will return whatever element of a sequence we wish - in this way it is like a subscript, but in function format (so it can be called by the Python sorting algorithm).

    import operator
    mydict = { 'a': 5, 'b': 2, 'c': 1, 'z': 0 }
    mydict_items = mydict.items()                        # [(a, 5), (c, 1), (b, 2), (z, 0)]
    mydict_items.sort(key=operator.itemgetter(1))
    print(mydict_items)                                  # [(z, 0), (c, 1), (b, 2), (a, 5)]
    for key, val in mydict_items:
        print(f"{key} = {val}")
    
                        ## z = 0
                        ## c = 1
                        ## b = 2
                        ## a = 5

    The above can be conveniently combined with looping, effectively allowing us to loop through a "sorted" dict:

    for key, val in sorted(mydict.items(), key=operator.itemgetter(1)):
        print(f"{key} = {val}")

    Database results come as a list of tuples. Perhaps we want our results sorted in different ways, so we can store as a list of tuples and sort using operator.itemgetter. This example sorts by the third field, then by the second field (last name, then first name):

    import operator
    items =[ (123, 'Joe', 'Wilson', 35, 'mechanic'),
             (124, 'Sam', 'Jones', 22, 'mechanic'),
             (125, 'Pete', 'Jones', 40, 'mechanic'),
             (126, 'Irina', 'Bibi', 31, 'mechanic'),
           ]
    items.sort(key=operator.itemgetter(2,1)) # sorts by last, first name
    for this_pair in items:
      print(f"{this_pair[1]} {this_pair[2]}")
    
           ## Irina Bibi
           ## Pete Jones
           ## Sam Jones
           ## Joe Wilson

    Multi-dimensional structures: sorting with lambda custom function

    Functions are useful but they require that we declare them separately, elsewhere in our code. A lambda is a function in a single statement, and can be placed in data structures or passed as arguments in function calls. The advantage here is that our function is used exactly where it is defined, and we don't have to maintain separate statements.


    A common use of lambda is in sorting. The format for lambdas is lambda arg: return_val. Compare each pair of regular function and lambda, and note the argument and return val in each.


    def by_lastname(name):
      fname, lname = name.split()
      return lname
    
    names = [ 'Josh Peschko', 'Gabriel Feghali', 'Billy Woods', 'Arthur Fischer-Zernin' ]
    sortednames = sorted(names, key=lambda name:  name.split()[1])
    
    
    list_of_dicts = [
      { 'id': 123,
        'fname': 'Joe',
        'lname': 'Wilson',
      },
      { 'id': 124,
        'fname': 'Sam',
        'lname': 'Jones',
      },
      { 'id': 125,
        'fname': 'Pete',
        'lname': 'abbott',
      },
    ]
    
    def by_dict_lname(this_dict):
      return this_dict['lname'].lower()
    
    sortedlenstrs = sorted(list_of_dicts, key=lambda this_dict:  this_dict['lname'].lower())

    In each, the label after lambda is the argument, and the expression that follows the colon is the return value. So in the first example, the lambda argument is name, and the lambda returns name.split()[1]. See how it behaves exactly like the regular function itself? Again, what is the advantage of lambdas? They allow us to design our own functions which can be placed inline, where a named function would go. This is a convenience, not a necessity. But they are in common use, so they must be understood by any serious programmer.


    Lambda expressions: breaking them down

    Many people have complained that lambdas are hard to grok (absorb), but they're really very simple - they're just so short they're hard to read. Compare these two functions, both of which add/concatenate their arguments:


    def addthese(x, y):
      return x + y
    
    addthese2 = lambda x, y:  x + y
    
    print(addthese(5, 9))        # 14
    print(addthese2(5, 9))       # 14

    The function definition and the lambda statement are equivalent - they both produce a function with the same functionality.


    Lambda expression example: dict.get and operator.itemgetter

    Here are our standard methods to sort a dictionary:


    import operator
    mydict = { 'a': 5, 'b': 2, 'c': 1, 'z': 0 }
    for key, val in sorted(mydict.items(), key=operator.itemgetter(1)):
        print(f"{key} = {val}")
    
    for key in sorted(mydict, key=mydict.get):
        print(f"{key} = {mydict[key]}")

    Imagine we didn't have access to dict.get and operator.itemgetter. What could we do?

    mydict = { 'a': 5, 'b': 2, 'c': 1, 'z': 0 }
    for key, val in sorted(mydict.items(), key=lambda keyval:  keyval[1]):
        print(f"{key} = {val}")
    
    for key in sorted(mydict, key=lambda key:  mydict[key]):
        print(f"{key} = {mydict[key]}")

    These lambdas do exactly what their built-in counterparts do: in the case of operator.itemgetter, take a 2-element tuple as an argument and return the 2nd element in the case of dict.get, take a key and return the associated value from the dict


    Sorting Multidimensional Structures

    Having built multidimensional structures in various configurations, we should now learn how to sort them -- for example, to sort the keys in a dictionary of dictionaries by one of the values in the inner dictionary (in this instance, the last name):


    def by_last_name(key):
        return dod[key]['lname']
    
    dod = {
             'db13':  {
                         'fname': 'Joe',
                         'lname': 'Wilson',
                         'tel':   '9172399895'
                      },
             'mm23':  {
                         'fname': 'Mary',
                         'lname': 'Doodle',
                         'tel':   '2122382923'
                      }
           }
    
    sorted_keys = sorted(dod, key=by_last_name)
    print(sorted_keys)                             # ['mm23', 'db13']

    The trick here will be to put together what we know about obtaining the value from an inner structure with what we have learned about custom sorting.


    Sorting review

    A quick review of sorting: recall how Python will perform a default sort (numeric or ASCII-betical) depending on the objects sorted. If we wish to modify this behavior, we can pass each element to a function named by the key= parameter:


    mylist = ['Alpha', 'Gamma', 'episilon', 'beta', 'Delta']
    
    print(sorted(mylist))                      # ASCIIbetical sort
                                              # ['Alpha', 'Gamma', 'Delta', 'beta', 'epsilon']
    
    mylist.sort()                             # sort mylist in-place
    
    print(sorted(mylist, key=str.lower))       # alphabetical sort
                                              # (lowercasing each item by telling Python to pass it
                                              # to str.lower)
                                              # ['Alpha', 'beta', 'Delta', 'epsilon', 'Gamma']
    
    print(sorted(mylist, key=len))             # sort by length
                                              # ['beta', 'Alpha', 'Gamma', 'Delta', 'epsilon']

    Sorting review: sorting dictionary keys by value: dict.get

    When we loop through a dict, we can loop through a list of keys (and use the keys to get values) or loop through items, a list of (key, value) tuple pairs. When sorting a dictionary by the values in it, we can also choose to sort keys or items.


    To sort keys, mydict.get is called with each key - and get returns the associated value. So the keys of the dictionary are sorted by their values.

    mydict = { 'a': 5, 'b': 2, 'c': 1, 'z': 0 }
    mydict_sorted_keys = sorted(mydict, key=mydict.get)
    for i in mydict_sorted_keys:
        print("{0} = {1}".format(i, mydict[i]))
    
                     ## z = 0
                     ## c = 1
                     ## b = 2
                     ## a = 5

    Multi-dimensional structures: sorting with custom function

    Similar to itemgetter, we may want to sort a complex structure by some inner value - in the case of itemgetter we sorted a whole tuple by its third value. If we have a list of dicts to sort, we can use the custom sub to specify the sort value from inside each dict:


    def by_dict_lname(this_dict):
      return this_dict['lname'].lower()
    
    list_of_dicts = [
      { 'id': 123,
        'fname': 'Joe',
        'lname': 'Wilson',
      },
      { 'id': 124,
        'fname': 'Sam',
        'lname': 'Jones',
      },
      { 'id': 125,
        'fname': 'Pete',
        'lname': 'abbott',
      },
    ]
    list_of_dicts.sort(key=by_dict_lname)      # custom sort function (above)
    for this_dict in list_of_dicts:
      print("{0} {1}".format(this_dict['fname'], this_dict['lname']))
    
    # Pete abbot
    # Sam Jones
    # Joe Wilson

    So, although we are sorting dicts, our sub says "take this dictionary and sort by this inner element of the dictionary".


    Multi-dimensional structures: sorting with lambda custom function

    Functions are useful but they require that we declare them separately, elsewhere in our code. A lambda is a function in a single statement, and can be placed in data structures or passed as arguments in function calls. The advantage here is that our function is used exactly where it is defined, and we don't have to maintain separate statements.


    A common use of lambda is in sorting. The format for lambdas is lambda arg: return_val. Compare each pair of regular function and lambda, and note the argument and return val in each.


    def by_lastname(name):
      fname, lname = name.split()
      return lname
    
    names = [ 'Josh Peschko', 'Gabriel Feghali', 'Billy Woods', 'Arthur Fischer-Zernin' ]
    sortednames = sorted(names, key=lambda name:  name.split()[1])
    
    
    list_of_dicts = [
      { 'id': 123,
        'fname': 'Joe',
        'lname': 'Wilson',
      },
      { 'id': 124,
        'fname': 'Sam',
        'lname': 'Jones',
      },
      { 'id': 125,
        'fname': 'Pete',
        'lname': 'abbott',
      },
    ]
    
    def by_dict_lname(this_dict):
      return this_dict['lname'].lower()
    
    sortedlenstrs = sorted(list_of_dicts, key=lambda this_dict:  this_dict['lname'].lower())

    In each, the label after lambda is the argument, and the expression that follows the colon is the return value. So in the first example, the lambda argument is name, and the lambda returns name.split()[1]. See how it behaves exactly like the regular function itself? Again, what is the advantage of lambdas? They allow us to design our own functions which can be placed inline, where a named function would go. This is a convenience, not a necessity. But they are in common use, so they must be understood by any serious programmer.


    Lambda expressions: breaking them down

    Many people have complained that lambdas are hard to grok (absorb), but they're really very simple - they're just so short they're hard to read. Compare these two functions, both of which add/concatenate their arguments:


    def addthese(x, y):
      return x + y
    
    addthese2 = lambda x, y:  x + y
    
    print(addthese(5, 9))        # 14
    print(addthese2(5, 9))       # 14

    The function definition and the lambda statement are equivalent - they both produce a function with the same functionality.


    Lambda expression example: dict.get and operator.itemgetter

    Here are our standard methods to sort a dictionary:


    import operator
    mydict = { 'a': 5, 'b': 2, 'c': 1, 'z': 0 }
    for key, val in sorted(list(mydict.items()), key=operator.itemgetter(1)):
        print("{0} = {1}".format(key, val))
    
    for key in sorted(mydict, key=mydict.get):
        print("{0} = {1}".format(key, mydict[key]))

    Imagine we didn't have access to dict.get and operator.itemgetter. What could we do?

    mydict = { 'a': 5, 'b': 2, 'c': 1, 'z': 0 }
    for key, val in sorted(list(mydict.items()), key=lambda keyval:  keyval[1]):
        print("{0} = {1}".format(key, val))
    
    for key in sorted(mydict, key=lambda key:  mydict[key]):
        print("{0} = {1}".format(key, mydict[key]))

    These lambdas do exactly what their built-in counterparts do: in the case of operator.itemgetter, take a 2-element tuple as an argument and return the 2nd element in the case of dict.get, take a key and return the associated value from the dict




    Functions and Proper Code Organization

    Summary Statement: Functions (user-defined)

    A user-defined function is simply a named code block that can be called and executed any number of times.


    def print_hello():
        print("Hello, World!")
    
    print_hello()             # prints 'Hello, World!'
    print_hello()             # prints 'Hello, World!'
    print_hello()             # prints 'Hello, World!'

    Function argument(s)


    A function's arguments are renamed in the function definition, and the function refers to them by these names.

    def print_hello(greeting, person):
      full_greeting = greeting + ", " + person + "!"
      print(full_greeting)
    
    print_hello('Hello', 'World')         # prints 'Hello, World!'
    print_hello('Bonjour', 'Python')      # prints 'Bonjour, Python!'
    print_hello('squawk', 'parrot')       # prints 'squawk, parrot!'

    (The argument objects are copied to the argument names -- they are the same objects.) Function return value


    A function's return value is passed back from the function with the return statement.

    def print_hello(greeting, person):
      full_greeting = greeting + ", " + person + "!"
      return full_greeting
    
    msg = print_hello('Bonjour', 'parrot')
    print(msg)                                       # 'Bonjour, parrot!'

    Coding all Python scripts as modules

    All Python scripts should be coded as modules: able to be both imported and executed.


    #!/usr/bin/env python
    
    """ helloworld.py:  print the "Hello! message """
    
    PI = 3.14
    LOGFILE = 'mylog.txt'
    
    def print_hello(arg):
        """ greets the user """
        print(f"Hello, {arg}!")
    
    def logmsg(msg):
        wfh = open(LOGFILE, 'w')
        wfh.write(f'{msg}\n')
        wfh.close()
    
    def main():
        """ the main event """
        print_hello('World')
    
    if __name__ == '__main__':
        logmsg('starting...')
        main()
        logmsg('ending...')
    

    If we run the above script, we'll see "Hello, World!". But if we import the above script, we won't see anything. Why? Because the if statement above will be true only if the script is executed. This is important behavior, because there are some scripts that we may want to run directly, but also allow others to import (in order to use the script's functions). Whether we intend to import a script or not, it is considered a "best practice" to build all of our programs in this way -- with a "main body" of statements collected under function main() and the call to main() inside the if __name__ == '__main__' gate.


    Raising exceptions

    Causing an exception to be raised is the principal way to signal errors from a function.


    A file called mylib.py

    def get_yearsum(user_year):
    
        user_year = int(user_year)
        if user_year < 1929 or user_year > 2013:
          raise ValueError(f'year {user_year} out of range')
    
        # calculate value for the year
    
        return 5.9         # returning a sample value (for testing purposes only)

    An exception raised by us is indistinguishable from one raised by Python, and we can raise any exception type we wish. This allows the user of our function to handle the error if needed (rather than have the script fail):


    import mylib
    
    while True:
    
        year = input('please enter a year:  ')
    
        try:
            mysum = mylib.get_yearsum(year)
            break
        except ValueError:
            print('invalid year:  try again')
    
    print('mysum is', mysum)



    SQL Part 3: CREATE TABLE and INSERT INTO

    Special Note: sqlite3 client column formatting

    Issue these two commands at the start of your sqlite> session


    At the start of your session, issue the following two commands -- these will format your sqlite3 output so it is clearer, and add columns headers.

    sqlite> .mode column
    sqlite> .headers on
    sqlite> SELECT * FROM students;
    
                  company     state       cost
                  ----------  ----------  ----------
                  Haddad's    PA          239.5
                  Westfield   NJ          53.9
                  The Store   NJ          211.5
                  Hipster's   NY          11.98
                  Dothraki F  NY          5.98
                  Awful's     PA          23.95
                  The Clothi  NY          115.2

    You may note that columns are 10 characters wide and that longer fields are cut off. You can set the width with values for each column for example .width 5 13 11 5 5 for the above table. Unfortunately this must be done separately for each table.


    Relational Table Structure and CREATE TABLE

    Table columns specify a data type.


    From the command line:

    $ sqlite3
    
    sqlite> .schema students
    CREATE TABLE revenue (company TEXT, state TEXT, cost FLOAT);

    .schema shows us the statement used to create this table. (In other databases, the DESC [tablename] statement shows table columns and types.) As you can see, each column is paired with a column type, which describes what kind of data can be stored in that column. To create a new table, we must specify a type for each column.

    type data stored
    INTEGER integers
    FLOAT floating-point values
    REAL numbers larger than INT
    TEXT string values
    BLOB binary or other data


    INSERT INTO statement and conn.commit()

    This statement adds a row (or rows) to a table. conn.commit() commits the transaction.


    # names fields to match values
    c.execute("INSERT INTO revenue (company, state, cost) VALUES ('IBM', 'NY', 3.50)")
    conn.commit()
    
    # values only -- assumes column order in table
    c.execute("INSERT INTO revenue VALUES ('IBM', 'NY', 3.50)")
    conn.commit()

    As with code, proper syntax is essential. Careful with quotes! The main challenge with SQL queries is in proper use of quotation marks. TEXT fields must be enquoted while numeric feels must not be. Lastly, the statement can only be committed when we call commit() on the connection object.


    sqlite3.OperationalError

    This exception is generated by SQLite3, usually when our query syntax is incorrect.


    When you receive this error, it means you have asked sqlite3 to do something it is unable to do. In these cases, you should print the query just before the execute() statement.


    query = "insert into revenue values ('Acme', 'CA')"
    c.execute(query)
    
    Traceback (most recent call last):
      File "", line 1, in 
    sqlite3.OperationalError: table revenue has 3 columns but 2 values were supplied

    A common issue is the use of single quotes inside a quoted value:

    query = "insert into revenue values ('Awful's', 'NJ', 20.39)"
    c.execute(query)
    
    Traceback (most recent call last):
      File "", line 1, in 
    sqlite3.OperationalError: near "A": syntax error

    Looking closely at the query above, you see that the name "Awful's" has a quote in it. So when SQL attempts to parse this string, it becomes confused by the quote in the text. There are 3 possible solutions to this particular problem: 1. escape the quote (use 2 quotes instead of one in the quoted string) 2. use double-quotes around the string in the query 3. use parameterized arguments (preferred, see next)


    INSERT INTO with parameterized arguments

    We can 'inject' data into our INSERT query dynamically, similar to .format().


    co = 'IBM'
    state = 'NY'
    rev = 3.50
    
    # names fields to match values
    query = "INSERT INTO revenue (company, state, cost) VALUES (?, ?, ?)"
    c.execute(query, (co, state, rev))
    conn.commit()

    This accomplishes two things: we can develop a string template (similar to a .format() template) into which we can insert data that is dynamically accessed (during program run); more importantly, we can avoid any syntax issues that would occur for the use of quotes in the data being used.


    DELETE FROM

    Delete some or all rows with this query (take care!)


    DELETE FROM removes rows from a table.

    DELETE FROM students WHERE student_id = 'jk43'

    Take special care -- DELETE FROM with no critera will empty the table!

    DELETE FROM students

    WARNING -- the above statement clears out the table!




    [pr]