list object

# subscript a list of lines
lines = [ 'id,fname,lname,city,state,tel',
          'jw234,Joe,Wilson,Smithtown,NJ,2015585894',
          'ms15,Mary,Smith,Wilsontown,NY,5185853892',
          'pk669,Pete,Krank,Darkling,VA,8044894893'   ]

header = lines[0]           # 'id,fname,lname,city,state,tel'
last_line = lines[-1]       # 'pk669,Pete,Krank,Darkling,VA,8044894893'


# slice a list
data = lines[1:]            # from first line to end of file

print(data)

   # [ 'jw234,Joe,Wilson,Smithtown,NJ,2015585894',
   #   'ms15,Mary,Smith,Wilsontown,NY,5185853892',
   #   'pk669,Pete,Krank,Darkling,VA,8044894893'   ]


# get the length of a list of lines (# of lines in a file)
x = len(lines)               # 4

Summary: File Object

3 ways to read strings from a file.

for: read (newline ('\n') marks the end of a line)

fh = open('students.txt')                 # file object allows looping
                                          # through a series of strings
for my_file_line in fh:                   # my_file_line is a string
    print(my_file_line)                   # prints each line of students.txt

fh.close()                                # close the file

read(): read entire file as a single string

fh = open('students.txt')  # file object allows reading
text = fh.read()                          # read() method called on file
                                          # object returns a string
fh.close()                                # close the file

print(text)

The above prints:

jw234,Joe,Wilson,Smithtown,NJ,2015585894
ms15,Mary,Smith,Wilsontown,NY,5185853892
pk669,Pete,Krank,Darkling,NJ,8044894893

readlines(): read as a list of strings (each string a line)

fh = open('students.txt')
file_lines = fh.readlines()               # file.readlines() returns
                                          # a list of strings
fh.close()                                # close the file
print(file_lines)

The above prints:

['jw234,Joe,Wilson,Smithtown,NJ,2015585894\n', 'ms15,Mary,Smith,Wilsontown,
NY,5185853892\n', 'pk669,Pete,Krank,Darkling,NJ,8044894893\n']

Summary: String Object

Strings: 4 ways to manipulate strings from a file.

split() a string into a list of strings

mystr = 'jw234,Joe,Wilson,Smithtown,NJ,2015585894'
elements = mystr.split(',')
print(elements)                           # ['jw234', 'Joe', 'Wilson',
                                          #  'Smithtown', 'NJ', '2015585894']

(included for completeness): join() a list of strings into a string

mylist = ['jw234', 'Joe', 'Wilson', 'Smithtown', 'NJ', '2015585894']

line = ','.join(mylist)          # 'jw234,Joe,Wilson,Smithtown,NJ,2015585894'

slice a string

mystr = '2014-03-13 15:33:00'
year =  mystr[0:4]               # '2014'
month = mystr[5:7]               # '03'
day =   mystr[8:10]              # '13'

rstrip() a string

xx = 'this is a line with a newline at the end\n'

yy = xx.rstrip()                 # return a new string without the newline

print(yy)                        # 'this is a line with a newline at the end'

splitlines() a multiline string

fh = open('students.txt') # open the file, return
                                         # a file object
text = fh.read()                         # read the entire file into a string
                                         # (of course this includes newlines)

lines = text.splitlines()                # returns a list of strings
                                         # (similar to fh.readlines(),
                                         # except without newlines)

Reading a file: options

(Note that the remaining slides repeat some of the same material, but from a more practical perspective.)

for loop: loop line-by-line The for loop repeats execution of its block until the file is completely read. Note that the for block is very similar to the while. The difference is that while relies on a test to continue executing, but for continues until it reaches the end of the file.

fh = open('students.txt')              # file object allows
                                       # looping through a
                                       # series of strings

for xx in fh:                          # xx is a string
    print(xx)                          # prints each line of students.txt
fh.close()                             # close the file

"xx" is called a control variable, and it is automatically reassigned each line in the file as a string. break and continue work with for as well as while loops. readlines(): work with the file as a list of string lines

To capture the entire file into a list of lines, use the file readlines() method:

fh = open('students.txt')

lines = fh.readlines()

for line in lines:
    line = line.rstrip()
    print(line)

print(lines[0])                        # the first line from the file

print(len(lines))                      # the number of lines in the file

We can then loop through the list, or perform other operations (select a single line or slice, get the number of lines with len() of the list, etc.) read() with splitlines(): an easy way to drop the newlines

A handy trick is to read() the file into a string, then call splitlines() on the string to split on newlines.

fh = open('students.txt')

text = fh.read()
lines = text.splitlines()

for line in lines:
    print(line)

This has the effect of delivering the entire file as a list of lines, but with the newlines removed (because the string was split on them with splitlines()).

Containers: Lists, Sets and Tuples

Introduction: Collections of values can be used for various types of analysis.

With a collection of numeric values, we can perform many types of analysis that would not be possible with a simple count or sum.

We will summarize a year's worth of data in the Fama-French file as we did previously, but be able to say much more about it.

var = [1, 4.3, 6.9, 11, 15]                            # a list container

print(f'count is {len(var)}')              # count is 5
print(f'sum is {sum(var)}      ')              # sum is 38.2
print(f'average is {sum(var)}')                # average is
                                                         # 7.640000000000001

print(f'max val is {max(var)}')                # max val is 15
print(f'min val is {min(var)}')                # min val is 1

print(f'top two: {var[3]}, {var[4]}')      # top two: 11, 15

print(f'median is {var[int(len(var) / 2)]}')   # median is 6.9')

Introduction: Collections of values can be used to determine membership.

Checking one list against another is a core task in data analysis. We can validate arguments to a program, see if a user id is in a "whitelist" of valid users, see if a product is in inventory, etc.

We will apply membership testing to a spell checker, which simply checks every word in a file against a "whitelist" of correctly spelled words.

valid_actions = ['run', 'stop', 'search', 'reset']

uinput = input('please enter an action:  ')

if uinput in valid_actions:          # if string can be found in list
    print(f'great, I will "{uinput}"')
else:
    print('sorry, action not found')

Objectives for this Unit (Containers: Lists, Sets and Tuples)

Containers broaden our data analysis powers significantly over simple looping and summing. We can:

Container Objects: List, Set, Tuple

Compare and contrast the characteristics of each container.

def myfunc():
    len = 'inside myfunc'   # local scope:
                            # len is initialized in the function
    print(len)

print(len)                  # built-in scope:
                            # prints '<built-in function len>'

len = 'in global scope'     # assigned in global scope:  a global variable
print(len)                  # global scope:  prints 'in global scope'

myfunc()                    # prints 'inside myfunc'
                            # (i.e. the function executes)

print(len)                  # prints 'in global scope' (the local
                            # len is gone, so we see the global

del len                     # 'deletes' the global len
print(len)                  # prints '<built-in function len>'

Summary exception: UnboundLocalError

An UnboundLocalError exception signifies a local variable that is "read" before it is defined.

x = 99
def selector():
    x = x + 1         # "read" the value of x; then assign to x
selector()

# Traceback (most recent call last):
#   File "test.py", line 1, in <module>
#   File "test.py", line 2, in selector
# UnboundLocalError: local variable 'x' referenced before assignment

Remember that a local variable is one that is initialized or assigned inside a function. In the above example, x is a local variable. So Python sees x not as the global variable (with value 99) but as a local variable. However, in the process of initializing x Python attempts to read x, and realizes that is hasn't been initialized yet -- the code has attempted to reference (i.e., read the value of) x before it has been assigned.

Since we want Python to treat x as the global x, we need to tell it to do so. We can do this with the global keyword:

x = 99
def selector():
    global x
    x = x + 1
selector()
print(x)             # 100

Complex Sorting

Summary Function: sorted()

sorted() takes a sequence argument and returns a sorted list. The sequence items are sorted according to their respective types.

sorted() with numbers

mylist = [4, 3, 9, 1, 2, 5, 8, 6, 7]

sorted_list = sorted(mylist)
print(sorted_list)                # [1, 2, 3, 4, 5, 6, 7, 8, 9]

sorted() with strings

namelist = ['jo', 'pete', 'michael', 'zeb', 'avram']

print(sorted(namelist))           # ['avram', 'jo', 'michael', 'pete', 'zeb']

Summary Task (Review): sorting a dictionary's keys

Sorting a dict means sorting the keys -- sorted() returns a list of sorted keys.

bowling_scores = {'jeb': 123, 'zeb': 98, 'mike': 202, 'janice': 184}

sorted_keys = sorted(bowling_scores)

print(sorted_keys)                     # ['janice', 'jeb', 'mike', 'zeb']

Indeed, any "listy" sort of operation on a dict assumes the keys: for looping, subscripting, sorted(); even sum(), max() and min().

Summary Task (Review): sorting a dictionary's keys by its values

The dict get() method returns a value based on a key -- perfect for sorting keys by values.

bowling_scores = {'jeb': 123, 'zeb': 98, 'mike': 202, 'janice': 184}

sorted_keys = sorted(bowling_scores, key=bowling_scores.get)

print(sorted_keys)                     # ['zeb', 'jeb', 'janice', 'mike']


for player in sorted_keys:
      print(f"{player} scored {bowling_scores[player]}")

        ##  zeb scored 98
        ##  jeb scored 123
        ##  janice scored 184
        ##  mike scored 202

Summary Feature: Custom sort using an sorting helper function

A sorting helper function returns to python the value by which a given element should be sorted.

Here is the same dict sorted by value in the same way as previously, through a custom sorting helper function.

def by_value(dict_key):                       # a key to be sorted
                                              # (for example, 'mike'

    dict_value = bowling_scores[dict_key]     # retrieving the value based on
                                              #  'mike':  202

    return dict_value                         # returning the value 202

bowling_scores = {'jeb': 123, 'zeb': 98, 'mike': 202, 'janice': 184}
sorted_keys = sorted(bowling_scores, key=by_value)

print(sorted_keys)          # ['zeb', 'jeb', 'janice', 'mike']

The dict's keys are sorted by value because of the by_value() sorting helper function: 1. sorted() sees by_value referenced in the function call. 2. sorted() calls the by_value() four times: once with each key in the dict. 3. by_value() is called with 'jeb' (which returns 123), 'zeb' (which returns 98), 'mike' (which returns 202), and 'janice' (which returns 184). 4. The return value of the function is the value by which the key will be sorted Therefore because of the return value of the sorting helper function, jeb will be sorted by 123, zeb by 98, etc.

Summary Task: sort a numeric string by its numeric value

Numeric strings (as we might receive from a file) sort alphabetically:

numbers_from_file = ['1', '10', '3', '20', '110', '1000' ]
sorted_numbers = sorted(numbers_from_file)

print(sorted_numbers)    # ['1', '1000', '110', '20', '3'] (alphabetic sort)

To sort numerically, the sorting helper function can convert to int or float.

def by_numeric_value(this_string):
    return int(this_string)

numbers_from_file = ['1', '10', '3', '20', '110', '1000' ]
sorted_numbers = sorted(numbers_from_file, key=by_numeric_value)

print(sorted_numbers)    # ['1', '3', '10', '20', '110', '1000']

Note that the values returned do not change; they are simply sorted by their integer equivalent.

Summary Task: sort a string by its case-insensitive value

Python string sorting sorts uppercase before lowercase:

namelist = ['Jo', 'pete', 'Michael', 'Zeb', 'avram']
print(sorted(namelist))            # ['Jo', 'Michael', 'Zeb', 'avram', 'pete']

To sort "insensitively", the sorting helper function can lowercase each string.

def by_lowercase(my_string):
    return my_string.lower()

namelist = ['Jo', 'pete', 'michael', 'Zeb', 'avram']

print(sorted(namelist, key=by_lowercase))

                                  # ['avram', 'Jo', 'michael', 'pete', 'Zeb']

Summary Task: sort a string by a portion of the string

To sort a string by a portion of the string (for example, the last name in these 2-word names), we can split or slice the string and return the portion.

full_names = ['Jeff Wilson', 'Abe Zimmerman', 'Zoe Apple', 'Will Jefferson']

def by_last_name(fullname):
    fname, lname = fullname.split()
    return lname

sfn = sorted(full_names, key=by_last_name)

print(sfn)                                     #  ['Zoe Apple',
                                               #   'Will Jefferson',
                                               #   'Jeff Wilson',
                                               #   'Abe Zimmerman']

Summary Task: sort a file line by a field within the line

To sort a string of fields (for example, a CSV line) by a field within the line, we can split() and return a field from the split.

def by_third_field(this_line):
    els = this_line.split(',')
    return els[2]

lines = open('students.txt')
sorted_lines = sorted(lines, key=by_third_field)
print(sorted_lines)


        # [ 'pk669,Pete,Krank,Darkling,NJ,8044894893\n',
        #   'ms15,Mary,Smith,Wilsontown,NY,5185853892\n',
        #   'jw234,Joe,Wilson,Smithtown,NJ,2015585894\n'    ]

Summary Task: custom sort using a built-in function as the sorting helper function

Built-in functions can be used to help sorted() decide how to sort in the same way as custom functions -- by telling Python to pass an element and sort by the return value.

len() returns string length - so it can be used to sort strings by length

mystrs = ['angie', 'zachary', 'zeb', 'annabelle']

print(sorted(mystrs, key=len))      # ['zeb', 'angie', 'zachary', 'annabelle']

Using a builtin function

os.path.getsize() returns the byte size of any file based on its name (in this example, in the present working directory):

import os

print(os.path.getsize('test.txt'))  # return 53, the byte size of test.txt

To sort files by their sizes, we can simply pass this function to sorted()

import os

files = ['test.txt', 'myfile.txt', 'data.csv', 'bigfile.xlsx']

                                    # some files in my current dir

size_files = sorted(files, key=os.path.getsize)
                                    # pass each file to getsize()

for this_file in size_files:
      print("{this_file}:  {os.path.getsize(this_file)} bytes")

(Please note that this will only work if your terminal's present working directory is the same as the files being sorted. Otherwise, you would have to prepend the path -- see File I/O, later in this course.)

Using methods

namelist = ['Jo', 'pete', 'michael', 'Zeb', 'avram']
print(sorted(namelist, key=str.lower))

                                  # ['avram', 'Jo', 'michael', 'pete', 'Zeb']

Using methods called on existing objects

companydict = {'IBM': 18.68, 'Apple': 50.56, 'Google': 21.3}

revc = sorted(companydict, key=companydict.get)   # [ 'IBM',
                                                         #   'Google',
                                                         #   'Apple'   ]

You can use a method here in the same way you would use a function, except that you won't be specifying the specific object as you would normally with a method. To refer to a method "in the abstract", you can say str.upper or str.lower. However, make sure not to actually call the method (which is done with the parentheses). Instead, you simply refer to the method, i.e., mention the method without using the parentheses.)

Summary task: sorting dict of dicts with custom function

Having built multidimensional structures in various configurations, we should now learn how to sort them -- for example, to sort the keys in a dictionary of dictionaries by one of the values in the inner dictionary (in this instance, the last name):

(The "thing to sort" is a dict key. The "value by which it should be sorted" is a value within the dict associated with that key, in this case the 'lname' value.)

def by_last_name(key):
    return dod[key]['lname']

dod = {
         'db13':  {
                     'fname': 'Joe',
                     'lname': 'Wilson',
                     'tel':   '9172399895'
                  },
         'mm23':  {
                     'fname': 'Mary',
                     'lname': 'Doodle',
                     'tel':   '2122382923'
                  }
       }

sorted_keys = sorted(dod, key=by_last_name)
print(sorted_keys)                             # ['mm23', 'db13']

The trick here will be to put together what we know about obtaining the value from an inner structure with what we have learned about custom sorting.

Summary task: sorting list of dicts with custom function

Similar to itemgetter, we may want to sort a complex structure by some inner value - If we have a list of dicts to sort, we can use the custom sub to specify the sort value from inside each dict.

(The "thing to sort" is a dict. The "value by which it should be sorted" is the 'lname' in the dict.)

def by_dict_lname(this_dict):
    return this_dict['lname'].lower()

list_of_dicts = [
  { 'id': 123,
    'fname': 'Joe',
    'lname': 'Wilson',
  },
  { 'id': 124,
    'fname': 'Sam',
    'lname': 'Jones',
  },
  { 'id': 125,
    'fname': 'Pete',
    'lname': 'abbott',
  },
]
list_of_dicts.sort(key=by_dict_lname)      # custom sort function (above)
for this_dict in list_of_dicts:
    print(f"{this_dict['fname']} {this_dict['lname']}")

# Pete abbot
# Sam Jones
# Joe Wilson

So, although we are sorting dicts, our sub says "take this dictionary and sort by this inner element of the dictionary".

Sidebar: cascading sort

Sort a list by multiple criteria by having your sorting helper function return a 2-element tuple.

def by_last_first(name):
  fname, lname = name.split()
  return (lname, fname)

names = ['Zeb Will', 'Deb Will', 'Joe Max', 'Ada Max']

lnamesorted = sorted(names, key=by_last_first)

                             # ['Ada Max', 'Joe Max', 'Deb Will', 'Zeb Will']

Sorting review

A quick review of sorting: recall how Python will perform a default sort (numeric or ASCII-betical) depending on the objects sorted. If we wish to modify this behavior, we can pass each element to a function named by the key= parameter:

mylist = ['Alpha', 'Gamma', 'episilon', 'beta', 'Delta']

print(sorted(mylist))                      # ASCIIbetical sort
                                          # ['Alpha', 'Gamma', 'Delta', 'beta', 'epsilon']

mylist.sort()                             # sort mylist in-place

print(sorted(mylist, key=str.lower))       # alphabetical sort
                                          # (lowercasing each item by telling Python to pass it
                                          # to str.lower)
                                          # ['Alpha', 'beta', 'Delta', 'epsilon', 'Gamma']

print(sorted(mylist, key=len))             # sort by length
                                          # ['beta', 'Alpha', 'Gamma', 'Delta', 'epsilon']

Sorting review: sorting dictionary keys by value: dict.get

When we loop through a dict, we can loop through a list of keys (and use the keys to get values) or loop through items, a list of (key, value) tuple pairs. When sorting a dictionary by the values in it, we can also choose to sort keys or items.

To sort keys, mydict.get is called with each key - and get returns the associated value. So the keys of the dictionary are sorted by their values.

mydict = { 'a': 5, 'b': 2, 'c': 1, 'z': 0 }
mydict_sorted_keys = sorted(mydict, key=mydict.get)
for i in mydict_sorted_keys:
    print(f"{i} = {mydict[i]}")

                 ## z = 0
                 ## c = 1
                 ## b = 2
                 ## a = 5

Sorting dictionary items by value: operator.itemgetter

Recall that we can render a dictionary as a list of tuples with the dict.items() method:

mydict = { 'a': 5, 'b': 2, 'c': 1, 'z': 0 }
mydict_items = list(mydict.items())                        # [(a, 5), (c, 1), (b, 2), (z, 0)]

To sort dictionary items by value, we need to sort each two-element tuple by its second element. The built-in module operator.itemgetter will return whatever element of a sequence we wish - in this way it is like a subscript, but in function format (so it can be called by the Python sorting algorithm).

import operator
mydict = { 'a': 5, 'b': 2, 'c': 1, 'z': 0 }
mydict_items = mydict.items()                        # [(a, 5), (c, 1), (b, 2), (z, 0)]
mydict_items.sort(key=operator.itemgetter(1))
print(mydict_items)                                  # [(z, 0), (c, 1), (b, 2), (a, 5)]
for key, val in mydict_items:
    print(f"{key} = {val}")

                    ## z = 0
                    ## c = 1
                    ## b = 2
                    ## a = 5

The above can be conveniently combined with looping, effectively allowing us to loop through a "sorted" dict:

for key, val in sorted(mydict.items(), key=operator.itemgetter(1)):
    print(f"{key} = {val}")

Database results come as a list of tuples. Perhaps we want our results sorted in different ways, so we can store as a list of tuples and sort using operator.itemgetter. This example sorts by the third field, then by the second field (last name, then first name):

import operator
items =[ (123, 'Joe', 'Wilson', 35, 'mechanic'),
         (124, 'Sam', 'Jones', 22, 'mechanic'),
         (125, 'Pete', 'Jones', 40, 'mechanic'),
         (126, 'Irina', 'Bibi', 31, 'mechanic'),
       ]
items.sort(key=operator.itemgetter(2,1)) # sorts by last, first name
for this_pair in items:
  print(f"{this_pair[1]} {this_pair[2]}")

       ## Irina Bibi
       ## Pete Jones
       ## Sam Jones
       ## Joe Wilson

Multi-dimensional structures: sorting with lambda custom function

Functions are useful but they require that we declare them separately, elsewhere in our code. A lambda is a function in a single statement, and can be placed in data structures or passed as arguments in function calls. The advantage here is that our function is used exactly where it is defined, and we don't have to maintain separate statements.

A common use of lambda is in sorting. The format for lambdas is lambda arg: return_val. Compare each pair of regular function and lambda, and note the argument and return val in each.

def by_lastname(name):
  fname, lname = name.split()
  return lname

names = [ 'Josh Peschko', 'Gabriel Feghali', 'Billy Woods', 'Arthur Fischer-Zernin' ]
sortednames = sorted(names, key=lambda name:  name.split()[1])


list_of_dicts = [
  { 'id': 123,
    'fname': 'Joe',
    'lname': 'Wilson',
  },
  { 'id': 124,
    'fname': 'Sam',
    'lname': 'Jones',
  },
  { 'id': 125,
    'fname': 'Pete',
    'lname': 'abbott',
  },
]

def by_dict_lname(this_dict):
  return this_dict['lname'].lower()

sortedlenstrs = sorted(list_of_dicts, key=lambda this_dict:  this_dict['lname'].lower())

In each, the label after lambda is the argument, and the expression that follows the colon is the return value. So in the first example, the lambda argument is name, and the lambda returns name.split()[1]. See how it behaves exactly like the regular function itself? Again, what is the advantage of lambdas? They allow us to design our own functions which can be placed inline, where a named function would go. This is a convenience, not a necessity. But they are in common use, so they must be understood by any serious programmer.

Lambda expressions: breaking them down

Many people have complained that lambdas are hard to grok (absorb), but they're really very simple - they're just so short they're hard to read. Compare these two functions, both of which add/concatenate their arguments:

def addthese(x, y):
  return x + y

addthese2 = lambda x, y:  x + y

print(addthese(5, 9))        # 14
print(addthese2(5, 9))       # 14

The function definition and the lambda statement are equivalent - they both produce a function with the same functionality.

Lambda expression example: dict.get and operator.itemgetter

Here are our standard methods to sort a dictionary:

import operator
mydict = { 'a': 5, 'b': 2, 'c': 1, 'z': 0 }
for key, val in sorted(mydict.items(), key=operator.itemgetter(1)):
    print(f"{key} = {val}")

for key in sorted(mydict, key=mydict.get):
    print(f"{key} = {mydict[key]}")

Imagine we didn't have access to dict.get and operator.itemgetter. What could we do?

mydict = { 'a': 5, 'b': 2, 'c': 1, 'z': 0 }
for key, val in sorted(mydict.items(), key=lambda keyval:  keyval[1]):
    print(f"{key} = {val}")

for key in sorted(mydict, key=lambda key:  mydict[key]):
    print(f"{key} = {mydict[key]}")

These lambdas do exactly what their built-in counterparts do: in the case of operator.itemgetter, take a 2-element tuple as an argument and return the 2nd element in the case of dict.get, take a key and return the associated value from the dict

Higher-Order Functions and Decorators

Object References

Everything is an Object; Objects Assigned by Reference

object: a data value of a particular type variable: a name bound to an object

When we create a new object and assign it to a name, we call it a variable. This simply means that the object can now be referred to by that name.

a = 5                 # bind an int object to name a

b = 'hello'           # bind a str object to name b

Here are three classic examples demonstrating that objects are bound by reference and that assignment from one variable to another is simply binding a 2nd name to the same object (i.e. it simply points to the same object -- no copy is made).

Aliasing (not copying) one object to another name:

a = ['a', 'b', 'c']   # bind a list object to a (by reference)

b = a                 # 'alias' the object to b as well -- 2 references

b.append('d')

print(a)               # ['a', 'b', 'c', 'd']

a was not manipulated directly, but it changed. This underscores that a and b are pointing to the same object.

Passing an object by reference to a function:

def modifylist(this_list):   # The object bound to a
    this_list.append('d')    # Modify the object bound to a

a = ['a', 'b', 'c']

modifylist(a)                # Pass object bound to a by reference

print(a)                       # ['a', 'b', 'c', 'd']

The same dynamics at work: a is pointing at the same list object as this_list, so a change through one name is a change to the one object -- and the other name pointing to the same object will see the same changed object.

Alias an object as a container item:

a = ['a', 'b', 'c']          # list bound to a

xx = [1, 2, a]               # 3rd item is reference to list bound to a

xx[2].append('d')            # appending to list object referred to in list item

print(a)                       # ['a', 'b', 'c', 'd']

The same dynamic applied to a container item: the only difference here is that a name and a container item are pointing to the same object.

Function References: Renaming Functions

Functions are variables (objects bound to names) like any other; thus they can be renamed.

def doubleit(val):
    val2 = val * 2
    return val2

print(doubleit)         # <function doubleit at 0x105554d90>

newfunc = doubleit

xx = newfunc(5)        # 10

The output <function doubleit at 0x105554d90> is Python's way of visualizing a function (i.e. showing its printed value). The hex code refers to the function object's location in memory (this can be used for debugging to identify the individual function).

Function References: Functions in Containers

Functions are "first-class" objects, and as such can be stored in containers, or passed to other functions.

Functions can be stored in containers the same way any other object (int, float, list, dict) can be:

def doubleit(val):
    val2 = val * 2
    return val2

def tripleit(val):
    return val * 3

funclist = [ doubleit, tripleit ]

print(funclist[0](10))   # 20
print(funclist[1](10))   # 30

Higher-Order Built-in Functions: map(), grep() and sorted()

These functions allow us to pass a function as an argument to enable its behavior.

We can pass a function name (or a lambda, which is also a reference to a function) to any of these built-in functions. The function controls the built-in function's behavior.

One example is the function passed to sorted():

def by_last(name):
    first, last = name.split()
    return last

names = ['Joe Wilson', 'Zeb Applebee', 'Abe Zimmer']

snames = sorted(names, key=by_last)   # ['Zeb Applebee, 'Joe Wilson', 'Abe Zimmer']

In this example, we are passing the function by_last to sorted(). sorted() calls the function once with each item in the list as argument, and sorts that item by the return value from the function.

In the case of map() (apply a function to each item and return the result) and filter() (apply a function to each item and include it in the returned list if the function returns True), the function is required:

def make_intval(val):
    return int(val)

def over9(val):
    if int(val) > 99:
        return True
    else:
        return False

x = ['1', '100', '11', '10', '110']

# apply make_intval() to each item and sort by the return value
sx = sorted(x, key=make_intval)     # ['1', '10', '11', '100', '110']

# apply make_intval() to each item and store the return value in the resulting list
mx = map(make_intval, x)
print(list(mx))                      # [ 1, 100, 11, 10, 110 ]

# apply over9() to each item and if the return value is True, store in resulting list
fx = filter(over9, x)
print(list(fx))                      # [ '100', '110' ]

Higher-Order Functions: Functions as Arguments and as Return Values

A "higher order" function is one that can be passed to another function, or returned from another function.

The charge() function takes a function as an argument, and uses it to calculate its return value:

def charge(func, val):
    newval = func(val) + val
    return '${}'.format(newval)

def tax_ny(val):
    val2 = val * 0.085
    return val2

def tax_ca(val):
    val2 = val * 0.065
    return val2

nyval = charge(tax_ny, 10)          # pass tax_ny to charge():  $10.85
caval = charge(tax_ca, 10)          # pass tax_ca to charge():  $10.65

Any function that takes a function as an argument or returns one as a return value is called a "higher order" function.

Using a function to build another function

A function can be kind of a "factory" of functions.

This function returns a function as return value, after "seeding" it with a value:

def makemul(mul):
    def times(startval):
        return mul * startval
    return times

doubler = makemul(2)
tripler = makemul(3)

print(doubler(5))      # 10
print(tripler(5))      # 15

In the two examples above, the values 2 and 3 are made part of the returning function -- "seeded" as built-in values .

Decorators

A decorator accepts a function as an argument and returns a replacement function as a return value.

Python decorators return a function to replace the function being decorated -- when the original function is called in the program, Python calls the replacement. A decorator can be added to any function through the use of the @ sign and the decorator name on the line above the function.

Here's a simple example of a function that returns another function (from RealPython blog):

def my_decorator(some_function):

    def wrapper():
        print("Something is happening before some_function() is called.")

        some_function()

        print("Something is happening after some_function() is called.")
    return wrapper


def this_function():
    print("Wheee!")


# now the same name points to a replacement function
this_function = my_decorator(this_function)

# calling the replacement function
this_function()

This is not a decorator yet, but it shows the concept and mechanics

If we wished to use this as a Python decorator, we can simply use the @ decorator notation:

def my_decorator(some_function):

    def wrapper():
        print("Something is happening before some_function() is called.")

        some_function()

        print("Something is happening after some_function() is called.")
    return wrapper

@my_decorator
def this_function():
    print('Wheee!')

this_function()
                       # Something is happening before...
                       # Whee!
                       # Something is happening after...

The benefit here is that, rather than requiring the user to explicitly pass a function to a processing function, we can simply decorate each function to be processed and it will behave as advertised.

*args and **kwargs to capture all arguments

To allow a decorated function to accept arguments, we must accept them and pass them to the decorated function.

Here's a decorator function that adds integer 1 to the return value of a decorated function:

def addone(oldfunc):
    def newfunc(*args, **kwargs):
        return oldfunc(*args, **kwargs) + 1
    return newfunc

@addone
def sumtwo(val1, val2):
    return val1 + val2

y = sumtwo(5, 10)
print(y)                  # 16

Now look closely at def newfunc(*args, **kwargs): *args in a function definition means that all positional arguments passed to it will be collected into a tuple called args. **kwargs in a function definition means that all keyword arguments passed to it will be collected into a dictionary called kwargs. (The * and ** are not part of the variable names; they are notations that allow the arguments to be passed into the tuple and dict.) Then on the next line, look at return oldfunc(*args **kwargs) + 1 *args in a function call means that the tuple args will be passed as positional arguments (i.e., the reverse of what happened above) **kwargs in a function call means that the dict kwargs will be passed as keyword arguments (i.e., the reverse of what happened above) This means that if we wanted to decorate a function that takes different arguments, *args and **kwargs would faithfully pass on those arguments as well.

Here's another example, adapted from the Jeff Knupp blog:

def currency(f):                              # decorator function
    def wrapper(*args, **kwargs):
        return '$' + str(f(*args, **kwargs))
    return wrapper

@currency
def price_with_tax(price, tax_rate_percentage):
    """Return the price with *tax_rate_percentage* applied.
    *tax_rate_percentage* is the tax rate expressed as a float, like
    "7.0" for a 7% tax rate."""

    return price * (1 + (tax_rate_percentage * .01))

print(price_with_tax(50, .10))           # $50.05

In this example, *args and **kwargs represent "unlimited positional arguments" and "unlimited keyword arguments". This is done to allow the flexibility to decorate any function (as it will match any function argument signature).

Modules

Introduction: Modules

Modules are files that contain reusable Python code: we often refer to them as "libraries" because they contain code that can be used in other scripts. It is possible to import such library code directly into our programs through the import statement -- this simply means that the functions in the module are made available to our program. Modules consist principally of functions that do useful things and are grouped together by subject. Here are some examples:

• The sys module has functions that let us work with python's interpreter and how it interacts with the operating system
• The os module has functions that let us work with the operating system's files, folders and other processes
• The datetime module has functions that let us easily calculate date into the future or past, or compare two dates
• The urllib2 module has functions that let us easily make HTTP requests over the internet

So when we import a module in our program, we're simply making other Python code (in the form of functions) available to our own programs. In a sense we're creating an assemblage of Python code -- some written by us, some by other people -- and putting it together into a single program. The imported code doesn't literally become part of our script, but it is part of our program in the sense that our script can call it and use it. We can also define our own modules -- collections of Python functions and/or other variables that we would like to make available to our other Python programs. We can even prepare modules designed for others to use, if we feel they might be useful. In this way we can collaborate with other members of our team, or even the world, by using code written by others and by providing code for others to use.

Objectives for the Unit: Modules

• save a file of Python code and import it into and use it in another Python program
• import individual functions from a module into our Python program
• access a module's functions as its attributes
• manipulate the module's search path
• write Python code that can act both as a module and a script
• raise exceptions in our module code, to be handled by the calling code
• design modules with an eye toward reuse by others
• install modules with pip or easy_install

Summary Statement: import modulename

Using import, we can import an entire Python module into our own code.

messages.py: a Python module that prints messages

import sys

def print_warning(msg):
    """write a message to STDOUT"""
    sys.stdout.write(f'warning:  {msg}\n')

def log_message(msg):
    """write a message to the log file"""
    try:
        fh = open('log.txt', 'a')
        fh.write(str(msg) + '\n')
    except FileNotFoundError:
        print_warning('log file not readable')

test.py: a Python script that imports messages.py

#!/usr/bin/env python

import messages

print("test program running...")

messages.log_message('this is an important message')
messages.print_warning("I think we're in trouble.")

The global variables in the module become attributes of the module. The module's variables are accessible through the name of the module, as its attributes.

Summary statement: import modulename as convenientname

A module can be renamed at the point of import.

import pandas as pd
import datetime as dt

users = pd.read_table('myfile.data', sep=',', header=None)

print("yesterday's date:  {dt.date.today() - dt.timedelta(days=1)}")

Summary statement: from modulename import variablename

Individual variables can be imported by name from a module.

#!/usr/bin/env python

from messages import print_warning, log_message

print("test program running...")

log_message('this is an important message')
print_warning("I think we're in trouble.")

Summary: module search path

Python must be told where to find our own custom modules.

Python's standard module directories When it encounters an import, Python searches for the module in a selected list of standard module directories. It does not search through the entire filesystem for modules. Modules like sys and os are located in one of these standard directories. Our own custom module directories Modules that we create should be placed in one or more directories that we designate for this purpose. In order to let Python know about our own module directories, we have a couple of options: PYTHONPATH environment variable The standard approach to adding our own module directories to the list of those that Python searches is to create or modify the PYTHONPATH environment variable. This colon-separated list of paths indicates any paths to search in addition to the ones Python normally searches.

In your Unix .bash_profile file in your home directory, you would place the following command:

export PYTHONPATH=$PYTHONPATH:/path/to/my/pylib:/path/to/my/other/pylib

...where /path/to/my/pylib and /path/to/my/other/pylib are paths In Windows, you can set the PYTHONPATH environment variable through the Windows GUI.

manipulating sys.path

import sys

print(sys.path)

    # ['', '/Users/dblaikie/lib', '//anaconda/lib/python2.7',
    #  '//anaconda/lib/python2.7/plat-darwin',
    #  '//anaconda/lib/python2.7/plat-mac',
    #  '//anaconda/lib/python2.7/plat-mac/lib-scriptpackages',
    #  '//anaconda/lib/python2.7/lib-tk',
    #  '//anaconda/lib/python2.7/lib-old',
    #  '//anaconda/lib/python2.7/lib-dynload',
    #  '//anaconda/lib/python2.7/site-packages',
    #  '//anaconda/lib/python2.7/site-packages/PIL',
    #  '//anaconda/lib/python2.7/site-packages/Sphinx-1.3.1-py2.7.egg',
    #  '//anaconda/lib/python2.7/site-packages/aeosa',
    #  '//anaconda/lib/python2.7/site-packages/setuptools-19.1.1-py2.7.egg']

sys.path.append('/path/to/my/pylib')

import mymod     # if mymod.py is in /path/to/my/pylib, it will be found

Once a python script is running, Python makes the PYTHONPATH search path available in a list called sys.path. Since it is a list, it can be manipulated; you are free to add whatever paths you wish

However, please note that this kind of manipulation is rare because the needed changes are customarily made to the PYTHONPATH environment variable.

Summary: coding all Python scripts as modules

All Python scripts should be coded as modules: able to be both imported and executed.

#!/usr/bin/env python


""" helloworld.py:  print the "Hello! message """

def print_hello(arg):
    print(f"Hello, {arg}!")

def main():
    print_hello('World')

if __name__ == '__main__':              # value is '__main__'
                                        # if this script is executed
                                        # value is 'helloworld'
                                        # if this script is imported
    main()

If we run the above script, we'll see "Hello, World!". But if we import the above script, we won't see anything. Why? Because the if statement above will be true only if the script is executed. This is important behavior, because there are some scripts that we may want to run directly, but also allow others to import (in order to use the script's functions). Whether we intend to import a script or not, it is considered a "best practice" to build all of our programs in this way -- with a "main body" of statements collected under function main() and the call to main() inside the if __name__ == '__main__' gate.

Summary: raising exceptions

Causing an exception to be raised is the principal way a module signals an error to the importing script.

A file called mylib.py

def get_yearsum(user_year):

    user_year = int(user_year)
    if user_year < 1929 or user_year > 2013:
      raise ValueError(f'year {user_year} out of range')

    # calculate value for the year

    return 5.9         # returning a sample value (for testing purposes only)

An exception raised by us is indistinguishable from one raised by Python, and we can raise any exception type we wish. This allows the user of our function to handle the error if needed (rather than have the script fail):

import mylib

while True:

    year = input('please enter a year:  ')

    try:
        mysum = mylib.get_yearsum(year)
        break
    except ValueError:
        print('invalid year:  try again')

print('mysum is', mysum)

Summary: installing modules

Third-party modules must be downloaded and installed into our Python distribution.

Unix

$ sudo pip search pandas         # searches for pandas in the PyPI repository
$ sudo pip install pandas        # installs pandas

Installation on Unix requires something called root permissions, which are permissions that the Unix system administrator uses to make changes to the system. The below commands include sudo, which is a way to temporarily be granted root permissions.

Windows

C:\\Windows > pip search pandas   # searches for pandas in the PyPI repo
C:\\Windows > pip install pandas  # installs pandas

PyPI: the Python Package Index The Python Package Index at https://pypi.python.org/pypi is a repository of software for the Python programming language. There are more than 70,000 projects uploaded there, from serious modules used by millions of developers to half-baked ideas that someone decided to share prematurely. Usually, we encounter modules in the field -- shared through blog posts and articles, word of mouth and even other Python code. But the PPI can be used directly to try to find modules that support a particular purpose.

Summary: the Python standard distribution of modules

Modules included with Python are installed when Python is installed -- they are always available.

Python provides hundreds of supplementary modules to perform myriad tasks. The modules do not need to be installed because they come bundled in the Python distribution, that is they are installed at the time that Python itself is installed. The documentation for the standard library is part of the official Python docs.

• various string-related services
• specialized containers (type-specific lists and dicts, pseudohashes, etc.)
• math calculations and number generation
• file and directory manipulation
• persistence (saving data on disk)
• data compression and archiving (e.g., creating zip files)
• encryption
• networking and interprocess (program-to-program) communication
• internet tasks: web server, web client, email, file transfer, etc.
• XML and HTML parsing
• multimedia: audio and image file manipulation
• GUI (graphical user interface) development
• code testing
• etc.

From the Standard Distribution: the time module

time gives us simple access to the current time or any other time in terms of epoch seconds, or seconds since the epoch began, which was set by Unix developers at January 1, 1970 at midnight!

The module has a simple interface - it generally works with seconds (which are often a float in order to specify milliseconds) and returns a float to indicate the time -- this float value can be manipulated and then passed back into time to see the time altered and can be formatted into any display.

import time

epochsecs = time.time()                     # 1450818009.925441
                                            # (current time in epoch seconds)

print(time.ctime(epochsecs))                 # 'Tue Apr 12 13:29:19 2016'

epochsecs = epochsecs + (60 * 60 *24)       # adding one day!

print(time.ctime(epochsecs))                 # 'Wed Apr 13 13:29:19 2016'

time.sleep(10)                              # pause execution 10 seconds

From the standard distribution: datetime.date, datetime.datetime and datetime.timedelta

Python uses the datetime library to allow us to work with dates. Using it, we can convert string representations of date and time (like "4/1/2001" or "9:30") to datetime objects, and then compare them (see how far apart they are) or change them (advance a date by a day or a year).

import datetime as dt            # load the datetime library
dt = dt.datetime.now()           # create a new date object set to now
dt = dt.date.today()             # same

dt = datetime(2011, 7, 14, 14, 22, 29)
                                 # create a new date object by setting
                                 # the year, month, day, hour, minute,
                                 # second (milliseconds if desired)

dt = datetime.strptime("21/11/06 16:30", "%d/%m/%y %H:%M")
                                 # create a new date object
                                 # by giving formatted date and time,
                                 # then telling datetime what format we used

Once a datetime object has been created, we can view the date in a number of ways

print(dt)                        # print formatted (ISO) date
                                 # 2011-07-14 14:22:29.045814

print(dt.strftime("%m/%d/%Y %H:%M:%S"))
                                # print formatted date/time
                                # using string format tokens
                                # '07/14/2011 14:22:29'

print(dt.year)                   # 2011
print(dt.month)                  # 7
print(dt.day)                    # 14
print(dt.hour)                   # 14
print(dt.minute)                 # 22
print(dt.second)                 # 29
print(dt.microsecond)
print(dt.weekday())              # 'Tue'

Comparing datetime objects

Often we may want to compare two dates. It's easy to see whether one date comes before another by comparing two date objects as if they were numbers:

d1 = datetime(2011, 7, 14, 9, 40, 15)
                                 # new date object:  July 14, 2011  9:40:15am

d2 = datetime(2011, 6, 14, 9, 30, 00)
                                 # new date object:  June 14, 2011  9:30:00am

print(d1 < d2)                   # False

print(d1 > d2)                   # True

"Delta" means change. If we want to measure the difference between two dates, we can subtract one from the other. The result is a timedelta object:

td = d1 - d2                     # a new timedelta object
print(td.days)                   # 30
print(td.seconds)                # 615

Between June 14, 2011 at 9:30am and July 14, 2011 at 9:45am, there is a difference of 30 days, 10 minutes and 15 seconds. timedelta doesn't show difference in minutes, however: instead, it shows the number of seconds -- seconds can easily be converted between seconds and minutes/hours with simple arithmetic.

Changing a date using the timedelta object

Timedelta can also be constructed and used to change a date. Here we create timedelta objects for 1 day, 1 hour, 1 minute, 1 second, and then change d1 to subtract one day, one hour, one minute and one second from the date.

from datetime import timedelta
add_day = timedelta(days=1)
add_hour = timedelta(hours=1)
add_minute = timedelta(minutes=1)
add_second = timedelta(seconds=1)

print(d1)                                  # 2011-07-14 09:40:15.678331

d1 = d1 - add_day
d1 = d1 - add_hour
d1 = d1 - add_minute
d1 = d1 - add_second

print(d1.strftime("%m/%d/%Y %H:%M:%S"))    # 2011-07-13 08:39:14.678331

Packages

Introduction: Packages

A package is a directory of files that work together as a Python application or library module.

Many applications or library modules consist of more than one file. A script may require configuration files or data files; some applications combine several .py files that work together. In addition, programs need unit tests to ensure reliability. A package groups all of these files (scripts, supporting files and tests) together as one entity. In this unit we'll discover Python's structure and procedures for creating packages. Some of the steps here were taken from this very good tutorial on packages: https://python-packaging.readthedocs.io/en/latest/minimal.html

Package Structure and the __init__.py File

The base of a package is a directory with an __init__.py file.

Folder structure for package davehello:

davehello/                      # base package folder - name is discretionary
    davehello/                  # module folder -  usually same name
        __init__.py             # initial script -- this is run first
    setup.py                    # setup file -- discussed below

The initial code for our program: __init__.py

def greet():
    return 'hello, world!'

The names of the folders are up to you. The "outer" davehello/ is the name of the base package folder. The "inner" davehello/ is the name of your module. These can be the same or different. setup.py is discussed next.

setup.py: the installation script

This file describes the script and its authorship.

Inside setup.py put the following code, but replace the name, author, author_email and packages (this list should reflect the name in name):

from setuptools import setup

setup( name='davehello',
       version='0.1',
       description='This module greets the user.  ',
       url='',                # usually a github URL
       author='David Blaikie',
       author_email='david@davidbpython.com',
       license='MIT',
       packages=['davehello'],
       install_requires=[ ],
       zip_safe=False )

setuptools is a Python module for preparing modules. The setup() function establishes meta information for the package.

url can be left blank for now. Later on we will commit this package to github and put the github URL here. packages should be a list of packages that are part this package (as there can be sub-packages within a package); however we will just work with the one package.

Again, folder structure for package davehello with two files:

davehello/                      # base package folder - name is discretionary
    davehello/                  # module folder -  usually same name
        __init__.py             # initial script -- this is run first
    setup.py                    # setup file -- discussed below

Please doublecheck your folder structure and the placement of files -- this is vital to being able to run the files.

Installing Locally

pip install can install your module into your own local Python module directories.

First, make sure you're in the same directory as setup.py. Then from the Unix/Mac Terminal, or Windows Command Prompt :

$ pip install .                            # $ means Unix/Mac prompt
Processing /Users/david/Desktop/davehello
Installing collected packages: davehello
  Running setup.py install for davehello ... done
Successfully installed davehello-0.1

The install module copies your package files to a Python install directory that is part of your Python installation's sys.path. Remember that the sys.path holds a list of directories that will be searched when you import a module. If you get an error when you try to install, double check your folder structure and placement of files, and make sure you're in the same directory as setup.py.

If successful, you should now be able to open a new Terminal or Command Prompt window (on Windows use the Anaconda Prompt), cd into your home directory, launch a Python interactive session, and import your module:

$ cd /Users/david                # moving to my home directory, to make sure
                                 # we're running the installed version

$ python
Python 3.6.1 |Anaconda custom (64-bit)| (default, May 11 2017, 13:04:09)
[GCC 4.2.1 Compatible Apple LLVM 6.0 (clang-600.0.57)] on darwin
Type "help", "copyright", "credits" or "license" for more information.

>>> import davehello
>>> davehello.greet()
'hello, world!'
>>>

If you get a ModuleNotFound error when you try to import: 1. It is possible that the files are not in their proper places, for example if __init__.py is in the same directory as setup.py. 2. It is possible that the pip you used installed the module into a different distribution than the one you are running.

$ pip --version
pip 9.0.1 from /Users/david/anaconda/lib/python3.6/site-packages (python 3.6)

Davids-MacBook-Pro-2:~ david$ python -V
Python 3.6.1 :: Anaconda custom (64-bit)
Davids-MacBook-Pro-2:~ david$

Note that my pip --version path indicates that it's running under Anaconda, and my python -V also indicates Anaconda.

Package Development Directory vs. Installation Directory

Development Directory is where you created the files; Installation Directory is where Python copied them upon install.

Keep in mind that when you import a module, the current directory will be searched before any directories on the sys.path. So if your command line / Command Prompt / Terminal session is currently in the same directory as setup.py (as we had been before we did a cd to my home directory), you'll be reading from your local package, not the installed one. So you won't be testing the installation until you move away from the package directory.

To see which folder the module was installed into, make sure you're not in the package directory; then read the module's __file__ attribute:

$ cd /Users/david                # moving to my home directory, to make sure
                                 # we're running the installed version

$ python
Python 3.6.1 |Anaconda custom (64-bit)| (default, May 11 2017, 13:04:09)
[GCC 4.2.1 Compatible Apple LLVM 6.0 (clang-600.0.57)] on darwin
Type "help", "copyright", "credits" or "license" for more information.

>>> import davehello
>>> davehello.__file__

'/Users/david/anaconda/lib/python3.6/site-packages/davehello/__init__.py'

Note that this is not one of my directories or one that I write to; it is a common install directory for Anaconda Python.

Contrast this with the result if you're importing the module from the package directory (the same directory as setup.py):

$ cd /Users/david/Desktop/davehello/          # my package location

$ python

>>> import davehello
>>> davehello.__file__

'/Users/david/Desktop/davehello/davehello/__init__.py'

Note that this is one of my local directories.

Making Changes and Reinstalling

Changes to the package will not be reflected in the installed module unless we reinstall.

If you make a change to the package source files, it won't be reflected in your Python installation until you reinstall with pip install. (The exception to this is if you happen to be importing the module from within the package itself -- then the import will read from the local files.)

To reinstall a previously installed module, we must include the --upgrade flag:

$ pip install . --upgrade
Processing /Users/david/Desktop/davehello
Installing collected packages: davehello
  Found existing installation: davehello 0.1
    Uninstalling davehello-0.1:
      Successfully uninstalled davehello-0.1
  Running setup.py install for davehello ... done
Successfully installed davehello-0.1

Adding files

__init__.py is the "gateway" file; the bulk of code may be in other .py files in the package.

Many packages are made of up several .py files that work together. They may be files that are only called internally, or they may be intended to be called by the user. Your entire module could be contained within __init__.py, but I believe this is customarily used only as the gateway, with the bulk of module code in another .py file. In this step we'll move our function to another file.

hello.py (new file -- this can be any name)

def greet():
    return 'hello, new file!'

__init__.py

from .hello import greet       # .hello refers to hello.py in the base package directory

New folder structure for package davehello:

davehello/                      # base folder - name is discretionary
    davehello/                  # package folder -  usually same name
        __init__.py             # initial script -- this is run first
        hello.py                # new file
    setup.py                    # setup file -- discussed below

Don't forget to reinstall the module once you've finalized changes. However you can run the package locally (i.e., from the same directory as setup.py) without reinstalling. When the package is imported, Python reads and executes the ___init___.py program. This file is now importing greet from hello.py into the module's namespace, making it available to the user under the package name davehello.

The user can also reach the variable in hello.py directly, by using attribute syntax to reach the module -- so both of these calls to greet() should work:

>>> import davehello as dh
>>> dh.greet()                    # 'hello, new file!'
                                  # (because __init__.py imported greet from hello.py)

>>> dh.hello.greet()              # 'hello, new file!'
                                  # (calling it directly in hello.py)

>>> from davehello import hello
>>> hello.greet()                 # 'hello, new file!'

The mechanics of files and variables in packages

Packages provide for accessing variables within multiple files.

• during install, the package directory has been copied to a directory on the sys.path (the list of folders where Python searches for modules) so a simple import will find it
• when the package is imported, __init__.py is automatically run
• global variables in __init__.py are accessible through the package name (packagename.variablename)
• variables in other files can be imported by __init__.py and thus will also be accessible through the package name
• other .py files can be accesssed from the import through the package name through attribute chaining (for instance if we had a file at davehello/dir1/myprog.py with a dothis() function, we could access it with davehello.dir1.myprog.dothis())
• if there are any other subdirectories in the package, each must have an __init__.py file in it (even if __init__.py is empty)

Specifying Dependencies

Dependencies are other modules that your module may need to import.

If your module imports a non-standard module like splain, it is known as a dependency. Dependencies must be mentioned in the setup() spec. The installer will make sure any dependent modules are installed so your module works correctly.

setup(name='davehello',
      version='0.1',
      description='This module greets the user.  ',
      url='',                # usually a github URL
      author='David Blaikie',
      author_email='david@davidbpython.com',
      license='MIT',
      packages=['davehello'],
      install_requires=[ 'splain' ],
      zip_safe=False)

This would not be necessary if the user already had splain installed. However, if they didn't, we would want the install of our module to result in the automatic installation of the splain module. (Please note that splain.py has not yet been uploaded to PyPI, so the above dependency will not work.)

Adding Tests

Tests belong in the package; thus anyone who downloads the source can run the tests.

In a package, tests should be added to a tests/ directory in the package root (i.e., in the same directory as setup.py).

We will use pytest for our testing -- the following configuration values need to be added to setup() in the setup.py file:

test_suite='pytest'
setup_requires=['pytest-runner']
tests_require=['pytest']

Here's our updated setup.py:

from setuptools import setup

setup(name='davehello',
      version='0.1',
      description='This module greets the user.  ',
      url='',                # usually a github URL
      author='David Blaikie',
      author_email='david@davidbpython.com',
      license='MIT',
      packages=['davehello'],
      install_requires=[ ],
      test_suite='pytest',
      setup_requires=['pytest-runner'],
      tests_require=['pytest'],
      zip_safe=False)

As is true for most testing suites, pytest requires that our test filenames should begin with test_, and test function names begin with test_.

Here is our test program test_hello.py, with test_greet(), which tests the greet() function.

import pytest
import davehello as dh

def test_greet():
    assert dh.greet() == 'hello, world!'

Here's a new folder structure for package davehello:

davehello/                      # base folder - name is discretionary
    davehello/                  # package folder -  usually same name
        __init__.py             # initial script -- this is run first
        hello.py                # new file
    tests/
        test_hello.py
    setup.py                    # setup file -- discussed below

Now when we'd like to run the package's tests, we run the following at the command line:

$ python setup.py pytest
running pytest
running egg_info
writing davehello.egg-info/PKG-INFO
writing dependency_links to davehello.egg-info/dependency_links.txt
writing top-level names to davehello.egg-info/top_level.txt
reading manifest file 'davehello.egg-info/SOURCES.txt'
writing manifest file 'davehello.egg-info/SOURCES.txt'
running build_ext
------------------------------- test session starts -------------------------------

platform darwin -- Python 3.6.1, pytest-3.0.7, py-1.4.33, pluggy-0.4.0
rootdir: /Users/david/Desktop/davehello, inifile:
collected 1 items

tests/test_hello.py F
----------------------------------- FAILURES -----------------------------------

    def test_greet():
>       assert dh.greet() == 'hello, world!'
E       AssertionError: assert 'hello, new file!' == 'hello, world!'
E         - hello, new file!
E         + hello, world!

tests/test_hello.py:7: AssertionError
--------------------------- 1 failed in 0.03 seconds ---------------------------

oops, our test failed. We're not supplying the right value to assert -- the function returns hello, new file! and our test is looking for hello, world!. We go into test_hello.py and modify the assert statement; or alternatively, we could change the output of the function.

After change has been made to test_hello.py to reflect the expected output:

$ python setup.py pytest
running pytest
running egg_info
writing dblaikie_hello.egg-info/PKG-INFO
writing dependency_links to dblaikie_hello.egg-info/dependency_links.txt
writing top-level names to dblaikie_hello.egg-info/top_level.txt
reading manifest file 'dblaikie_hello.egg-info/SOURCES.txt'
writing manifest file 'dblaikie_hello.egg-info/SOURCES.txt'
running build_ext
------------------------------- test session starts -------------------------------
platform darwin -- Python 3.6.1, pytest-3.0.7, py-1.4.33, pluggy-0.4.0
rootdir: /Users/david/Desktop/davehello, inifile:
collected 1 items

davehello/tests/test_hello.py .

----------------------------- 1 passed in 0.01 seconds ----------------------------

The output first shows us what setup.py is doing in the background, then shows collected 1 items to indicate that it's ready to run tests. The final statement indicates how many tests passed (or failed).

With these basic steps you can create a package, install it in your Python distribution, and prepare it for distribution to the world. May all beings be happy.

Distributing Modules and the PyPI Package Repo

PyPI: The Python Package Index

All publicly available modules can be found here.

https://testpypi.python.org/pypi

Registering your Application on PyPI

setuptools can do this for us automatically.

Davids-MacBook-Pro-2:dblaikie_hello david$ python setup.py register
running register
running egg_info
writing dblaikie_hello.egg-info/PKG-INFO
writing dependency_links to dblaikie_hello.egg-info/dependency_links.txt
writing top-level names to dblaikie_hello.egg-info/top_level.txt
reading manifest file 'dblaikie_hello.egg-info/SOURCES.txt'
writing manifest file 'dblaikie_hello.egg-info/SOURCES.txt'
running check
We need to know who you are, so please choose either:
 1. use your existing login,
 2. register as a new user,
 3. have the server generate a new password for you (and email it to you), or
 4. quit
Your selection [default 1]:

twine for uploading to PyPI

$ pip install twine

Testing

Testing: Introduction

Code without tests is like driving without seatbelts.

All code is subject to errors -- not just ValueErrors and TypeErrors encountered during development, but errors related to unexpected data anomalies or user input, or the unforeseen effects of functions run in untested combinations. Unit testing is the front line of the effort to ensure code quality. Many developers say they won't take a software package seriously unless it comes with tests. testing: a brief rundown Unit testing is the most basic form and the one we will focus on here. Other styles of testing:

• unit test: testing individual units (function/method)
• integration test: testing multiple units together
• regression test: testing to see if changes have introduced errors
• end-to-end test: testing the entire program

Unit Testing

"Unit" refers to a function. Unit testing calls individual functions and validates the output or result of each.

The most easily tested scripts are made up of small functions that can be called and validated in isolation. Therefore "pure functions" (functions that do not refer or change "external state" -- i.e., global variables) are best for testing. Testing for success, testing for failure A unit test script performs test by importing the script to be tested and calling its functions with varying arguments, including ones intended to cause an error. Basically, we are hammering the code as many ways as we can to make sure it succeeds properly and fails properly. Test-driven development As we develop our code, we can write tests simultaneously and run them periodically as we develop. This way we can know that further changes and additions are not interfering with anything we have done previously. Any time in the process we can run the testing program and it will run all tests. In fact commonly accepted wisdom supports writing tests before writing code! The test is written with the function in mind: after seeing that the tests fail, we write a function to satisfy the tests. This called test-driven development.

The assert statement

assert raises an AssertionError exception if the test returns False

assert 5 == 5        # no output

assert 5 == 10       # AssertionError raised

We can incorporate this facility in a simple testing program: program to be tested: "myprogram.py"

import sys

def doubleit(x):
    var = x * 2
    return var

if __name__ == '__main__':
    input_val = sys.argv[1]
    doubled_val = doubleit(input_val)

    print("the value of {0} is {1}".format(input_val, doubled_val))

testing program: "test_myprogram.py"

import myprogram

def test_doubleit_value():
    assert myprogram.doubleit(10) == 20

If doubleit() didn't correctly return 20 with an argument of 10, the assert would raise an AssertionError. So even with this basic approach (without a testing module like pyttest or unittest), we can do testing with assert.

pytest Basics

All programs named test_something.py that have functions named test_something() will be noticed by pytest and run automatically when we run the pytest script py.test.

instructions for running tests 1. Download the testing program test_[name].py (where name is any name) and place in the same directory as your script. 2. Open up a command prompt. In Mac/Unix, open the Terminal program (you can also use the Terminal window in PyCharm). In Windows, you must launch the Anaconda Command Prompt, which should be accessible by searching for cmd on Windows 10 -- let me know if you have trouble finding the Anaconda prompt. 3. Make sure your homework script and the test script are in the same directory, and that your Command Prompt or Terminal window is also in that directory (let me know if you have any trouble using cd to travel to this directory in your Command Prompt or Terminal window.) 4. Execute the command py.test at the command line (keep in mind this is not the Python prompt, but your Mac/Unix Terminal program or your Anaconda cmd/Command Prompt window). py.test is a special command that should work from your command line if you have Anaconda installed. (py.test is not a separate file.) 5. If your program and functions are named as directed, the testing program will import your script and test each function to see that it is providing correct output. If there are test failures, look closely at the failure output -- look for the assert test showing what values were involved in the failure. You can also look at the testing program to see what it is requiring (look for the assert statements). 6. If you see collected 0 items it means there was no test_[something].py file (where [something] is a name of your choice) or there was no test_[something]() function inside the test program. These names are required by py.test. 7. If your run of py.test hangs (i.e., prints something out but then just waits), or if you see a lot of colorful error output saying not found here and there, it may be for the above reason.

running py.test from the command line or Anaconda command prompt:

$ py.test
 =================================== test session starts ====================================
platform darwin -- Python 2.7.10 -- py-1.4.27 -- pytest-2.7.1
rootdir: /Users/dblaikie/testpytest, inifile:
collected 1 items

test_myprogram.py .

 ================================= 1 passed in 0.01 seconds =================================

noticing failures

def doubleit(x):
    var = x * 2
    return x      # oops, returned the original value rather than the doubled value

Having incorporated an error, run py.test again:

$ py.test
 =================================== test session starts ====================================
platform darwin -- Python 2.7.10 -- py-1.4.27 -- pytest-2.7.1
rootdir: /Users/dblaikie/testpytest, inifile:
collected 1 items

test_myprogram.py F

 ========================================= FAILURES =========================================
___________________________________ test_doubleit_value ____________________________________

    def test_doubleit_value():
>       assert myprogram.doubleit(10) == 20
E       assert 10 == 20
E        +  where 10 = (10)
E        +    where  = myprogram.doubleit

test_myprogram.py:7: AssertionError
 ================================= 1 failed in 0.01 seconds =================================

Writing tests

Use assert to test values returned from a function against expected values

assert raises an AssertionError exception if the test returns False

assert 5 == 5        # no output

assert 5 == 10       # AssertionError raised

We can incorporate this facility in a simple testing program: program to be tested: "myprogram.py"

import sys

def doubleit(x):
    var = x * 2
    return var

if __name__ == '__main__':
    input_val = sys.argv[1]
    doubled_val = doubleit(input_val)

    print("the value of {0} is {1}".format(input_val, doubled_val))

testing program: "test_myprogram.py"

import myprogram

def test_doubleit_value():
    assert myprogram.doubleit(10) == 20

If doubleit() didn't correctly return 20 with an argument of 10, the assert would raise an AssertionError. So even with this basic approach (without a testing module like pyttest or unittest), we can do testing with assert.

Writing tests with pytest

All programs named test_something.py that have functions named test_something() will be noticed by pytest and run automatically when we run the pytest script py.test.

instructions for writing and running tests using pytest 1. Make sure your program, myprogram.py and your testing program test_myprogram.py are in the same directory. 2. Open up a command prompt. In Mac/Unix, open the Terminal program (you can also use the Terminal window in PyCharm). In Windows, you must launch the Anaconda Command Prompt, which should be accessible by searching for cmd on Windows 10 -- let me know if you have trouble finding the Anaconda prompt. 3. Use cd to change the present working directory for your Command Prompt or Terminal window session to the same directory (let me know if you have any trouble with this step.) 4. Execute the command py.test at the command line (keep in mind this is not the Python prompt, but your Mac/Unix Terminal program or your Anaconda cmd/Command Prompt window). py.test is a special command that should work from your command line if you have Anaconda installed. (py.test is not a separate file.) 5. If your program and functions are named as directed, the testing program will import your script and test each function to see that it is providing correct output. If there are test failures, look closely at the failure output -- look for the assert test showing what values were involved in the failure. You can also look at the testing program to see what it is requiring (look for the assert statements). 6. If you see collected 0 items it means there was no test_[something].py file (where [something] is a name of your choice) or there was no test_[something]() function inside the test program. These names are required by py.test. 7. If your run of py.test hangs (i.e., prints something out but then just waits), or if you see a lot of colorful error output saying not found here and there, it may be for the above reason.

running py.test from the command line or Anaconda command prompt:

$ py.test
 =================================== test session starts ====================================
platform darwin -- Python 2.7.10 -- py-1.4.27 -- pytest-2.7.1
rootdir: /Users/dblaikie/testpytest, inifile:
collected 1 items

test_myprogram.py .

 ================================= 1 passed in 0.01 seconds =================================

noticing failures

def doubleit(x):
    var = x * 2
    return x      # oops, returned the original value rather than the doubled value

Having incorporated an error, run py.test again:

$ py.test
 =================================== test session starts ====================================
platform darwin -- Python 2.7.10 -- py-1.4.27 -- pytest-2.7.1
rootdir: /Users/dblaikie/testpytest, inifile:
collected 1 items

test_myprogram.py F

 ========================================= FAILURES =========================================
___________________________________ test_doubleit_value ____________________________________

    def test_doubleit_value():
>       assert myprogram.doubleit(10) == 20
E       assert 10 == 20
E        +  where 10 = (10)
E        +    where  = myprogram.doubleit

test_myprogram.py:7: AssertionError
 ================================= 1 failed in 0.01 seconds =================================

Testing the expected raising of an exception

Many of our tests will deliberately pass bad input and test to see that an appropriate exception is raised.

import sys

def doubleit(x):
    if not isinstance(x, (int, float)):           # make sure the arg is the right type
        raise TypeError('must be int or float')   # if not, raise a TypeError
    var = x * 2
    return var

if __name__ == '__main__':
    input_val = sys.argv[1]
    doubled_val = doubleit(input_val)

    print("the value of {0} is {1}".format(input_val, doubled_val))

Note that without type testing, the function could work, but incorrectly (for example if a string or list were passed instead of an integer). To verify that this error condition is correctly raised, we can use with pytest.raises(TypeError).

import myprogram
import pytest

def test_doubleit_value():
    assert myprogram.doubleit(10) == 20

def test_doubleit_type():
    with pytest.raises(TypeError):
        myprogram.doubleit('hello')

with is the same context manager we have used with open(): it can also be used to detect when an exception occured inside the with block.

Grouping tests into a class

We can organize related tests into a class, which can also include setup and teardown routines that are run automatically (discussed next).

""" test_myprogram.py -- test functions in a testing class """

import myprogram
import pytest

class TestDoubleit(object):

    def test_doubleit_value(self):
        assert myprogram.doubleit(10) == 20

    def test_doubleit_type(self):
        with pytest.raises(TypeError):
            myprogram.doubleit('hello')

So now the same rule applies for how py.test looks for tests -- if the class begins with the word Test, pytest will treat it as a testing class.

Mock data: setup and teardown

Tests should not be run on "live" data; instead, it should be simulated, or "mocked" to provide the data the test needs.

""" myprogram.py -- makework functions for the purposes of demonstrating testing """

import sys

def doubleit(x):
    """ double a number argument, return doubled value """
    if not isinstance(x, (int, float)):
        raise TypeError('arg to doublit() must be int or float')
    var = x * 2
    return var

def doublelines(filename):
    """open a file of numbers, double each line, write each line to a new file"""
    with open(filename) as fh:
        newlist = []
        for line in fh:                   # file is assumed to have one number on each line
            floatval = float(line)
            doubleval = doubleit(floatval)
            newlist.append(str(doubleval))
    with open(filename, 'w') as fh:
        fh.write('\n'.join(newlist))

if __name__ == '__main__':
    input_val = sys.argv[1]
    doubled_val = doubleit(input_val)

    print("the value of {0} is {1}".format(input_val, doubled_val))

For this demo I've invented a rather arbitrary example to combine an external file with the doubleit() routine: doublelines() opens and reads a file, and for each line in the file, doubles the value, writing each value as a separate line to a new file (supplied to doublelines()).

""" test_myprogram.py -- test the doubleit.py script """

import myprogram
import os
import pytest
import shutil

class TestDoubleit(object):

    numbers_file_template = 'testnums_template.txt'  # template for test file (stays the same)
    numbers_file_testor = 'testnums.txt'             # filename used for testing
                                                     # (changed during testing)

    def setup_class(self):
        shutil.copy(TestDoubleit.numbers_file_template, TestDoubleit.numbers_file_testor)

    def teardown_class(self):
        os.remove(TestDoubleit.numbers_file_testor)

    def test_doublelines(self):
        myprogram.doublelines(TestDoubleit.numbers_file_testor)
        old_vals = [ float(line) for line in open(TestDoubleit.numbers_file_template) ]
        new_vals = [ float(line) for line in open(TestDoubleit.numbers_file_testor) ]
        for old_val, new_val in zip(old_vals, new_vals):
            assert float(new_val) == float(old_val) * 2

    def test_doubleit_value(self):
        assert myprogram.doubleit(10) == 20

    def test_doubleit_type(self):
        with pytest.raises(TypeError):
            myprogram.doubleit('hello')

setup_class and teardown_class run automatically. As you can see, they prepare a dummy file and when the testing is over, delete it. In between, tests are run in order based on the function names.

Writing tests

Use assert to test values returned from a function against expected values

assert raises an AssertionError exception if the test returns False

assert 5 == 5        # no output

assert 5 == 10       # AssertionError raised

We can incorporate this facility in a simple testing program: program to be tested: "myprogram.py"

import sys

def doubleit(x):
    var = x * 2
    return var

if __name__ == '__main__':
    input_val = sys.argv[1]
    doubled_val = doubleit(input_val)

    print("the value of {0} is {1}".format(input_val, doubled_val))

testing program: "test_myprogram.py"

import myprogram

def test_doubleit_value():
    assert myprogram.doubleit(10) == 20

If doubleit() didn't correctly return 20 with an argument of 10, the assert would raise an AssertionError. So even with this basic approach (without a testing module like pyttest or unittest), we can do testing with assert.

Writing tests with pytest

All programs named test_something.py that have functions named test_something() will be noticed by pytest and run automatically when we run the pytest script py.test.

instructions for writing and running tests using pytest 1. Make sure your program, myprogram.py and your testing program test_myprogram.py are in the same directory. 2. Open up a command prompt. In Mac/Unix, open the Terminal program (you can also use the Terminal window in PyCharm). In Windows, you must launch the Anaconda Command Prompt, which should be accessible by searching for cmd on Windows 10 -- let me know if you have trouble finding the Anaconda prompt. 3. Use cd to change the present working directory for your Command Prompt or Terminal window session to the same directory (let me know if you have any trouble with this step.) 4. Execute the command py.test at the command line (keep in mind this is not the Python prompt, but your Mac/Unix Terminal program or your Anaconda cmd/Command Prompt window). py.test is a special command that should work from your command line if you have Anaconda installed. (py.test is not a separate file.) 5. If your program and functions are named as directed, the testing program will import your script and test each function to see that it is providing correct output. If there are test failures, look closely at the failure output -- look for the assert test showing what values were involved in the failure. You can also look at the testing program to see what it is requiring (look for the assert statements). 6. If you see collected 0 items it means there was no test_[something].py file (where [something] is a name of your choice) or there was no test_[something]() function inside the test program. These names are required by py.test. 7. If your run of py.test hangs (i.e., prints something out but then just waits), or if you see a lot of colorful error output saying not found here and there, it may be for the above reason.

running py.test from the command line or Anaconda command prompt:

$ py.test
 =================================== test session starts ====================================
platform darwin -- Python 2.7.10 -- py-1.4.27 -- pytest-2.7.1
rootdir: /Users/dblaikie/testpytest, inifile:
collected 1 items

test_myprogram.py .

 ================================= 1 passed in 0.01 seconds =================================

noticing failures

def doubleit(x):
    var = x * 2
    return x      # oops, returned the original value rather than the doubled value

Having incorporated an error, run py.test again:

$ py.test
 =================================== test session starts ====================================
platform darwin -- Python 2.7.10 -- py-1.4.27 -- pytest-2.7.1
rootdir: /Users/dblaikie/testpytest, inifile:
collected 1 items

test_myprogram.py F

 ========================================= FAILURES =========================================
___________________________________ test_doubleit_value ____________________________________

    def test_doubleit_value():
>       assert myprogram.doubleit(10) == 20
E       assert 10 == 20
E        +  where 10 = (10)
E        +    where  = myprogram.doubleit

test_myprogram.py:7: AssertionError
 ================================= 1 failed in 0.01 seconds =================================

Testing the expected raising of an exception

Many of our tests will deliberately pass bad input and test to see that an appropriate exception is raised.

import sys

def doubleit(x):
    if not isinstance(x, (int, float)):           # make sure the arg is the right type
        raise TypeError('must be int or float')   # if not, raise a TypeError
    var = x * 2
    return var

if __name__ == '__main__':
    input_val = sys.argv[1]
    doubled_val = doubleit(input_val)

    print("the value of {0} is {1}".format(input_val, doubled_val))

Note that without type testing, the function could work, but incorrectly (for example if a string or list were passed instead of an integer). To verify that this error condition is correctly raised, we can use with pytest.raises(TypeError).

import myprogram
import pytest

def test_doubleit_value():
    assert myprogram.doubleit(10) == 20

def test_doubleit_type():
    with pytest.raises(TypeError):
        myprogram.doubleit('hello')

with is the same context manager we have used with open(): it can also be used to detect when an exception occured inside the with block.

Grouping tests into a class

We can organize related tests into a class, which can also include setup and teardown routines that are run automatically (discussed next).

""" test_myprogram.py -- test functions in a testing class """

import myprogram
import pytest

class TestDoubleit(object):

    def test_doubleit_value(self):
        assert myprogram.doubleit(10) == 20

    def test_doubleit_type(self):
        with pytest.raises(TypeError):
            myprogram.doubleit('hello')

So now the same rule applies for how py.test looks for tests -- if the class begins with the word Test, pytest will treat it as a testing class.

Mock data: setup and teardown

Tests should not be run on "live" data; instead, it should be simulated, or "mocked" to provide the data the test needs.

""" myprogram.py -- makework functions for the purposes of demonstrating testing """

import sys

def doubleit(x):
    """ double a number argument, return doubled value """
    if not isinstance(x, (int, float)):
        raise TypeError('arg to doublit() must be int or float')
    var = x * 2
    return var

def doublelines(filename):
    """open a file of numbers, double each line, write each line to a new file"""
    with open(filename) as fh:
        newlist = []
        for line in fh:                   # file is assumed to have one number on each line
            floatval = float(line)
            doubleval = doubleit(floatval)
            newlist.append(str(doubleval))
    with open(filename, 'w') as fh:
        fh.write('\n'.join(newlist))

if __name__ == '__main__':
    input_val = sys.argv[1]
    doubled_val = doubleit(input_val)

    print("the value of {0} is {1}".format(input_val, doubled_val))

For this demo I've invented a rather arbitrary example to combine an external file with the doubleit() routine: doublelines() opens and reads a file, and for each line in the file, doubles the value, writing each value as a separate line to a new file (supplied to doublelines()).

""" test_myprogram.py -- test the doubleit.py script """

import myprogram
import os
import pytest
import shutil

class TestDoubleit(object):

    numbers_file_template = 'testnums_template.txt'  # template for test file (stays the same)
    numbers_file_testor = 'testnums.txt'             # filename used for testing
                                                     # (changed during testing)

    def setup_class(self):
        shutil.copy(TestDoubleit.numbers_file_template, TestDoubleit.numbers_file_testor)

    def teardown_class(self):
        os.remove(TestDoubleit.numbers_file_testor)

    def test_doublelines(self):
        myprogram.doublelines(TestDoubleit.numbers_file_testor)
        old_vals = [ float(line) for line in open(TestDoubleit.numbers_file_template) ]
        new_vals = [ float(line) for line in open(TestDoubleit.numbers_file_testor) ]
        for old_val, new_val in zip(old_vals, new_vals):
            assert float(new_val) == float(old_val) * 2

    def test_doubleit_value(self):
        assert myprogram.doubleit(10) == 20

    def test_doubleit_type(self):
        with pytest.raises(TypeError):
            myprogram.doubleit('hello')

setup_class and teardown_class run automatically. As you can see, they prepare a dummy file and when the testing is over, delete it. In between, tests are run in order based on the function names.

Regular Expressions: Text Matching and Extraction

Text Matching: Why is it so Useful?

Short answer: validation and extraction of formatted text. Either we want to see whether a string contains the pattern we seek, or we want to pull selected information from the string.

In the case of fixed-width text, we have been able to use a slice.

line = '19340903  3.4 0.9'
year = line[0:4]                 # year == 1934

In the case of delimited text, we have been able to use split()

line = '19340903,3.4,0.9'
els = line.split(',')
yearmonthday = els[0]            # 193400903
MktRF = els[1]                   # 3.4

In the case of formatted text, there is no obvious way to do it.

# how would we extract 'Jun' from this string?
log_line = '66.108.19.165 - - [09/Jun/2003:19:56:33 -0400] "GET /~jjk265/cd.jpg HTTP/1.1" 200 175449'

We may be able to use split() and slicing in some combination to get what we want, but it would be awkward and time consuming. So we're going to learn how to use regular expressions.

Preview: a complex example

Just as an example to show you what we're doing, the following regex pattern could be used to pull 'Jun' from the log line:

import re

log_line = '66.108.19.165 - - [09/Jun/2003:19:56:33 -0400] "GET /~jjk265/cd.jpg HTTP/1.1" 200 175449'

reg = re.search(r'\d{2,3}\.\d{2,3}\.\d{2,3}\.\d{2,3} - - \[\d\d\/(\w{3})\/\d{4}', log_line)

print(reg.group(1))   # Jun

Reading from left to right, the pattern (shown in the r'' string) says this: "2-3 digits, followed by a period, followed by 2-3 digits, followed by a period, followed by 2-3 digits, followed by a period, followed by 2-3 digits, followed by a space, dash, space, dash, followed by a square bracket, followed by 2 digits, followed by a forward slash, followed by 3 word characters (and this text grouped for extraction), followed by a slash, followed by 4 digit characters." Now, that may seem complex. Many of our regexes will be much simpler, although this one isn't terribly unusual. But the power of this tool is in allowing us to describe a complex string in terms of the pattern of the text -- not the specific text -- and either pull parts of it out or make sure it matches the pattern we're looking for. This is the purpose of regular expressions.

Python's re module and re.search()

import re makes regular expressions available to us. Everything we do with regular expressions is done through re.

re.search() takes two arguments: the pattern string which is the regex pattern, and the string to be searched. It can be used in an if expression, and will evaluate to True if the pattern matched.

# weblog contains string lines like this:
  '66.108.19.165 - - [09/Jun/2003:19:56:33 -0400] "GET /~jjk265/cd.jpg HTTP/1.1" 200 175449'
  '66.108.19.165 - - [09/Jun/2003:19:56:44 -0400] "GET /~dbb212/mysong.mp3 HTTP/1.1" 200 175449'
  '66.108.19.165 - - [09/Jun/2003:19:56:45 -0400] "GET /~jjk265/cd2.jpg HTTP/1.1" 200 175449'

# script snippet:
for line in weblog.readlines():
  if re.search(r'~jjk265', line):
    print line                      # prints 2 of the above lines

not for negating a search

As with any if test, the test can be negated with not. Now we're saying "if this pattern does not match".

# again, a weblog:
  '66.108.19.165 - - [09/Jun/2003:19:56:33 -0400] "GET /~jjk265/cd.jpg HTTP/1.1" 200 175449'
  '66.108.19.165 - - [09/Jun/2003:19:56:44 -0400] "GET /~dbb212/mysong.mp3 HTTP/1.1" 200 175449'
  '66.108.19.165 - - [09/Jun/2003:19:56:45 -0400] "GET /~jjk265/cd2.jpg HTTP/1.1" 200 175449'

# script snippet:
for line in weblog.readlines():
  if not re.search(r'~jjk265', line):
    print line                      # prints 1 of the above lines -- the one without jjk265

The raw string (r'')

The raw string is like a normal string, but it does not process escapes. An escaped character is one preceded by a backslash, that turns the combination into a special character. \n is the one we're famliar with - the escaped n converts to a newline character, which marks the end of a line in a multi-line string.

A raw string wouldn't process the escape, so r'\n' is literally a backslash followed by an n.

var = "\n"            # one character, a newline
var2 = r'\n'          # two characters, a backslash followed by an n

The re Bestiary

We call it a "bestiary" because it contains many strange animals. Each of these animals takes the shape of a special character (like $, ^, |); or a regular text character that has been escaped (like \w, \d, \s); or a combination of characters in a group (like {2,3}, [aeiou]). Our bestiary can be summarized thusly:

Anchor Characters and the Boundary Character: $, ^, \b Character Classes: \w, \d, \s, \W, \S, \D Custom Character Classes: [aeiou], [a-zA-Z] The Wildcard: . Quantifiers: +, *, ? Custom Quantifiers: {2,3}, {2,}, {2} Groupings: (parentheses groups)

Patterns can match anywhere, but must match on consecutive characters

Patterns can match any string of consecutive characters. A match can occur anywhere in the string.

import re

str1 = 'hello there'
str2 = 'why hello there'
str3 = 'hel lo'

if re.search(r'hello', str1):  print('matched')   # matched
if re.search(r'hello', str2):  print('matched')   # matched
if re.search(r'hello', str3):  print('matched')   # does not match

Note that 'hello' matches at the start of the first string and the middle of the second string. But it doesn't match in the third string, even though all the characters we are looking for are there. This is because the space in str3 is unaccounted for - always remember - matches take place on consecutive characters.

Anchors and Boundary

Our match may require that the search text appear at the beginning or end of a string. The anchor characters can require this.

This program lists only those files in the directory that end in .txt:

import os, re
for filename in os.listdir(r'/path/to/directory'):
  if re.search(r'\.txt$', filename):     # look for '.txt' at end of filename
    print(filename)

This program prints all the lines in the file that don't begin with a hash mark:

for text_line in open(r'/path/to/file.py'):
  if not re.search(r'^#', text_line):        # look for '#' at start of filename
    print(text_line)

When they are used as anchors, we will always expect ^ to appear at the start of our pattern, and $ to appear at the end.

Character Classes

A character class is a special regex entity that will match on any of a set of characters. The three built-in character classes are these:

\d: [0-9] (Digits) \w: [a-zA-Z0-9_] (Word characters -- letters, numbers or underscores) \s: [ \n\t] ('Whitespace' characters -- spaces, newlines, or tabs) So a \d will match on a 5, 9, 3, etc.; a \w will match on any of those, or on a, Z, _ (underscore). Keep in mind that although they match on any of several characters, a single instance of a character class matches on only one character. For example, a \d will match on a single number like '5', but it won't match on both characters in '55'. To match on 55, you could say \d\d.

Built-in Character Classes: digits

The \d character class matches on any digit. This example lists only those files with names formatted with a particular syntax -- YYYY-MM-DD.txt:

import re
dirlist = ('.', '..', '2010-12-15.txt', '2010-12-16.txt', 'testfile.txt')
for filename in dirlist:
  if re.search(r'^\d\d\d\d-\d\d-\d\d\.txt$', filename):
    print(filename)

Here's another example, validation: this regex uses the pattern ^\d\d\d\d$ to check to see that the user entered a four-digit year:

import re
answer = input("Enter your birth year in the form YYYY\n")
if re.search(r'^\d\d\d\d$', answer):
  print("Your birth year is ", answer)
else:
  print("Sorry, that was not YYYY")

Built-in Character Classes: "word" characters

A word character casts a wider net: it will match on any number, letter or underscore.

In this example, we require the user to enter a username with any word characters:

username = input()
if not re.search(r'^\w\w\w\w\w$', username):
  print("use five numbers, letters, or underscores\n")

As you can see, the anchors prevent the input from exceeding 5 characters.

Built-in Character Classes: spaces

A space character class matches on any of three characters: a space (' '), a newline ('\n') or a tab ('\t'). This program searches for a space anywhere in the string and if it finds it, the match is successful - which means the input isn't successful:

new_password = input()
if re.search(r'\s', new_password):
  print("password must not contain spaces")

Note in particular that the regex pattern \s is not anchored anywhere. So the regex will match if a space occurs anywhere in the string. You may also reflect that we treat spaces pretty roughly - always stripping them off. They always get in the way! And they're invisible, too, and still we feel the need to persecute them. What a nuisance.

Inverse Character Classes

These are more aptly named inverse character classes - they match on anything that is not in the usual character set.

Not a digit: \D

So \D matches on letters, underscores, special characters - anything that is not a digit. This program checks for a non-digit in the user's account number:

account_number = input()
if re.search(r'\D', account_number):
  print("account number must be all digits!")

Not a word character: \W

Here's a username checker, which simply looks for a non-word:

account_number = input()
if re.search(r'\W', account_number):
  print("account number must be only letters, numbers, and underscores")

Not a space character: \S

These two regexes check for a non-space at the start and end of the string:

sentence = input()
if re.search(r'^\S', sentence) and re.search(r'\S$', sentence):
  print("the sentence does not begin or end with a space, tab or newline.")

Custom Character Classes

Consider this table of character classes and the list of characters they match on:

\d: [0123456789] \w: [abcdefghijklmnopqrstuvwxyzABCDEFGHIJKLMNOOPQRSTUVWXYZ0123456789_] or
[a-zA-Z0-9_] \s: [ \t\n] In fact, the bracketed ranges can be used to create our own character classes. We simply place members of the class within the brackets and use it in the same way we might use \d or the others.

A custom class can contain a range of characters. This example looks for letters only (there is no built-in class for letters):

import re
input = input("please enter a username, starting with a letter:  ")
if not re.search(r'^[a-zA-Z]', input):
  exit("invalid user name entered")

This custom class [.,;:?!] matches on any one of these punctuation characters, and this example identifies single punctuation characters and removes them:

import re
text_line = 'Will I?  I will.  Today, tomorrow; yesterday and before that.'
for word in text_line.split():
  while re.search(r'[.,;:?! -]$', word):
    word = word[:-1]
  print(word)

Inverse Custom Character Classes

Like \S for \s, the inverse character class matches on anything not in the list. It is designated with a carrot just inside the open bracket:

import re
for text_line in open('unknown_text.txt'):
  for word in text_line.split():
    while re.search(r'[^a-zA-Z]$', word):
      word = word[:-1]
    print(word)

It would be easy to confuse the carrot at the start of a string with the carrot at the start of a custom character class -- just keep in mind that one appears at the very start of the string, and the other at the start of the bracketed list.

The Wildcard (.)

The ultimate character class, it matches on every character except for a newline. (We might surmise this is because we are often working with line-oriented input, with pesky newlines at the end of every line. Not matching on them means we never have to worry about stripping or watching out for newlines.)

import re
username = input()
if not re.search(r'^.....$', username):   # five dots here
  print("you can use any characters except newline, but there must \
  be five of them.\n")

Quantifiers: specifies how many to match on

A quantifier appears immediately after a character, character class, or grouping (coming up). It describes how many of the preceding characters there may be in our matched text.

We can say three digits (\d{3}), between 1 and 3 word characters (\w{1,3}), one or more letters [a-zA-Z]+, zero or more spaces (\s*), one or more x's (x+). Anything that matches on a character can be quantified.

+      : 1 or more
*      : 0 or more
?      : 0 or 1
{3,10} : between 3 and 10

In this example directory listing, we are interested only in files with the pattern config_ followed by an integer of any size. We know that there could be a config_1.txt, a config_12.txt, or a config_120.txt. So, we simply specify "one or more digits":

import re
filenames = ['config_1.txt', 'config_10.txt', 'notthis.txt', '.', '..']
wanted_files = []
for file in filenames:
  if re.search(r'^config_\d+\.txt$', file):
    wanted_files.append(file)

Here, we validate user input to make sure it matches the pattern for valid NYU ID. The pattern for an NYU Net ID is: two or three letters followed by one or more numbers:

import re
input = input("please enter your net id:  ")
if not re.search(r'^[A-Za-z]{2,3}\d+$', input):
  print("that is not valid NYU Net ID!")

A simple email address is one or more word characters followed by an @ sign, followed by a period, followed by 2-4 letters:

import re
email_address = input()
if re.search(r'^\w+@\w+\.[A-Za-z]{2,}$', email_address):
  print("email address validated")

Of course email addresses can be more complicated than this - but for this exercise it works well.

flags: re.IGNORECASE

We can modify our matches with qualifiers called flags. The re.IGNORECASE flag will match any letters, whether upper or lowercase. In this example, extensions may be upper or lowercase - this file matcher doesn't care!

import re
dirlist = ('thisfile.jpg', 'thatfile.txt', 'otherfile.mpg', 'myfile.TXT')
for file in dirlist:
  if re.search(r'\.txt$', file, re.IGNORECASE):   #'.txt' or '.TXT'
    print(file)

The flag is passed as the third argument to search, and can also be passed to other re search methods.

re.search(), re.compile() and the compile object

re.search() is the one-step method we've been using to test matching. Actually, regex matching is done in two steps: compiling and searching. re.search() conveniently puts the two together.

In some cases, a pattern should be compiled first before matching begins. This would be useful if the pattern is to be matched on a great number of strings, as in this weblog example:

import re
access_log = '/home1/d/dbb212/public_html/python/examples/access_log'
weblog = open(access_log)
patternobj = re.compile(r'edg205')
for line in weblog.readlines():
  if patternobj.search(line):
    print(line, end=' ')
weblog.close()

The pattern object is returned from re.compile, and can then be called with search. Here we're calling search repeatedly, so it is likely more efficient to compile once and then search with the compiled object.

Grouping for Alternates: Vertical Bar

We can group several characters together with parentheses. The parentheses do not affect the match, but they do designate a part of the matched string to be handled later. We do this to allow for alternate matches, for quantifying a portion of the pattern, or to extract text.

Inside a group, the vertical bar can indicate allowable matches. In this example, a string will match on any of these words, and because of the anchors will not allow any other characters:

import re
import sys

program_arg = sys.argv[1]
if not re.search(r'^Q(1|2|3|4)\-\d{4}$', program_arg):
    exit("quarter argument must match the pattern 'Q[num]-YYYY' "
         "where [num] is 1-4 and YYYY is a 4-digit year")

Grouping for Quantifying

Let's expand our email address pattern and make it possible to match on any of these examples:

good_emails = [
    'joe@apex.com',
    'joe.wilson@apex.com',
    'joe.wilson@eng.apex.com',
    'joe.g.zebulon.wilson@my.subdomain.eng.apex.com'
]

And let's make sure our regex fails on any of these:

bad_emails = [
    '.joe@apex.com',          # leading period
    'joe.wilson@apex.com.',   # trailing period
    'joe..wilson@apex.com'    # two periods together
]

How can we include the period while making sure it doesn't appear at the start or end, or repeated, as it does in the bad_emails list?

Look for a repeating pattern of groups of characters in the good_emails. In these combinations, we are attempting to account for subdomains, which could conceivably be chained togtehter. In this case, there is a pattern joe., that we can match with \w+\. (a period, the wildcard, must be escaped). If we see that this may repeat, we can group the pattern and apply a quantifier to it:

import re
for address in good_emails + bad_emails:                # concatenates two lists
    if re.search(r'^(\w+\.)*\w+@(\w+\.)+[A-Za-z]{2,}$', address):
        print("{0}:  good".format(address))
    else:
        print("{0}:  bad".format(address))

Grouping for Extraction: the matchobject group() method.

We use the group() method of the match object to extract the text that matched the group.

Here's an example, using our log file. What if we wanted to capture the last two numbers (the status code and the number of bytes served), and place the values into structures?

log_lines = [
'66.108.19.165 - - [09/Jun/2003:19:56:33 -0400] "GET /~jjk265/cd.jpg HTTP/1.1" 200 175449',
'216.39.48.10 - - [09/Jun/2003:19:57:00 -0400] "GET /~rba203/about.html HTTP/1.1" 200 1566',
'216.39.48.10 - - [09/Jun/2003:19:57:16 -0400] "GET /~dd595/frame.htm HTTP/1.1" 400 1144'
]

import re
bytes_sum = 0
for line in log_lines:
  matchobj = re.search(r'(\d+) (\d+)$', line) # last two numbers in line
  status_code = matchobj.group(1)
  bytes = matchobj.group(2)
  bytes_sum += int(bytes)                     # sum the bytes

groups()

groups() returns all grouped matches.

If you wish to grab all the matches into a tuple rather than call them by number, use groups(). You can then read variables from the tuple, or assign groups() to named variables.

In this example, the Salesforce Customer Relationship Management system has a field in one of its objects that discounts certain revenue and explains the reason. Our job is to extract the code and the reason from the string:

import re
my_GL_codes = [
    '12520 - Allowance for Customer Concessions',
    '12510 - Allowance for Unmet Mins',
    '40000 - Platform Revenue',
    '21130 - Pre-paid Revenue',
    '12500 - Allowance for Doubtful Accounts'
]
for field in my_GL_codes:
    codeobj = re.search(r'^(\d+)\s*\-\s*(.+)$', field)      # GL code
    name_tuple = codeobj.groups()
    print('tuple from groups:  ', name_tuple)
    code, reason = name_tuple
    print("extracted:  '{0}':  '{1}'".format(code, reason))
    print()

findall() for multiple matches

findall() matches the same pattern repeatedly, returning all matched text within a string.

findall() with a groupless pattern

Usually re tries to a match a pattern once -- and after it finds the first match, it quits searching. But we may want to find as many matches as we can -- and return the entire set of matches in a list. findall() lets us do that:

text = "There are seven words in this sentence";
words = re.findall(r'\w+', text)
print(words)  # ['There', 'are', 'seven', 'words', 'in', 'this', 'sentence']

This program prints each of the words on a separate line. The pattern \b\w+\b is applied again and again, each time to the text remaining after the last match. This pattern could be used as a word counting algorithm (we would count the elements in words), except for words with punctuation. findall() with groups

When a match pattern contains more than one grouping, findall returns multiple tuples:

text = "High: 33, low: 17"
temp_tuples = re.findall(r'(\w+):\s+(\d+)', text)
print(temp_tuples)                       # [('High', '33'), ('low', '17')]

re.sub() for substitutions

re.sub() replaces matched text with replacement text.

Regular expressions are used for matching so that we may inspect text. But they can also be used for substitutions, meaning that they have the power to modify text as well.

This example replaces Microsoft '\r\n' line ending codes with Unix '\n'.

text = re.sub(r'\r\n', '\n', text)

Here's another simple example:

string = "My name is David"
string = re.sub('David', 'John', string)

print(string)                            # 'My name is John'

Practical: matching on an entire file string

An entire file as a single string opens up additional matching possibilities.

This example opens and reads a web page (which we might have retrieved with a module like urlopen), then looks to see if the word "advisory" appears in the text. If it does, it prints the page:

file = open('weather-ny.html')
text = file.read()
if re.search(r'advisory', text, re.I):
  print("weather advisory:  ", text)

re.MULTILINE: ^ and $ can match at start or end of line

Within a file of many lines, we can specify start or end of a single line.

We have been working with text files primarily in a line-oriented (or, in database terminology, record-oriented way, and regexes are no exception - most file data is oriented in this way. However, it can be useful to dispense with looping and use regexes to match within an entire file - read into a string variable with read().

In this example, we surely can use a loop and split() to get the info we want. But with a regex we can grab it straight from the file in one line:

# passwd file:
nobody:*:-2:-2:Unprivileged User:/var/empty:/usr/bin/false
root:*:0:0:System Administrator:/var/root:/bin/sh
daemon:*:1:1:System Services:/var/root:/usr/bin/false

# python script:
import re
passwd_text = open('/etc/passwd').read()
mobj =  re.search(r'^root:[^:]+:[^:]+:[^:]:([^:]+):([^:]+)', passwd_text, re.MULTILINE)
if mobj:
    info = mobj.groups()
    print("root:  Name %s, Home Dir %s" % (info[0], info[1]))

We can even use findall to extract all the information rfrom a file - keep in mind, this is still being done in two lines:

import re
passwd_text = open('/etc/passwd').read()
lot = re.findall(r'^(\w+):[^:]+:[^:]+:[^:]+:[^:]+:([^:]+)', passwd_text, re.MULTILINE)

mydict = dict(lot)

print(mydict)

re.DOTALL -- allow the wildcard (.) to match on newlines

Matching the wildcard on newlines may be needed for a multi-line file string.

Normally, the wildcard doesn't match on newlines. When working with whole files, we may want to grab text that spans multiple lines, using a wildcard.

# search file sample.txt
some text we don't want
==start text==
this is some text that we do want.
the extracted text should continue,
including just about any character,
until we get to
==end text==
other text we don't want

# python script:
import re
text = open('sample.txt').read()
matchobj = re.search(r'==start text==(.+)==end text==', text, re.DOTALL)
print(matchobj.group(1))

flags: re.IGNORECASE

re.I provides a case-insensitive match.

We can modify our matches with qualifiers called flags. The re.IGNORECASE flag will match any letters, whether upper or lowercase. In this example, extensions may be upper or lowercase - this file matcher doesn't care!

import re
dirlist = ('thisfile.jpg', 'thatfile.txt', 'otherfile.mpg', 'myfile.TXT')
for file in dirlist:
  if re.search(r'\.txt$', file, re.IGNORECASE):   #'.txt' or '.TXT'
    print(file)

The flag is passed as the third argument to search, and can also be passed to other re search methods.

Web Clients and Scraping

Basic HTTP

Headers, Cookie Headers and Response Codes

As we discussed in our CGI session, HTTP (HyperText Transport Protocol) is the protocol for sending and receiving messages from a browser (the client) to a web server (the server) and back again. We call the browser's message the request and the server's response the response. request A request consists of two parts: the URL (along with any parameters) and (optionally) any content. A request is generally one of two methods: GET (for retrieving data) or POST (for sending data, like form input). (Note in this context method is unrelated to a Python method.) HTTP Headers are meta information sent along with a request. This may include session (cookie) information. response The content of a response is the HTML, text or other data (can be binary data, or anything else a browser can send or receive). The response may also include headers. These include the response code, the size of the response, and may also contain cookies. The response code of a response indicates whether the request was handled without error (200), whether there was a server error (500), etc.

Python on the Client side: the requests module

requests is highly popular for web client functionality

A web client is any program that can send HTTP requests to a web server. Your browser is a web client. The requests module lets us issue HTTP requests in the same way a browser would. We can therefore have our Python program behave like a browser, i.e. visit web pages and download data. Requesting a web page through requests.get()

A page is usually requested as a GET request

import requests

response = requests.get('http://www.nytimes.com')

page_text =   response.text
status_code = response.status_code

page_text = page_text.encode('utf-8')      # if necessary

print('status code:  {}'.format(status_code))
print('======================= page text =======================')
print(page_text)

Decoding a JSON response

API calls are also made over HTTP, and if the call returns JSON, this can easily be decoded through requests.

import requests

response = requests.get('http://api.wunderground.com/api/d2e101aa48faa661/conditions/q/CA/San_Francisco.json')
conditions_json = response.json()

print(conditions_json["current_observation"]["temp_f"])        # 84.5 (accessing data within the JSON)

Requesting a web page with parameters

Many web requests include parameters specifying what content or action is desired. Here's a link to my homework application:

http://homework-davidbpython.rhcloud.com/route_view?assignment_id=1.1&student_id=bill_hanson

The parameters here are assignment_id (value 1.1) and student_id (value bill_hanson)

To pass parameters, we simply include a dict keyed to params.

param_dict = {'assignment_id': '1.1', 'student_id': 'bill_hanson'}
response = requests.get('http://homework-davidbpython.rhcloud.com/route_view', params=param_dict)

posting data to a web address, with parameter input

A form submission is usually sent as a POST request and includes parameter data. Here is a sample form:

<FORM ACTION="http://www.mywebsite.com/user" METHOD="POST">
  <INPUT NAME="firstname"><BR>
  <INPUT NAME="lastname"><BR>
  <INPUT NAME="password" TYPE="password">
  <INPUT TYPE="submit">
</FORM>

This form produces key/value data in the body of the request.

We can replicate this kind of request by using request.post() and passing a dict of param keys and values:

userdata = {"firstname": "John", "lastname": "Doe", "password": "jdoe123"}

resp = requests.post('http://www.mywebsite.com/user', params=userdata)

Some other features of requests: * International Domains and URLs * Keep-Alive & Connection Pooling * Sessions with Cookie Persistence * Browser-style SSL Verification * Basic/Digest Authentication * Elegant Key/Value Cookies * Automatic Decompression * Unicode Response Bodies * Multipart File Uploads * Connection Timeouts

Python as a web client: the urllib module

urllib is another module for making web requests.

Although the requests module is strongly favored by some for its simplicity, it has not yet been added to the Python builtin distribution.

The urlopen method takes a url and returns a file-like object that can be read() as a file:

import urllib.request
my_url = 'http://www.google.com'
readobj = urllib.request.urlopen(my_url)
text = readobj.read()
print(text)
readobj.close()

Alternatively, you can call readlines() on the object (keep in mind that many objects that can deliver file-like string output can be read with this same-named method:

for line in readobj.readlines():
  print(line)
readobj.close()

The text that is downloaded is CSV, HTML, Javascript, and possibly other kinds of data. TypeError: can't use a string pattern on a bytes-like object This error may occur with some websites. It indicates that an undecoded unicode response was received.

The response usually comes to us as a special object called a byte string. In order to work with the response as a string, we may need to use the decode() method:

text = text.decode('utf-8')

SSL Certificate Error Many websites enable SSL security and require a web request to accept and validate an SSL certificate (certifying the identity of the server). urllib by default requires SSL certificate security, but it can be bypassed (keep in mind that this may be a security risk).

import ssl
ctx = ssl.create_default_context()
ctx.check_hostname = False
ctx.verify_mode = ssl.CERT_NONE

my_url = 'http://www.nytimes.com'
readobj = urllib.request.urlopen(my_url, context=ctx)

Download binary files: images and other files can be saved locally using urllib.requests.urlretrieve().

import urllib.request

urllib.requests.urlretrieve('http://www.azquotes.com/picture-quotes/quote-python-is-an-experiment-in-how-much-freedom-programmers-need-too-much-freedom-and-nobody-guido-van-rossum-133-51-31.jpg', 'guido.jpg')

Note the two arguments to urlretrieve(): the first is a URL to an image, and the second is a filename -- this file will be saved locally under that name.

Encoding Parameters: urllib.requests.urlencode()

When including parameters in our requests, we must encode them into our request URL. The urlencode() method does this nicely:

import urllib.request, urllib.parse

params = urllib.parse.urlencode({'choice1': 'spam and eggs', 'choice2': 'spam, spam, bacon and spam'})
print("encoded query string: ", params)
f = urllib.request.urlopen("http://www.google.com?{}".format(params))
print(f.read())

this prints:

encoded query string: choice1=spam+and+eggs&choice2=spam%2C+spam%2C+bacon+and+spam

choice1:  spam and eggs<BR>
choice2:  spam, spam, bacon and spam<BR>

Web Scraping with Beautiful Soup (bs4)

Beautiful Soup parses XML or HTML documents, making text and attribute extraction a snap.

Here we are passing the text of a web page (obtained by requests) to the BS parser:

from bs4 import BeautifulSoup
import requests

response = requests.get('http://www.nytimes.com')

soup = BeautifulSoup(response.text, 'html.parser')

# show HTML in "pretty" form
print(soup.prettify())

# show all plain text in a page
print(soup.get_text())

The result is a BeautifulSoup object which we can use to search for tags and data.

For the following examples, let's use the HTML provided on the Beatiful Soup Quick Start page:

<!doctype html>
<html>
  <head>
    <title>The Dormouse's story</title>
  </head>
  <body>
    <p class="story_title"><b>The Dormouse's story</b></p>

    <p class="story">Once upon a time there were three little sisters; and their names were
        <a href="http://example.com/elsie" class="sister eldest" id="link1">Elsie</a>,
        <a href="http://example.com/lacie" class="sister" id="link2">Lacie</a> and
        <a href="http://example.com/tillie" class="sister" id="link3">Tillie</a>;
    and they lived at the bottom of a well.</p>

    <p class="story">They were happy, and eventually died.  The End.</p>
  </body>
</html>

bs4: finding a tag with attributes, find() and find_all()

Finding the first tag by name using soup.attribute

The BeautifulSoup object's attributes can be used to search for a tag. The first tag with a name will be returned.

# first (and only) <title> tag
print(soup.title)              # <title>The Dormouse's story</title>

# first (of several) <p> tags
print(soup.p)                  # <p class="title"><b>The Dormouse's story</b></p>

Attributes can be chained to drill down to a particular tag:

print(soup.body.p.b)           # <b>The Dormouse's story</b>

However keep in mind these represent the first of each tag listed.

Finding the first tag by name: find()

find() works similarly to an attribute, but filters can be applied (discussed shortly).

print soup.find('a')          <a class="sister eldest" href="http://example.com/elsie" id="link1">Elsie</a>

Finding all tags by name: find_all()

findall() retrieves a list of all tags with a particular name.

tags = soup.find_all('a')

bs4: finding a tag using varying criteria

Tag criteria can focus on a tag's name, its attributes, or text within the tag.

SEARCHING NAME, ATTRIBUTE OR TEXT Finding a tag by name

Links in a page are marked with the <A> tag (usually seen as <A HREF="">). This call pulls out all links from a page:

link_tags = soup.find_all('a')

Finding a tag by tag attribute and/or name and tag attribute

# all <a> tags with an 'id' attribute of link1
link1_a_tags = soup.find_all('a', id="link1")

# all tags (of any name) with an 'id' attribute of link1
link1_tags = soup.find_all(id="link1")

"multi-value" tag attribute

CSS allows multiple values in an attribute:

<a href="http://example.com/elsie" class="sister eldest" id="link1">Elsie</a>

If we'd like to find a tag through this value, we pass a list:

link1_elsie_tag = soup.find(class_=['sister', 'eldest'])

Finding a tag by string within the tag's text

All <a> tags containing text 'Dormouse'

elsie_tags = soup.find_all('a', text='Dormouse')

FILTER TYPES: STRING, LIST, REGEXP, FUNCTION string: filter on the tag's name

tags = soup.find_all('a')          # return a list of all <a> tags

list: filter on tag names

tags = soup.find_all(['a', 'b'])   # return a list of all <a> or <b> tags

regexp: filter on pattern match against name

import re
tags = soup.find_all(re.compile('^b'))      # a list of all tags whose names start with 'b'

re.compile() produces a pattern object that is applied to tag names using re.match()

function: filter if function returns True

soup.find_all(lambda tag: tag.name == 'a' and 'mysite.com' in tag.get('href'))

bs4: the Tag object

Tags' attributes and contents can be read; they can also be queried for tags and text within

body_text = """
    <BODY class="someclass otherclass">
        <H1 id='mytitle'<This is a headings</H1>
        <A href="mysite.com"<This is a link</A>
    </BODY>
"""

An HTML tag has four types of data: 1. The tag's name ('BODY' 'H1' or 'A') 2. The tag's attributes (<BODY class=, H1 id= or <A href=) 3. The tag's text ('This is a header' or 'This is a link') 4. The tag's contents (i.e., tags within it -- for <BODY>, the <H1> and <A> tags)

from bs4 import BeautifulSoup
soup = BeautifulSoup(body_text, 'html.parser')

h1 = soup.body.h1        # h1 is a Tag object
print(h1.name)            # u'h1'
print(h1.get('id'))       # u'mytitle'
print(h1.attrs)           # {u'id': u'mytitle'}
print(h1.text)            # u'This is a heading'

body = soup.body         # body is a Tag object
print(body.name)          # u'body'
print(body.get('class'))  # ['someclass', 'otherclass']
print(body.attrs)         # {'class': ['someclass', 'otherclass']}
print(body.text)          # u'\nThis is a heading\nThis is a link\n'

A tag's child tags can be searched the same as the BeautifulSoup object

body = soup.body         # find the <body> tag in this document

atag = body.find('a')    # find first <a> tag in this <body> tag

Sidebar: Sending email with smtplib

Sending mail is simple when you have an SMTP server running and available on your host computer. Python's smtplib module makes this easy:

#!/usr/bin/env python

# Import smtplib for the actual sending function
import smtplib

# Import the email modules we'll need
from email.mime.text import MIMEText

# Create a text/plain message formatted for email
msg = MIMEText('Hello, email.')

from_address = 'dbb212@nyu.edu'
to_address = 'david.beddoe@gmail.com'
subject = 'Test message from a Python script'

msg['Subject'] = subject
msg['From'] =    from_address
msg['To'] =      to_address

s = smtplib.SMTP('localhost')
s.sendmail(from_address, [to_address], msg.as_string())
s.quit()

Flask

Web Frameworks

Introduction

A "web framework" is an application or package that facilitates web programming. Server-side apps (for example: a catalog, content search and display, reservation site or most other interactive websites) use a framework to handle the details of the web network request, page display and database input/output -- while freeing the programmer to supply just the logic of how the app will work. Full Stack Web Frameworks A web application consists of layers of components that are configured to work together (i.e., built into a "stack"). Such components may include: * authenticating and identifying users through cookies * handling data input from forms and URLs * reading and writing data to and from persistant storage (e.g. databases) * displaying templates with dynamic data inserted * providing styling to templates (e.g., with css) * providing dynamic web page functionality, as with AJAX The term "full stack developer" used by schools and recruiters refers to a developer who is proficient in all of these areas. Django is possibly the most popular web framework. This session would probably focus on Django, but its configuration and setup requirements are too lengthy for the available time. "Lightweight" Web Frameworks A "lightweight" framework provides the base functionality needed for a server-side application, but allows the user to add other stack components as desired. Such apps are typically easier to get started with because they require less configuration and setup. Flask is a popular lightweight framework with many convenient defaults allows us to get our web application started quickly.

The Flask app object, the app.run() method and @app.route dispatch decorators

"@app.route()" functions describe what happens when the user visits a particular "page" or URL shown in the decorator.

hello_flask.py

Here is a basic template for a Flask app.

#!/usr/bin/env python

import flask
app = flask.Flask(__name__)           # a Flask object

@app.route('/hello')                 # called when visiting web URL 127.0.0.1:5000/hello/
def hello_world():
    print('*** DEBUG:  inside hello_world() ***')
    return '<PRE>Hello, World!</PRE>'            # expected to return a string (usu. the HTML to display)

if __name__ == '__main__':
    app.run(debug=True, port=5000)    # app starts serving in debug mode on port 5000

The first two lines and last two lines will always be present (production apps will omit the debug= and port= arguments). app is an object returned by Flask; we will use it for almost everything our app does. We call it a "god object" because it's always available and contains most of what we need. app.run() starts the Flask application server and causes Flask to wait for new web requests (i.e., when a browser visit the server). @app.route() functions are called when a particular URL is requested. The decorator specifies the string to be found at the end of the URL. For example, the above decorator @app.route('/hello') specifies that the URL to reach the function should be http://localhost:5000/hello/ The string returned from the function is passed on to Flask and then to the browser on the other end of the web request.

Running a Flask app locally

Flask comes complete with its own self-contained app server. We can simply run the app and it begins serving locally. No internet connection is required.

$ python hello_flask.py
 * Running on http://127.0.0.1:5000/ (Press CTRL+C to quit)
 * Restarting with stat
127.0.0.1 - - [13/Nov/2016 15:58:16] "GET / HTTP/1.1" 200 -
*** DEBUG:  inside hello_world() ***

The Flask app prints out web server log messages showing the URL requested by each visitor. You can also print error messages directly from the application, and they will appear in the log (these were printed with *** strings, for visibility.) Changes to Flask code are detected and cause Flask to restart the server; errors cause it to exit

Whenever you make a change and save your script, the Flask server will restart -- you can see it issue messages to this effect:

 * Detected change in '/Users/dblaikie/Dropbox/tech/nyu/advanced_python/solutions/flask/guestbook/guestbook_simple.py', reloading
 * Restarting with stat
 * Debugger is active!
 * Debugger pin code: 161-369-356

If there is an error in your code that prevents it from running, the code will raise an exception and exit.

 * Detected change in '/Users/dblaikie/Dropbox/tech/nyu/advanced_python/solutions/flask/guestbook/guestbook_simple.py', reloading
 * Restarting with stat
  File "./guestbook_simple.py", line 17
    return hello, guestbook_id ' + guestbook_id
                                              ^
SyntaxError: EOL while scanning string literal

At that point, the browser will simply report "This site can't be reached" with no other information.

Therefore we must keep an eye on the window to see if the latest change broke the script -- fix the error in the script, and then re-run the script in the Terminal.

Redirection and Event Flow

Once an 'event' function is called, it may return a string, call another function, or redirect to another page.

Return a plain string

A plain string will simply be displayed in the browser. This is the simplest way to display text in a browser.

@app.route('/hello')
def hello_world():
    return '<PRE>Hello, World!</PRE>'            # expected to return a string (usu. the HTML to display)

Return HTML (also a string)

HTML is simply tagged text, so it is also returned as a string.

@app.route('/hello_template')
def hello_html():
    return """
<HTML>
  <HEAD>
    <TITLE>My Greeting Page</TITLE>
  </HEAD>
  <BODY>
    <H1>Hello, world!</H1>
  </BODY>
</HTML>"""

Return an HTML Template (to come)

This method also returns a string, but it is returned from the render_template() function.

@app.route('/hello_html')
def hello_html():
    return flask.render_template('response.html')    # found in templates/response.html

Return another function call

Functions that aren't intended to return strings but to perform other actions (such as making database changes) can simply call other functions that represent the desired destination:

def hello_html():
    return """
<HTML>
  <HEAD>
    <TITLE>My Greeting Page</TITLE>
  </HEAD>
  <BODY>
    <H1>Hello, world (from another function)  !</H1>
  </BODY>
</HTML>"""

@app.route('/hello_template')
def hello():
    return hello_html()

Because hello() is calling hello_html() in its return statement, whatever is returned from there will be returned from hello().

Redirecting to another program URL with flask.redirect() and flask.url_for()

At the end of a function we can call the flask app again through a page redirect -- that is, to have the app call itself with new parameters.

from datetime import date

@app.route('/sunday_hello')
def sunday_hello():
    return "It's Sunday!  Take a rest!"

@app.route('/shello')
def hello():
    if date.today().strftime('%a') == 'Sun':
        return flask.redirect(flask.url_for('sunday_hello'))
    else:
        return 'Hello, workday (or Saturday)!'

redirect() issues a redirection to a specified URL; this can be http://www.google.com or any desired URL.

url_for() simply produces the URL that will call the flask app with /login.

Building a Multi-Page Application

Use flask.url_for() to build links to other apps

This app has three pages; we can build a "closed system" of pages by having each page link to another within the site.

happy_image = 'http://davidbpython.com/advanced_python/python_data/happup.jpg'
sad_image = 'http://davidbpython.com/advanced_python/python_data/sadpup.jpg'

@app.route('/question')
def ask_question():
    return """
<HTML>
  <HEAD><TITLE>Do you like puppies?</TITLE></HEAD>
  <BODY>
      <H3>Do you like puppies?</H3>
      <A HREF="{}">arf!</A><BR>
      <A HREF="{}">I prefer cats...</A>
  </BODY>
</HTML>""".format(flask.url_for('yes'), flask.url_for('no'))


@app.route('/yes')
def yes():
    return """
<HTML>
  <HEAD><TITLE>C'mere Boy!</TITLE></HEAD>
  <BODY>
      <H3>C'mere, Boy!</H3>
      <IMG SRC="{}"><BR>
      <BR>
      Change your mind?  <A HREF="{}">Let's try again.</A>
  </BODY>
</HTML>""".format(happy_image, flask.url_for('ask_question'))


@app.route('/no')
def no():
    return """
<HTML>
  <HEAD><TITLE>Aww...</TITLE></HEAD>
  <BODY>
      <H3>Aww...really?</H3>
      <IMG SRC="{}"><BR>
      <BR>
      Change your mind?  <A HREF="{}">Let's try again.</A>
  </BODY>
</HTML>""".format(sad_image, flask.url_for('ask_question'))

Simple Templating

Use {{ varname }} to create template tokens and flask.render_template() to insert to them.

HTML pages are rarely written into Flask apps; instead, we use standalone template files. The template files are located in a templates directory placed in the same directory as your Flask script.

question.html

<HTML>
  <HEAD><TITLE>Do you like puppies?</TITLE></HEAD>
  <BODY>
      <H3>Do you like puppies?</H3>
      <A HREF="{{ yes_link }}">arf!</A><BR>
      <A HREF="{{ no_link }}">I prefer cats...</A>
  </BODY>
</HTML>

puppy.html

<HTML>
  <HEAD><TITLE><{{ title_message }}</TITLE></HEAD>
  <BODY>
      <H3>{{ title_message }}</H3>
      <IMG SRC="{{ puppy_image }}"><BR>
      <BR>
      Change your mind?  <A HREF="{{ question_link }}">Let's try again.</A>
  </BODY>
</HTML>

puppy_question.py

happy_image = 'http://davidbpython.com/advanced_python/python_data/happup.jpg'
sad_image =   'http://davidbpython.com/advanced_python/python_data/sadpup.jpg'


@app.route('/question')
def ask_question():
    return flask.render_template('question.html',
                                  yes_link=flask.url_for('yes'),
                                  no_link=flask.url_for('no'))

@app.route('/yes')
def yes():
    return flask.render_template('puppy.html',
                                  puppy_image=happy_image,
                                  question_link=flask.url_for('ask_question'),
                                  title_message='Cmere, boy!')

@app.route('/no')
def no():
    return flask.render_template('puppy.html',
                                  puppy_image=sad_image,
                                  question_link=flask.url_for('ask_question'),
                                  title_message='Aww... really?')

Embedding Python Code in Templates

Use {% %} to embed Python code for looping, conditionals and some functions/methods from within the template.

Template document: "template_test.html"

<!DOCTYPE html>
<html lang="en">
  <head>
    <title>Important Stuff</title>
  </head>
  <body>

  <h1>Important Stuff</h1>

  Today's magic number is {{ number }}<br><br>

  Today's strident word is {{ word.upper() }}<br><br>

  Today's important topics are:<br>
  {% for item in mylist %}
      {{ item }}<br>
  {% endfor %}
  <br><br>

  {% if reliability_warning %}
      WARNING:  this information is not reliable
  {% endif %}

  </body>
</html>

Flask code

@app.route('template_test')
def template_test():
    return flask.render_template('template_test.html', number=1035,
                                                       word='somnolent',
                                                       mylist=['children', 'animals', 'bacteria'],
                                                       reliability_warning=True)

As before, {{ variable }} can used for variable insertions, as well as instance attributes, method and function calls
{% for this in that %} can be used for 'if' tests, looping with 'for' and other basic control flow

Reading args from URL or Form Input

Input from a page can come from a link URL, or from a form submission.

name_question.html

<HTML>
  <HEAD>
  </HEAD>
  <BODY>
    What is your name?<BR>
    <FORM ACTION="{{ url_for('greet_name') }}" METHOD="post">
      <INPUT NAME="name" SIZE="20">
      <A HREF="{{ url_for('greet_name') }}?no_name=1">I don't have a name</A>
      <INPUT TYPE="submit" VALUE="tell me!">
    </FORM>
  </BODY>
</HTML>

flaskapp.py

@app.route('/name_question')
def ask_name():
    return flask.render_template('name_question.html')

@app.route('/greet', methods=['POST', 'GET'])
def greet_name():
    name =    flask.request.form.get('name')     # from a POST (form with 'method="POST"')
    no_name = flask.request.args.get('no_name')  # from a GET (URL)

    if name:
        msg = 'Hello, {}!'.format(name)
    elif no_name:
        msg = 'You are anonymous.  I respect that.'
    else:
        raise ValueError('\nraised error:  no "name" or "no_name" params passed in request')

    return '<PRE>{}</PRE>'.format(msg)

Base templates

Many times we want to apply the same HTML formatting to a group of templates -- for example the <head> tag, which my include css formatting, javascript, etc.

We can do this with base templates:

{% extends "base.html" %}         # 'base.html' can contain HTML from another template
    <h1>Special Stuff</h1>
    Here is some special stuff from the world of news.

The base template "surrounds" any template that imports it, inserting the importing template at the {% block body%} tag:

<html>
  <head>
  </head>
  <body>
    <div class="container">
      {% block body %}
         <H1>This is the base template default body.</H1>
      {% endblock %}
    </div>
  </body>
</html>

There are many other features of Jinja2 as well as ways to control the API, although I have found the above features to be adequate for my purposes.

Sessions

Sessions (usually supported by cookies) allow Flask to identify a user between requests (which are by nature "anonymous").

When a session is set, a cookie with a specific ID is passed from the server to the browser, which then returns the cookie on the next visit to the server. In this way the browser is constantly re-identifying itself through the ID on the cookie. This is how most websites keep track of a user's visits.

flask_session.py

import flask
app = flask.Flask(__name__)

app.secret_key = 'A0Zr98j/3yX R~XHH!jmN]LWX/,?RT'  # secret key

@app.route('/index')
def hello_world():

    # see if the 'login' link was clicked:  set a session ID
    user_id = flask.request.args.get('login')
    if user_id:
        flask.session['user_id'] = user_id
        is_session = True

    # else see if the 'logout' link was clicked:  clear the session
    elif flask.request.args.get('logout'):
        flask.session.clear()

    # else see if there is already a session cookie being passed:  retrieve the ID
    else:
        # see if a session cookie is already active between requests
        user_id = flask.session.get('user_id')

    # tell the template whether we're logged in (user_id is a numeric ID, or None)
    return flask.render_template('session_test.html', is_session=user_id)


if __name__ == '__main__':
    app.run(debug=True, port=5001)    # app starts serving in debug mode on port 5001

session_test.html

<!DOCTYPE html>
<html lang="en">
  <head>
    <title>Session Test</title>
  </head>
  <body>

  <h1>Session Test</h1>

  {% if is_session %}
  <font color="green">Logged In</font>
  {% else %}
  <font color="red">Logged Out</font>
  {% endif %}

  <br><br>

  <a href="index?login=True">Log In</a><br>
  <a href="index?logout=True">Log Out</a><br>

  </body>
</html>

Config Values

Configuration values are set to control how Flask works as well as to be set and referenced by an individual application.

Flask sets a number of variables for its own behavior, among them DEBUG=True to display errors to the browser, and SECRET_KEY='!jmNZ3yX R~XWX/r]LA098j/,?RTHH' to set a session cookie's secret key. A list of Flask default configuration values is here. Retrieving config values

value = app.config['SERVER_NAME']

Setting config values individually

app.config['DEBUG'] = True

Setting config values from a file

app.config.from_pyfile('flaskapp.cfg')

Such a file need only contain python code that sets uppercased constants -- these will be added to the config. Setting config values from a configuration Object Similarly, the class variables defined within a custom class can be read and applied to the config with app.config.from_object(). Note in the example below that we can use inheritance to distribute configs among several classes, which can aid in organization and/or selection:

In a file called configmodule.py:

class Config(object):
    DEBUG = False
    TESTING = False
    DATABASE_URI = 'sqlite://:memory:'

class ProductionConfig(Config):
    DATABASE_URI = 'mysql://user@localhost/foo'

class DevelopmentConfig(Config):
    DEBUG = True

class TestingConfig(Config):
    TESTING = True

In the flask script:

app.config.from_object('configmodule.ProductionConfig')

Environment Variables

Environment Variables are system-wide values that are set by the operating system and apply to all applications. They can also be set by individual applications.

The OpenShift web container sets a number of environment variables, among them OPENSHIFT_LOG_DIR for log files and OPENSHIFT_DATA_DIR for data files. Flask employs jinja2 templates. A list of Openshift environment variables can be found here.

Flask and security

An important caveat regarding web security: Flask is not considered to be a secure approach to handling sensitive data.

...at least, that was the opinion of a former student, a web programmer who worked for Bank of America, about a year ago -- they evaluated Flask and decided that it was not reliable and could have security vulnerabilities. His team decided to use CGI -- the baseline protocol for handling web requests. Any framework is likely to have vulnerabilities -- only careful research and/or advice of a professional can ensure reliable privacy. However for most applications security is not a concern -- you will simply want to avoid storing sensitive data on a server without considering security.

Flask-Specific Errors

Keep these Flask-specific errors in mind.

Page not found

URLs specified in @app.route() functions should not end in a trailing slash, and URLs entered into browsers must match

@app.route(/'hello')

If you omit a trailing slash from this path but include one in the URL, the browser will respond that it can't find the page.

What's worse, some browsers sometimes try to 'correct' your URL entries, so if you type a URL with a trailing slash and get "Page not found", the next time you type it differently (even if correctly) the browser may attempt to "correct" it to the way you typed it the first time (i.e., incorrectly). This can be extremely frustrating; the only remedy I have found is to clear the browser's browsing data.

Functions must return strings (or redirect to another function or URL); they do not print page responses.

@app.route(/'hello')
def hello():
    return 'Hello, world!'        # not print 'Hello, world!'

Each routing function expects a string to be returned -- so the function must do one of these:

   1) return a string (this string will be displayed in the browser)    2) call another @app.route() function that will return a string    3) issue a URL redirect (described later) Method not allowed usually means that a form was submitting that specifies method="POST" but the @app.route decorator doesn't specify methods=['POST']. See "Reading args from URL or Form Input", above.

name_question.html

<HTML>
  <HEAD>
  </HEAD>
  <BODY>
    What is your name?<BR>
    <FORM ACTION="{{ url_for('greet_name') }}" METHOD="post">
      <INPUT NAME="name" SIZE="20">
      <A HREF="{{ url_for('greet_name') }}?no_name=1">I don't have a name</A>
      <INPUT TYPE="submit" VALUE="tell me!">
    </FORM>
  </BODY>
</HTML>

If the form above submits data as a "post", the app.route() function would need to specify this as well:

@app.route('/greet', methods=['POST', 'GET'])
def greet_name():
   ...

Jinja2 Templating

Jinja2: Getting Started

Jinja2 is a module for inserting data into templates.

Last week we used the simple .format() method to insert "dynamic" data (that is, variables that may be any value) into "static" templates (text that does not change). Jinja2 offers a full-featured templating system, including the following features:

• Variable insertion (like .format())
• Attribute access (like .format())
• Container subscripting (mylist[0], mydict['thiskey'], etc.)
• Method calls
• if/elif/else conditionals
• for looping over containers or iterable objects

Here is a basic example showing these features:

test.html, stored the test_templates/ directory

Hello, {{ name }}.

Please say it loudly:  {{ compliment.upper() }}!

Must I tell you:
{% for item in pronouncements %}
   {{ item }}
{% endfor %}

Or {{ pronouncements[0] }}?

{% if not reconciled: %}
  We have work left to do.
{% else %}
  I'm glad we worked that out.
{% endif %}

test.py, in the same dir as test_templates

import jinja2

env = jinja2.Environment()
env.loader = jinja2.FileSystemLoader('test_templates')

template = env.get_template('test.html')

print(template.render(name='Joe', compliment="you're great",
                      pronouncements=['over', 'over', 'over again'],
                      reconciled=False))

The rendered template!

Hello, Joe.

Please say it loudly:  YOU'RE GREAT!

Must I tell you:

   over
   over
   and over again

Or over?


  We have work left to do.

pandas: Introduction

pandas: Introduction

pandas is a Python module used for manipulation and analysis of tabular data. * Excel-like numeric calculations, particularly column-wise and row-wise calculations (vectorization) * SQL-like merging, grouping and aggregating * visualizing (line chart, bar chart, etc.) * emphasis on: - aligning data from multiple sources - "slicing and dicing" by rows and columns - concatenating and joining - cleaning and normalizing missing or incorrect data - working with time series - categorizing * ability to read and write to CSV, XML, Excel, database queries, etc. numpy is a data analysis library upon which pandas is built. We sometimes make direct calls to numpy - some of its variables (such as np.nan), variable-generating functions (such as np.arange or np.linlist) and some processing functions.

pandas Reference

Use the docs for an ongoing study of pandas' rich feature set.

pandas official documentation

full docs (HTML, pdf)

http://pandas.pydata.org/pandas-docs/stable
http://pandas.pydata.org/pandas-docs/version/0.19.0/pandas.pdf

"10 minutes to pandas"

https://pandas.pydata.org/pandas-docs/stable/10min.html

pandas cookbook

http://pandas.pydata.org/pandas-docs/stable/cookbook.html

matplotlib official documentation

http://matplotlib.org/api/pyplot_api.html

Tutorials and Learning Aids

Online resources are many; when you find one you like, stick with it

pandas textbook "Python for Data Analysis" by Wes McKinney

http://astronomi.erciyes.edu.tr/wp-content/uploads/astronom/pdf/OReilly%20Python%20for%20Data%20Analysis.pdf

(If the above link goes stale, simply search Python for Data Analysis pdf.)

The Second Edition is available from O'Reilly on Safari Bookshelf:

http://shop.oreilly.com/product/0636920050896.do

Please keep in mind that pandas is in active development, which means that features may be added, removed and changed (latest version: 0.25.2)

blog tutorials

These often provide the kind of "insider view" that is most helpful when getting oriented.

Tom Augspurger blog (6-part series)

http://tomaugspurger.github.io/modern-1.html

Greg Reda blog (3-part series)

http://gregreda.com/2013/10/26/intro-to-pandas-data-structures/

cheat sheet (Treehouse)

https://s3.amazonaws.com/assets.datacamp.com/blog_assets/PandasPythonForDataScience.pdf

Inspecting attributes and docs on pandas and pandas objects

An object type that is new to us can be explored through attribute inspection -- we can list the object's attributes with dir() and see brief documentation on an attribute with help().

import pandas as pd

# list of pandas attributes (global vars)
print(dir(pd))

# list of pandas functions (filtering for <class 'function'> only)
import types
print([ attr for attr in dir(pd) if isinstance(getattr(pd, attr), types.FunctionType) ])

# short doc on read_csv() function
help(pd.read_csv)       # help on the read_csv function of pandas


# list of Series attributes
s1 = pd.Series()
print((dir(s1)))

# list of DataFrame attributes
df = pd.DataFrame()
print(dir(df))

# list of DataFrame methods (filtering for <class 'method'> only)
print([ attr for attr in dir(df)
        if isinstance(getattr(df, attr), types.MethodType) ])

# short doc on DataFrame join() method
help(df.join)           # help on the join() method of a DataFrame

pandas Object Types

DataFrame: rows and columns; Series: a single column or single row; Index: column or row labels.

Series * an ordered sequence of values indexed by labels * items addressable by index label (dict-like) * items addressable by sequence ineger (list-like) * has a dtype attribute that holds its objects' common type DataFrame: * is the core pandas structure -- a 2-dimensional array * is like a "dict of dicts" in that columns and rows can be indexed by label * is like a "list of lists" in that columns and rows can be indexed by integer index * is also like an Excel spreadsheet - rows, columns, and row and column labels Index * an object that provides indexing for both the Series (its item index) and the DataFrame (its column and row indices).

The Series: Ordered Sequence of Values Indexed by Label

Like a list, but with item labels... so like a dict too.

• ordered sequence with integer position of each item (list-like)
• index attribute labels each item (dict-like)
• name attribute

Initialize a single series

import pandas as pd

s1 = pd.Series([10, 20, 30], index=['r1', 'r2', 'r3'], name='a')

print(s1)
            # r1    10
            # r2    20
            # r3    30
            # Name: b, dtype: int64

s2 = pd.Series(['x', 'y', 'z'], index=['r1', 'r2', 'r3'], name='b')


# combine Series to make a DataFrame
df = pd.DataFrame(s1, s2)

The DataFrame

The DataFrame is the pandas workhorse structure. It is a 2-dimensional structure with columns and rows (i.e., a lot like a spreadsheet).

Initializing

import pandas as pd

# initialize a new, empty DataFrame
df = pd.DataFrame()


# init with dict of lists (keyed to columns) and index
df = pd.DataFrame( {'a': [1, 2, 3],
                    'b': [1.0, 1.5, 2.0],
                    'c': ['a', 'b', 'c']  },
                    index=['r1', 'r2', 'r3'] )

print(df)
            #     a    b  c
            # r1  1  1.0  a
            # r2  2  1.5  b
            # r3  3  2.0  c

previewing the DataFrame

print(len(df))     # 3 (# of rows)

print(df.head(2))  # 1st 2 rows

print(df.tail(2))  # last 2 rows

column attribute / subscripting: delivers a Series

sa = df.a         # or df['a']

print(sa)          # pandas.Series([1, 2, 3], index=['r1', 'r2', 'r3'], name='a')

label and positional access

print(df.loc['r2', 'b'])    # 1.5
print(df.iloc[1, 1])        # 1.5

.columns and .index attributes

Columns and rows can be accessed through the DataFrame's attributes:

print(df.columns)        # Index(['a', 'b', 'c'], dtype='object')
print(df.index)          # Index(['r1', 'r2', 'r3'], dtype='object')

DataFrame as a list of Series objects Again, any DataFrame's columns or rows can be sliced out as a Series:

# read a column as a Series (use DataFrame subscript)
bcol = df['b']
print(bcol)
            # r1    1.0
            # r2    1.5
            # r3    2.0
            # Name: b, dtype: float64



# read a row as a Series (use subscript of df.loc[])
oneidx = df.loc['r2']
print(oneidx)
            # a      2
            # b    1.5
            # c      b
            # Name: r2, dtype: object

Note: df is a common variable name for pandas DataFrame objects; you will see this name used frequently in these examples.

The Index: DataFrame Column or Index Labels

An Index object is used to specify a DataFrame's columns or index, or a Series index.

Columns and Indices

A DataFrame makes use of two Index objects: one to represent the columns, and one to represent the rows.

df = pd.DataFrame( {'a': [1, 2, 3, 4],
                    'b': [1.0, 1.5, 2.0, 2.5],
                    'c': ['a', 'b', 'c', 'd'],
                    'd': [100, 200, 300, 400] },
                    index=['r1', 'r2', 'r3', 'r4'] )

print((df.index))    # Index(['r1', 'r2', 'r3', 'r4'], dtype=object)

print((df.columns))  # Index(['a', 'b', 'c', 'd'], dtype=object)

# set name for index and columns
df.index.name = 'year'
df.columns.name = 'state'

s_index = s1.index     # Index(['r1', 'r2', 'r3'])
columns = df.columns   # Index(['a',  'b',  'c'],  dtype='object')
idx = df.index         # Index(['r1', 'r2', 'r3'], dtype='object')

There are a number of "exotic" Index object types as well - all come from Index and behave in the same ways: Index (standard, default and most common Index type) RangeIndex (index built from an integer range) Int64Index, UInt64Index, Float64Index (index values of specific types) DatetimeIndex, TimedeltaIndex, PeriodIndex, IntervalIndex (datetime-related indices) CategoricalIndex (index related to the Categorical type)

pandas Objects are Like Python Objects

DataFrames behave as you might expect when converted to any Python container

df = pd.DataFrame( {'a': [1, 2, 3, 4],
                    'b': [1.0, 1.5, 2.0, 2.5],
                    'c': ['a', 'b', 'b', 'a'] }, index=['r1', 'r2', 'r3', 'r4'] )


print(len(df))             # 4

print(len(df.columns))     # 3

print(max(df['a']))        # 4

print(list(df['a']))       # [1, 2, 3, 4]     (column for 'a')

print(list(df.loc['r2']))  # [2, 1.5, 'b']   (row for 'r2')

print(set(df['c']))        # {'b', 'a'}       (a set of unique values)

DataFrame .values -- convert to a list of numpy arrays

An numpy array is a list-like object. simple list comprehension could convert these to a list of lists:

print((df.values))

    # array([[1, 1.0, 'a'],
    #        [2, 1.5, 'b'],
    #        [3, 2.0, 'b'],
    #        [4, 2.5, 'a']], dtype=object)


lol = list( [ list(item) for item in df.values ])

print(lol)

                          # [ [1, 1.0, 'a'],
                          #   [2, 1.5, 'b'],
                          #   [3, 2.0, 'b'],
                          #   [4, 2.5, 'a'] ]

looping - loops through columns

for colname in df:
    print('{}:  {}'.format(colname, type(df[colname])))

                          # a:  <pandas.core.series.Series>
                          # b:  <pandas.core.series.Series>
                          # c:  <pandas.core.series.Series>


# looping with iterrows -- loops through rows
for row_index, row_series in df.iterrows():
    print('{}:  {}'.format(row_index, type(row_series)))

                          # r1:  <pandas.core.series.Series>
                          # r2:  <pandas.core.series.Series>
                          # r3:  <pandas.core.series.Series>
                          # r4:  <pandas.core.series.Series>

Although keep in mind that we generally prefer vectorized operations across columns or rows to looping (discussed later).

DataFrame: Initializing from Data Source

DataFrame can be read from CSV, JSON, Excel and XML formats.

CSV

# read from file
df = pd.read_csv('quarterly_revenue_2017Q4.csv')


# write to file
wfh = open('output.csv', 'w')
df.to_csv(wfh, na_rep='NULL')


# reading from Fama-French file (the abbreviated file, no header)
# sep= indicates the delimiter on which to split() the fields
# names= indicates the column heads
df = pd.read_csv('FF_abbreviated.txt', sep='\s+',
                                       names=['date', 'MktRF', 'SMB', 'HML', 'RF'])


# reading from Fama-French non-abbreviated (the main file including headers and footers)
# skiprows=5:  start reading 5 rows down
df = pd.read_csv('F-F_Research_Data_Factors_daily.txt', skiprows=5, sep='\s+',
                                                        names=['date', 'MktRF', 'SMB', 'HML', 'RF'])

df.to_csv('newfile.csv')

Excel

# reading from excel file to DataFrame
df = pd.read_excel('revenue.xlsx', sheet_name='Sheet1')

# optional:  produce a 'reader' object used to obtain sheet names, etc.
xls_file = pd.ExcelFile('data.xls')    # produce a file 'reader' object
df = xls_file.parse('Sheet1')          # parse a selected sheet


# write to excel
df.to_excel('data2.xls', sheet_name='Sheet1')

JSON

# sample df for demo purposes
df = pd.DataFrame( {'a': [1, 2, 3, 4],
                    'b': [1.0, 1.5, 2.0, 2.5],
                    'c': ['a', 'b', 'c', 'd'] }, index=['r1', 'r2', 'r3', 'r4'] )



# write dataframe to JSON
pd.json.dump(df, open('df.json', 'w'))

mydict = pd.json.load(open('df.json'))
new_df = pd.DataFrame(mydict)

Relational Database

import sqlite3                        # file-based database format
conn = sqlite3.connect('example.db')  # a db connection object

df = pd.read_sql('SELECT this FROM that', conn)

The above can be used with any database connection (MySQL, Oracle, etc.)

From Clipboard: this option is excellent for cutting and pasting data from websites

df = pd.read_clipboard(skiprows=5, sep='\s+',
                       names=['date', 'MktRF', 'SMB', 'HML', 'RF'])

Dataframe and Series dtypes

pandas infers a column type based on values and applies it to the column automatically.

Pandas is built on top of numpy, a numeric processing module, compiled in C for efficiency. Unlike core Python containers (but similar to a database table), numpy cares about object type. Wherever possible, numpy will assign a type to a column of values and attempt to maintain the type's integrity. This is done for the same reason it is done with database tables: speed and space efficiency. In the below DataFrame, numpy/pandas "sniffs out" the type of a column Series. It will set the type most appropriate to the values.

import pandas as pd

df = pd.DataFrame( {'a': [1, 2, 3, 4],
                    'b': [1.0, 1.5, 2.0, 2.5],
                    'c': ['a', 'b', 'c', 'd'],
                    'd': ['2016-11-01', '2016-12-01', '2017-01-01', '2018-02-01'] },
                    index=['r1', 'r2', 'r3', 'r4'] )

print(df)
                  #     a    b  c          d
                  # r1  1  1.0  a 2016-11-01
                  # r2  2  1.5  b 2016-12-01
                  # r3  3  2.0  c 2017-01-01
                  # r4  4  2.5  d 2017-02-01

print(df.dtypes)

                  # a      int64       # note special pandas types int64 and float64
                  # b    float64
                  # c     object       # 'object' is general-purpose type,
                  # d     object       #     covers strings or mixed-type columns
                  # dtype: object

You can use the regular integer index to set element values in an existing Series. However, the new element value must be the same type as that defined in the Series; if not, pandas may refuse, or it may upconvert or cast the Series column to a more general type (usually object, because numpy is focused on the efficiency of numeric and datetime types).

print(df.b.dtype)       # float64
df.loc['b'] = 'hello'
print(df.b.dtype)       # object

Note that we never told pandas to store these values as floats. But since they are all floats, pandas decided to set the type.

We can change a dtype for a Series ourselves with .astype():

df.a = df.a.astype('object')     # or df['a'] = df['a'].astype('object')

         #  df.loc[0, 'a'] = 'hello'

The numpy dtypes you are most likely to see are:

int64
float64
datetime64
object

Checking the memory usage of a DataFrame

.info() provides approximate memory size of a DataFrame

df.info()   # on the original example at the top

         #  
         #  Index: 4 entries, r1 to r4
         #  Data columns (total 4 columns):
         #  a    4 non-null int64
         #  b    4 non-null float64
         #  c    4 non-null object
         #  d    4 non-null object
         #  dtypes: float64(1), int64(1), object(2)
         #  memory usage: 160.0+ bytes

'+' means "probably larger" -- info() only sizes numeric types, not 'object'

With memory_usage='deep', size includes type'object'

df.info(memory_usage='deep')

         # memory usage: 832 bytes

Line Chart on a Series

A Series object (usually a column from a DataFrame) can be easily plotted as a line.

import pandas as pd

df = pd.read_csv('weather_newyork.csv')

print(len(df))   # 365   (weather information for each day)

df.mean_temp.plot()

pandas: Subscripting, Slicing, Joining, Appending