Notes if you are using Jupyter Notebook: to call exit() from a notebook, please use sys.exit() (requires import sys). If a strange error occurs, it may be because Jupyter retains variables from all executed cells. To reset the notebook, click 'Restart Kernel' (the circular arrow) -- this will reset variables, but will not undo any changes made.

REQUESTS: VIEWING RESPONSE

You can also loop through the dict-like response.headers to see each key/value pair clearly.

Next, change one of the parameters to see how the code changes; also, change the spelling of the word 'product'. In addition, use requests.status_codes._codes[response.status_code] with the status code to see the meaning of the response code.

REQUESTS: ENCODED (text) and UNENCODED (bytes) RESPONSE

With the first URL, check the type of response.text and the type of response.content. Each contains the response content, but one is decoded as a string and the other is encoded as bytes. Next, check the value of response.encoding to see what encoding the yahoo page uses. The check the same with the Microsoft English page and the Microsoft French page.

Check the folder where this script or notebook is located; you should find the image file there.

LAB

Download the following page and save to disk in a new file (write response.text to the file). Make sure to close the file. After writing the file, please open the file in a browser (use your browser's File > Open File menu command). In another tab on your browser, open the url directly. Compate the two pages. (The locally saved file will not have correct formatting or colors - this is because these are supplied by a separate CSS file that was not downloaded.)

Continuing the previous exercise, use the .status_code attribute to view the response status. Now change the URL to something you wouldn't expect to be correct, run and view the status code.

Continuing the previous exercise, use the .headers attribute to retrieve the headers coming back as a dict from the response. Loop through and print each key/value pair in the dict.

Use the below URL to retrieve an image from the internet, and save it locally to a filename of your choice (make sure it has a .jpg file extension so it is recognized properly by your image viewer). The data must be read and written as binary data (i.e., not plaintext). Therefore when writing, use 'wb' (write binary) instead of 'w'.

REQUESTS: CONFIGURING THE REQUEST

First, issue the following request and view the headers reflected back. Next, uncomment the 'headers' lines to see how http_reflect "sees" you (i.e., what browser and platform it thinks you are requesting from). Finally, change 'text/plain' to 'text/html' and see http_reflect respond with HTML instead of plain text.

In the below program, add the params= argument with the dict 'my_params' to send key/value pairs to the http_reflect service as part of the query string. In the response, find the parameters you sent.

In the below program, add the data= argument with the dict 'my_data' to send key/value pairs to the http_reflect service as part of the body of the request. In the response, find the parameters you sent.

REQUESTS: UPLOADING A FILE

Note that the file has been opened with 'rb', which stands for 'read binary': we are uploading encoded bytes.

LAB

Please note below the approach for setting headers, parameters and data in a request, and note that each have been successfully sent to (and reflected back from) the server.

Complete a request from the below URL by using the parameter term with a value that is a term you would like to search on urbandictionary.com. (Warning: urbandictionary.com definitions can use offensive language. For a "clean" set of definitions, try the defition for squee as shown below.) Read and save the resulting JSON text to a file.

Continue the previous program by using the .json() method of the response object to read the data as a JSON object. You should find that the object is a dict, that it has one key list with a value that is a list, and that if you loop through this list of dicts, the key author is the author's name, and definition is the definition.

Request the following data and retrieve as text. Load the text as a csv.reader object, then read line-by-line and print out each parsed line. (Hint: to load the data into text into csv.reader, split the text using .splitlines() first, and pass this list to csv.reader().)

WEB SCRAPING

Explore the following attributes of the object named 'soup':

• print the type of the object
• print the object itself
• print the .text attribute

Note that you may occasionally encounter a UnicodeDecodeError when you attempt to read a file from the internet. In these cases you should tell Python which encoding to use:

Print the type of 1 of these objects. Print the .text attribute of each of these objects.

LAB

The following exercises will parse the test_scrape.html page in this week's session folder.

<html>
  <head>
    <title>This is a page title.</title>
  </head>
  <body>
    <h3 class="heading">This is a page heading.</h3>
    <p>This is some text.</p>
    <p>This is some more text.</p>
    <h3 class="midpage">This is a midpage heading.</h3>
    <p>This is even more text.</p>
    <div content="Some div parameter content we want!">
    Some div text!
    </div>
  </body>
</html>

Expected output:

Note that you may occasionally encounter a UnicodeDecodeError when you attempt to read a file from the internet. In these cases you should tell Python which encoding to use:

Expected Output:

This is a midpage heading.

Expected Output:

Some div parameter content we want!

    Some div text!

Expected Output:

This is some text.
This is some more text.
This is even more text.

WORKING WITH ENCODINGS AND UNICODE

Use the ord() function to convert this character to an integer; then use the chr() function to convert back from integer to character.

You can use integers to retrieve characters. Run the following program:

greet.encode() should include an encoding ('ascii', 'latin-1' or 'utf-8'):

The below string contains a non-ascii character. Encode into the following encodings: 'latin-1', 'utf-8' and 'ascii'.

optional: FLASK

In these exercises, we'll re-create a Flask application that takes form input and displays an image. You can supplement these instructions with your own application idea -- one that uses user input to choose and display images.

Use the basic hello_flask.py example. Start the application by running it from the command line (the Terminal or Command Prompt window). Expected Output:

$ ./hello.py
 * Running on http://127.0.0.1:5000/ (Press CTRL+C to quit)
 * Restarting with stat
 * Debugger is active!
 * Debugger PIN: 323-629-461

Call the application from a browser with the following URL; you should see "Hello, World!" displayed in the browser.

http://localhost:5000/hello

Expected Output:

Hello, world!

The sample code from the hello world app contains a print() statement. Look in the Terminal or Command Prompt window to see the print output displayed. Refresh the page several times so you can see that the log message is displayed. Sample program run:

*** DEBUG:  inside hello_world() ***
127.0.0.1 - - [24/Mar/2018 13:45:59] "GET /hello HTTP/1.1" 200 -
*** DEBUG:  inside hello_world() ***
127.0.0.1 - - [24/Mar/2018 13:45:59] "GET /hello HTTP/1.1" 200 -
*** DEBUG:  inside hello_world() ***
127.0.0.1 - - [24/Mar/2018 13:46:00] "GET /hello HTTP/1.1" 200 -

Add another @app.route() function that says "Goodbye, now! Come on back, y'hear?". This function should be called when the user adds /goodbye to the URL. Call the new URL:

http://localhost:5000/goodbye

Expected Output:

Goodbye, now!  Come on back, y'hear?

The templates must be placed within a folder called template/ that is in the same directory as the script. Here is the template to add for hello (hello.html, included in this session's files):

<html>
  <head><title>A Greeting</title></head>
  <body>
      <h1>Hello, Template!</h1>
  </body>
</html>

Expected Output:

Hello, Template!

Here is the template to add for goodbye (goodbye.html, included in this session's files):

<html>
  <head><title>A Farewell</title></head>
  <body>
      <h1>Goodbye, now!  Come on back, y'hear?!</h1>
  </body>
</html>

Expected Output:

Goodbye, now!  Come on back, y'hear?!

This page has been added as launch.html, also in the templates/ directory.

<html>
  <head><title>Launch</title></head>
  <body>
    <h1>Launch!</h1>
    <A HREF="http://localhost:5000/hello">say hello</A><br><br>
    <A HREF="http://localhost:5000/goodbye">say goodbye</A>
  </body>
</html>

As you can see, each link points to another place in the app. After saving this page as launch.html, use your browser's File > Open File... to navigate to the page and display it. You'll see the two links displayed. Expected output: Click on each one to see that the page calls the Flask app and displays the proper template.

However, don't use the <A HREF= URL as was done in the launch page. Instead, use the flask.url_for() function and allow Flask to build the URL for you:

<A HREF="{{ url_for('hello_world') }}">say goodbye</A>

Also make special note that the argument to url_for() is the name of the function, not the @app.route() URL string. Expected output: You should be able to click back and forth between one page and another by clicking the link on each page.

    <form action="http://localhost:5000/goodbye">

    Please enter a caption:  <input type="text" name="caption">
    <br>

    Please select a mood:
    <select name="mood">
        <option>happy</option>
        <option>sad</option>
    </select>
    <br>

        <input type="submit">
    </form>

The form sends field values for "caption" and "mood" to the app when it calls the goodbye/ URL; retrieve these values using flask.request.args.get('caption') and flask.request.args.get('mood'). Use the {{ [token name] }} token (where [token_name] is the token name) to insert a value into the template. Add parameter arguments (for example caption=[caption variable], where [caption variable] is whatever variable you used to retrieve the value from the form) to render_template() to insert the value. Expected Output:

Goodbye, now! Come on back, y'hear?!

caption: Hi there (or whatever you typed into form)
mood: happy (or whichever mood you chose)
say hello

Note that Hi there and happy are values that I entered into the form when I submitted it.

Add the happup.jpg image to the goodbye.html template. In order to reference images in an HTML page served by flask, we must put the images in static/images and reference them using <IMG SRC="{{ url_for('static', filename='images/[name of image file]') }}">. Expected Output: You should see the happup.jpg image along with the form input values

Now add the sadpup.jpg image to the static/images/ directory and modify the script so that it sends either image name to the template based on the user's "mood" choice. Use the {% if [varname] == [value] %} {% else %} {% endif %} construct to have the template show either the happup.jpg or sadpup.jpg image. Also, place the caption string beneath the image. Expected Output: You should see the happup.jpg or sadpup.jpg image displayed along with a caption beneath.