Introduction to Python

davidbpython.com

Interacting with External Processes

the operating system (OS) manages files and processes

Through interacting with the OS, we can manage files and launch other programs.

• the "wider world" of our program is the world of the OS (operating system)
• the OS is responsible for managing the files that are stored on our computer's disk
• Python has functions to copy, delete, and move files
• the OS is also responsible for managing the many programs that run on our computer simultaneously (processes)
• Python has functions to launch and communicate with external processes
• Python is a favored language among sysops (systems administrators) because of its wide feature set in this area

the subprocess module

This module can launch external programs from your Python script.

The subprocess module allows us to:

• spawn new processes (i.e., run external programs)
• connect to the processes' input/output/error pipes (STDOUT, STDIN, etc.)
• obtain the processes' return codes when finished

subprocess.call()

Executes a command and outputs to STDOUT.

for Mac/Linux, using ls:

import subprocess

subprocess.call(['ls', '-l'])      # -l means 'long listing'

for Windows, using dir:

import subprocess

subprocess.call(['dir', '/b'], shell=True)  # /b means 'bare listing'

• here, the .call() function is executing the ls and dir commands
• the first string is the command, and additional strings specify arguments to these commands
• the programs ls and dir output file listings
• the output of these calls will go to STDOUT (usually, the screen)
• the program name and argument(s) are placed in a list for security purposes

subprocess.call(): redirecting output

The output of the called program can be directed to a file or other process.

sending output of command to a write file (and error output to STDOUT)

import subprocess
import sys

wfh = open('outfile.txt', 'w')
subprocess.call(['ls', '-l'], stdout=wfh, stderr=sys.stdout)
wfh.close()

reading the contents of a file to the input of command wc

fh = open('pyku.txt')
subprocess.call(['wc'], stdin=fh)
fh.close()

• as mentioned, wc is the 'word count' program on Unix: it returns the number of lines, words and characters in the file

subprocess.call(): executing through the shell with shell=True

The shell means the program that runs your Command or Terminal Prompt.

import subprocess

subprocess.call('dir /b', shell=True)

• shell=True will execute directly through the operating system's shell (think of the shell as the "public face" of the os)
• this means that certain behaviors of the Command Prompt, like glob expansion (replacing the * with all files in the directory) or environment variables will be activated
• when using shell=True, the command must appear in a single string
• However, if there is any chance the string could contain inputs from the user or the internet, do not use shell=True. This has been the source of some notable hacks in the past.
• (note that dir is Windows-specific -- use ls on Mac/Linux)

subprocess.check_output()

This command executes a command and returns the output to a byte string rather than STDOUT.

(using the dir Windows command)

import subprocess

var = subprocess.check_output(["dir", "."])
var = var.decode('utf-8')
print(var)                   # prints the file listing for the current directory

(using the wc Mac/Linux command)

out = subprocess.check_output(['wc', 'pyku.txt'])
out = out.decode('utf-8')
print(out)                  #        3     15     80 pyku.txt

• the output of .check_output() is a bytestring, or 'encoded' string of bytes
• working with a bytestring means we must decode it to characters to use it as a conventional string

forking child processes with multiprocessing

forking allows a running program to execute multiple copies of itself simultaneously.

• when a program forks, a new process is created that is a duplicate of the original process
• the "child" process can be dispatched by the "parent" process to do some additional work while the parent continues with its own work
• this technique is used in situations in which a single script process would take too long to complete, for example sending email messages (which requires the program to wait on the email sending program)
• by dividing up the work, multiple processes can get some jobs done much more quickly

multiprocessing example

forking allows a running program to execute multiple copies of itself simultaneously.

from multiprocessing import Process
import os
import time

def info(title):                # function for a process to identify itself
    print(title)
    if hasattr(os, 'getppid'):  # only available on Unix
        print('parent process:', os.getppid())
    print('process id:', os.getpid())

def func(childnum):                # function for child to execute
    info(f'|Child Process {childnum}|')
    print('now taking on time consuming task...')
    time.sleep(3)
    print(f'{childnum}:  done')
    print()

if __name__ == '__main__':

    info('|Parent Process|')
    print(); print()
    procs = []
    for num in range(3):
        p = Process(target=func, args=(num,))
                                         # a new process object
                                         # target is function f
        p.start()                        # new process is spawned
        procs.append(p)                  # collecting list of Process objects

    for p in procs:
        p.join()                         # parent waits for child to return
    print('parent concludes')

multiprocessing output and discussion

Because multiple processes are spawned, we must imagine each one as a separately running program.

• the if __name__ == '__main__' line ensures that any new process spawned will not execute the code inside the 'if' block
• Process() defines a new child process, i.e. a new program
• each new child process is an identical copy of the original process, except it begins executing at a different point
• the target=func argument names the function that should be run by the child process, along with any arguments to be passed to that function ((num,) is a 1-item tuple)
• p.start() starts ("spawns") a child process
• after all 3 processes have been spawned, the program calls .join() on each of them. This is a method that causes the parent wait until each process concludes before exiting.
• the net result is that the parent will not terminate execution until all of the child processes have concluded. This is considered good housekeeping - the parent should always wait for the child (otherwise a "zombie process" may be created).

Looking closely at the output, note that all three processes executed and that the parent didn't continue until it had heard back from each of them

|Parent Process|
parent process: 92180
process id: 92316

|Child Process 0|
parent process: 92316
process id: 92318
now taking on time consuming task...
|Child Process 2|
parent process: 92316
process id: 92320
now taking on time consuming task...
|Child Process 1|
parent process: 92316
process id: 92319
now taking on time consuming task...
0:  done
2:  done
1:  done
parent concludes