Python 3home |
A package is a directory of files that work together as a Python application or library module.
Many applications or library modules consist of more than one file. A script may require configuration files or data files; some applications combine several .py files that work together. In addition, programs need unit tests to ensure reliability. A package groups all of these files (scripts, supporting files and tests) together as one entity. In this unit we'll discover Python's structure and procedures for creating packages. Some of the steps here were taken from this very good tutorial on packages: https://python-packaging.readthedocs.io/en/latest/minimal.html
The base of a package is a directory with an __init__.py file.
Folder structure for package pyhello:
pyhello/ # base package folder - name is discretionary
pyhello/ # module folder - usually same name
__init__.py # initial script -- this is run first
setup.py # setup file -- discussed below
The initial code for our program: __init__.py
def greet():
return 'hello, world!'
The names of the folders are up to you. The "outer" pyhello/ is the name of the base package folder. The "inner" pyhello/ is the name of your module. These can be the same or different. setup.py is discussed next.
This file describes the script and its authorship.
Inside setup.py put the following code, but replace the name, author, author_email and packages (this list should reflect the name in name):
from setuptools import setup
setup( name='pyhello',
version='0.1',
description='This module greets the user. ',
url='', # usually a github URL
author='David Blaikie',
author_email='david@davidbpython.com',
license='MIT',
packages=['pyhello'],
install_requires=[ ],
zip_safe=False )
setuptools is a Python module for preparing modules. The setup() function establishes meta information for the package. url can be left blank for now. Later on we will commit this package to github and put the github URL here. packages should be a list of packages that are part this package (as there can be sub-packages within a package); however we will just work with the one package.
Again, folder structure for package pyhello with two files:
pyhello/ # base package folder - name is discretionary
pyhello/ # module folder - usually same name
__init__.py # initial script -- this is run first
setup.py # setup file -- discussed below
Please doublecheck your folder structure and the placement of files -- this is vital to being able to run the files.
pip install can install your module into your own local Python module directories.
First, make sure you're in the same directory as setup.py. Then from the Unix/Mac Terminal, or Windows Command Prompt :
$ pip install . # $ means Unix/Mac prompt Processing /Users/david/Desktop/pyhello Installing collected packages: pyhello Running setup.py install for pyhello ... done Successfully installed pyhello-0.1
The install module copies your package files to a Python install directory that is part of your Python installation's sys.path. Remember that the sys.path holds a list of directories that will be searched when you import a module. If you get an error when you try to install, double check your folder structure and placement of files, and make sure you're in the same directory as setup.py.
If successful, you should now be able to open a new Terminal or Command Prompt window (on Windows use the Command Prompt), cd into your home directory, launch a Python interactive session, and import your module:
$ cd /Users/david # moving to my home directory, to make sure
# we're running the installed version
$ python
Python 3.12.5 (v3.12.5:ff3bc82f7c9, Aug 7 2024, 05:32:06)
[Clang 13.0.0 (clang-1300.0.29.30)] on darwin
Type "help", "copyright", "credits" or "license"
for more information.
>>> import pyhello
>>> pyhello.greet()
'hello, world!'
>>>
If you get a ModuleNotFound error when you try to import: 1. It is possible that the files are not in their proper places, for example if __init__.py is in the same directory as setup.py. 2. It is possible that the pip (or on Mac, pip3) you used installed the module into a different distribution than the one you are running.
$ pip --version pip 24.2 from /Library/Frameworks/Python.framework/Versions/3.12/ lib/python3.12/site-packages/pip (python 3.12) Davids-MacBook-Pro-2:~ david$ python -V PPython 3.12.5 Davids-MacBook-Pro-2:~ david$
Note that my pip --version path indicates that it's running under 3.12. If you are on Mac and see 2.7, you must use pip3 and not pip
Development Directory is where you created the files; Installation Directory is where Python copied them upon install.
Keep in mind that when you import a module, the current directory will be searched before any directories on the sys.path. So if your command line / Command Prompt / Terminal session is currently in the same directory as setup.py (as we had been before we did a cd to my home directory), you'll be reading from your local package, not the installed one. So you won't be testing the installation until you move away from the package directory.
To see which folder the module was installed into, make sure you're not in the package directory; then read the module's __file__ attribute:
$ cd /Users/david # moving to my home directory, to make sure
# we're running the installed version
$ python
Python 3.12.5 (v3.12.5:ff3bc82f7c9, Aug 7 2024, 05:32:06)
[Clang 13.0.0 (clang-1300.0.29.30)] on darwin
Type "help", "copyright", "credits" or "license"
for more information.
>>> import pyhello
>>> pyhello.__file__
/Library/Frameworks/Python.framework/Versions/3.12/lib/python3.12/site-packages/pyhello/__init__.py'
Note that this is not one of my directories or one that I write to; it is a common install directory for Python.
Contrast this with the result if you're importing the module from the package directory (the same directory as setup.py):
$ cd /Users/david/Desktop/pyhello/ # my package location $ python >>> import pyhello >>> pyhello.__file__ '/Users/david/Desktop/pyhello/pyhello/__init__.py'
Note that this is one of my local directories.
Changes to the package will not be reflected in the installed module unless we reinstall.
If you make a change to the package source files, it won't be reflected in your Python installation until you reinstall with pip install. (The exception to this is if you happen to be importing the module from within the package itself -- then the import will read from the local files.)
To reinstall a previously installed module, we must include the --upgrade flag (make sure you're in the same directory as .setup.py:
$ pip install . --upgrade
Processing /Users/david/Desktop/pyhello
Installing collected packages: pyhello
Found existing installation: pyhello 0.1
Uninstalling pyhello-0.1:
Successfully uninstalled pyhello-0.1
Running setup.py install for pyhello ... done
Successfully installed pyhello-0.1
__init__.py is the "gateway" file; the bulk of code may be in other .py files in the package.
Many packages are made of up several .py files that work together. They may be files that are only called internally, or they may be intended to be called by the user. Your entire module could be contained within __init__.py, but I believe this is customarily used only as the gateway, with the bulk of module code in another .py file. In this step we'll move our function to another file.
hello.py (new file -- this can be any name)
def greet():
return 'hello, new file!'
__init__.py
from .hello import greet # .hello refers to hello.py in the base package directory
New folder structure for package pyhello:
pyhello/ # base folder - name is discretionary
pyhello/ # package folder - usually same name
__init__.py # initial script -- this is run first
hello.py # new file
setup.py # setup file -- discussed below
Don't forget to reinstall the module once you've finalized changes. However you can run the package locally (i.e., from the same directory as setup.py) without reinstalling. When the package is imported, Python reads and executes the ___init___.py program. This file is now importing greet from hello.py into the module's namespace, making it available to the user under the package name pyhello.
The user can also reach the variable in hello.py directly, by using attribute syntax to reach the module -- so both of these calls to greet() should work:
>>> import pyhello as dh
>>> dh.greet() # 'hello, new file!'
# (because __init__.py imported greet from hello.py)
>>> dh.hello.greet() # 'hello, new file!'
# (calling it directly in hello.py)
>>> from pyhello import hello
>>> hello.greet() # 'hello, new file!'
Packages provide for accessing variables within multiple files.
Dependencies are other modules that your module may need to import.
If your module imports a non-standard module like my own module splain, it is known as a dependency. Dependencies must be mentioned in the setup() spec. The installer will make sure any dependent modules are installed so your module works correctly.
setup(name='pyhello',
version='0.1',
description='This module greets the user. ',
url='', # usually a github URL
author='David Blaikie',
author_email='david@davidbpython.com',
license='MIT',
packages=['pyhello'],
install_requires=[ 'splain' ],
zip_safe=False)
This would not be necessary if the user already had splain installed. However, if they didn't, we would want the install of our module to result in the automatic installation of the splain module.
Tests belong in the package; thus anyone who downloads the source can run the tests.
In a package, tests should be added to a tests/ directory in the package root (i.e., in the same directory as setup.py).
We will use pytest for our testing -- the following configuration values need to be added to setup() in the setup.py file:
test_suite='pytest' setup_requires=['pytest-runner'] tests_require=['pytest']
Here's our updated setup.py:
from setuptools import setup
setup(name='pyhello',
version='0.1',
description='This module greets the user. ',
url='', # usually a github URL
author='David Blaikie',
author_email='david@davidbpython.com',
license='MIT',
packages=['pyhello'],
install_requires=[ 'splain' ],
test_suite='pytest',
setup_requires=['pytest-runner'],
tests_require=['pytest'],
zip_safe=False)
As is true for most testing suites, pytest requires that our test filenames should begin with test_, and test function names begin with test_.
Here is our test program test_hello.py, with test_greet(), which tests the greet() function.
import pytest
import pyhello as dh
def test_greet():
assert dh.greet() == 'hello, world!'
Here's a new folder structure for package pyhello:
pyhello/ # base folder - name is discretionary
pyhello/ # package folder - usually same name
__init__.py # initial script -- this is run first
hello.py # new file
tests/
test_hello.py
setup.py # setup file -- discussed below
Now when we'd like to run the package's tests, we run the following at the command line:
$ python setup.py pytest
running pytest
running egg_info
writing pyhello.egg-info/PKG-INFO
writing dependency_links to pyhello.egg-info/dependency_links.txt
writing top-level names to pyhello.egg-info/top_level.txt
reading manifest file 'pyhello.egg-info/SOURCES.txt'
writing manifest file 'pyhello.egg-info/SOURCES.txt'
running build_ext
------------------------------- test session starts -------------------------------
platform darwin -- Python 3.6.1, pytest-3.0.7, py-1.4.33, pluggy-0.4.0
rootdir: /Users/david/Desktop/pyhello, inifile:
collected 1 items
tests/test_hello.py F
----------------------------------- FAILURES -----------------------------------
def test_greet():
> assert dh.greet() == 'hello, world!'
E AssertionError: assert 'hello, new file!' == 'hello, world!'
E - hello, new file!
E + hello, world!
tests/test_hello.py:7: AssertionError
--------------------------- 1 failed in 0.03 seconds ---------------------------
oops, our test failed. We're not supplying the right value to assert -- the function returns hello, new file! and our test is looking for hello, world!. We go into test_hello.py and modify the assert statement; or alternatively, we could change the output of the function.
After change has been made to test_hello.py to reflect the expected output:
$ python setup.py pytest running pytest running egg_info writing dblaikie_hello.egg-info/PKG-INFO writing dependency_links to dblaikie_hello.egg-info/dependency_links.txt writing top-level names to dblaikie_hello.egg-info/top_level.txt reading manifest file 'dblaikie_hello.egg-info/SOURCES.txt' writing manifest file 'dblaikie_hello.egg-info/SOURCES.txt' running build_ext ------------------------------- test session starts ------------------------------- platform darwin -- Python 3.6.1, pytest-3.0.7, py-1.4.33, pluggy-0.4.0 rootdir: /Users/david/Desktop/pyhello, inifile: collected 1 items pyhello/tests/test_hello.py . ----------------------------- 1 passed in 0.01 seconds ----------------------------
The output first shows us what setup.py is doing in the background, then shows collected 1 items to indicate that it's ready to run tests. The final statement indicates how many tests passed (or failed). With these basic steps you can create a package, install it in your Python distribution, and prepare it for distribution to the world. May all beings be happy.
All publicly available modules can be found here.
The repo contains every module or script that anyone has cared to upload. It is not a curated or vetted library of modules. One must be careful when choosing and installing, as there have been known instances of spoof modules with malicious code.
https://testpypi.python.org/pypi
The twine module can do this for us automatically. The module will ask us for our registration info for pypi.
Davids-MacBook-Pro-2:dblaikie_hello david$ python setup.py twine Uploading distributions to https://upload.pypi.org/legacy/ Enter your username: YOUR_USER_NAME Enter your password: YOUR_PASSWORD Uploading test_hello-0.6.tar.gz 100%|██████████████████████████████████████████████████████████████| 3.49k/3.49k [00:01<00:00, 2.17kB/s] View at: https://pypi.org/project/test_hello/0.1/