Python on Benny Simmonds

Python Packaging (2024)

Fri, 29 Dec 2023 17:29:33 +1100

sysconfig

The process of bundling Python code into a format that eases distribution and sharing. First up, I find it helps to get a concrete understanding of how the specific python distro I’m working with is configured, of paricular interest are the various system paths that will be visited for package dependencies. The built-in sysconfig module neatly manages and surfaces this information.

python -m sysconfig

On a Windows system:

Python PR Checklist

Mon, 30 Oct 2023 20:44:33 +1100

A modified version of excellent original checklist by Paul Wolf

General

Code is blackened with black
ruff has been run with no errors
mypy has been run with no errors
Function complexity problems have been resolved using the default complexity index of flake8.
Important core code can be loaded in iPython, ipdb easily.
There is no dead code
Comprehensions or generator expressions are used in place of for loops where appropriate
Comprehensions and generator expressions produce state but they do not have side effects within the expression.
Use zip(), any(), all(), filter(), etc. instead of for loops where appropriate
Functions that take as parameters and mutate mutable variables don’t return these variables. They return None.
Return immutable copies of mutable types instead of mutating the instances themselves when mutable types are passed as parameters with the intention of returning a mutated version of that variable.
Avoid method cascading on objects with methods that return self.
Function and method parameters never use empty collection or sequence instances like list [] or dict {}. Instead they must use None to indicate missing input
Variables in a function body are initialised with empty sequences or collections by callables, list(), dict(), instead of [], {}, etc.
Always use the Final type hint for class instance parameters that will not change.
Context-dependent variables are not unnecessarily passed between functions or methods
View functions either implement the business rules the view is repsonsible for or it passes data downstream to have this done by services and receives non-context dependent data back.
View functions don’t pass request to called functions
Functions including class methods don’t have too many local parameters or instance variables. Especially a class’ __init__() should not have too many parameters.
Profiling code is minimal
Logging is the minimum required for production use
There are no home-brewed solutions for things that already exist in the PSL (python standard library)

n00b habbits

Bare except clause, Python uses exceptions to flag system level interupts such as sigkills. Don’t do this.
Argument default mutatable arguments such as def foo(bar=[]) are defined when the function is defined, not when its run, and will result in a all function calls sharing the same instance of bar
Checking for equality using ==. Due to inheritance this is not desirable as it pins to a concrete type and not potentially it descendents. In other words the Liskov substitution principle. Instead isinstance(p, tuple)
Explicit bool or length checks, such as if bool(x) or if len(x) > 0 is redundant, as Python has sane truthy evaluation.
Use of range over the for in idiom
If you really need the index, always use enumerate
Not using items() on a dict for k, v in dict.items())
Using time.time to measure code performance. Use time.perf_counter instead.
Using print statements over logging
Using import * will normally liter the namespace with variable. Dont be lazy, be specific. from itertools import count

Imports and modules

Imports are sorted by isort or according to some standard that is consistent within the team
Import packages or modules to qualify the use of functions or classes so that unqualified function calls can be assumed to be to functions in the current module

Documentation

Modules have docstrings
Classes have docstrings unless their purpose is immediately obvious
Methods and functions have docstrings
Comments and docstrings add non-obvious and helpful information that is not already present in the naming of functions and variables

General Complexity

Functions as complex as they need to be but no more (as defined by flake8\ ’s default complexity threshold)
Classes have only as many methods as required and have a simple hierarchy

Context Freedom

All important functionality can be loaded easily in ipython without having to construct dummy requests, etc.
All important functionality can be loaded in pdb (or a variant, ipdb, etc.)

Types

Use immutable types ()tuple, frozenset, Enum, etc) over mutable types whenever possible

Functions

Functions are pure wherever possible, i.e. they take input and provide a return value with no side-effects or reliance on hidden state.

Modules

Module level variables do not take context-dependent values like connection clients to remote systems unless the client is used immediately for another module level variable and not used again

Classes

Every class has a single well-defined purpose. That is, the class does not mix up different tasks, like remote state acquisition, web sockets notification, data formatting, etc.
Classes manage state and do not just represent the encapsulation of behaviour
All methods access either cls or self in the body. If a method does not access cls or self, it should be a function at module level.
@classmethod is used in preference to @staticmethod but only if the method body accesses cls otherwise the method should be a module level function.
Constants are declared at module level not in methods or class level
Constants are always upper case
Abstract classes are derived from abc: from abc import ABC
Abstract methods use the @abstractmethod decorator
Abstract class properties use both @abstractmethod and @property decorators
Classes do not use multiple inheritance
Classes do not use mixins (use composition instead) except in rare cases
Class names do not use the word “Base” to signal they are the single ancestor, like “BaseWhatever”
Decorators are not used to replace classes as a design pattern
__init__() does not define too many local variables. Use the Parameter Consolidation pattern instead.
A factory class or function at module level is used for complex class construction (see Design Patterns) to achieve composition
Classes are not dynamically created from strings except where forward reference requires this

Design Patterns

Do not use designs that cause a typical Python developer to have to learn new semantics that are unexpected in Python
Classes primarily use composition in preference to inheritance
Beyond a very small number of simple variables, a class’ purpose is to acquire state for another class or it uses another class to acquire state in particular if the state is from a remote service.
If you use the Context Parameter pattern, it is critical that the state of the context does not change after calling its __init__(), i.e. it should be immutable
If a class’ purpose is to represent an external integration, you probably want numerous classes to compose the service: RemoteDataClient, DomainManager, ContextManager, Factory, NotificationController, DomainResponse, DataFormatter and so on.

Async Python

Wed, 09 Aug 2023 21:12:33 +1100

Background

Using asyncio will not make your code multi-threaded. That is, it will not cause multiple Python instructions to be executed at the same time, and it will not in any way allow you to side step the so-called “global interpreter lock” (GIL).

Some processes are CPU-bound: they consist of a series of instructions which need to be executed one after another until the result has been computed. Most of their time is spent making heavy use of the processor.

Python Type Annotations

Wed, 09 Aug 2023 20:44:33 +1100

Start with the docs and the Type hints cheat sheet

Topics for consideration:

syntax shorthands e.g. | for Union or Optional
Self
If you are using the typing library then there is an abstract type class provided for asynchronous context managers AsyncContextManager[T], where T is the type of the object which will be bound by the as clause of the async with statement.
mypy

If you are using typing then there is an abstract class Awaitable which is generic, so that Awaitable[R] for some type R means anything which is awaitable, and when used in an await statement will return something of type R.

Python Standard Libraries

Fri, 04 Aug 2023 18:01:33 +1100

An important part of becoming “good” at a language is becoming familiar with its library eco-system.

The official Python Standard Library reference manual rocks.

Module	Category	Description
`argparse`		functions for parsing command line arguments
`atexit`		allows you to register functions for your program to call when it exits
`bisect`		bisection algorithms for sorting lists (see Chapter 10)
`calendar`		a number of date-related functions
`codecs`		functions for encoding and decoding data
`collections`		a variety of useful data structures
`concurrent`		asynchronous computation
`copy`		functions for copying data
`csv`		functions for reading and writing CSV files
`datetime`		classes for handling dates and times
`fileinput`	file access	iterate over lines from multiple files or input streams
`fnmatch`		functions for matching Unix-style filename patterns
`glob`		functions for matching Unix-style path patterns
`io`		functions for handling I/O streams and StringIO, which allows you to treat strings as files.
`json`		functions for reading and writing data in JSON format
`logging`		access to Python’s own built-in logging functionality
`multiprocessing`		allows you to run multiple subprocesses, while providing an API that makes them look like threads
`operator`		functions implementing the basic Python operators, instead of writing your own lambda expressions
`os`	swiss army knife	access to basic OS functions
`pprint`	data types	data pretty printer
`random`		functions for generating pseudorandom numbers
`re`		regular expression functionality
`sched`		an event scheduler without using multithreading
`select`		access to the select() and poll() functions for creating event loops
`shutil`	file access	access to high-level file functions
`signal`		functions for handling POSIX signals
`tempfile`	file access	functions for creating temporary files and directories
`threading`		access to high-level threading functionality
`urllib`		provides functions for handling and parsing URLs
`uuid`		allows you to generate Universally Unique Identifiers (UUIDs)

Objects in Python

Thu, 03 Aug 2023 16:28:33 +1100

As I learn more about Pythons idioms reflect on its unique approach to object based programming. In combination with duck typing its approach to objects feels distrubingly flexible.

Testing in Python

Thu, 03 Aug 2023 16:28:33 +1100

There are many ways to write unit tests in Python.

unittest

Here the focus is living off the land with built-in unittest.

unittest is both a framework and test runner, meaning it can execute your tests and return the results. In order to write unittest tests, you must:

Write your tests as methods within classes
These TestCase classes must subclass unittest.TestCase
Names of test functions must begin with test_
Import the code to be tested
Use a series of built-in assertion methods

Basic example

import unittest

class TestStringMethods(unittest.TestCase):

    def test_upper(self):
        self.assertEqual('foo'.upper(), 'FOO')

    def test_isupper(self):
        self.assertTrue('FOO'.isupper())
        self.assertFalse('Foo'.isupper())

    def test_split(self):
        s = 'hello world'
        self.assertEqual(s.split(), ['hello', 'world'])
        # check that s.split fails when the separator is not a string
        with self.assertRaises(TypeError):
            s.split(2)

if __name__ == '__main__':
    unittest.main()

Assertions

The TestCase class provides several assert methods to check for and report failures.

Python quick reference

Sun, 05 Jun 2022 13:57:33 +1100

Forked from 101t/python-cheatsheet.md

RTFM

The Python Standard Library
Built-in Functions
Python Enhancement Propsoals (PEPs)
The Zen of Python never far away in the REPL import this

Getting started: CPython, Easter eggs, Import paths, venv
Collections: List, Dictionary, Set, Tuple, Range, Enumerate, Iterator, Generator
Functions: Functions, Modules
Types: Type, String, Regular_Exp, Format, Numbers, Combinatorics, Datetime
Syntax: Args, Splat, Inline, Closure, Decorator, Class, Duck_Type, Enum, Exception
System: Exit, Print, Input, Command_Line_Arguments, Open, Path, OS_Commands
Data: JSON, Pickle, CSV, SQLite, Bytes, Struct, Array, Memory_View, Deque
Advanced: Threading, Operator, Introspection, Metaprograming, Eval, Coroutines
Libraries: Progress_Bar, Plot, Table, Curses, Logging, Scraping, Web, Profile, NumPy
Packaging and Tools: Real app, Bytecode disassembler, Poetry, Gems, Resources

CPython

Most distros lag behind the latest releases of python. Its quite a pleasant experience to just build CPython from source, as per the docs:

sudo apt update
sudo apt install build-essential gdb lcov pkg-config \
      libbz2-dev libffi-dev libgdbm-dev libgdbm-compat-dev liblzma-dev \
      libncurses5-dev libreadline6-dev libsqlite3-dev libssl-dev \
      lzma lzma-dev tk-dev uuid-dev zlib1g-dev
./configure
sudo make
sudo make install
sudo python3 -m pip install --upgrade pip setuptools wheel

Easter eggs

import this
import antigravity
from __future__ import braces

Import paths

When importing modules, Python relies on a list of paths to know where to look for the module. This list is stored in the sys.path variable.