software/python
Python
iPython tricks
Use the cpaste
command to copy in blocks of raw python code, even if the indenting is a little weird.
Style
Python PEP-008: Style Guide for Python Code
pylint, a Python syntax checker. Very verbose, use pylint -E (errors only) or at least pylint -r no
(no report). Eg, pylint -r no file.py -d W0614 -d C -d R
.
For docstring documentation, refer to PEP-257 and the Sphinx documentation; specifically, document script functionality in a top level (above imports, below any hashbang) docstring.
Use leading #:
style comments to document important non-object/non-function element definitions (eg, static variables) in a way that will get pulled out into Sphinx. Use "Google-style" function argument/return documentation instead of "Sphinx style". For example:
def public_fn_with_googley_docstring(name, state=None):
"""This function does something.
Args:
name (str): The name to use.
Kwargs:
state (bool): Current state to be in.
Returns:
int. The return code::
0 -- Success!
1 -- No good.
2 -- Try again.
Raises:
AttributeError, KeyError
A really great idea. A way you might use me is
>>> print public_fn_with_googley_docstring(name='foo', state=None)
0
BTW, this always returns 0. **NEVER** use with :class:`MyPublicClass`.
"""
return 0
autopep8 is a tool to automatically pep8-ify a file. Use it like:
sudo pip install autopep8
autopep8 --in-place --select=W293,W191,W291 *.py
pep8radius is sort of similar, but only applies to code that you are going to commit (using VCS info).
Packaging
Flask app packaging advice, including MANIFEST.in
and non-PyPi dependancy advice: http://flask.pocoo.org/docs/patterns/distribute/
Use console_scripts
in setup.py
to install system-wide scripts: http://packages.python.org/distribute/setuptools.html#automatic-script-creation
For debian packaging, use stdeb (via stackoverflow thread).
For notes on pip vs. setup.py dependencies: https://caremad.io/blog/setup-vs-requirement/
"Fucking" String Encoding
(str/unicode errors in python are very prevalent and give me the rage)
The codecs
package has some helpers; see for example open(f,mode,encoding)
.
ASCII
'ord' is the function that takes a single ASCII character and returns the value number (as an int).
RunSnakeRun
Runsnake seems to be unmaintained... snakeviz is the new thing?
Example session:
$ python -m cProfile -o ./dump.profile myscript.py --script-option blah
$ # run to completion or Ctrl-C, then
$ runsnakerun ./dump.profile
# or
$ snakeviz ./dump.profile
nosetests
NOTE: by default I now use pytest instead of nose
To do minimal tests without wrapping everything in a class, import assert functions from nose.tools, eg:
from nose.tools import assert_raises, assert_equal
To do interactive pdb debugging, simply:
$ nosetests --pdb
# or sometimes:
$ nosetests --pdb-failures
pdb
To debug a script (interactive prompt on exception):
python -m pdb myscript.py
or in-line, always at a particular point:
import pdb; pdb.set_trace()
Use ipdb (debian: python-ipdb) instead of pdb to get a nicer IPython prompt.
Python 3 Porting
To help port and support both Python 2.7 and 3.0+, start with an import:
from __future__ import absolute_import, division, print_function
str/bytes/unicode is indeed the major porting challenge. Using bytearrays helps. Use b'asdf'
style byte array definitions for most low-level constants.
struct.unpack()
wants doesn't allow bytearray()
; use bytes()
instead.
Make sure rase Exception ("Message here")
style is used everywhere, instead of raise Exception, "Message here"
.
There was some change in comparison between None and integers, which makes if args.verbose > 0
break. Use if args.verbose and args.verbose > 1
instead.
Debian Default Python
Keep in mind that this can break some system things (eg, arandr, some cups things).
To make Python 3.4 the default for python system-wide, do something like:
sudo update-alternatives --install /usr/bin/python python /usr/bin/python3.4 2
For an individual user you can also just create a python alias. See also:
http://linuxconfig.org/how-to-change-from-default-to-alternative-python-version-on-debian-linux
Nice Packages
subprocess/pexpect: https://github.com/kennethreitz/delegator.py
time/datetime: https://github.com/crsmithdev/arrow
tsv: https://github.com/kennethreitz/tablib
simple HTML scraping: https://github.com/michaelhelmick/lassie
sqlite3dbm is a library to back a python dict with sqlite3 on disk
pytest
pytest sort of "just works" if you put test files under ./tests/. If you want crude file imports, skipping directories, and including test_* functions from any python file (not just those with test_*.py, install pytest-pythonpath and create a pytest.ini like:
[pytest] python_paths = . python_files = .py norecursedirs = .svn _build tmp
Need to mock? <https://blog.fugue.co/2016-02-11-python-mocking-101.html>
Debugging Memory Usage
Most helpful tools I found were psutil and pympler (both need to be installed).
import os, psutil process = psutil.Process(os.getpid()) print(process.memory_info().rss) # ... do some stuff ... print(process.memory_info().rss)
and
from pympler import tracker tr = tracker.SummaryTracker() tr.print_diff()
# ... do some stuff ... tr.print_diff()