Mnet coding standards

This document is current as of Mnet Mnet v0.7.

CVS revision: $Id: coding_standards.html,v 1.124 2004/07/17 11:33:25 arnowa Exp $

Here are some Python code style guidelines. We also use some other official Python guidelines by reference (see official Python standards below). Our Mnet-specific standards override the official standards whenever there is a conflict.

Mnet-specific guidelines:

basic standards

compatibility

Mnet requires Python v2.3 or greater. No effort should be made to offer compatibility with versions of Python older than 2.3. Effort should be made to work with the most recent release of Python, and with every release between v2.3 and the most recent release.

naming and layout

Use readable_names for functions, shortnames for variables and CamelCapNames for classes.
Always put trailing commas in lists, tuples and dicts. For example (spam, eggs, bacon,), not (spam, eggs, bacon). This makes it easier to edit (for example, to re-arrange the elements), it makes it more visually distinct from function invocation, and it prevents the error where you meant to write the 1-element tuple (spam,), but you wrote (spam), and the parser thinks you mean just spam (with precedence-forcing parenthesis).
Always put parenthesis around tuples. Write (spam, eggs,) = fetch_breakfast() instead of spam, eggs, = fetch_breakfast().

comments, idioms, miscellany, license

Here is a useful template for starting new Python files:

# Copyright (c) 2003 Bryce "Zooko" Wilcox-O'Hearn
# mailto:zooko@zooko.com
# See the end of this file for the free software, open source license (BSD-style).

"""
XXX doc string describing the module here
"""

__version__ = "$Revision: 1.124 $"
# $Source: /cvsroot/mnet/www/coding_standards.html,v $

# XXX import Python Standard Library modules here

from pyutil.assertutil import _assert, precondition, postcondition
from pyutil.debugprint import debugprint, debugstream
from pyutil.humanreadable import hr
# XXX import other pyutil modules here

# XXX import from other libraries, with a blank line between each library

# XXX your code here

# Copyright (c) 2003 Bryce "Zooko" Wilcox-O'Hearn
# Permission is hereby granted, free of charge, to any person obtaining a copy
# of this software to deal in this software without restriction, including
# without limitation the rights to use, copy, modify, merge, publish,
# distribute, sublicense, and/or sell copies of this software, and to permit
# persons to whom this software is furnished to do so, subject to the following
# condition:
#
# THIS SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
# IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
# FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
# AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
# LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
# OUT OF OR IN CONNECTION WITH THIS SOFTWARE OR THE USE OR OTHER DEALINGS IN
# THIS SOFTWARE.

Files should begin with a copyright notice, a docstring about that module, then a __version__ var and CVS source comment. It must be in that order for epydoc to find the docstring.
You can use any suitable free software license you want. We recommend something like this one. (Actually, it would be nice if there were a shorter version of this license.)

truths and falsehoods

Always use True or False if that is what you mean -- never write a literal 0, 1, or None to indicate a boolean value.
Never use the literals True or False in conditional expressions -- instead just write the expression which will evaluate to true or false. For example, write if expr: instead of if expr == True: and if not expr: instead of if expr == False:."
Use the fact that empty sequences, empty strings, empty dicts, and None all evaluate to false. Write if not items: instead of if len(items) == 0:.

modules

Put imports from the same library on a single line e.g.: from pyutil import dictutil, strutil, randutil. This over-rides PEP 8.
Import modules rather than classes, functions, or other contents of a module (especially not variables). For example, import BlockWrangler with "import BlockWrangler" (the module) and then reference the BlockWrangler class with "BlockWrangler.BlockWrangler".
- exception 1: ubiquitous functions which are really "part of the language" more than part of the application such as humanreadable.hr and debugprint.debugprint can be imported directly for terseness.
- exception 2: if it looks a lot better to you, for readability, to import the contents of a module, then go ahead and make an exception to this rule, but still do not do "from Thingie import *".
To import other modules from the same package do a simple import module instead of a from package import module, e.g. inside pyutil.humanreadable we write import dictutil instead of from pyutil import dictutil.

advanced idioms

preconditions and assertions

basic preconditions and assertions

Make sure you have from pyutil.assertutil import _assert, precondition, postcondition in your imports (as shown in the template above). Now design preconditions for your methods and functions, and assert them like this:

def oaep(m, emLen, p=""):
    precondition(emLen >= (2 * SIZE_OF_UNIQS) + 1, "emLen is required to be big enough.", emLen=emLen, SIZE_OF_UNIQS=SIZE_OF_UNIQS)
    ...

Notice how you pass in any values that ought to be printed out in the error message if the assertion fails -- in the example, we pass the values emLen and SIZE_OF_UNIQS. You can pass these as normal args or keyword args. If you use keyword args then the name of the argument will also appear in the error message, which can be helpful. For example, if the assertion above fails, then a debug message will appear at the end of the stack trace, like this:

>>> oaep("some secret thingie", 20)
Traceback (most recent call last):
  File "<stdin>", line 1, in ?
  File "<stdin>", line 2, in oaep
  File "/home/zooko/playground/pyutil_new/pyutil/assertutil.py", line 47, in precondition
    raise preconditionfailureexception
AssertionError: precondition: emLen is required to be big enough. -- emLen: 20 :: <type 'int'>, 'SIZE_OF_UNIQS': 20 :: <type 'int'>

The "error message" that will accompany a failed expression should be a statement of what is required for correct operation. Don't write something like "Spam isn't firm.", because that is ambiguous: the error could be that the spam is supposed to be firm and it isn't, or the error could be that spam isn't supposed to be firm and it is! A good practice is to always use the words "required to" in your message, for example "Spam is required to be firm.".

class invariants

If your class has internal state which is complicated enough that a bug in the class's implementation could lead to garbled internal state which could in turn lead to misbehavior, then you should have a class invariant. A class invariant is a method like this (an actual example from BlockWrangler, but truncated for space):

def _assert_consistency(self):
    # All of the keys in all of these dicts must be ids.
    for d in (self.bId2chunkobj, self.bId2peers, self.Idsofwantedblocks, self.Idsoflocatedblocks,):
        _assert(not [key for key in d.keys() if not idlib.is_id(key)], "All of the keys in these dicts are required to be ids.", listofnonIds=[key for key in d.keys() if not idlib.is_id(key)])

    # For each (peer, blockId,) tuple in peerclaimedblock, if the peer *has*
    # claimed the block, then the blockId must appear in bId2peers[blockId],
    # and if the peer has claimed *not* to have the block then the blockId
    # must *not* appear in bId2peers[blockId].
    for ((peer, blockId,), claim,) in self.peerclaimedblock.items():
        _assert((claim == 1) == (peer in self.bId2peers.get(blockId, ())), "The blockId must appear in bId2peers if and only if the peer has claimed the block.", claim=claim, peer=peer, bId2peersentry=self.bId2peers.get(blockId, ()))

Now you can put assert self._assert_consistency() everywhere in your class where the class ought to be in an internally consistent state. For example, at the beginning of every externally-callable method. This technique can be very valuable in developing a complex class -- it catches bugs early while isolating them into specific code paths and it disambiguates the internal structure of the class so that other developers can hack on it without subtle misunderstandings.

list comprehensions

If you are writing code to generate a list, consider using Python's list comprehension syntax. Consider writing resultlist = [x for x in inputlist if x > 3] instead of

resultlist = []
for x in inputlist:
    if x > 3:
        resultlist.append(x)

configuration

minimizing configuration

Do not implement configuration files for modules or libraries -- code that is going to be used by other code. Only applications -- code that is going to be used by humans -- have configuration files. Modules and libraries get "configured" by the code that calls them, for example by passing arguments to their constructors.
If there are constant values which end-users do not need to modify, then do not make them configurable, but put them in all-caps variables at the beginning of the Python file in which they are used.
Design your algorithms so that they have as few "voodoo constants" and "tweakable parameters" as possible. If you added a few of those parameters then you would have a combinatorial explosion of possibilities. If that happened, then (a) the code would never get tested in all its possible configurations, and (b) nobody would be able to understand how the algorithm behaves in general. The end result would be that users run algorithms that have never been tested and that nobody understands. In addition, this is an indicator that the basic algorithm ought to be replaced. If you find yourself needing to add more and more special cases to handle failures of the basic algorithm, this might indicate that the basic algorithm ought to be replaced with one that doesn't have so many edge cases.

how to implement configuration

Whether in application code or in library code, never pass configuration values via a "conf dict". Instead use Python parameters. For example, do not write

class BlockStore:
    def __init__(self, confdict={}, recoverdb=True, name='*unnamed*'):
        if confdict.has_key('MAX_MEGABYTES'):
            self.maxspace = (2**20) * int(confdict.get('MAX_MEGABYTES'))
        else:
            self.maxspace = None
        self.basepath = os.path.abspath(confdict.get("PATH", ""))
        self.maintainertype = confdict.get("MAINTAINER", "rnd").lower()
        self.backendtype = confdict.get("BACKEND", "flat").lower()

blockstore = BlockStore(confdict)

, but instead

class BlockStore:
    def __init__(self, maxspace=None, path="", maintainertype="rnd", backendtype="flat", recoverdb=True, name='*unnamed*'):
        self.basepath = os.path.abspath(path)
        self.maintainertype = maintainertype
        self.backendtype = backendtype

maxspace = confdict.get('MAX_MEGABYTES')
if maxspace is not None:
    maxspace = int(maxspace) * 2**20
blockstore = BlockStore(maxspace, confdict.get('PATH', ""), confdict.get('MAINTAINER', 'rnd').lower(), confdict.get('BACKEND', 'flat').lower())

official Python standards

These are listed in decreasing order of priority, so if a point in one of the latter guidelines contradicts a point in one of the earlier ones, then go with the earlier. The Mnet-specific guidelines above override all else, of course.

Last modified: Sun Mar 7 12:33:46 EST 2004