Category Archives: Python 3

python-rrdtool now available on PyPI

python-rrdtool is now available on PyPI. This simplifies the installation of the rrdtool bindings for Python when using pip or easy_install.

To install, you can simply use the following command:

# pip install rrdtool

python-rrdtool is a Python binding for rrdtool. It’s based on the original Python rrdtool binding distributed with the source code of rrdtool. This bindings work with Python 2.6+ and 3.3+.

python-rrdtool 0.1.1 released

An update to python-rrdtool was released today. Version 0.1.1 adds support for Python 2.x and fixes some minor issues that came up when porting the original Python binding for Python 3.

You can find more information at the project page on Github.

Installation

The installation of the module is pretty easy. All you need is to have rrdtool installed (and its headers) and of course the header files for your Python version.

Assuming you want to install it via pip, use

# pip install https://github.com/commx/python-rrdtool/tarball/master

You can also install it from sources by downloading the source and run

# python setup.py install

About python-rrdtool

python-rrdtool is a Python binding for rrdtool. It works with both Python 2.x and 3.x and comes with an object-oriented interface as well.

AES encryption with Python

For a current project I had to encrypt arbitary data with AES. AES is a symmetric-key algorithm that is even used by the U.S. government. That cipher is approved to be used for top secret information at the National Secret Agency (NSA). It offers key sizes of 128, 192 and 256 bits.

To encrypt a message with AES, you’ll first have to install PyCrypto. Its considered as the “standard” cryptography library for Python and is widely used. The easiest way to install PyCrypto is using pip:

# pip install pycrypto

Encryption of a message

You first have to decide which key size you’d like to use. The higher the key size, the more secure is the encryption, but it takes more computing resources to encrypt your message. While this is almost irrelevant for small usage, it might have a big impact if you plan to encrypt large amount of data with AES. In the following example we’ll use AES-256 with a key size of 256 bits.

from Crypto import Random
from Crypto.Cipher import AES

def encrypt(message, key=None, key_size=256):
def pad(s):
x = AES.block_size - len(s) % AES.block_size
return s + (bytes([x]) * x)

padded_message = pad(message)

if key is None:
key = Random.OSRNG.posix.new().read(key_size // 8)

iv = Random.OSRNG.posix.new().read(AES.block_size)
cipher = AES.new(key, AES.MODE_CBC, iv)

return (iv + cipher.encrypt(padded_message), key)

The encrypt function expects the message as parameter, which is a bytes object. If your message is a string, you can convert it to a bytes object by using ‘my string’.encode(‘utf-8’). The second parameter is optional: The key you’d like to use for encryption. It must be a bytes object as well; if no key is specified, a random one will be generated. The third and last parameter specifies the key size in bits.

Since we’re using the CBC mode for AES, the message must be a multiple of key size / 8 . If your message is of length 3 and the key size is 256 bits, it must be padded with 29 bytes (256 bits / 8 = 32 bytes – 3 bytes of the message length). The padding is performed by appending the integer representation of the number of bytes that have to be appended to the message. In case if the message is 3 bytes long, we’ll append 29 times the byte x1d (hexadecimal representation of 29) to the message to get the padded message.

If no key has been specified in the function arguments, a random one will be generated. The key must be of size key size / 8  (32 bytes for AES-256, 16 bytes for AES-128). If you have a key that is shorter or longer than this, you could hash your key using SHA to get the appropriate number of bytes you need.

Next, we need an initialization vector (IV). We’ll use a random IV here. The IV must be block_size  bytes long (which is always 16 bytes). Once we’ve got the IV, we can create an AES object with the key, mode and IV. That AES object must be created later again when you want to decipher the message.

In the last step, we’ll encrypt the padded message. The encrypt function above will return a tuple that contains two items:

  1. The ciphertext of the padded message. The IV has been prepended to the ciphertext as we need the IV to decipher the message again
  2. The key that was used to encrypt the message

Decryption of a previously encrypted message

To decrypt AES encrypted ciphertext, you need to know how it was encrypted (which AES mode was being used, the IV and of course the key). For messages that were encrypted using the above function, this is easy, since we know all these parameters. For AES ciphertext that was generated by other programs, its rather hard to find out how it was encrypted to get the original message from it – unless you know the encryption parameters.

def decrypt(ciphertext, key):
unpad = lambda s: s[:-s[-1]]
iv = ciphertext[:AES.block_size]
cipher = AES.new(key, AES.MODE_CBC, iv)
plaintext = unpad(cipher.decrypt(ciphertext))[AES.block_size:]

return plaintext

Start decrypting the ciphertext by passing the ciphertext (from encrypt()’s first return parameter)  and the key that was used for encryption (from encrypt()’s second return parameter).

We first define the unpad lambda function: It will basically remove the padding bytes from the original message. The length of bytes to remove from end has been specified in the padding function when the message was encrypted.

The IV has to be extracted from the ciphertext, as its a required parameter for the AES object initialized when using the CBC mode. The IV is the first 16 bytes of the ciphertext. Now we can reconstruct the AES object that was used to encrypt the original message.

To get the original message (also called plaintext in the code above), we have to decrypt the ciphertext with the reconstructed AES object. Then, the message is un-padded to remove the bytes that have been appended when the message was encrypted. Finally, only use bytes after the IV to get the original message.

A complete example

#!/usr/bin/env python3.3

from Crypto import Random
from Crypto.Cipher import AES

def encrypt(message, key=None, key_size=256):
def pad(s):
x = AES.block_size - len(s) % AES.block_size
return s + (bytes([x]) * x)

padded_message = pad(message)

if key is None:
key = Random.OSRNG.posix.new().read(key_size // 8)

iv = Random.OSRNG.posix.new().read(AES.block_size)
cipher = AES.new(key, AES.MODE_CBC, iv)

return (iv + cipher.encrypt(padded_message), key)

def decrypt(ciphertext, key):
unpad = lambda s: s[:-s[-1]]
iv = ciphertext[:AES.block_size]
cipher = AES.new(key, AES.MODE_CBC, iv)
plaintext = unpad(cipher.decrypt(ciphertext))[AES.block_size:]

return plaintext

if __name__ == '__main__':
message = b'my secret message'

encrypted = encrypt(message)
decrypted = decrypt(*encrypted)

assert decrypted == message

The example will encrypt a message and validate whether the decrypted version of the encrypted message will match the original message.

Python 2.x

Please note that the code will only run on Python 3. To use it with Python 2, you’ll have to use other (un)padding functions to achieve that. Below is the full program example as from above that will use with Python 2.x.

#!/usr/bin/env python2.7

from Crypto import Random
from Crypto.Cipher import AES

def encrypt(message, key=None, key_size=256):
def pad(s):
x = AES.block_size - len(s) % AES.block_size
return s + (chr(x) * x)

padded_message = pad(message)

if key is None:
key = Random.OSRNG.posix.new().read(key_size // 8)

iv = Random.OSRNG.posix.new().read(AES.block_size)
cipher = AES.new(key, AES.MODE_CBC, iv)

return (iv + cipher.encrypt(padded_message), key)

def decrypt(ciphertext, key):
unpad = lambda s: s[:-ord(s[-1])]
iv = ciphertext[:AES.block_size]
cipher = AES.new(key, AES.MODE_CBC, iv)
plaintext = unpad(cipher.decrypt(ciphertext))[AES.block_size:]

return plaintext

if __name__ == '__main__':
message = b'my secret message'

encrypted = encrypt(message)
decrypted = decrypt(*encrypted)

assert decrypted == message

 Useful links

  1. http://en.wikipedia.org/wiki/Advanced_Encryption_Standard
  2. https://www.dlitz.net/software/pycrypto/
  3. http://kyleisom.net/downloads/crypto_intro.pdf

Upcoming release of Django 1.5

The release of Django 1.5 was originally scheduled to happen around Christmas last year. However, blocking bugs delayed the release and it is now expected to happen within the next days. One major advantage of Django 1.5 for me is the fact that it works with Python 3. Most people will argue that I should proceed using Python 2.x, but I don’t think that this is a good idea for new projects, because Python 3 got some useful additions and uses Unicode strings everywhere. I’ve got some problems with character encodings in the past and with Python 3, these problems are gone.

But that’s not all. Django 1.5 introduces some cool features, which are outlined here. It includes a configurable User model, HTTP streaming and loads of small improvements and bug fixes. But be warned: Just upgrading your project to use Django 1.5 may cause problems. For example, the Syntax for the {% url %}  tag has changed. Rather than specifying the name or path of the view you want to link to, you have to encapsulate it into single quotes, e.g: {% url ‘my.project.view’ %}  because Django interpreted the name as a literal view, not a template variable. It’s always good to have a look at the release notes mentioned above, as it might give you an useful advice on how to transform your code to be working with the new version of Django.

Update 2013-02-26: Django 1.5 was finally released today. Some release blockers came up in the last days  which delayed the final release for some days.

rrdtool-py3k: Python 3 bindings for rrdtool

As I started to port some code from Python 2 to Python 3 I had a problem that probably thousands of developers may have: A third-party library they require is not ported to Python 3 yet.

To support the community I’ve ported the Python bindings for rrdtool to use with Python 3. Its based on the original binding and should work almost the same as the old one, but got some improvements:

  • Updated documentation contents (__doc__) of each of the functions
  • The graph function can now return the actual image from function rather requiring to write it to a file. This is possible because the rrdtool program has the ability to specify “-” as the output file which will print the image bytes to stdout. The binding stores the data written to stdout and returns it along with the graph geometry as a bytes object.
  • An object-oriented interface was added. While the C extension can be imported withimport rrdtool, you can import the object-oriented interface with import RRDtool. The interface offers a class that can work with RRD files and execute all the rrdtool functions directly on arbitrary instances. Again, the graph method of the RRD class supports writing the output into files or to a BytesIO object.

The library is not well tested and considered being buggy. I’d like to encourage everyone interested into it to test it and submit bugs to the issue tracker on the Github project page.

Pre-compiled Windows binaries

The binaries are built with rrdtool 1.4.7 and MSVC 10.0. Please note that the output redirection mentioned in the feature list above is not available under Windows yet. Additionally, the flushcached function will have no effect, since I wasn’t able to compile the libraries when using that function from rrdlib. Use the appropriate architecture and Python version listed in the list below.

Django and Python 3

Many people use the Django Web Framework for turning their Web Project ideas into production websites. I did that myself a while ago, because it not only saves a lot of time – but I also like it because I can write the backend directly in Python code. Django is getting more and more attention and it has a large user base.

People which are new to Python have to decide first whether they want to use Python 2.x or 3.x for their projects. While on the one hand its better to go straight with 3.x because its actively developed and offers a lot of cool new features, many people are forced to use 2.x as many important Libraries have not been ported to Python 3.x yet on the other. This includes Django as well as the Twisted networking Library. All these projects have one in common: They are maintained for several years now and they have got a large user base. Turning 2.x code into code that runs on 3.x is a long term process which may take several months to years for projects of that size. Porting the code is definitely required: Python 2.x is not developed anymore. Python 2.7  is the latest release of the 2.x branch and is only updated with bug fixes and security patches. That’s why some Linux distributions, such as Ubuntu, have decided to port all their tools to use Python 3 in the future.

However, Django is on a good way. The upcoming 1.5 release will support experimental Python 3 support using the six compability layer. The other day I cloned the development code of Django on Github and tried it to run with Python 3.2.3, which worked like a charm. I have not tested everything yet, but at least it feels to be faster. The Django development team encourages everyone to test whether their Django projects work with a latest version of Django’s development snapshot and help to fix issues that are found by doing so.

I would recommend to start using the development snapshot along with Python 3 for new (large-scale) projects, because it will save you a lot of time if you have to port the code later.

The final release of Django 1.5 is scheduled for 24th of December 2012 (or even a few days before). Its release schedule was posted today in the django-developers mailing list.

Useful resources: