Category Archives: Software Development

AWS Codedeploy not working on current Ubuntu release

For one of our clients, I had to get provision new AMI images based on Ubuntu 18.04 (“bionic”). Some of the services require the Codedeploy agent installed to perform deployments on those machines. This works pretty well on current Debian releases (eg. “stretch”), but no on Ubuntu (which is pretty widely used IMO).

To get the Codedeploy agent running on the current Ubuntu release, I’ve headed to its official Github repository, which already contains a pull request with a fix. Unfortunately, that fix does not fix the agent itself, but parts of the installation routine. The installer script will download a prebuilt .deb image from an S3 bucket, which then will refuse to install because it has dependencies which cannot be met on current Ubuntu versions. The Codedeploy agent is written in Ruby and requires Ruby 2.0, 2.1, 2.2 or 2.3. Newer versions are not (officially) supported; I’ve modified some checks to let the version checks pass, and it works so far. I’ve opened a pull request on Github to get it eventually merged.

Weeks have passed since then. My PR has not been merged, and no useful proposal from upstream was given. They want additional tests for the code, which have not existed before. It frustrates me, because I had to implement some ugly hack to install Ruby 2.3 from previous Ubuntu LTS version (“Xenial”) to get it work on the current Ubuntu bionic. This only works if you don’t have any other projects that depend on Ruby 2.4 or higher (because I’ve pinned the Xenial repository for ruby packages).

Update: I have closed my PR. AWS has merged the very same set of changes into master, and has been rolling out updated packages in most regions.

python-rrdtool 0.1.13 released

I’m proud to announce the availability of python-rrdtool 0.1.13. It contains backported bug fixes and improvements from the upstream repository.

Since python-rrdtool have found its way into the original rrdtool distribution, some people have asked about the future of the project. In fact, the Python bindings part of the official rrdtool distribution and the python-rrdtool are maintained separately, but they share improvements from each other. Due to the nature of release engineering, Linux distributions may see the inclusion of the updated bindings with the rrdtool bindings packages sooner or later, and users who wish to use a more pythonic way to install the bindings, python-rrdtool is the way to go for users that prefer installing packages via pip or similar utilities.

Links

python-rrdtool 0.1.3 released

Today I’ve released python-rrdtool version 0.1.3, which fixes some bugs when building the module in build environments that do not have specific rrdtool build headers included. Now all the graph and xport functions are available again.

To update or install, use pip:

pip install --upgrade rrdtool

There are some discussions going on about the number of Python bindings for rrdtool. The version I’ve created is just one of a few, and was probably the first one that offered Python 3.x support. Some people were asking if it may be possible to get this merged into the regular rrdtool distribution. It might be. But from a certain point of view I think it’s a better approach to distribute the bindings via PyPI, because it’s more pythonic, and it allows building for multiple Python versions at once.

python-rrdtool 0.1.1 released

An update to python-rrdtool was released today. Version 0.1.1 adds support for Python 2.x and fixes some minor issues that came up when porting the original Python binding for Python 3.

You can find more information at the project page on Github.

Installation

The installation of the module is pretty easy. All you need is to have rrdtool installed (and its headers) and of course the header files for your Python version.

Assuming you want to install it via pip, use

# pip install https://github.com/commx/python-rrdtool/tarball/master

You can also install it from sources by downloading the source and run

# python setup.py install

About python-rrdtool

python-rrdtool is a Python binding for rrdtool. It works with both Python 2.x and 3.x and comes with an object-oriented interface as well.

AES encryption with Python

For a current project I had to encrypt arbitary data with AES. AES is a symmetric-key algorithm that is even used by the U.S. government. That cipher is approved to be used for top secret information at the National Secret Agency (NSA). It offers key sizes of 128, 192 and 256 bits.

To encrypt a message with AES, you’ll first have to install PyCrypto. Its considered as the “standard” cryptography library for Python and is widely used. The easiest way to install PyCrypto is using pip:

# pip install pycrypto

Encryption of a message

You first have to decide which key size you’d like to use. The higher the key size, the more secure is the encryption, but it takes more computing resources to encrypt your message. While this is almost irrelevant for small usage, it might have a big impact if you plan to encrypt large amount of data with AES. In the following example we’ll use AES-256 with a key size of 256 bits.

from Crypto import Random
from Crypto.Cipher import AES

def encrypt(message, key=None, key_size=256):
def pad(s):
x = AES.block_size - len(s) % AES.block_size
return s + (bytes([x]) * x)

padded_message = pad(message)

if key is None:
key = Random.OSRNG.posix.new().read(key_size // 8)

iv = Random.OSRNG.posix.new().read(AES.block_size)
cipher = AES.new(key, AES.MODE_CBC, iv)

return (iv + cipher.encrypt(padded_message), key)

The encrypt function expects the message as parameter, which is a bytes object. If your message is a string, you can convert it to a bytes object by using ‘my string’.encode(‘utf-8’). The second parameter is optional: The key you’d like to use for encryption. It must be a bytes object as well; if no key is specified, a random one will be generated. The third and last parameter specifies the key size in bits.

Since we’re using the CBC mode for AES, the message must be a multiple of key size / 8 . If your message is of length 3 and the key size is 256 bits, it must be padded with 29 bytes (256 bits / 8 = 32 bytes – 3 bytes of the message length). The padding is performed by appending the integer representation of the number of bytes that have to be appended to the message. In case if the message is 3 bytes long, we’ll append 29 times the byte x1d (hexadecimal representation of 29) to the message to get the padded message.

If no key has been specified in the function arguments, a random one will be generated. The key must be of size key size / 8  (32 bytes for AES-256, 16 bytes for AES-128). If you have a key that is shorter or longer than this, you could hash your key using SHA to get the appropriate number of bytes you need.

Next, we need an initialization vector (IV). We’ll use a random IV here. The IV must be block_size  bytes long (which is always 16 bytes). Once we’ve got the IV, we can create an AES object with the key, mode and IV. That AES object must be created later again when you want to decipher the message.

In the last step, we’ll encrypt the padded message. The encrypt function above will return a tuple that contains two items:

  1. The ciphertext of the padded message. The IV has been prepended to the ciphertext as we need the IV to decipher the message again
  2. The key that was used to encrypt the message

Decryption of a previously encrypted message

To decrypt AES encrypted ciphertext, you need to know how it was encrypted (which AES mode was being used, the IV and of course the key). For messages that were encrypted using the above function, this is easy, since we know all these parameters. For AES ciphertext that was generated by other programs, its rather hard to find out how it was encrypted to get the original message from it – unless you know the encryption parameters.

def decrypt(ciphertext, key):
unpad = lambda s: s[:-s[-1]]
iv = ciphertext[:AES.block_size]
cipher = AES.new(key, AES.MODE_CBC, iv)
plaintext = unpad(cipher.decrypt(ciphertext))[AES.block_size:]

return plaintext

Start decrypting the ciphertext by passing the ciphertext (from encrypt()’s first return parameter)  and the key that was used for encryption (from encrypt()’s second return parameter).

We first define the unpad lambda function: It will basically remove the padding bytes from the original message. The length of bytes to remove from end has been specified in the padding function when the message was encrypted.

The IV has to be extracted from the ciphertext, as its a required parameter for the AES object initialized when using the CBC mode. The IV is the first 16 bytes of the ciphertext. Now we can reconstruct the AES object that was used to encrypt the original message.

To get the original message (also called plaintext in the code above), we have to decrypt the ciphertext with the reconstructed AES object. Then, the message is un-padded to remove the bytes that have been appended when the message was encrypted. Finally, only use bytes after the IV to get the original message.

A complete example

#!/usr/bin/env python3.3

from Crypto import Random
from Crypto.Cipher import AES

def encrypt(message, key=None, key_size=256):
def pad(s):
x = AES.block_size - len(s) % AES.block_size
return s + (bytes([x]) * x)

padded_message = pad(message)

if key is None:
key = Random.OSRNG.posix.new().read(key_size // 8)

iv = Random.OSRNG.posix.new().read(AES.block_size)
cipher = AES.new(key, AES.MODE_CBC, iv)

return (iv + cipher.encrypt(padded_message), key)

def decrypt(ciphertext, key):
unpad = lambda s: s[:-s[-1]]
iv = ciphertext[:AES.block_size]
cipher = AES.new(key, AES.MODE_CBC, iv)
plaintext = unpad(cipher.decrypt(ciphertext))[AES.block_size:]

return plaintext

if __name__ == '__main__':
message = b'my secret message'

encrypted = encrypt(message)
decrypted = decrypt(*encrypted)

assert decrypted == message

The example will encrypt a message and validate whether the decrypted version of the encrypted message will match the original message.

Python 2.x

Please note that the code will only run on Python 3. To use it with Python 2, you’ll have to use other (un)padding functions to achieve that. Below is the full program example as from above that will use with Python 2.x.

#!/usr/bin/env python2.7

from Crypto import Random
from Crypto.Cipher import AES

def encrypt(message, key=None, key_size=256):
def pad(s):
x = AES.block_size - len(s) % AES.block_size
return s + (chr(x) * x)

padded_message = pad(message)

if key is None:
key = Random.OSRNG.posix.new().read(key_size // 8)

iv = Random.OSRNG.posix.new().read(AES.block_size)
cipher = AES.new(key, AES.MODE_CBC, iv)

return (iv + cipher.encrypt(padded_message), key)

def decrypt(ciphertext, key):
unpad = lambda s: s[:-ord(s[-1])]
iv = ciphertext[:AES.block_size]
cipher = AES.new(key, AES.MODE_CBC, iv)
plaintext = unpad(cipher.decrypt(ciphertext))[AES.block_size:]

return plaintext

if __name__ == '__main__':
message = b'my secret message'

encrypted = encrypt(message)
decrypted = decrypt(*encrypted)

assert decrypted == message

 Useful links

  1. http://en.wikipedia.org/wiki/Advanced_Encryption_Standard
  2. https://www.dlitz.net/software/pycrypto/
  3. http://kyleisom.net/downloads/crypto_intro.pdf

yDNS: A free dynamic DNS service

I am proud to announce to beta operation of yDNS, a free dynamic DNS service. It allows you to define a unlimited number of hosts for use. With the time being, new domains may be added for the use with the service. The service works with the Dynamic DNS of AVM Fritz!Box routers without any problems.

I’ve created this service because the free service of DynDNS ceased. They also changed their SLA last week that requires you to login each 30 days to keep your host, otherwise it expires. Other services like afraid.org do not offer the functionality to work with Fritz!Box routers.

Please test this service and report bugs. You can also recommend new features if you like.

Get your free account at https://ydns.eu

Update 2013/05/21: API docs and a updater script is now available. You’ll find the at my github repository for yDNS.

Upcoming release of Django 1.5

The release of Django 1.5 was originally scheduled to happen around Christmas last year. However, blocking bugs delayed the release and it is now expected to happen within the next days. One major advantage of Django 1.5 for me is the fact that it works with Python 3. Most people will argue that I should proceed using Python 2.x, but I don’t think that this is a good idea for new projects, because Python 3 got some useful additions and uses Unicode strings everywhere. I’ve got some problems with character encodings in the past and with Python 3, these problems are gone.

But that’s not all. Django 1.5 introduces some cool features, which are outlined here. It includes a configurable User model, HTTP streaming and loads of small improvements and bug fixes. But be warned: Just upgrading your project to use Django 1.5 may cause problems. For example, the Syntax for the {% url %}  tag has changed. Rather than specifying the name or path of the view you want to link to, you have to encapsulate it into single quotes, e.g: {% url ‘my.project.view’ %}  because Django interpreted the name as a literal view, not a template variable. It’s always good to have a look at the release notes mentioned above, as it might give you an useful advice on how to transform your code to be working with the new version of Django.

Update 2013-02-26: Django 1.5 was finally released today. Some release blockers came up in the last days  which delayed the final release for some days.

rrdtool-py3k: Python 3 bindings for rrdtool

As I started to port some code from Python 2 to Python 3 I had a problem that probably thousands of developers may have: A third-party library they require is not ported to Python 3 yet.

To support the community I’ve ported the Python bindings for rrdtool to use with Python 3. Its based on the original binding and should work almost the same as the old one, but got some improvements:

  • Updated documentation contents (__doc__) of each of the functions
  • The graph function can now return the actual image from function rather requiring to write it to a file. This is possible because the rrdtool program has the ability to specify “-” as the output file which will print the image bytes to stdout. The binding stores the data written to stdout and returns it along with the graph geometry as a bytes object.
  • An object-oriented interface was added. While the C extension can be imported withimport rrdtool, you can import the object-oriented interface with import RRDtool. The interface offers a class that can work with RRD files and execute all the rrdtool functions directly on arbitrary instances. Again, the graph method of the RRD class supports writing the output into files or to a BytesIO object.

The library is not well tested and considered being buggy. I’d like to encourage everyone interested into it to test it and submit bugs to the issue tracker on the Github project page.

Pre-compiled Windows binaries

The binaries are built with rrdtool 1.4.7 and MSVC 10.0. Please note that the output redirection mentioned in the feature list above is not available under Windows yet. Additionally, the flushcached function will have no effect, since I wasn’t able to compile the libraries when using that function from rrdlib. Use the appropriate architecture and Python version listed in the list below.