Posts tagged ‘python’

ctypes and flexible (zero-length) arrays

Dear lazyweb, I’ve been racking my brain trying to get information out of the following C struct in a nice way when using ctypes:

struct Foo
{
    unsigned int length;
    char name[];
};

I wrote a little C function to get instance of the struct easily into python:

struct Foo *
put_name(char *name)
{
    struct Foo *foo = malloc(sizeof(struct Foo) + strlen(name));
    foo->length = strlen(name);
    memcpy(foo->name, name, foo->length);
    return(foo);
}

Now, how do I actually get the full contents of Foo.name in python? The only way I could think of that actually works is to create a dummy-struct type in order to get the length out and then use it to dynamically create a new sub-class of ctypes.Structure, then create an instance based on the address of what was returned. I think the following shows it pretty clearly:

class FOO(Structure):
    _fields_ = [('length', c_uint), ('name', c_char * 0)]

liblib = cdll.LoadLibrary(’./_build/liblib.so’)
c_put_name = liblib.put_name
c_put_name.argtypes = [c_char_p]
c_put_name.restype = POINTER(FOO)

def put_name(str):
    f = c_put_name(str)
    K = type(’FOO%s’ % f.contents.length, (Structure,),
                        {’_fields_’ : [('length', c_uint), ('name', c_char * f.contents.length)]})
    return K.from_address(addressof(f.contents))

I still think there ought to be some other way that ‘feels’ nicer. I mean the use of “short arrays” (declared as here with [], or zero size as supported by GCC, or the more portable array of size one) seems to be common enough to warrant some support from ctypes, right?

Any suggestions for improvements?

Decorator pattern in Python

The other day I was talking to a mate and former colleague of mine, he’s been doing a lot of Java and C# before but recently he got hired by a small company to do Python work. Anyway he related a funny part of the interview where he said he’d done design patterns and they asked him to explain one that he’s used. He chose Decorator. After he was done explaining the interviewer commented that surely he meant Proxy. The interviewer was wrong and my mate suspects this might be something that’s common in the Python world due to the built-in support for function/method decorators in the language. I suspect he’s right. Anyway, he showed me what he was playing with and I couldn’t help but play a bit on my own afterwards.

Here’s the class of the core object, a simple self-explanatory piece of code:

class Writer(object):
    def write(self, s):
        print s

Here’s a not very exciting example of using it:

> w = Writer()
> w.write('hello')
hello

We want to decorate it by modifying the string passed to write in different ways. First here’s a base decorator class:

class WriterDecorator(object):
    def __init__(self, wrappee):
        self.wrappee = wrappee

    def write(self, s):
        self.wrappee.write(s)

Using it is straight forward, and still not very exciting:

> wd = WriterDecorator(w)
> wd.write('hello')
hello

The constructor requires a wrappee object and the implementation of write is straight forward. Strictly speaking this class is unnecessary, but it’s convenient once we implement “real” decorators. Here’s the first one, it converts the string to upper case before passing it on down the chain:

class UpperWriter(WriterDecorator):
    def write(self, s):
        self.wrappee.write(s.upper())

This is where it gets a little more exciting, not much though:

> uw = UpperWriter(w)
> uw.write('hello')
HELLO

Here’s a nice detail about Python that I’ve never reflected over myself—constructors are inherited in Python. Here’s another decorator, one that makes the string “shouty”:

class ShoutWriter(WriterDecorator):
    def write(self, s):
        self.wrappee.write('!'.join([t for t in s.split(' ') if t]) + ‘!’)

Now it’s getting a little more interesting, because the decorators can be combined:

> sw1 = ShoutWriter(w)
> sw1.write('hello again')
hello!again!
> sw2 = ShoutWriter(uw)
> sw2.write('hello again')
HELLO!AGAIN!

Some of these combinations are more useful than others, and if they’re used very often then it might be worth creating a convenience class for them. Here’s one that I imagine could be useful if you’re a writer for The Register:

class YahooWriter(WriterDecorator):
    def __init__(self, wrappee):
        self.wrappee = UpperWriter(ShoutWriter(wrappee))

Using it is simple:

> yw = YahooWriter(w)
> yw.write('hello again')
HELLO!AGAIN!

Well, so far it’s been child’s play and I wouldn’t have bothered writing about this unless I took this a little further. I thought something was familiar about how the convenience class worked. I vaguely remembered reading something about super being harmful and there seemed to be similarities between behaviour described there and the desired behaviour when nesting decorators. Rewriting the basic decorator classes using super like this retains their behaviour:

class UpperWriter(WriterDecorator):
    def write(self, s):
        super(UpperWriter, self).write(s.upper())


class ShoutWriter(WriterDecorator):
    def write(self, s):
        super(ShoutWriter, self).write('!'.join([t for t in s.split(' ') if t]) + ‘!’)

What this does though is allow implementing YahooWriter like this:

class YahooWriter(UpperWriter, ShoutWriter):
    pass

I think that’s pretty cute.

Here’s where I have to stop though. I don’t know if this is even useful, is it? Maybe it has some serious draw-backs my inexperience and ignorance prevents me from seeing, does it? Has super been used like this somewhere? I’d love pointers to that code :-)

[Edited 16-06-2007 00:34 BST] Bloody hell, can’t believe I had a spelling error in the title all this time. Embarrassing really!

Comment to “A Look at Lua”

In the June issue of Linux Journal (#158) there’s an article on Lua in which the author compares it to Python. The article is only available to subscribers at the moment, so I decided to post my comment here as well.

  1. “There is no need to worry about different types of integers.” There is non need to do that in Python either nowadays. The conversion between “regular integers” and long ones is done automagically. Take the following function:

    def frac(n):
        if n == 0: return 1
        else: return n * frac(n - 1)
    

    Evaluating frac(10) gives the result 3628800 while frac(30) returns 265252859812191058636308480000000L.

  2. “You can add a float, long integer or a double to any other type of integer or number without a hitch in Lua. In contrast, doing this can cause programs written in Python 2.4 or older versions to crash.” Really? I’ve never heard of this problem. Do you have any references?

  3. “In contrast to Python, Lua does not focus on 100% backward compatibility.” That Python would focus on 100% backward compatibility is just plain wrong. Code I’ve written broke when moving from version 2.4 to 2.5 due to changes in generators. This is the only time I’ve personally been bitten by it but every version of Python has broken something from earlier versions. AFAIU Python 3000 will break a whole lot of older Python programs.

  4. “Unlike Python, global variables do not need to be declared.” They don’t need to be declared at all, not on a global level. If a function wants to access a global variable then it has to declare it as global. Contrast that to the following point.

  5. “Exactly the opposite of Python, most variables in Lua are global by default, and you must declare the variable “local” to make it a local variable rather than assuming that all variables are local.” Variables in Python are local by default, which in my experience better reflect how variables are used by programmers.

  6. “The for loop is a little more complex in Python than it is in Lua, because Python needs to know the key to be able to get the key’s value.” It’s possible to iterate over both key and value in Python as well:

    h = {1:'hello', 2:'world'}
    for k, v in h.iteritems():
        print k, v
    

In general it’s a good article, it shows off some interesting points in Lua. In order to make a comparison between two languages it’s necessary to have very good knowledge of both languages, otherwise there’s a real risk that one language is the favourite simply because the author knows that one the best. This piece definitely suffers from that.

I believe it would have been better without the author’s personal opinions on aesthetics sprinkled across the piece.

Unescaping URLs in Python and Haskell

A while ago I decided that in order to learn Haskell I should make an attempt to stop reaching for Python whenever I needed to solve a “small problem” and use Haskell instead. This morning I found a need to unescape URLs inside Vim and I didn’t catch myself until after I had written the following Python code:

#! /usr/bin/env python

import urllib
import sys

for l in sys.stdin:
    print urllib.unquote(l),

After a little bit of cursing I started browsing Hoogle and after bit of searching I found the Network.URI module. My Haskell solution ended up looking like this:

module Main where

import Network.URI

main :: IO ()
main = interact $ unlines . (map unEscapeString) . lines

Short, sweet, and easy to understand, I think.

In case you want to go the other direction, i.e. to escape strings in Haskell then combining escapeURIString and isUnreserved is the right thing to do.

I am…

I’m not too surprised by this actually…

You are Python You are slower than others, but easier to understand. You are a minimalist, who doesn't like clutter.
Which Programming Language are You?

PaiMei on Python 2.5

The PaiMei page says that you need Python 2.4, which turns out to be true due to it shipping with a compiled for Python 2.4 version of pydasm. Of course it’s possible to compile pydasm yourself, it’s even fairly easy just as long as you have the correct version of Visual Studio installed. You could also use the utterly unofficial build I’ve made available here. Not even I know if I’m to be trusted though ;-). Use at own risk, and all that.

So, the steps are:

  1. Install Python 2.5 off python.org
  2. Install pydasm
  3. Install PaiMei
  4. Remove PaiMei’s version of pydasm (c:\python25\site-packages\paimei\pydasm.pyd) to be sure the correct one is used.

Oh, I probably should say that I’ve only been using the core functionality of PaiMei (pydbg and pydbg_core). There may be other dependencies on Python 2.4 in PaiMei that I haven’t stumbled upon!

Python, web forms and cookies

Just the other day I finally got around to something that I’ve wanted to play around with for a fairly long time—posting web forms using python. As an added bonus I also took a look at dealing with cookies in Python.

For posting forms there is of course a module that makes things a lot easier, mechanize, but I wanted to first of all understand how to do it myself and secondly to avoid using anything but the standard Python modules. It turns out there isn’t much to understand. Say that we have a very simple form, say it’s a login form containing two text entries:

<form method="post" action="/login">
<label for="user_name">username</label>
<input type="text" name="user_name" id="user_name" value="" />
<label for="password">password</label>
<input type="password" name="password" id="password" class="sized" />
<input type="submit" class="button" name="login" value="log in" />
</form>

One way to post this form would be the following:

import urllib
import urllib2

login_data = urllib.urlencode({'user_name' : 'foo', 'password' : 'bar'})
resp = urllib2.urlopen('http://url.for.my.site/login', login_data)

Simple enough, I’d say. urllib2.urlopen automatically switches from GET to POST on the existance of some data.

On most sites a cookie is used to track whether a user is logged in or not. Extending the example above to deal with this and enable subsequent requests to the site as a logged-in user leads us to the CookieJar:

import urllib
import urllib2
import cookielib

cj = cookielib.CookieJar()
opener = urllib2.build_opener(urllib2.HTTPCookieProcessor(cj))
login_data = urllib.urlencode({'user_name' : 'foo', 'password' : 'bar'})
resp = opener.open('http://url.for.my.site/login', login_data)

After this cj will hold all the cookies returned in the response. You can enumerate over them like this:

for c in enumerate(cj):
    print c.name, c.value

Making requests with a cookie c is simple as well, just add c to the cookie jar before making the request:

cj.set_cookie(c)

The cookie jar also has a policy object and a method, set_cookie_if_ok that will set a cookie for a specific request only if the policy allows it. I.e. it seems fairly simple to make sure there is no cookie leakage when making requests to multiple sites. I’ll leaving playing with that for another day though.

Django is cute

I’ve been playing around a bit more with django, and writing simple web services in it. I have to say it’s cute!

Building on my simple authentication middleware I added a second decorator that extract argument out of the URL and passes them to the function as regular arguments:

def extract_arguments(required, optional):
    def _wrapper(func):
        def _iwrapper(request):
            kwargs = {}
            try:
                for r in required:
                    kwargs[r] = request.GET[r]
            except MultiValueDictKeyError, e:
                return HttpResponse(’Failure’)
            for o in optional:
                if request.GET.get(o, ”):
                    kwargs[o] = request.GET[o]
            return func(request, **kwargs)
        return _iwrapper
    return _wrapper

This allows me to write functions like this

@extract_arguments(['req1', 'req2'], ['opt1', 'opt2'])
def exposed_func(req1, req2, opt1=’default’, opt2=”):
    # perform magic

Yes it’s lacking in type information, but it shouldn’t be too difficult to add that if the need arises. For now string are enough.

I’m also impressed with the documentation. It didn’t take me long to dig out unique_together in the model reference. I have to admit I was stuck at the thought of a generic unique_for field option, luckily I scrolled down and read through the meta options as well.

Some stuff (30/08/2006)

I enjoyed reading this article on how an XSS attack works. I’ve always just done the alert("Game over!") XSS which isn’t really an attack at all, just a proof that there’s a possibility for an attack.

That trusted computing is bad for consumers is something I’ve known for a while, but apparently TC is bad for security as well. Every security measure has its side effects, I’m not convinced this one is unintended though.

Here’s a prime example of just how bad laws like DMCA can be. If this holds up we basically allow the law to force us backwards in time. (I just have to sneak in Cory’s excellent write-up on Europe’s broadcasting flag here.)

Network neutrality is a complicated subject. Ed Felten has done a lot lately to clarify things for me with his Nuts and Bolts of Network Neutrality. I still have to find the time to look through his blog a bit more carefully.

Why is this such big news? An update for Ubuntu broke X. Boohoo! I bet most people complaining don’t have a shadow of a leg to stand on in this. They don’t pay, they don’t contribute, they only bitch in the forums/mailinglists/blogs/etc. It only took 8 hours to fix!

Just in case the UK government wants a good reason to not introduce bloody ID cards and national databases to keep records of everything everyone does—here it is!

Looking to replace M$ Office? Here are a few MS Office killers.

I wouldn’t mind having my desktop look like this!

I’ve actually wondered how to uniquify a list in Python for a long time. ;-)

Python is moving up, or maybe it’s down, I don’t know.

Want to learn Python and PyGTK? This blog on learning Python seems like a good place to get inspired.

Rolling your own authentication in Django

For various reasons I wanted to see just what was required to roll my own basic authentication using Django. I am aware there’s a built-in authentication module, django.contrib.auth, but it’s overkill for my ultimate goal and it depends on the session module in ways that only further study can reveal. I heard rumours that the authentication module would be pluggable. The only pluggable aspect of it that I found was the backend (i.e. the data model).

In the end the most relevant reason for doing this was that I wanted to :-)

So, what’s involved in it? It turns out, not much!

First I created a middleware class that extracts the authentication credentials (read username and password) and sticks a ‘user object’ in the request depending on just how successfully it does this:

from base64 import b64decode
from models import User, AnonUser, BadUser

class MiniAuthMiddleWare(object):
    def process_request(self, request):
        auth = request.META.get('Authorization', '') or \
                request.META.get('HTTP_AUTHORIZATION', '')
        if not auth:
            request.user = AnonUser()
            return

        name, pwd = b64decode(auth[6:]).split(’:')
        try:
            u = User.objects.get(name=name)
            if pwd != u.passwd:
                request.user = BadUser()
                return
            request.user = u
            return
        except User.DoesNotExist, e:
            request.user = BadUser()
        except AssertionError, e:
            request.user = BadUser()

        return

The model related to this is rather minimal:

from django.db import models

class User(models.Model):
    name = models.CharField(maxlength=50, primary_key=True)
    passwd = models.CharField(maxlength=50)

    def is_authenticated(self):
        return True

class AnonUser:
    name = 'Anonymous'
    passwd = 'NoPass'

    def is_authenticated(self):
        return False

class BadUser:
    name = 'BadUser'
    passwd = 'NoPass'

    def is_authenticated(self):
        return False

Then I decorated the functions/urls that required authentication with the following:

def _RequireAuthorization(func):
    def _wrapper(request):
        if not request.user.is_authenticated():
            response = HttpResponse()
            response.status_code = 401
            response['WWW-Authenticate'] = ‘Basic Realm=”Silly Realm”‘
            return response
        else:
            return func(request)
    return _wrapper

That all seems to work just fine :-)