therning.org/ magnus

Dear Lazyweb,

Please tell me there’s an easier way of building Python C/C++ extensions (read shared objects) using automake than by using libtool. Please! Please!

I just want a frickin’ .so file, nothing complicated. Is the only way, besides using libtool, to write custom targets?

I just came across this post about a coding dojo in Cambridge. Sounds interesting.

I bet a lot of people would consider this to be fairly obvious, but today, while solving the fourth of Google’s Treasure Hunt problems I realised just how similar in essence laziness+infinate lists in Haskell is to generators in Python. It’s just much easier to work with the former

Dear lazyweb, I’ve been racking my brain trying to get information out of the following C struct in a nice way when using ctypes:

struct Foo
{
    unsigned int length;
    char name[];
};

I wrote a little C function to get instance of the struct easily into python:

struct Foo *
put_name(char *name)
{
    struct Foo *foo = malloc(sizeof(struct Foo) + strlen(name));
    foo->length = strlen(name);
    memcpy(foo->name, name, foo->length);
    return(foo);
}

Now, how do I actually get the full contents of Foo.name in python? The only way I could think of that actually works is to create a dummy-struct type in order to get the length out and then use it to dynamically create a new sub-class of ctypes.Structure, then create an instance based on the address of what was returned. I think the following shows it pretty clearly:

class FOO(Structure):
    _fields_ = [('length', c_uint), ('name', c_char * 0)]
 
liblib = cdll.LoadLibrary('./_build/liblib.so')
c_put_name = liblib.put_name
c_put_name.argtypes = [c_char_p]
c_put_name.restype = POINTER(FOO)
 
def put_name(str):
    f = c_put_name(str)
    K = type('FOO%s' % f.contents.length, (Structure,),
                        {'_fields_' : [('length', c_uint), ('name', c_char * f.contents.length)]})
    return K.from_address(addressof(f.contents))

I still think there ought to be some other way that ‘feels’ nicer. I mean the use of “short arrays” (declared as here with [], or zero size as supported by GCC, or the more portable array of size one) seems to be common enough to warrant some support from ctypes, right?

Any suggestions for improvements?

The other day I was talking to a mate and former colleague of mine, he’s been doing a lot of Java and C# before but recently he got hired by a small company to do Python work. Anyway he related a funny part of the interview where he said he’d done design patterns and they asked him to explain one that he’s used. He chose Decorator. After he was done explaining the interviewer commented that surely he meant Proxy. The interviewer was wrong and my mate suspects this might be something that’s common in the Python world due to the built-in support for function/method decorators in the language. I suspect he’s right. Anyway, he showed me what he was playing with and I couldn’t help but play a bit on my own afterwards.

Here’s the class of the core object, a simple self-explanatory piece of code:

class Writer(object):
    def write(self, s):
        print s

Here’s a not very exciting example of using it:

> w = Writer()
> w.write('hello')
hello

We want to decorate it by modifying the string passed to write in different ways. First here’s a base decorator class:

class WriterDecorator(object):
    def __init__(self, wrappee):
        self.wrappee = wrappee

    def write(self, s):
        self.wrappee.write(s)

Using it is straight forward, and still not very exciting:

> wd = WriterDecorator(w)
> wd.write('hello')
hello

The constructor requires a wrappee object and the implementation of write is straight forward. Strictly speaking this class is unnecessary, but it’s convenient once we implement “real” decorators. Here’s the first one, it converts the string to upper case before passing it on down the chain:

class UpperWriter(WriterDecorator):
    def write(self, s):
        self.wrappee.write(s.upper())

This is where it gets a little more exciting, not much though:

> uw = UpperWriter(w)
> uw.write('hello')
HELLO

Here’s a nice detail about Python that I’ve never reflected over myself—constructors are inherited in Python. Here’s another decorator, one that makes the string “shouty”:

class ShoutWriter(WriterDecorator):
    def write(self, s):
        self.wrappee.write('!'.join([t for t in s.split(' ') if t]) + '!')

Now it’s getting a little more interesting, because the decorators can be combined:

> sw1 = ShoutWriter(w)
> sw1.write('hello again')
hello!again!
> sw2 = ShoutWriter(uw)
> sw2.write('hello again')
HELLO!AGAIN!

Some of these combinations are more useful than others, and if they’re used very often then it might be worth creating a convenience class for them. Here’s one that I imagine could be useful if you’re a writer for The Register:

class YahooWriter(WriterDecorator):
    def __init__(self, wrappee):
        self.wrappee = UpperWriter(ShoutWriter(wrappee))

Using it is simple:

> yw = YahooWriter(w)
> yw.write('hello again')
HELLO!AGAIN!

Well, so far it’s been child’s play and I wouldn’t have bothered writing about this unless I took this a little further. I thought something was familiar about how the convenience class worked. I vaguely remembered reading something about super being harmful and there seemed to be similarities between behaviour described there and the desired behaviour when nesting decorators. Rewriting the basic decorator classes using super like this retains their behaviour:

class UpperWriter(WriterDecorator):
    def write(self, s):
        super(UpperWriter, self).write(s.upper())


class ShoutWriter(WriterDecorator):
    def write(self, s):
        super(ShoutWriter, self).write('!'.join([t for t in s.split(' ') if t]) + '!')

What this does though is allow implementing YahooWriter like this:

class YahooWriter(UpperWriter, ShoutWriter):
    pass

I think that’s pretty cute.

Here’s where I have to stop though. I don’t know if this is even useful, is it? Maybe it has some serious draw-backs my inexperience and ignorance prevents me from seeing, does it? Has super been used like this somewhere? I’d love pointers to that code

[Edited 16-06-2007 00:34 BST] Bloody hell, can’t believe I had a spelling error in the title all this time. Embarrassing really!

In the June issue of Linux Journal (#158) there’s an article on Lua in which the author compares it to Python. The article is only available to subscribers at the moment, so I decided to post my comment here as well.

1. “There is no need to worry about different types of integers.” There is non need to do that in Python either nowadays. The conversion between “regular integers” and long ones is done automagically. Take the following function:

   def frac(n):
       if n == 0: return 1
       else: return n * frac(n - 1)
   

   Evaluating frac(10) gives the result 3628800 while frac(30) returns 265252859812191058636308480000000L.

2. “You can add a float, long integer or a double to any other type of integer or number without a hitch in Lua. In contrast, doing this can cause programs written in Python 2.4 or older versions to crash.” Really? I’ve never heard of this problem. Do you have any references?

3. “In contrast to Python, Lua does not focus on 100% backward compatibility.” That Python would focus on 100% backward compatibility is just plain wrong. Code I’ve written broke when moving from version 2.4 to 2.5 due to changes in generators. This is the only time I’ve personally been bitten by it but every version of Python has broken something from earlier versions. AFAIU Python 3000 will break a whole lot of older Python programs.

4. “Unlike Python, global variables do not need to be declared.” They don’t need to be declared at all, not on a global level. If a function wants to access a global variable then it has to declare it as global. Contrast that to the following point.

5. “Exactly the opposite of Python, most variables in Lua are global by default, and you must declare the variable “local” to make it a local variable rather than assuming that all variables are local.” Variables in Python are local by default, which in my experience better reflect how variables are used by programmers.

6. “The for loop is a little more complex in Python than it is in Lua, because Python needs to know the key to be able to get the key’s value.” It’s possible to iterate over both key and value in Python as well:

   h = {1:'hello', 2:'world'}
   for k, v in h.iteritems():
       print k, v
   

In general it’s a good article, it shows off some interesting points in Lua. In order to make a comparison between two languages it’s necessary to have very good knowledge of both languages, otherwise there’s a real risk that one language is the favourite simply because the author knows that one the best. This piece definitely suffers from that.

I believe it would have been better without the author’s personal opinions on aesthetics sprinkled across the piece.

A while ago I decided that in order to learn Haskell I should make an attempt to stop reaching for Python whenever I needed to solve a “small problem” and use Haskell instead. This morning I found a need to unescape URLs inside Vim and I didn’t catch myself until after I had written the following Python code:

#! /usr/bin/env python

import urllib
import sys

for l in sys.stdin:
    print urllib.unquote(l),

After a little bit of cursing I started browsing Hoogle and after bit of searching I found the Network.URI module. My Haskell solution ended up looking like this:

module Main where

import Network.URI

main :: IO ()
main = interact $ unlines . (map unEscapeString) . lines

Short, sweet, and easy to understand, I think.

In case you want to go the other direction, i.e. to escape strings in Haskell then combining escapeURIString and isUnreserved is the right thing to do.

I’m not too surprised by this actually…

Which Programming Language are You?

The PaiMei page says that you need Python 2.4, which turns out to be true due to it shipping with a compiled for Python 2.4 version of pydasm. Of course it’s possible to compile pydasm yourself, it’s even fairly easy just as long as you have the correct version of Visual Studio installed. You could also use the utterly unofficial build I’ve made available here. Not even I know if I’m to be trusted though . Use at own risk, and all that.

So, the steps are:

1. Install Python 2.5 off python.org
2. Install pydasm
3. Install PaiMei
4. Remove PaiMei’s version of pydasm (c:\python25\site-packages\paimei\pydasm.pyd) to be sure the correct one is used.

Oh, I probably should say that I’ve only been using the core functionality of PaiMei (pydbg and pydbg_core). There may be other dependencies on Python 2.4 in PaiMei that I haven’t stumbled upon!

Just the other day I finally got around to something that I’ve wanted to play around with for a fairly long time—posting web forms using python. As an added bonus I also took a look at dealing with cookies in Python.

For posting forms there is of course a module that makes things a lot easier, mechanize, but I wanted to first of all understand how to do it myself and secondly to avoid using anything but the standard Python modules. It turns out there isn’t much to understand. Say that we have a very simple form, say it’s a login form containing two text entries:

<form method="post" action="/login">
<label for="user_name">username</label>
<input type="text" name="user_name" id="user_name" value="" />
<label for="password">password</label>
<input type="password" name="password" id="password" class="sized" />
<input type="submit" class="button" name="login" value="log in" />
</form>

One way to post this form would be the following:

import urllib
import urllib2

login_data = urllib.urlencode({'user_name' : 'foo', 'password' : 'bar'})
resp = urllib2.urlopen('http://url.for.my.site/login', login_data)

Simple enough, I’d say. urllib2.urlopen automatically switches from GET to POST on the existance of some data.

On most sites a cookie is used to track whether a user is logged in or not. Extending the example above to deal with this and enable subsequent requests to the site as a logged-in user leads us to the CookieJar:

import urllib
import urllib2
import cookielib

cj = cookielib.CookieJar()
opener = urllib2.build_opener(urllib2.HTTPCookieProcessor(cj))
login_data = urllib.urlencode({'user_name' : 'foo', 'password' : 'bar'})
resp = opener.open('http://url.for.my.site/login', login_data)

After this cj will hold all the cookies returned in the response. You can enumerate over them like this:

for c in enumerate(cj):
    print c.name, c.value

Making requests with a cookie c is simple as well, just add c to the cookie jar before making the request:

cj.set_cookie(c)

The cookie jar also has a policy object and a method, set_cookie_if_ok that will set a cookie for a specific request only if the policy allows it. I.e. it seems fairly simple to make sure there is no cookie leakage when making requests to multiple sites. I’ll leaving playing with that for another day though.