ctypes and flexible (zero-length) arrays

Dear lazyweb, I’ve been racking my brain trying to get information out of the following C struct in a nice way when using ctypes:

struct Foo
{
    unsigned int length;
    char name[];
};

I wrote a little C function to get instance of the struct easily into python:

struct Foo *
put_name(char *name)
{
    struct Foo *foo = malloc(sizeof(struct Foo) + strlen(name));
    foo->length = strlen(name);
    memcpy(foo->name, name, foo->length);
    return(foo);
}

Now, how do I actually get the full contents of Foo.name in python? The only way I could think of that actually works is to create a dummy-struct type in order to get the length out and then use it to dynamically create a new sub-class of ctypes.Structure, then create an instance based on the address of what was returned. I think the following shows it pretty clearly:

class FOO(Structure):
    _fields_ = [('length', c_uint), ('name', c_char * 0)]
 
liblib = cdll.LoadLibrary('./_build/liblib.so')
c_put_name = liblib.put_name
c_put_name.argtypes = [c_char_p]
c_put_name.restype = POINTER(FOO)
 
def put_name(str):
    f = c_put_name(str)
    K = type('FOO%s' % f.contents.length, (Structure,),
                        {'_fields_' : [('length', c_uint), ('name', c_char * f.contents.length)]})
    return K.from_address(addressof(f.contents))

I still think there ought to be some other way that ‘feels’ nicer. I mean the use of “short arrays” (declared as here with [], or zero size as supported by GCC, or the more portable array of size one) seems to be common enough to warrant some support from ctypes, right?

Any suggestions for improvements?

Kid’s play with HTML in Haskell

In my ever-continuing attempts to replace Python by Haskell as my language of first choice I’ve finally managed to dip a toe in the XML/HTML sea. I decided to use the Haskell XML Toolkit (HXT) even though it’s not packaged for Debian (something I might look into doing one day). HXT depends on tagsoup which also isn’t packaged for Debian. Both packages install painlessly thanks to Cabal.

As the title suggests my itch wouldn’t require anything complicated, but when I’ve previously have looked at any Haskell XML library I’ve always shied away. It all just looks so complicated. It turns out it looks worse than it is, and of course the documentation is poor when it comes to simple, concrete examples with adequate explanations. HXT would surely benefit from documentation at a level similar to what’s available for Parsec. I whish I were equipped to write it.

Anyway, this was my problem. I’ve found an interesting audio webcast. The team behind it has published around 90 episodes already and I’d like to listen to all of them. Unfortunately their RSS feed doesn’t include all episodes so I can’t simply use the trusted hpodder to get all episodes. After manually downloading about 20 of them I thought I’d better write some code to make it less labour-intensive. Here’s the complete script:

module Main where

import System.Environment
import Text.XML.HXT.Arrow

isMp3Link = (==) "3pm." . take 4 . reverse

myReadDoc = readDocument [(a_parse_html, "1"), (a_encoding, isoLatin1),
    (a_issue_warnings, "0"), (a_issue_errors, "0")]

myProc src = (runX $ myReadDoc src >>> deep selectMp3Links >>> getAttrValue “href”)
    >>= mapM_ putStrLn

selectMp3Links = hasName “a” >>> hasAttrValue “href” isMp3Link

main = do
    [src] <- getArgs
    myProc src

The thing that took by far the most time was finding out that hasAttrValue exists. I’m currently downloading episodes using the following command line:

curl -L $(for h in $(runhaskell get_mp3links.hs 'http://url.of.page/'); do \
    echo '-O' $h; done)

Yet another set of itches where Haskell has displaced Python as the utensil used for scratching. :-)

Saddle, two SDDL related tools

I’ve just uploaded two small tools that makes it a little easier to deal with SDDL (Security Descriptor Description Language, this is a good resource for SDDL):

  1. saddle-ex - “extract” the security descriptor for a number of different kinds of objects
  2. saddle-pp - a pretty printer for an SDDL string

Full build instructions are included. Download the archive here.

The tools are written (mostly) in Haskell and I’ll probably upload it all to hackage at some point in the future. I’m holding out for a fix to ticket 2097 since a fix for it will allow me to add a third tool which I feel belong in the package.

Suggestions for improvements are always welcome, patches even more so :-)

Omnicodec released

Four days ago I uploaded omnicodec to Hackage. It’s a collection of two simple command-line utilities for encoding and decoding data built on the dataenc package. I’m planning on building a Debian package for it as soon as I find the time.

Haskell on Windows

Recently I’ve had reason to write a few tools for Windows. Nothing complicated, and since there was nothing “mission critical” (there hardly ever is for me :-) ) I decided to try to write them in Haskell rather than Python or C/C++. This post contains some sort of recollection of my experience so far.

First off I should probably disclose that I don’t particularly like Windows. Besides feeling more at home on a Unix (Linux) system I find that Windows also upsets me on a very basic level. If I spend a whole day at work in Windows I tend to go home more stressed and irritated than after a day in Linux. Part of this is probably due to my familiarity with Linux. My hope was that the combination of the immense joy of programming Haskell and my loath of the Windows platform I would end up with something that at least was bearable. It turns out I did.

My setup

Besides installing the pre-built GHC I also installed Visual Studio Express more later to explain the need for it. I don’t like the Visual Studio environment so I installed CMake and jEditi. Last but not least I installed Console, it addresses the fact that Windows’ “DOS-box” is an amazing piece of crap.

FFI on Windows

First, GHC comes with a number of Windows-specific modules and they cover an impressive number of the Win32 API functions. The Win32 API is simply huge and one will sooner or later come find the need to add to what those modules offer. Second, GHC is able to compile C code by delegation to MinGW, however MinGW doesn’t completely cover the Win32 API either and if one is unlucky MinGW isn’t enough for the task at hand. Of course that’s exactly what happened to me. So, I installed Visual Studio Express, but since I don’t like VS that much and I didn’t want VS solutions in my darcs repository I decided to use CMake to build a library that I then instruct GHC, through a Cabal file, to link with my Haskell code. The end result is that building requires a few extra steps before the familiar configure/build sequence. It works remarkably well really, and I’ll definately use that strategy again if I need to.

There is a benign warning thrown when linking the final executable though:

Warning: .drectve `/manifestdependency:"type='win32' name='Microsoft.VC90.CRT'
version='9.0.21022.8' processorArchitecture='x86' publicKeyToken='1fc8b3b9a1e18e3b'"
/DEFAULTLIB:"uuid.lib" /DEFAULTLIB:"uuid.lib" /DEFAULTLIB:"MSVCRT" /DEFAULTLIB:"OLDNAMES" '
unrecognized

It seems that it might be possible to skip the extra steps in the future if Cabal ticket #229 is addressed.

Windows System modules

Admittedly I haven’t been using much of the functionality in these modules, but it happens that the only function I needed had a bug that results in a segmentation fault, sometimes. See GHC ticket #2097 for some details. I guess this confirms my impression of most Haskell hackers being Linux/BSD/Unix users.

Conclusions

I won’t drop Haskell on Linux anytime soon, but I’ve found Haskell on Windows to be possible. As I suspected when I started out, Haskell does make Windows somewhat easier to bear.

  1. ordinarily I would install Vim but I thought I’d try out another editor. Vim on Windows is somewhat “leaky”, i.e. it sometimes shows behaviour that betrays its Unix roots.[back]

Rotonyms in Haskell

I have to say rotonyms seem kind of pointless, but since there is a post with a Python solution I thought I’d make an attempt at a solution in Haskell. Of course I didn’t manage on my own, I received some much needed help from Cale and dmwit on #haskell.

First some rotation functions, as far as I know there’s no built-in rot13 encoding in Haskell:

rot13Char c = chr ( (ord c - ord 'a' + 13) `mod` (ord 'z' - ord 'a' + 1) + ord 'a')
rot13 = map rot13Char

In order to ease working with rotonyms I introduced a type to represent one. This turned out to pay off later on.

data Rotonym = R String

It would be easy to let the compiler derive Show, but I wanted to mimick the output produced by Will’s solution in Python:

instance Show Rotonym where
    show (R w) =
        let r = rot13 w
        in (min w r) ++ "\t" ++ (max w r)

Now I’m ready for the meat of the matter, reading the word list, pick out the rotonyms and finally print them. Following Will’s cue I represent the word list as a set of strings (importing Data.Set as S), this also eliminates duplicates.

main = do
    words <- liftM lines $ readFile "WORD.LST"
    let wordset = S.fromList words
        isRotonym w = S.member (rot13 w) wordset
        rots = S.fromList [R w | w <- words, isRotonym w]
    mapM_ (putStrLn . show) (S.elems rots)

Sticking Rotonym instances in a set requires them to be instances of Ord and Eq:

instance Eq Rotonym where
    (R w1) == (R w2) =
        let r1 = rot13 w1
            r2 = rot13 w2
        in (w1 == w2 && r1 == r2) || (w1 == r2 && w2 == r1)

instance Ord Rotonym where
    compare (R w1) (R w2) =
        let r1 = rot13 w1
            r2 = rot13 w2
        in compare (length w1, (min w1 r1)) (length w2, (min w2 r2))

dmwit pointed out that what is actually going on here is taking an intersection of the the words and their rotated counterparts. This means the main could be written

main = readFile "WORD.LST" >>=
        mapM_ (putStrLn . show) . S.toList . S.map R . findRotonyms . S.fromList . lines
    where findRotonyms ws = ws `S.intersection` (S.map rot13 ws)

Another idea that dmwit came up with was to introduce a class for rot13:

class Rot13 a where
    rot13 :: a -> a

Char is an instance of that class with the expected implementation of rot13:

instance Rot13 Char where
    rot13 c = chr ((ord c - ord 'a' + 13) `mod` (ord 'z' - ord 'a' + 1) + ord 'a')

If a is an instance of Rot13 then list of a is as well, again with the expected implementation:

instance (Rot13 a) => Rot13 [a] where
    rot13 = map rot13

It is possible to intersect lists (Data.List.intersect) but it turned out to be painfully slow, so back to sets again. A set of a is an instance of Rot13:

instance (Ord a, Rot13 a) => Rot13 (S.Set a) where
    rot13 = S.map rot13

Finding all rotonyms is then straight forward:

main = readFile "WORD.LST" >>=
        mapM_ (putStrLn . show) . S.toList . S.map R . findRotonyms . S.fromList . lines
    where findRotonyms ws = ws `S.intersection` (rot13 ws)

Back to listening to pauldotcom

I just began listening to episode 95 of pauldotcom and was glad to hear that they enjoyed my email. Here’s the complete email I sent them:

Well, something must have changed since I last communicated with you (see http://therning.org/magnus/archives/257). I’m not sure what though. I heard you when you were on the Linux Reality audio cast and thought I’d check you out again, just to see what you were up to. Well, episode 92 (both parts) was bloody brilliant, episode 93 was good too, and now I’m halfway into episode 94. I have no recollection of the earlier episodes being this organised and good. At some point when I wasn’t listening you must have learnt to rock!

I enjoy the tech segment. The amount of banter is down and the episodes move along a lot more than I remember. No offence to Twitchy, but I’m not sad he isn’t as involved any more, you know, Kramer is brilliant but Seinfeld just wouldn’t be a good show if he were in each and every scene. Twitchy has more of a “celebrity guest” personality… The only criticism I have, and this is pushing it I know; given my walk to work I’d prefer each episode to be around 60 minutes, rather than 80-90 ;-)

Keep it up!

/M

PS I’m planning on posting this email on my blog. I’ll put any reply from you on there as well.

Reading my email on the show sure beats any reply they could have sent by email :-) At some point I have to go back and check out the other podcast I stopped listening to

BSD or GPL?

Today I had a long and interesting discussion about what is “more free”, GPL or the BSD license. I did a little searching on the internet afterwards and found the following posts quite illuminating:

I’ll refrain from saying where I come down on this issue :-)

The KLM way of respecting privacy…

I just received an email from KLM UK. I have probably given them my email address and thereby my consent to them sending me “informative emails”. Still, I think their words sound a bit hollow when the link to unsubscribe takes me to doubleclick:

If you do not wish to receive future communication from us click here to unsubscribe. KLM is firmly committed to respecting your privacy. We don’t share your information with any third party without your consent.

Shame on you!

A visit to the golden prison

My wife managed to win an iPod Touch a the office Christmas party. I got more excited than her; she actually suggested we put it on eBay unopened, of course I simply had to open it and play a little. After a weekend I came to the conclusion that, to me, it was a piece of useless crap that we should try to sell ASAP. Luckily we managed to sell it on to someone who’s recently had a large sip of the Apple Kool-Aid. Here’s what I found out anyway, let’s start with the least bad.

The UI

It’s cute, very cute. Clicking icons, typing in the “keyboard” all very enjoyable. I tried out the zooming in Safari, beautiful. Then the cracks started showing, I can’t zoom everywhere, suggesting that it’s implemented on an application level. Why not provide that on a system level instead, so that all applications have it automatically? I also found it really irritating that there’s no copy-paste (apparently a common gripe). After I broke the jail of the 1.1 firmware and installed a audio cast client I had to use a pen and paper to copy URLs from Safari in order to subscribe. Poor!

Browsing the web

Getting network connectivity was a breeze. The UI of Safari is cute and the zoom is really handy. The first thing that irritated me was that there’s no way of saving things. Yes, there’s WiFi available in many large cities nowadays, but there’s still a lack of wireless connectivity outside of cities. So, my hope of being able to look for things while online and read while offline was a no-starter. I also would have liked to download videos and watch them later, on the other hand, and this is such a ridiculous mistake by Apple since it basically breaks the web, there’s no support for flash. Yes, the format that basically everybody uses to post videos online is not supported! Oh, but there’s YouTube you say? Well, on to that then…

The YouTube application

The only way of watching videos on YouTube is through this application. Surprisingly it’s not possible to use Safari to browse around using YouTube’s well-known web interface and find interesting things to watch that way. No flash support, remember?

On top of this there is no way of saving a video to watch it later. Again, listen Apple, there are areas that don’t have WiFi connectivity in the world!

Loading things on to the iPod Touch

I found this to be most disappointing. The only way to get stuff, music/video/audio cast, on to it one must use iTunes. There are rumours of support in bleeding edge libgpod, but by this time I didn’t bother with trying that out. What I did try was breaking the jail, installing OpenSSH, mounting it using SSHFS just to find out I only had read access in the versions of gtkpod/libgpod that’s in Debian Sid. Then I tried installing iTunes in WINE, but gave up after a few hours of trying to get that working. The next attempt was to use VMWare, using an evaluation license for Workstation I got as far as installing iTunes just to find out that it wouldn’t accept the iPod Touch when I plugged it into the USB port. After all of this, and my other findings, I didn’t think it’d be worth compiling libgpod from source.

Other bits and pieces

I’ve heard the calendar and contacts can only be synced to a Mac. Of course I couldn’t try it, since I don’t have a Mac.

The iTunes application on the device itselfi only contains just enough to spend money on the iTunes store. No way to subscribe to audio casts that I could findii.

Conclusions

I’ve never thought quite as much about the words “proprietary” and “technological silo” as I did while playing with the iPod Touch. The experience has put me off all Apple products, sure they are beautiful and cute, but to me they aren’t usable.

  1. I can’t remember if it were on there when I first opened it of if it only appeared after upgrading to firmware version 1.2.[back]
  2. I could always set up a personal audio cast containing my music had it been possible.[back]