therning.org/ magnus

Yesterday I released a new version of dataenc. It’s available on Hackage of course. Summary of (visible) changes:

• implementation of a bunch of new encodings:
  • xxencode
  • quoted-printable
  • python escaping
  • url encoding
• squashing of a bug in the yEncoding implementation that only manifested on 32-bit systems
• an attempt to conform to the guidelines for the Haskell Platform regarding versioning and dependencies

I have written a few posts on debianising haskell packages in the past. Back then I stopped looking at it because of the age of the ghc-related packages in Debian Sid. Things have since improved and I’ve finally found some time to revisit the problem. This isn’t however a technical post, but rather a post on the strategy I think would be best for making Debian the best Haskell development environment in the universe.

In my opinion there is no point in trying to get all packages on Hackage into Debian proper. The few Debian developers (DD) who are interested in Haskell and maintain the existing packages are already terribly short on time and adding further burden would improve the chance of timely availability of GHC in the future. Sure, it would be possible to recruit more DDs with an interest in Haskell (I’ve considered in the past to try to become a DD, but the timing was off on both sides and I decided to postpone it) but it would take time. Maintaining 1000 packages is a huge task, and would require quite a large team I think. I suspect the recruiting would be such that if one were to plot the number of packages the team could maintain as it grew over time, compared to the number of packages on Hackage, then the lines would diverge rather than converge

Instead I think all packages on Hackage should first be debianised and put in a source-only repository. As some packages grow in importance and get added into Debian proper (e.g. a killer-app written in Haskell) then it and its dependencies are removed.

I believe that with this solution it’d be possible to quickly make a large part of Hackage available on Debian while still giving the DDs time to grow the number of “Haskell-friendlies” within Debian.

This of course rests on a few assumptions:

1. It would require a smaller team to maintain source packages than binary packages. At least it would require less time from DDs.
2. There is some space to store the source packages, ideally non-Debian space since that would mean even less involvement required by DDs.
3. There’s a tool available in Debian that makes it easy for end-users to download, compile, and install source packages.
4. There’s a tool that automates the creation of a Debian source package from a Cabal package.

In my optimism I’ve decided to not worry about the two first points and instead focus on the last two. Besides the optimism they also feel like the points I can actually address on my own, I’ll need help with the first two.

The tool in Debian for handling source packages is apt-src. In my opinion it will suffice, but if the developer would address one issue then it would be perfect.

The tool for automating the creation of Debian source packages could be cabal-debian. It is a very good tool, and it seems to be suited for maintaining Debian packages. I only have two objections to it. The first objection I have is that it is almost too well engineered; it does more than is necessary, e.g. there is no need to maintain proper changelogs in the source repo. The second is that it requires all dependencies of a package to be installed when it’s debianised. It is almost certainly the tool for the team charged with maintaining a large set of Cabal-based packages in Debian proper, but for the source-repo it might be too much. Because of that I’ve started hacking on a simplistic tool that I feel would be better suited for the source-repo.

avsm pointed me to ubigraph. So after a bit of XML-RPC hacking (haxr really rocks, it is just so amazingly easy to use) I now have code that sends the graph data over to a ubigraph server.

Of course I had to create a video of it. The video shows the dependencies of just two packages (dataenc and datetime). I tried creating a second video of the dependencies of more packages, but xvidcap isn’t very reliable it seems. One irritating thing is that I run out of filehandles fairly soon, so I couldn’t create a 3d graph of more than about 75 packages (all packages starting with ‘a’ and ‘b’ on Hackage).

It wasn’t why I started playing with Cabal, but after extracting dependencies from a single package I thought struck me that I could extract dependencies from many packages, e.g. hackage, and draw a dependency graph of the result.

The basic idea is to use the code from my earlier post, accumulate dependency information by mapping it over several cabal files. Then convert that information into nodes and edges suitable for building a graph (Data.Graph). That graph is then “graphviz’ed” using Data.Graph.Inductive.Graphviz. Not that since this is performed on Debian Sid I’m using some rather old versions of packagesⁱ.

First off a shortcut for reading cabal files:

readCabalFile = readPackageDescription silent

Once I have a GenericPackageDescription I want to collapse it into a regular PackageDescription (see the comments in my previous post for some details regarding this). Then I extract the package name and its dependencies and package them into a tuple. by mapping this function over a list of GenericPackageDescription I end up with an association list where the key is the package and the value is a list of all its dependencies.

processFile gpd = let
        finPkgDesc = finalizePackageDescription [] Nothing
            "Linux" "X86_64" ("GHC", Version [6, 8, 2] [])
        (Right (pd, _)) = finPkgDesc gpd
        getPackageName (Dependency name _) = name
        nameNDeps = (pkgName . package) &&& (nub . map getPackageName . buildDepends)
    in
        nameNDeps pd

In order to create the graph later on I need a complete list of all nodes. To do this I take all the keys and all the values in the association list, collapse them into a single list, and remove duplicates. To turn this resulting list of packages into a list of LNode I then zip it with a list of integers.

getNodes = let
        assocKeys = map fst
        assocVals = concat . map snd
    in zip [1..] . nub . uncurry (++) . (assocKeys &&& assocVals)

Building the edges is straight forward, but a bit more involved. An edge is a tuple of two integers and something else, I don’t need to label the edges so in my case it is (Int, Int, ()). The list of nodes is basically an association list (an integer for key and a string for value), but I need to flip keys and values since I know the package name and need its node number.

getEdges deps = let
        nodes = getNodes deps
        nodesAssoc = map (\ (a, b) -> (b, a)) nodes
        buildEdges (name, dep) = let
                getNode n = fromJust $ lookup n nodesAssoc
                fromNode = getNode name
            in map (\ t -> (fromNode, getNode t, ())) dep
    in concat $ map buildEdges deps

Now that that’s done I can put it all together in a main function. The trickiest bit of that was to find the size of A4 in incehs

main = do
    files <- getArgs
    gpds <- mapM readCabalFile files
    let deps = map processFile gpds
    let (nodes, edges) = (getNodes &&& getEdges) deps
    let depGraph = mkGraph nodes edges :: Gr String ()
    putStrLn $ graphviz depGraph "Dependency_Graph" (11.7, 16.5) (1,1) Landscape

I downloaded all cabal files from Hackage and ran this code over them. I bumped into a few that use Cabal features not supported in the ancient version I’m using. I was a bit disappointed that Cabal wouldn’t let me handle that kind of errors myself (as I already expressed in an earlier post) so I was forced to delete them manually.

Here’s the output, completely unreadable, I know, but still sort of cool.

1. Yes, I’d be really happy if the transition to GHC 6.10 would finish soon.[back]

A recent “find” in Cabal made me think of this:

In my opinion the final decision of what to do in case of an error belongs in the appliation, not in libraries.

I’m sure there are exceptions to this, but I believe it’s good as a guiding principle when defining APIs.

Here’s the code in Cabal that breaks this rule. Basically, if you write an application and call readPackageDescription on a cabal file with errors then your application is terminated (the call to dieWithLocation ends up calling exitWith). I would much rather it returned an error (e.g. based on Either) so I can decide what to do with bad cabal files myself.

I’ve raised a ticket for it of course

Since the question comes up every now and then on haskell-cafe; why bother with distro-specific packages when we have Cabal? This problem is trivially solved with packaging like Debian’s.

This is going to be a silly post, simply because it is so amazingly simple to do some basic parsing of Cabal files. It turns out to be true even for old versions of Cabal, the code below is for 1.2.3.0 since that’s the version still found in Debian Unstableⁱ

First import the required modules:

import Distribution.PackageDescription
import Data.Version
import Distribution.Verbosity
import Distribution.Version

The a simplistic shortcut, I don’t care about setting flags or pretty much anything else

finPkgDesc = finalizePackageDescription [] Nothing "Linux" "X86_64" ("GHC", Version [6, 8, 2] [])

and I’m only interested in listing the package names of dependencies

getPackageName (Dependency name _) = name

After this it’s possible to do the following in ghci (ex01.cabal is a copy of the Cabal file for dataenc):

> gpd <- readPackageDescription silent "ex01.cabal"
...
> let pd = finPkgDesc gpd
> let deps = either (\_ -> []) (\ (p, _) -> buildDepends p) pd
> map getPackageName deps
["array","base","containers"]

1. That’ll change when GHC 6.10 makes it onto my system. It’s already in Debian, just not on my system since some Xmonad-related packages haven’t been rebuilt yet.[back]

First of all I decided to make this a new post rather then answer in the comment section in my previous post. I think that this way the chance is greater that both David and Jeremy will read this

So, first let’s set the stage… It’s almost midnight and I’ve just spent somewhere between 30 and 45 minutes hunting down packages on Haskell and navigating the directory listings on src.seereason.com. I’ve been sent in the wrong direction once (downloading a Debian package that seems to be from Linspire instead of grabbing the identically named package from the seereason site. I’m tired and slightly frustrated. I run cabal-debian, it works, then I run debuild just to be told that I really should have build cabal-debian itself into a Debian package and installed it that way and I also need another package that isn’t in Debian at all.

Anyway, having slept and let almost 24 pass by I’ve come to my senses somewhat

Here’s what think is a slightly more tempered whishlist:

1. Remove haskell-devscripts-cdbs and include hlibrary.mk in haskell-devscripts.
2. Modify dh_haskell* so that the two snippets for -doc are automatically generated instead (just like the script snippets for -dev already are).

I think it would be easy to stick that in seereason’s repos for public consumption. Then interested people, like me, can start thinking about how to use that as a base to make Debian the best Haskell platform in the world, or at least making it better than Arch

Completely unrelated question for seereason people

I have the following line in my source.list:

deb     http://deb.seereason.com/debian sid-seereason main

Why don’t I see the seereason-keyring package?

So, after receiving several pointers to seereason’s cabal-debian tool I thought I’d take it for a spin.ⁱ

After about 30 minutes of browsing through HackageDB and seereason’s source repos, building and installing, I had finally satisfied all dependencies and the build of cabal-debian succeeded. (Oh, BTW, seereason people, it’s really confusing that you have a package called debian among your source repos, when there already is a different debian package on HackageDB. Please consider renaming it!) I decided to take the tool for a spin on my own package, dataenc.

The result was fairly good. It seems the generated files depend on some packages that either aren’t in Debian or that the people at seereason have modified. With the following changes to the generated files I was happy with the contents of ./debian and I successfully built an (almost) autogenerated Debian package:

1. Download hlibrary.mk into ./debian.
2. Modify ./debian/rules so that hlibrary.mk is loaded from $(CURDIR)/debian/hlibrary.mk.
3. Delete ./debian/libghc6-dataenc-doc.post*.
4. Remove cabal-debian and haskell-devscripts-cdbs from the build dependencies in ./debian/control.

I’ll try to find the time to put those changes into cabal-debian itself. Then I’d have a rather nice tool for building Debian packages automatically.

1. I did pull down autobuilder as well, but didn’t feel I had enough time to explore it tonight.[back]

I’ve just spent the last hour or so ironing out the details required to automate the building of cabalised Haskell packages for Debian. At the same time I also built Debian packages for 5 Haskell packages (test-framework and its dependencies).

These are the basic steps I’ve followed:

1. Extract the tar-ball in the working directory.
2. Rename the directory to match the form haskell-X-0.1.
3. Re-tar the directory into an archive named haskell-X_0.1.orig.tar.gz.
4. Change into the package directory.
5. Run Cabal’s configure command followed by register --gen-pkg-conf=pkg.conf.

6. Modify the generated file and turn it into a .ini-style file:

   echo '[conf]' > pkg.conf.tmp; cat pkg.conf >> pkg.conf.tmp; mv pkg.conf.tmp pkg.conf
   

7. Use Python to extract the dependencies:

   python << EOF
   from ConfigParser import ConfigParser
   cfg = ConfigParser()
   cfg.read('pkg.conf')
   deps = cfg.get('conf', 'depends').split()
   for d in map(lambda s: s.rsplit('-', 1)[0].lower(), deps):
       print d
   EOF
   

8. Clean up by rm pkg.conf and running Cabal’s clean command.

9. Create the debian directory.
10. Create debian/compat: echo 5 > debian/compat

11. Create debian/changelog, it should be something like:

    haskell-X (0.1-1) unstable; urgency=low
    
      * Automatic build.
    
     -- Magnus Therning <magnus@therning.org>  Mon, 19 Jan 2009 22:35:00 +0000
    

12. Create debian/control. This is by far the most time consuming part to do manually. It’s also a bit long so I’ll leave it out of this post. Take a look in my APT repo if you are interested in the details.

13. Copy hlibrary.mk into debian/. At some point it will be included in some Debian package (bug #462482) but for now it can be found here.

14. Create debian/rules with

    #!/usr/bin/make -f
    include $(CURDIR)/debian/hlibrary.mk
    

    and make it executable, chmod +x debian/rules.

15. Done. Run debuild.

My intention is to at some point encode these steps in a tool so that I can spew out Debian packages for all interesting packages on HackageDB at the press of a button. Of course that tool will be written in Haskell, and it sure won’t rely on Python for any part of the process

Readers who are familiar with the rules for Debian packages will notice that this won’t create well-formed packages. i’m not particularly interested in getting these packages into Debian proper, I see the package generated this way living in an unofficial repo. So there is no need to make sure they are acceptable for inclusion in the distribution itself. The resulting packages might offer a good starting point for someone who feels that a package deserves to become official. I also suspect that a tool that generates fully compliant packages, and is able to deal with keeping them up-to-date, is too big a task for me. I’m also not sure it really is worth it, better to keep things as simple as possible in order to deal with the amazing rate of new releases on Hackage.