therning.org/ magnus

I bet you have at least one silly little thing at work that, whenever it happens, you let out a sigh, maybe roll your eyes and whish that everyone would use a proper operating system. A few days I finally decided to do something about one of my things like that. At work, Windows users will at times for some strange reason, manually create directories inside their work area, even though the directories actually are under version control. Invariably they get the case wrong and due to an onfortunate combination of case insensitive filesystem on the client (Windows) and a version control system the cares about case (Perforce). This results in files ending up all over the place even though they belong in the same directory. The Windows users are none the wiser, they simply don’t see the problem. Since I use a sane system (Linux) I do notice, and when I see it I sigh and roll my eyes.

Here’s my take on solving the problem:

module Main where

import Control.Monad
import Data.Char
import System.Directory
import System.Environment
import System.FilePath
import System.IO.HVFS.Utils
import System.Posix.Files

data IFile = IFile
    { iFileIPath :: FilePath
    , iFileIName :: FilePath
    , iFileFull :: FilePath
    } deriving (Show)

toIFile :: FilePath -> IFile
toIFile fp =  IFile path file fp
    where
        path = map toLower $ takeDirectory fp
        file = map toLower $ takeFileName fp

listFilesR :: FilePath -> IO [IFile]
listFilesR path =
    recurseDir SystemFS path >>=
    filterM doesFileExist >>=
    mapM (return . toIFile)

linkFile :: FilePath -> IFile -> IO ()
linkFile dest ifile = do
        createDirectoryIfMissing True newDir
        createLink (iFileFull ifile) newFile
    where
        newDir = normalise $ dest </> (iFileIPath ifile)
        newFile = newDir </> (iFileIName ifile)

main :: IO ()
main = do
    args <- getArgs
    listFilesR (args !! 0) >>= mapM_ (linkFile $ args !! 1)

Yes, this is the code I wrote in Literate Haskell, but I think I’d better not disclose my rant against clueless Windows users publically

I’ve just uploaded two small tools that makes it a little easier to deal with SDDL (Security Descriptor Description Language, this is a good resource for SDDL):

1. saddle-ex - “extract” the security descriptor for a number of different kinds of objects
2. saddle-pp - a pretty printer for an SDDL string

Full build instructions are included. Download the archive here.

The tools are written (mostly) in Haskell and I’ll probably upload it all to hackage at some point in the future. I’m holding out for a fix to ticket 2097 since a fix for it will allow me to add a third tool which I feel belong in the package.

Suggestions for improvements are always welcome, patches even more so

Recently I’ve had reason to write a few tools for Windows. Nothing complicated, and since there was nothing “mission critical” (there hardly ever is for me ) I decided to try to write them in Haskell rather than Python or C/C++. This post contains some sort of recollection of my experience so far.

First off I should probably disclose that I don’t particularly like Windows. Besides feeling more at home on a Unix (Linux) system I find that Windows also upsets me on a very basic level. If I spend a whole day at work in Windows I tend to go home more stressed and irritated than after a day in Linux. Part of this is probably due to my familiarity with Linux. My hope was that the combination of the immense joy of programming Haskell and my loath of the Windows platform I would end up with something that at least was bearable. It turns out I did.

My setup

Besides installing the pre-built GHC I also installed Visual Studio Express more later to explain the need for it. I don’t like the Visual Studio environment so I installed CMake and jEditⁱ. Last but not least I installed Console, it addresses the fact that Windows’ “DOS-box” is an amazing piece of crap.

FFI on Windows

First, GHC comes with a number of Windows-specific modules and they cover an impressive number of the Win32 API functions. The Win32 API is simply huge and one will sooner or later come find the need to add to what those modules offer. Second, GHC is able to compile C code by delegation to MinGW, however MinGW doesn’t completely cover the Win32 API either and if one is unlucky MinGW isn’t enough for the task at hand. Of course that’s exactly what happened to me. So, I installed Visual Studio Express, but since I don’t like VS that much and I didn’t want VS solutions in my darcs repository I decided to use CMake to build a library that I then instruct GHC, through a Cabal file, to link with my Haskell code. The end result is that building requires a few extra steps before the familiar configure/build sequence. It works remarkably well really, and I’ll definately use that strategy again if I need to.

There is a benign warning thrown when linking the final executable though:

Warning: .drectve `/manifestdependency:"type='win32' name='Microsoft.VC90.CRT'
version='9.0.21022.8' processorArchitecture='x86' publicKeyToken='1fc8b3b9a1e18e3b'"
/DEFAULTLIB:"uuid.lib" /DEFAULTLIB:"uuid.lib" /DEFAULTLIB:"MSVCRT" /DEFAULTLIB:"OLDNAMES" '
unrecognized

It seems that it might be possible to skip the extra steps in the future if Cabal ticket #229 is addressed.

Windows System modules

Admittedly I haven’t been using much of the functionality in these modules, but it happens that the only function I needed had a bug that results in a segmentation fault, sometimes. See GHC ticket #2097 for some details. I guess this confirms my impression of most Haskell hackers being Linux/BSD/Unix users.

Conclusions

I won’t drop Haskell on Linux anytime soon, but I’ve found Haskell on Windows to be possible. As I suspected when I started out, Haskell does make Windows somewhat easier to bear.

1. ordinarily I would install Vim but I thought I’d try out another editor. Vim on Windows is somewhat “leaky”, i.e. it sometimes shows behaviour that betrays its Unix roots.[back]

The PaiMei page says that you need Python 2.4, which turns out to be true due to it shipping with a compiled for Python 2.4 version of pydasm. Of course it’s possible to compile pydasm yourself, it’s even fairly easy just as long as you have the correct version of Visual Studio installed. You could also use the utterly unofficial build I’ve made available here. Not even I know if I’m to be trusted though ;-). Use at own risk, and all that.

So, the steps are:

1. Install Python 2.5 off python.org
2. Install pydasm
3. Install PaiMei
4. Remove PaiMei’s version of pydasm (c:\python25\site-packages\paimei\pydasm.pyd) to be sure the correct one is used.

Oh, I probably should say that I’ve only been using the core functionality of PaiMei (pydbg and pydbg_core). There may be other dependencies on Python 2.4 in PaiMei that I haven’t stumbled upon!

I was disappointed to find the Debugging Tools for Windows contains precious few code examples that helps in understanding how to use the API. Especially if one is interested in using a symbol server to get symbols. Here are my findings:

1. Make sure to use the correct version of dbhelp.{dll,h}. The ones that ship with Visual Studio or the Platform SDK aren’t the same as the one in the Debugging Tools (they lack quite a few functions).

2. SymSetSearchPath doesn’t seem to accept the same syntax as _NT_SYMBOL_PATH. I gave up on using the former and resigned to having to set the latter in the environment.

3. You want to use SymSrvGetFileIndexes to find id, two and three that you pass to SymFindFileInPath.

4. Microsoft’s public symbol server only offers public symbols, what this means is that your SymEnumSymbolsProc will recieve SYMBOL_INFO structs with Tag set to SymTagPublicSymbol and Flags set to 0. Not very helpful if you want to find all functions in a DLL.

Added 2007-04-17 / 09:13

You also want to use the following little snippet to include dbghelp.h:

#define __out_xcount(x)
#define _NO_CVCONST_H
#include <dbghelp.h> 

I always find myself going through the same motions when trying to programming on Windows: excitement, bewilderedness, frustration, relief. It’s exciting to find new libraries and frameworks that seem to deliver exactly the functionality I require. I feel bewildered because I don’t think I’ve ever come across a Windows C/C++ API that immediately makes sense to me. Then follows a time of frustration, often a rather long period too, when I try to use the library/framework to solve the problem I have. At some point I hit that stage where my little project is debugged into behaving properlyⁱ and a sense of relief comes over me.

One thing that never ceases to amaze me is how many small surprising things there are lurking just under the hood in Windows:

1. Want to print an error message? GetLastError gives you an error value and FormatMessage whips it into a nice printable string. Take a good long look at FormatMessage. Where is the convenience function a lá strerror?

2. Another thing is the surprising order of paths that is searched for DLLs. By putting PATH so far down the list and completely leaving out an equivalent of LD_LIBRARY_PATH they actively encourage developers to copy DLLs into the home dirs of executables. I suspect this is inevitable given the DLL-hell phenomenon on Windows. It’s nonetheless extremely irritating when developing against a non-standard DLL (i.e. one that isn’t installed in \windows\system32).

3. The utter confusion I experience when trying to figure out just where to find the correct framework to use. There is considerable overlap between Visual Studio and the Windows Platform SDK. To add more confusion there are sometimes other frameworks that overlap the both of them, e.g. Debugging Tools for Windows provides dbghelp.{dll,h}, both of which are provided in slightly different versions in the other placesⁱⁱ.

4. The lack of fixes for known issues, e.g. the version of dbghelp.h included in Debugging Tools for Windows can’t be included as is because it lacks the definition of a macro. The webpage announcing version 6.6.7.5 was updated 18 July, 2006. One would think that gives Microsoft ample time to address the issue, but no such luck.

Well, that’s enough of ranting for one night…

1. Through experience I’ve come to the conclusion that it isn’t worth the time and effort to try to fit Microsoft solutions into some logical framework. I’d argue that’s true for most closed-source solutions.[back]
2. A tip, make sure to use the ones that comes from Debugging Tools for Windows![back]

An excellent paper on the cost of DRM in Windows Vista has been making its rounds on the internet for a few weeks now. The topic’s been picked up by Security Now (episode 73 and episode 74). The former gives a nice background to the technical side and the latter has Peter Gutman, the author of the paper, as a guest.

This has triggered my writing something about my thoughts on the topic of DRM. Now, I don’t consider myself an expert on this topic. I’ve formed most of my opinions on DRM by reading things like Cory Doctorow’s excellent talk at Microsoft Research and the Darknet paper from Microsoft. I’ve also worked for a large consumer electronics company for almost 5 years, of which the last 18 months to 2 years on security-related issues in consumer devices.

Why do the consumer electronics companies do DRM?

For the last 50 years consumers have bought new devices for the simple reason that the newer devices had more features, were faster, had better resolution, better audio… in short this year’s models were better than last year’s models. So, why on earth would these companies be interested in bringing out models that are fundamentally flawed through DRM? The easy answer is Hollywood… but as a professor of mine used to say, “Every difficult problem has an easy answer… which is wrong”. Hollywood isn’t the answer in my opinion. Hollywood is only a convenient scape goat. The real answer is format control.

A company that controls a format makes money even when a competitor sells a device. Just think of the patent Philips and Sony had on CDs. That patent pulled in money on every CD sold, worldwide. Talk about a gravy train. Nowadays content formats involve a lot of companies and I guess it’s less lucrative because license money will have to be shared between more companies (just think of the MPEG group). DRM is still a fairly new area of standardisation and there’s a good chance of cashing in even more than on the format itself, especially if DRM is written into lawⁱ.

Consumer electronics companies and the broadcast flag

A while back the broadcast flag was beaten in the US. A court found that the FCC didn’t have the authority to introduce such a flag and the US was saved. At least for now. What wasn’t reported so widely on was the fact that when the broadcast flag was put on the table there was an outcry among the consumer electronics companies (a few other companies joined in as well). No, don’t be fooled, they weren’t considering the consumer, they weren’t interested in keeping TV the way it was. No, they were outraged because the suggested broadcast flag allowed only one DRM system. A system controlled by 5C (if I remember correctly). Companies not in the core group were facing extortionate licenses (basically giving up all IP to the core companies). No wonder they were outraged. Intense lobbying of the FCC followed and the outcome was that a set of DRM technologies would be “legal” in the US. That’s where the consumer electronics industry spent their time and money. They were fighting the possibility of 5C gaining a strangling grip in the market rather then stand up and try to do the right thing which would have been to work to make it possible to bring to market the best possible devices.

Consumer electronics companies and security

When working on security at the research branch of a consumer electronics company I quickly found myself “attached” to DRM-related projects. That was the only place where they were interested in security at all. Of course they weren’t interested in keeping consumers safe in a future where tellys have internet connections. No, the interest was solely in keeping customers out of the telly, preventing them to do interesting things with the boxes they bought. So, who paid for this sort of research project? The IP/standardisation department, that’s who. They practically poured money into DRM projects while the parts of the company that actually made devices showed little interest. (On a personaly note I have to admit that this aspect of security was one of the reasons why I left the company.)

End note

Well, I hope I’ve made some sense and that I’ve added something to the discussion about DRM that currently is taking place. Peter Gutman has done a great job in making people aware of it and I’m looking forward to see what happens once Vista really hits the homes. I’m of course hoping that there is broad disapproval and that Vista does appallingly based solely on its DRM.

1. This brings me to a rather paranoid theory of mine, involving the “unholy trinity” of software patents, DMCA-like laws and DRM that can be used to explain some companies’ behaviour. That would probably have to be the topic of another post though.[back]

I stumbled on the function QueryWorkingSet today. It seems to be usable for solving a problem I have… What’s really amazing about this function is its totally braindead signature:

BOOL QueryWorkingSet(
  HANDLE hProcess,
  PVOID pv,
  DWORD cb
);

Please look beyond the weird Microsoft-isms with type names and the idiotic Hungarian notation. What does it do? Basically it takes a buffer (pv) of a certain length (cb) and tries to stuff some information in it. So, how big do we have to make the buffer? Who knows?

Microsoft seems to have a desire to force programmers to make 2 calls to each function that returns data in a buffer. Functions don’t take an int as the bufffer’s length, instead they take int *. The first call would pass in 0 as length (or rather a pointer to a variable that’s set to 0). The call would fail, but the value of the variable would be set to the required size of the buffer for a successful call.

So, back to QueryWorkingSet. How does one find out the required size of the buffer pv? The only way seems to be repeated calls with progressively larger buffers. Brilliant API Microsoft!

Some interesting articles on Net Neutrality by David Ernst and Andy Kessler. I’m slowly starting to lean towards letting the market sort this out without government intervention. The idea of labelling is especially appealing.

That something’s wrong in Microsoft is rather obvious given the amazing delay in releasing Vista/Longhorn (whatever it’s called this week). After listening to the LQ podcast I tracked down the MSDN blog entry mentioned in it, Broken Windows Theory. It’s an interesting look behind the curtain.

Look here to have a good laugh at RIAA and MPAA.

I complained earlier about VMWare Player not allowing me to replace my Windows boxen at work. It seems VMWare took my complaints seriously (yeah right), by releasing VMWare Server they’ve made my dream possible.

I have to admit that their releasing VMWare Server is surprising, and since it allows creation of new images it makes the limitation of VMWare Player rather strange. Well, I’m not complaining. I’ve finally been able to purge Windows from my desk. From today on all my Windows boxen will be VMWare images accessed from my Linux machine. Brilliant. Thank you VMWare!