therning.org/ magnus

I’ve read very little about Unicode before but today I had the questionable pleasure of delving a bit deeper into it. Mind you, it still feels like I’ve just dipped a foot in the water, but before today I had only dipped a single toe.

Especially I was interested in the URI encoding (“percentage encoding”) and Unicode. According to RFC 3986:

When a new URI scheme defines a component that represents textual data consisting of characters from the Universal Character Set [UCS], the data should first be encoded as octets according to the UTF-8 character encoding [STD63]; then only those octets that do not correspond to characters in the unreserved set should be percent- encoded.

Of course this particular document if fairly new (January 2005) so I bet there are quite a few URI codecs out there that don’t behave this way yet. Another interesting detail is that Microsoft long has supported a special URI encoding especially suited for dealing with UCS-2ⁱ which takes the form %uhhhh. E.g. the character ‘A’ would be %41 according to the standard encoding, using Microsoft’s encoding it looks like %u0041. So far it’s quite straight forward but then enters something strange in Unicode; compatibility characters. They do make certain sense when they are combinations of a base character and some sort of marker (I’m not sure I’m using the right terminology here), e.g. the character ‘å’ can be constructed in two ways, either using the code point U+00E5 or by combining an ‘a’ (U+0061) and the “combining diacritical mark” ‘ ̊’ (U+030A). Of course comparing these two characters which are completely differently encoded while still having exactly the same semantics is a bit of a problem. That’s solved by canonicalisation, which there are two standards for. I didn’t bother going further into that, because my real problem, the reason why I started all of this was that there are compatibility characters for something called “Halfwidth and Fullwidth Forms” (block FF01–FFEF). This block contains some non-latin characters and then it makes sense, but for some strange reason all printable characters in the Basic Latin block (0000–007f) is present as “fullwidth forms” as well. The reason for this is unclear to me and I’d really love an explanation. The result of this is that there apparently is some confusion just what to do with these “fullwidth forms” when decoding them, in some cases they are treated just like their “halfwidth form” cousins in the Basic Latin block. The end result is that on Microsoft products ‘A’ can also be encoded as %uff21.

While reading about Unicode I always have to remind myself that “for every complex problem, there is a solution that is simple, neat, and wrong”. I simply can’t help but think “this is so complicated, there must be an easier solution”…

Re-reading this post I realise there isn’t much of a point to it, besides possibly that writing (or talking) about something always helps my understanding of it. Please let me know if my understanding of Unicode or URI encoding is wrong…

1. I suspect this is connected to Microsoft’s love for UCS-2 in other areas of their operating system.[back]

Only a monopolist feeling the ground trembling under his/her feet resorts to this sort of “research”.

I always find myself going through the same motions when trying to programming on Windows: excitement, bewilderedness, frustration, relief. It’s exciting to find new libraries and frameworks that seem to deliver exactly the functionality I require. I feel bewildered because I don’t think I’ve ever come across a Windows C/C++ API that immediately makes sense to me. Then follows a time of frustration, often a rather long period too, when I try to use the library/framework to solve the problem I have. At some point I hit that stage where my little project is debugged into behaving properlyⁱ and a sense of relief comes over me.

One thing that never ceases to amaze me is how many small surprising things there are lurking just under the hood in Windows:

1. Want to print an error message? GetLastError gives you an error value and FormatMessage whips it into a nice printable string. Take a good long look at FormatMessage. Where is the convenience function a lá strerror?

2. Another thing is the surprising order of paths that is searched for DLLs. By putting PATH so far down the list and completely leaving out an equivalent of LD_LIBRARY_PATH they actively encourage developers to copy DLLs into the home dirs of executables. I suspect this is inevitable given the DLL-hell phenomenon on Windows. It’s nonetheless extremely irritating when developing against a non-standard DLL (i.e. one that isn’t installed in \windows\system32).

3. The utter confusion I experience when trying to figure out just where to find the correct framework to use. There is considerable overlap between Visual Studio and the Windows Platform SDK. To add more confusion there are sometimes other frameworks that overlap the both of them, e.g. Debugging Tools for Windows provides dbghelp.{dll,h}, both of which are provided in slightly different versions in the other placesⁱⁱ.

4. The lack of fixes for known issues, e.g. the version of dbghelp.h included in Debugging Tools for Windows can’t be included as is because it lacks the definition of a macro. The webpage announcing version 6.6.7.5 was updated 18 July, 2006. One would think that gives Microsoft ample time to address the issue, but no such luck.

Well, that’s enough of ranting for one night…

1. Through experience I’ve come to the conclusion that it isn’t worth the time and effort to try to fit Microsoft solutions into some logical framework. I’d argue that’s true for most closed-source solutions.[back]
2. A tip, make sure to use the ones that comes from Debugging Tools for Windows![back]

I think it’d be better if Microsoft’s security specialists concentrated on improving security in their products (and possibly write about how they do it) rather than trying to make the rest of the world feel sorry for them. I’m sorry, but full disclosure is the fairest we have at the moment.

Microsoft sits on a reported vulnerability for months, releases patch when it becomes a 0-day. As I write this Microsoft is sitting on a few publically known vulnerabilities in Office (0-days as well) that have been known for a while now.

So, until companies start behaving I think full disclosure is fair. It seems to be the only way of forcing delivery of security to paying customers at the moment. When there’s a sign that the business as a whole can function without FD I’ll be the first to argue against it; at the moment though it seems to be our only hope.

FD ⇒ bad PR ⇒ declining share price and sales ⇒ security fixes

Some companies seem to be on the verge of understanding this and taking it to heart. (Microsoft has understood it, but doesn’t seem to have found its heart just yet.)

Microsoft, if you need to pay people to use your product then it’s probably not a very good one!

An excellent paper on the cost of DRM in Windows Vista has been making its rounds on the internet for a few weeks now. The topic’s been picked up by Security Now (episode 73 and episode 74). The former gives a nice background to the technical side and the latter has Peter Gutman, the author of the paper, as a guest.

This has triggered my writing something about my thoughts on the topic of DRM. Now, I don’t consider myself an expert on this topic. I’ve formed most of my opinions on DRM by reading things like Cory Doctorow’s excellent talk at Microsoft Research and the Darknet paper from Microsoft. I’ve also worked for a large consumer electronics company for almost 5 years, of which the last 18 months to 2 years on security-related issues in consumer devices.

Why do the consumer electronics companies do DRM?

For the last 50 years consumers have bought new devices for the simple reason that the newer devices had more features, were faster, had better resolution, better audio… in short this year’s models were better than last year’s models. So, why on earth would these companies be interested in bringing out models that are fundamentally flawed through DRM? The easy answer is Hollywood… but as a professor of mine used to say, “Every difficult problem has an easy answer… which is wrong”. Hollywood isn’t the answer in my opinion. Hollywood is only a convenient scape goat. The real answer is format control.

A company that controls a format makes money even when a competitor sells a device. Just think of the patent Philips and Sony had on CDs. That patent pulled in money on every CD sold, worldwide. Talk about a gravy train. Nowadays content formats involve a lot of companies and I guess it’s less lucrative because license money will have to be shared between more companies (just think of the MPEG group). DRM is still a fairly new area of standardisation and there’s a good chance of cashing in even more than on the format itself, especially if DRM is written into lawⁱ.

Consumer electronics companies and the broadcast flag

A while back the broadcast flag was beaten in the US. A court found that the FCC didn’t have the authority to introduce such a flag and the US was saved. At least for now. What wasn’t reported so widely on was the fact that when the broadcast flag was put on the table there was an outcry among the consumer electronics companies (a few other companies joined in as well). No, don’t be fooled, they weren’t considering the consumer, they weren’t interested in keeping TV the way it was. No, they were outraged because the suggested broadcast flag allowed only one DRM system. A system controlled by 5C (if I remember correctly). Companies not in the core group were facing extortionate licenses (basically giving up all IP to the core companies). No wonder they were outraged. Intense lobbying of the FCC followed and the outcome was that a set of DRM technologies would be “legal” in the US. That’s where the consumer electronics industry spent their time and money. They were fighting the possibility of 5C gaining a strangling grip in the market rather then stand up and try to do the right thing which would have been to work to make it possible to bring to market the best possible devices.

Consumer electronics companies and security

When working on security at the research branch of a consumer electronics company I quickly found myself “attached” to DRM-related projects. That was the only place where they were interested in security at all. Of course they weren’t interested in keeping consumers safe in a future where tellys have internet connections. No, the interest was solely in keeping customers out of the telly, preventing them to do interesting things with the boxes they bought. So, who paid for this sort of research project? The IP/standardisation department, that’s who. They practically poured money into DRM projects while the parts of the company that actually made devices showed little interest. (On a personaly note I have to admit that this aspect of security was one of the reasons why I left the company.)

End note

Well, I hope I’ve made some sense and that I’ve added something to the discussion about DRM that currently is taking place. Peter Gutman has done a great job in making people aware of it and I’m looking forward to see what happens once Vista really hits the homes. I’m of course hoping that there is broad disapproval and that Vista does appallingly based solely on its DRM.

1. This brings me to a rather paranoid theory of mine, involving the “unholy trinity” of software patents, DMCA-like laws and DRM that can be used to explain some companies’ behaviour. That would probably have to be the topic of another post though.[back]

Are you running a huge corporation with the solid cash flow that follows from having your employees sending instant messages to each other?

Then Microsoft is the company to turn to for software. They will help you “hit the ground running”.

Just remember that hitting the ground running won’t do you much good if you’re falling head first!

Am I the only one who finds this absolutely terrifying?

If you haven’t seen DCLugi’s Snakes On A Plane auditions on YouTube you need to!

Bruce Schneier reported on this a while ago, I need to keep the link around. What to do when your neighbour is using your internet.

Our “honourable” EU politicians are finally worrying about the right things when it comes to Microsoft. Microsoft is all about lock-in, they’ll use security to achieve it if they can. Microsoft is of course responding. I wonder if they’re ever going to inhabit the same world as I am? Vista creating 100000 new jobs? Only if changing the title of a position from “Windows XP Developer/Administrator/Shithead” to “Windows Vista Developer/Administrator/Shithead” counts as “creating a new job”. I think Linux Journal is too kind when they call it Microsoft’s Masterpiece FUD. I think it only shows just how desperate Microsoft is to get Vista out the door.

Digital Rights Ireland are challenging the Data Retention Directive. I hope they are successful, I don’t there are many citizens in Europe who would be sad to see that invasive directive go away.

A good article on “open vs. closed” from the Financial Times—A closed mind about an open world. FT is increasingly “getting it”.

I couldn’t help but laugh out loud when reading this article trusted computing—Trusted computing a shield against worst attacks?. Let me see if I got it right. Phoneix Technologies, who happen to make a TPM module, pays analyst firm Trusted Strategies to have a report on digital attacks done. Then, surprisingly, the analysts come back and say that a TPM would have stopped most of those attacks. Who would ever read and put any faith in a report like this? Besides other analysts of course. I can’t help but think that the computer analyst firms are locked in a circle-jerk that’s paid for by computer companies. It is a very strange world we live in.

Thinking of writing a media app in Python? This seems like a good place to start.

Now, why are politicians repeating what the unrivaled masters of FUD say?

I thought our elected politicians were supposed to look out for our best interest, not suck up to foreign multi-nationals who have been found to indulge in nti-competitive behaviour.

It seems Microsoft is having problems getting the next version of their flagsship product into a state where it can be released and they’re grabbing for straws in order to pass blame. The business world seems fully capable of running on XP/2000—is has been for quite a while now—waiting a few extra months for Vista won’t hurt anyone but Microsoft.

Lately there’s been a lot of talk about Microsoft’s new Zune stuff. A lot of people seem to believe it’s never going to de-throne the iPod/iTunes monopoly Apple has managed to create. Maybe. I don’t really care. 79p is still too much to pay for music with DRM. The one thing that got me thinking was Microsoft’s strategy for breaking the Apple monopoly:

Make it easy for people to switch away from iTunes by offering the songs already bought there for free from Microsoft’s own store.

Brilliant idea? Possibly, but the devil’s in the details.

How are they going to do that, exactly? How do I prove that I’ve bought music from iTunes to Microsoft?

Remember, the iTunes music is under DRM. Apple has only sold one single license to that technology, Motorola. Reverse engineering of Apple’s DRM is not legal in the states, DMCA took care of that. (It’s probably not legal in the EU either.) And last, but not least, Apple and Microsoft are fierce competitors with nothing nice to say about each other.

So, how will Microsoft make sure that I don’t lie when saying that I bought song A from iTunes?

Will I have to give them my music files, they inspect them and then I’m rewarded? But, the file format is proprietary and Microsoft doesn’t have a license. Will they buy a license? Will Apple sell them one?

I’m sure that Cory Doctorow is right, if Microsoft goes ahead we’ll soon see a lot of “tools” that help users trick Microsoft into handing out free music…