Basically what the code does is passing a 4 byte PSAPI_WORKING_SET_INFORMATION structure to QueryWorkingSet(). The first 4 bytes of the structure is the NumberOfEntries member. If the buffer is 4 bytes or more but still too small he will still fill out the NumberOfEntries member.
Once you have NumberOfEntries you can easily calculate the full size of the structure. (NumberOfEntries+1)*4 and you have the number of bytes the structure needs.
This behaviour is not inconsistent with msdn’s documentation. But you’re right, they don’t document it at all. It’s a well know hack that’s been around at least since NT4. As with most hacks, the main drawback with it is that you don’t get to whine if they change this behaviour in future versions..

That is not consistent with the documentation of the function in MSDN. I’ll give it a shot and report on the results.

DWORD dw, dwNeededSize;
QueryWorkingSet(hProc, &dw, sizeof(DWORD));
dwNeededSize = (dw+1) * sizeof(DWORD_PTR);
…
QueryWorkingSet(hProc, some allocated buff, dwNeededSize);

you will want to check for correct error codes etc (like ERROR_BAD_LENGTH) after the first call..