Archive for November, 2006

New Version of SDbgExt released!

Saturday, November 25th, 2006

If you’re a WinDBG fanatic, then you should be using SDbgExt as part of your extension tools. It’s a great extension written by WinDBG Veteran Skywing.

And a new version has just been released, with 64-bit support for EH Records!

Interview

Wednesday, November 22nd, 2006

The ReactOS Project has published an interview with me, so if anyone’s interested in my work there, feel free to take a read.

Publications

Tuesday, November 21st, 2006

Just realized I forgot to post these in the last update, but perhaps it was better anyways since the post was already large enough.

NTFS On-Disk Structure – A fairly large reference to all the structures used on NTFS as well as some of the technical details behind some implementations. Was going to cover EFS and those structures, but I stopped it short there. This was written quite some time ago, and the structures are in Visual Basic format, but it should still be pretty useful. My explenation of NTFS runs was one of the best things in it.

Process Internals – Was going to become Part 1 of a series of 3 or 4 documents on each of the main executive components of NT, the Process Manger, the Object Manager and the Executive itself. Not very happy about this one in retrospect, since a lot of the fields I had documented aren’t used anymore or the information was wrong, but I still think it’s a good reference (especially the later sections). Again, done when I was younger and writing Visual Basic code.

Visual Basic File FormatOne of the articles I’m most proud of, this one was the result of several weeks of independent study into the Visual Basic file format for compiled executables. It explains every field, structure, relationship, etc, that the compiler inserts into the file that is then read by the runtime. Allowed me to write a simple runtime library that was only 20KB (for basic MsgBoxes). Highly graphical and easy to read.

Native API Compression and Introduction to NT DesignAn older article of mine again, gave a short primer on Native APIs, then presented a set of useful compression APIs burried into NT, and gave some interesting study on their performance and compressibility.

NTFS Alternate Data StreamsBack when alternate data streams weren’t very popular (I think I participated into making them popular, I was approached several times for inclusion of this article into books, magazines and other websites), I wrote code and an article exposing them and the dangers they presented, as well as a scanner that could find them. Again, Visual Basic code, and done when I was younger.

Subverting Windows 2003 Service Pack 1 Kernel Integrity ProtectionMy latest large presentation/project, this one was presented at REcon 2006. Shows a way to defeat the new protection mechanisms added in 2003 to disable access to kernel-mode from user-mode administrative applications, and how to access physical memory again. Exposed a flaw in VDM present in all released (at the time) versions of Windows NT.

Windows XP/2003 User-Mode Debugging Internals, Part 1Part of a series about the User-Mode Debugging framework in kernel32, ntdll and ntoskrnl. This part deals with Win32.

Windows XP/2003 User-Mode Debugging Internals, Part 2Part of a series about the User-Mode Debugging framework in kernel32, ntdll and ntoskrnl. This part deals with Native.

Blogroll and Publications

Thursday, November 16th, 2006

It’s been a pretty slow week until now, so I’m going to take this time to do the last “todo” for my blog. As I mentionned previously, I have been setting up my blogroll and publication page, and now that I’ve finished uploading everything, I wanted to give a more in-depth overview of the materials and links:

Blogs

  • Ken Johnson’s Blog Ken is a well-known NT expert known as Skywing. We met while he was still working on ReactOS, and we frequently discuss NT implementation details and security bugs. We both have a hate for crappy drivers, hooks, and other kinds of dangerous code. If there’s something I don’t post here, Ken probably has posted about it before or will :)
  • Ero Carrera’s Blog Ero is a fellow security researcher that I met while at Recon, and he is well known for his work at Sabre, especially in analysis of binaries, and he has published various papers and given talks at conferences, as well as trainings. If you have a chance to meet him, take it, because he knows a lot!
  • TinyKRNL Website TinyKRNL is my own personal project, and team of people working on it. You can find more information on its website. It is a Windows NT Kernel implementation for educational use.
  • Larry Osterman’s Blog Larry is one of my favorite bloggers at Microsoft. He has been working there for over two decades, and his experience (and anecdotes) are always insightful. He has interesing topics such as “What’s wrong with this code” and other optimization/algorithm questions sometimes.
  • ReactOS The other project that I work on, ReactOS attempts to create an open-sourced (GPL) version of the Windows NT (XP/2003) Operating System by implementing its own kernel and using Wine’s Win32 libraries.
  • Jason Geffner’s Blog Jason is a reverse engineer at Microsoft that I’m happy to call my friend. He’s one of the smartest guys I’ve met, and he has a lot of neat ideas that he gets the chance to work on. He’s also responsible for saving the world a couple of times.
  • Raymond Chen’s Blog Raymond chen is the iconic Microsoft blogger. Blogging on everything from languages, music to Windows 1.0 compatibility functions present in Vista, Raymond makes it hard for you to hate those annoying features in Windows, and explains just how stupid some 3rd-party developers really are, and how much trouble he and his team have to go through to keep those apps working, so that you don’t complain.
  • OpenRCE OpenRCE is the premier Reverse Code Engineering portal, with all the big names in the field, interesting articles (such as mine ;), forum posts, tools and diagrams, and a helpful bunch of users from all over the industry. It’s also the brainchild of a very good friend of mine, Pedram.
  • Steve Dispensa’s Blog Steve is another NT genius, especially in NDIS, that took his knowledge to the corporate level and now is a CTO for a very sucessful company. We met through Ken Johnson, who is now employed there, and I’m still hoping to finally be able to meet him in the future. His blog is of particular interest to NT driver developers, much like Ken’s.
  • Pedram’s Blog Pedram is another security researcher and big name, as well as the creator of OpenRCE and many other valuable tools, including PaiMei and tools such as PyDbg. He’s also a good friend of mine, and a very fun guy.

Win32 Calling Conventions Review

Saturday, November 11th, 2006

Ken Johnson (Skywing) from Nyaneve has posted a TOC for his great series on Win32 calling conventions. It seems every systems developer always posts a SEH and Calling Convention guide on his blog, but the best I’ve found until now is his, so make sure you take a read!

Using spaces in your TARGETLIBS path

Saturday, November 11th, 2006

I’ve been trying to find a solution on this annoying “sources” problems in the Build utility for ages. Thankfully, a post at NotAKernelGuy pointed a way to the solution. It’s in Russian, but the basic solution is simple:

LINKER_OPTIDATA = \\
"$(VS80COMNTOOLS)\\..\\..\\VC\\PlatformSDK\\lib\\mscoree.lib"

Replace that path by whatever you need, but the end result is the same: the library will be added to the response file, and spaces will be preserved and respected.

Part 2 of User-Mode Debugging Internals Article

Friday, November 10th, 2006

I’ve almost finished setting up the remaining parts of my blog. I’ve added an About page and pretty much filled my BlogRoll with the blogs I try to read daily. Thanks to everyone that’s visited/linked here in the last few days.

I’m currently working on the “Publications” page of the blog to have a central repository with all my data. I will also duplicate it on OpenRCE, but that site requires a login, and I wanted to make sure anyone could freely access my stuff. The links aren’t live yet, but they should be within the day.

Also, Part 2 of my article should appear soon, but for those that want to beat the clock, you can download it for now directly here. If you haven’t read Part 1 first, make sure you do here.

Brief overview of what’s discussed:

  • Part 1 – Win32: DebugActiveProcess, DebugBreakProcess, DebugSetProcessKillOnExit, CheckRemoteDebuggerPresent, WaitForDebugEvent, ContinueDebugEvent, DebugActiveProcessStop.
  • Part 1 – Win32: Teb->DbgSsReservedData[0] and DBGSS_THREAD_DATA. SaveProcessHandle, SaveThreadHandle, MarkThreadHandle, MarkProcessHandle, RemoveHandles, CloseAllProcessHandles.
  • Part 2 – Native: DbgUiConnectToDbg, DbgUiDebugActiveProcess, DbgUiStopDebugging, DbgUiIssueRemoteBreakin, DbgUiRemoteBreakin, DbgUiGetThreadDebugObject, DbgUiSetThreadDebugObject, DbgUiContinue, DbgUiWaitStateChange, DbgUiConvertStateChangeStructure.
  • Part 2 – Native: Teb->DbgSsReservedData[1], DBGUI_WAIT_STATE_CHANGE, Teb->Tib.ArbitraryUserPointer, DBG_STATE, DBGKM_EXCEPTION, DBGKM_CREATE_THREAD, DBGKM_CREATE_PROCESS, DBGKM_EXIT_THREAD, DBGKM_LOAD_DLL, DBGKM_UNLOAD_DLL.

Part 3 will cover Kernel Mode and the Nt* APIs when it’s out.

WDK RTM Changes

Thursday, November 9th, 2006

I consistently did diffs (differential changes) between each new release of the WDK. It was interesting to follow the evolution of certain APIs and structures, as well as APIs which were added by mistake.

The latter happens because kits like the WDK are built from a master header file. Suppose it looks like this:

// begin_ntddk

//
// Process Functions
//
NTKERNELAPI
NTAPI
KeSetProcess(IN PRKPROCESS Process);

//
// Thread Functions
//
NTKERNELAPI
NTAPI
KeStartThread(IN PRKTHREAD Thread);

// end_ntddk

NTKERNELAPI
NTAPI
KeSetThreadDrmProtection(IN PRKTHREAD Thread);

What would happen in the DDK is that the KeSetProcess and KeStartThread would be exported, and by definition, “legit” to be used in drivers. Now suppose the developers add a new API in Vista, and don’t properly take a look at the DDK tags, you could end up with this:

// begin_ntddk

//
// Process Functions
//
NTKERNELAPI
NTAPI
KeSetProcess(IN PRKPROCESS Process);

NTKERNELAPI
NTAPI
KeProtectProcessForDrm(IN PRKPROCESS Process);

//
// Thread Functions
//
NTKERNELAPI
NTAPI
KeStartThread(IN PRKTHREAD Thread);

// end_ntddk

NTKERNELAPI
NTAPI
KeSetThreadDrmProtection(IN PRKTHREAD Thread);

Notice that the tags weren’t properly updated to keep the DRM/internal/undocumented function out from the DDK, so it will appear in the WDK. Of course, at the next release, someone is bound to notice and fixup the tags. So by doing cumulative diffs, I was able to get the prototypes of quite a few new APIs that didn’t make it into the final WDK. Of course, I don’t condone their use in a driver, but they’re useful for ReactOS/TinyKRNL development and to better understand some of the changes done in Vista.

One of the more memorable API sets that were added allow drivers (well, at least, were supposed to!) to modify the size of the kernel stack. Typically MmCreateKernelStack was a way to do this, but these new Ke functions give a much greater degree of control as well as give you a Callout function:

#define MAXIMUM_EXPANSION_SIZE (KERNEL_LARGE_STACK_SIZE – (PAGE_SIZE / 2))

typedef
VOID
(NTAPI *PEXPAND_STACK_CALLOUT) (
__in_opt PVOID Parameter
);

#if (NTDDI_VERSION >= NTDDI_WS03SP1)
NTKERNELAPI
NTSTATUS
KeExpandKernelStackAndCallout (
__in PEXPAND_STACK_CALLOUT Callout,
__in_opt PVOID Parameter,
__in SIZE_T Size
);
#endif

#if (NTDDI_VERSION >= NTDDI_LONGHORN)
NTKERNELAPI
NTSTATUS
KeExpandKernelStackAndCalloutEx (
__in PEXPAND_STACK_CALLOUT Callout,
__in_opt PVOID Parameter,
__in SIZE_T Size,
__in BOOLEAN Wait,
__in_opt PVOID Context
);

NTKERNELAPI
PVOID
KeAllocateCalloutStack (
__in BOOLEAN LargeStack
);

NTKERNELAPI
VOID
KeFreeCalloutStack (
__in PVOID Context
);

#endif

Now here’s the ironic thing: http://msdn.microsoft.com/library/default.asp?url=/library/en-us/DevTest_g/hh/DevTest_g/t06_bugs_B0_77bda7e9-4f41-49e9-86db-04446dc9c7b7.xml.asp

“The driver switched stacks using a method that is not supported by the operating system. The only supported way to extend a kernel mode stack is by using KeExpandKernelStackAndCallout.”

I guess it’s either time for a WDK bug or an MSDN documentation bug to be opened!

However, perhaps the best WDK RTM change was this:

Original:

// This logic is a reasonable hack-o-rama to make BillG happy
// since his machine ran chkdsk after he installed Beta 3. Why?
// ’cause setup cracked a non-exclusive DASD handle near the
// end of setup, wrote some data, closed the handle and we
// set the verify bit … came back around and saw that other
// arbitrary activity had left the volume in a temporarily dirty
// state.
//
// Of course, the real problem is that we don’t have a journal.
//

RTM:

// This logic is a reasonable change. Why?
// ’cause setup cracked a non-exclusive DASD handle near the
// end of setup, wrote some data, closed the handle and we
// set the verify bit … came back around and saw that other
// arbitrary activity had left the volume in a temporarily dirty
// state.
//
// Of course, the real problem is that we don’t have a journal.

“hack-o-rama to make BillG happy” => “reasonable change”.

Got any more similar changes of your own? Feel free the post them!

Microsoft Tools for Power Users/Developers

Wednesday, November 8th, 2006

I haven’t fully finished up setting the blog yet, but I wanted to blog about some useful and not-very-well-known Microsoft projects, tools and technologies.

Unfortunately, Microsoft advertises way too much to regular users, and doesn’t take advantage of the influencial student/developer/power user market segment, which is usually the most vocal Anti-Microsoft. You can throw words like “SuperFetch” at an engineering student that uses Linux all you want, it probably won’t impress him much. Now show him PowerShell connecting to his NFS shares through a Windows Server 2003 R2 machine with Subsystem for Unix Applications and he might raise an eyebrow.

I’ve had to explain NT’s subsystem design twice this week and I always get a “wow? really? NT can run POSIX?” Yup, and NT 5.2 SP1 can even run 64-bit POSIX and debug them in Visual Studio 2005. NT 5.2 is also called Windows Server 2003. It’s basically a kernel that’s 30% faster then XP’s and was used as the core for Vista. If you want a fast OS with an optimized kernel and don’t want to take part of the “Vista Experience”, you should consider it.

Anyways, without further ado (visit the links for information, I could blog entire pages about these). All these are FREE!

News/Community

Downloads:

Source Code:

I’ll add more as I remember them, there’s really a lot of great stuff at Microsoft that’s untapped to!

Also, if you have anything to add, please comment!

First Post

Tuesday, November 7th, 2006

I’ve kept many blogs during recent times, often struggling to combine personal, private, profesional and random information into various places, usually without much success, and only fragmenting the pieces of my online life.

This site is now the definitive resource for anything that’s not of a private nature or not of general public interest, and it will contain news, rants, ramblings and otherwise hopefully informative data. I will also post up some biographical information and resume, and and detailed contact information.

I hope you’ll find this site a valuable resource to keep yourself informed of my activities and interests. I will probably blog on random topics, but you can expect to find:

  • Mirrors of my articles, presentations and blog-entries at OpenRCE.org.
  • NT Kernel/Native Mode or otherwise low-level information about Windows NT.
  • Development news from ReactOS and TinyKRNL.
  • Personal projects and achievements.
  • Reversing and coding hints and tips.
  • Links to valuable resources.

As always, don’t hesitate to contact me if you have any questions, or simply leave a comment.