Secrets of the Application Compatilibity Database (SDB) – Part 4

My apologies for the long delay until this fourth part was published. I have been teaching in Seattle for the previous two weeks, and have just started to settle in Cupertino for my Apple internship, and I had very few spare moments in my hands.

In Part 3, we discussed how generic shims modify key parts of the system, usually through API hooking or undocumented flags, in order to provide compatibility with a variety of applications. We looked at shims such as the Windows 9x Heap Manager implementation in NT, and several re-direction and reflection APIs, as well as even some security bypassing shims. Today, we’ll take a look at how certain applications have specific shims implemented specifically just for them. We can find these with CDD easily, by noticing that the Shim name is usually a program name, as well as looking in the DLL which implements it. Finally, specific shims never have any descriptive text describing them. While looking through the Shim dump, I’ve chosen this one (arbitrarly):

Dumping Entry:


Any continued analysis on this shim must be done through reverse engineering, since we have no hint as to what this shim is attempting to do. By using IDA on the DLL specified, one can notice it is a series of C++ classes, each which represent a specific shim (there are of course other classes such as CString and the generic Shim Engine initialization classes). The prefix for the specific shims seems to be “NS”, so it was easy to locate our target of interest: NS_CorelSiteBuilder. Every shim class also has an initialization function that gets called, and is responsible for initializing the class and its hooks. This is usually called IniitalizeHooksMulti. In the disassembly of this function, pay special attention to loc_714F3691. This is where this class initializes the API hooks that make up this specific shim (other specific shims can also have other types of hooks, such as patches or COM hooks). The tagHOOKAPI structure contains the information required to patch an API, and one can clearly see that SetWindowTextA inside user32.dll is being hooked, and re-directed to NS_CorelSiteBuilder::APIHook_SetWindowTextA.

Now the actual hook can be looked at, and I’ve provided an analyzed and commented disassembly here. This is a pretty simple hook, and seems to check on whether the window handlw and window text that are being sent as arguments match the previous window handle and window text that the shim had saved durinng the last call. If they do match, it will simply return TRUE (success) without actually calling the original API, otherwise, the hook will save the window text that’s being set as the “old” window text (so that when the hook is called again, it will compare against this name now), and then perform a call to the original API (in tagAPIHOOK+0xC) with the unmodified arguments.

In other words, the whole point of this shim is to “absorb” SetWindowTextA calls to the Corel Site Builder window if the new text that’s being set matches the previous text, and simply return success. The reason on why such a shim would be necessary is left as an excercise to the reader.

In the next article, I will release the first version of the CDD utility which I’ve used when showing some of the Shims available, and document some of its uses.

Secrets of the Application Compatilibity Database (SDB) – Part 3

Continuing over from where we left last time, today’s entry will look at how the loader interacts with the AppCompat/Shim Engine Interfaces to determine that a module requires shimming or not. Unfortunately, it seems like this process underwent several revisions inside Microsoft’s codebase, so it may be difficult to experiment on your own based on this information. I will however, present all the known implementations to me in a generic fashion, without going too much under the hood in terms of actual assembly code.

Like many Win32-specific features, the Shim Engine actually gets initialized by the parent process through kernel32.dll, and not by the actual PE Loader/Startup routines inside the NT System DLL (ntdll.dll), although it also plays an important role in the process. As CreateProcessInternalW executes, it eventually calls BasepCheckBadApp (which is actually an exported API). The first thing that immediately happens is a check on whether or not the Shim Engine is disabled, followed by a lookup inside the Application Compatibility Shim Cache (done through BaseCheckAppcompatCache).

This cache is implemented in 2 different ways depending on the OS. On pre-Windows 2003, kernel32 maintains a shared section which other instances can use for caching the information, and a lock/unlock is done each time the cache is accessed. On post 5.2 kernels, there is a new Native API, NtApphelpCacheControl which supports the following classes:


In both cases, if the cache lookup doesn’t find anything, a “long” lookup is performed. This is where the architectural differences are the largest. In Windows XP SP0, this is done by using CSRSS, and calling BaseSrvCheckApplicationCompatibility in basesrv.dll. In SP2, apphelp.dll is imported and ApphelpCheckExe is called directly. In Windows Server 2003, a connection to the LPC Port AELPort is made, and a lookup LPC message is sent. Finally, in Vista, we’re back to SP2’s method, albeit with a newer API, ApphelpCheckExeEx.

The end result, however, is that the Peb’s pShimData member is now filled out with Apphelp Information (we’ll see what happens with this later) if this is a “bad” application indeed, meaning that it needs to be shimmed. How are these checks actually made, however? Recall that one of the “constructs” or entries that an SDB can have is the Matching File entry. The checks first discover the “Executable” entry for the filename given, and if one is found, all Matching File entries are parsed. This can include the name of the application and its helper files, the publisher, vendor, company, version, file size, timestamp and even linker version and other obscure data. Several boolean operations are available which can be built on top of inclusion and exclusion rules. If the Matching File entries check out, then the pShimData is filled with an opaque Apphelp structure.

The next important part of the Shim Engine’s mingling with our application happens in LdrpInitializeProcess, which is part of the PE Loader inside ntdll.dll. Here, a check is made if pShimData is non-NULL. If this is the case, then this pointer is saved then cleared, and the Shim Engine DLL is loaded with a call to LdrpLoadShimEngine. A variety of callbacks are then setup through LdrpGetShimEngineInterface, which mostly consist of pre and post initialization, and DLL load/unload notifications.

A bit later during initialization, the pre-initialization hook is called, if the Shim Engine was previously loaded, and the old pShimData pointer is passed along to the Shim Engine, so that it may begin initialization. The routine is SE_InstallBeforeInit inside shimeng.dll, and most of the work is done by SeiGetShimData and SeiInit. The former unpacks the information from the PEB Shim Data pointer that it received. It also has a check to disable shimming for ntsd and windbg, as well as slsvc, on Vista (since this is a semi-protected process related to licensing). As for the latter, it will process all the compatibility layers, shims, flags and finally patches which are defined in the Apphelp entry for the executable.

Shims will usually consist of either internal flags that are saved inside shimeng.dll or inside the PEB (see +0x1d8 AppCompatFlags and +0x1e0 AppCompatFlagsUser), or by the IAT of the shimmed process to be hooked and redirected into one of the Ac***.dll files which contain an alternate, or hacked implementation. These contain two main exports, GetHookAPIs and NotifyShims which allow the Shim Engine to know which APIs should be hooked and to send notifications during loader events. The Shim Engine is smart and will also hook GetProcAddress to make sure that APIs are properly caught. Patches are done through a method that will be looked into more detail later.

During the next entry, we’ll take a look at an actual shimmed application in action, and future parts will cover patches/flags in more detail. It is my hope that this part was useful into giving some insight on how the hooking is performed. Many vendors of application/DLL hooking software risk of running into the Shim Engine during their testing and development process, so having a good handle on how and when everything happens is certainly helpful.

Part 3 of User-Mode Debugging Internals

The last part in my series on how Windows XP and higher support user-mode debugging is now up on OpenRCE; this part covers the kernel-mode side of things, aka the Dbgk module. Read it and find out how to use the native system calls in your debugger, which let you do things like debug multiple processes from a single debugger!

I will post the article on my Publications page as well, shortly.

Coming up shortly: the secrets of RtlRemoteCall!

Part 2 of User-Mode Debugging Internals Article

I’ve almost finished setting up the remaining parts of my blog. I’ve added an About page and pretty much filled my BlogRoll with the blogs I try to read daily. Thanks to everyone that’s visited/linked here in the last few days.

I’m currently working on the “Publications” page of the blog to have a central repository with all my data. I will also duplicate it on OpenRCE, but that site requires a login, and I wanted to make sure anyone could freely access my stuff. The links aren’t live yet, but they should be within the day.

Also, Part 2 of my article should appear soon, but for those that want to beat the clock, you can download it for now directly here. If you haven’t read Part 1 first, make sure you do here.

Brief overview of what’s discussed:

  • Part 1 – Win32: DebugActiveProcess, DebugBreakProcess, DebugSetProcessKillOnExit, CheckRemoteDebuggerPresent, WaitForDebugEvent, ContinueDebugEvent, DebugActiveProcessStop.
  • Part 1 – Win32: Teb->DbgSsReservedData[0] and DBGSS_THREAD_DATA. SaveProcessHandle, SaveThreadHandle, MarkThreadHandle, MarkProcessHandle, RemoveHandles, CloseAllProcessHandles.
  • Part 2 – Native: DbgUiConnectToDbg, DbgUiDebugActiveProcess, DbgUiStopDebugging, DbgUiIssueRemoteBreakin, DbgUiRemoteBreakin, DbgUiGetThreadDebugObject, DbgUiSetThreadDebugObject, DbgUiContinue, DbgUiWaitStateChange, DbgUiConvertStateChangeStructure.

Part 3 will cover Kernel Mode and the Nt* APIs when it’s out.