Secrets of the Application Compatilibity Database (SDB) – Part 3

Continuing over from where we left last time, today’s entry will look at how the loader interacts with the AppCompat/Shim Engine Interfaces to determine that a module requires shimming or not. Unfortunately, it seems like this process underwent several revisions inside Microsoft’s codebase, so it may be difficult to experiment on your own based on this information. I will however, present all the known implementations to me in a generic fashion, without going too much under the hood in terms of actual assembly code.

Like many Win32-specific features, the Shim Engine actually gets initialized by the parent process through kernel32.dll, and not by the actual PE Loader/Startup routines inside the NT System DLL (ntdll.dll), although it also plays an important role in the process. As CreateProcessInternalW executes, it eventually calls BasepCheckBadApp (which is actually an exported API). The first thing that immediately happens is a check on whether or not the Shim Engine is disabled, followed by a lookup inside the Application Compatibility Shim Cache (done through BaseCheckAppcompatCache).

This cache is implemented in 2 different ways depending on the OS. On pre-Windows 2003, kernel32 maintains a shared section which other instances can use for caching the information, and a lock/unlock is done each time the cache is accessed. On post 5.2 kernels, there is a new Native API, NtApphelpCacheControl which supports the following classes:


In both cases, if the cache lookup doesn’t find anything, a “long” lookup is performed. This is where the architectural differences are the largest. In Windows XP SP0, this is done by using CSRSS, and calling BaseSrvCheckApplicationCompatibility in basesrv.dll. In SP2, apphelp.dll is imported and ApphelpCheckExe is called directly. In Windows Server 2003, a connection to the LPC Port AELPort is made, and a lookup LPC message is sent. Finally, in Vista, we’re back to SP2’s method, albeit with a newer API, ApphelpCheckExeEx.

The end result, however, is that the Peb’s pShimData member is now filled out with Apphelp Information (we’ll see what happens with this later) if this is a “bad” application indeed, meaning that it needs to be shimmed. How are these checks actually made, however? Recall that one of the “constructs” or entries that an SDB can have is the Matching File entry. The checks first discover the “Executable” entry for the filename given, and if one is found, all Matching File entries are parsed. This can include the name of the application and its helper files, the publisher, vendor, company, version, file size, timestamp and even linker version and other obscure data. Several boolean operations are available which can be built on top of inclusion and exclusion rules. If the Matching File entries check out, then the pShimData is filled with an opaque Apphelp structure.

The next important part of the Shim Engine’s mingling with our application happens in LdrpInitializeProcess, which is part of the PE Loader inside ntdll.dll. Here, a check is made if pShimData is non-NULL. If this is the case, then this pointer is saved then cleared, and the Shim Engine DLL is loaded with a call to LdrpLoadShimEngine. A variety of callbacks are then setup through LdrpGetShimEngineInterface, which mostly consist of pre and post initialization, and DLL load/unload notifications.

A bit later during initialization, the pre-initialization hook is called, if the Shim Engine was previously loaded, and the old pShimData pointer is passed along to the Shim Engine, so that it may begin initialization. The routine is SE_InstallBeforeInit inside shimeng.dll, and most of the work is done by SeiGetShimData and SeiInit. The former unpacks the information from the PEB Shim Data pointer that it received. It also has a check to disable shimming for ntsd and windbg, as well as slsvc, on Vista (since this is a semi-protected process related to licensing). As for the latter, it will process all the compatibility layers, shims, flags and finally patches which are defined in the Apphelp entry for the executable.

Shims will usually consist of either internal flags that are saved inside shimeng.dll or inside the PEB (see +0x1d8 AppCompatFlags and +0x1e0 AppCompatFlagsUser), or by the IAT of the shimmed process to be hooked and redirected into one of the Ac***.dll files which contain an alternate, or hacked implementation. These contain two main exports, GetHookAPIs and NotifyShims which allow the Shim Engine to know which APIs should be hooked and to send notifications during loader events. The Shim Engine is smart and will also hook GetProcAddress to make sure that APIs are properly caught. Patches are done through a method that will be looked into more detail later.

During the next entry, we’ll take a look at an actual shimmed application in action, and future parts will cover patches/flags in more detail. It is my hope that this part was useful into giving some insight on how the hooking is performed. Many vendors of application/DLL hooking software risk of running into the Shim Engine during their testing and development process, so having a good handle on how and when everything happens is certainly helpful.

Secrets of the Application Compatilibity Database (SDB) – Part 2

As noted in the introductory article, Windows Vista (and XP) ship with a number of default shims which are not exposed through any control panel or dialog available to end-users. Running the CDD Utility however, one can see all the shims installed in the defalt system database (sysmain.sdb):

Compatibility Database Dumper (CDD) v1.0
Copyright (C) 2007 Alex Ionescu

usage: cdd.exe [-s][-e][-l][-f][-p][-d kernel-mode database file][-a user-mode database file]
  -s Show shims
  -e Show executables
  -l Show layers
  -f Show flags
  -p Show patches
  -d Use Blocked Driver Database from this path
  -a Use Application Compatibility Database from this path

NOTE: If no paths are given, the default system database is used.

Dumping Entry: SHIM

DESCRIPTION=”Our internal hook for GetProcAddress used to not check include/exclude list at all which means the GetProcAddress calls from all modules are shimmed. Then we added code to take include/exclude list into consideration and that “broke” apps that used to rely on the previous behavior. To compensate for this, you can specify this shim to get back the old behavior.”
Dumping Entry: SHIM


The utility continues to dump several dozen other shims. It’s still in beta for now, so the final output might not match, but it allows us to build a list of several interesting system shims, which I’ll enumerate below.  Caveat: my criteria was a mix between usefulness, interesting security implications, and completely out-of-this-world, bizare or uber-hack shims. The ones in bold are some of my favorite, but you should defintely read through them all. Once the tool is completed, you’ll be able to dump your own.

DESCRIPTION=”Add flags to Peb-ProcessParameter-Flags. The flags are a ULONG. Specify it as a hex number (so at most 8 digits).”
DESCRIPTION=”Logs API calls made by the application to an .LGV file in %windir%\AppPatch. You must copy LogExts.dll, LogViewer.exe and the Manifest directory to %windir%\AppPatch in order for this shim to function properly.”

DESCRIPTION=”Changes COM Security Level from RPC_C_AUTHN_LEVEL_NONE to RPC_C_AUTHN_LEVEL_COMMON. This enables temporary elevation of the security context for an application.

DESCRIPTION=”Some applications may use static DLLs, which could potentially issue calls to APIs before the application is ready. This compatibility fix provides a workaround for this behavior by causing a delay in the application’s static DLLs. This compatibility fix takes a command line containing a list of the DLLs affected. They will be loaded in the reverse order of the command line listing. Note that this compatibility fix is similar to InjectDll, which works with dynamically loaded DLLs.”

DESCRIPTION=”Some installation programs will create a randomly named executable when they are launched that is responsible for performing the actual setup. This compatibility fix takes a command line that specifies what random executable name is created, and upon creation, renames it to the new name specified on the command line. The command line is given as the source name followed by the desired name. For example: *.EXE;RANDOMSETUP.EXE.”

DESCRIPTION=”This compatibility fix disables execution protection (NX) for a process. This is useful for applications that decide to execute from memory region marked with NX attribute (like stack, heap etc).”

DESCRIPTION=”Disable safe exception handling.”

DESCRIPTION=”This compatibility fix causes Windows XP to return a significantly reduced environment block from the environment APIs. This reduces the chance of a buffer overrun causing corruption.”

DESCRIPTION=”This compatibility fix emulates the functionality of the Windows 9x heap manager. It is is full implementation of the Windows 9x heap manager ported to Windows XP.”

DESCRIPTION=”Fixes for known API differences between Win9x and NT: SetWindowsHookEx, SetWindowLong, RegisterClass, ChangeDisplaySettings/ChangeDisplaySettingsEx, ToAscii/ToAsciiEx, GetMessage/PeekMessage, ShowWindow. Also persists palette state through mode changes.”

DESCRIPTION=”In Windows 9x applications could restart the computer by calling the ExitWindowsEx API. Windows XP requires the application to run with adequate security privileges to successfully call the ExitWindowsEx API. This compatibility fix enables an application to call the ExitWindowsEx API with correct security privileges. Applies to: Windows 95, Windows 98″

DESCRIPTION=”A service startup circular dependency occurs when two or more installed services depend upon each other to start. That is, service ‘A’ cannot start until service ‘B’ starts, but service ‘B’ cannot start without service ‘A’ running. This compatibility fix attempts to counter this behavior.”

DESCRIPTION=”This compatibility fix addresses issues that may be encountered when an application uses the CheckTokenInformation API call to verify if the current user is part of the Administrators group. The fix intercepts calls to CheckTokenInformation and returns a value of true.”

DESCRIPTION=”This compatibility fix addresses issues with APIs that may not gracefully handle receiving bad parameters. Currently, this works with the BackupSeek, CreateEvent, and GetFileAttributes APIs.”

DESCRIPTION=”This compatibility fix provides a facility to convert the argument list from LPSTR into VA_LIST. Some native Windows 9x applications use LPSTR instead of VA_LIST. Without properly checking the return value, these applications may assume that it is safe to use Wvsprintf, but in Windows XP, this may cause an access violation. This compatibility fix takes one command line: “arglistfix” (case insensitive).”

DESCRIPTION=”This compatibility fix will clear out every heap allocation for the application with zeroes, or with a DWORD value that has been supplied in the command line.”

DESCRIPTION=”This compatibility fix will delay calls to LocalFree. This may help applications that are trying to free heap memory using LocalFree before all activities have been concluded.”

DESCRIPTION=”Prevent CRT shutdown routines from running.”

DESCRIPTION=”This compatibility fix will prevent specified DLLs from being loaded by the LoadLibrary API, specified on the command line. If specifying multiple DLLs on the command line, they should be seperated by spaces. This may be useful for applications that have fallback mechanisms for features that are not supported. In addition, it reduces the error mode so library problems won’t cause the system to generate an error dialog. Applies to: Windows 95, Windows 98″

DESCRIPTION=”This compatibility fix intercepts calls to the MessageBox API and, based on the supplied command line, prevents the message box from being displayed. Many applications display a message box with debugging or other extraneous content that can be confusing to users. These are normally the result of differences between Windows 9x and Windows XP.”

DESCRIPTION=”Some VB apps try to store win32 handles in WORD size variables. On Win9x this works because most handles are 16-bit. However, on NT, the VB type checking code throws a “Runtime Error 6″. The shim intercepts the type checking code and ignores the check.”

DESCRIPTION=”This compatibility fix calls WinExec on the passed command line, and then terminates the caller process. The command line can contain any environment variables that need to be passed to the executable.”

DESCRIPTION=”This compatibility fix fixes problems with any application that uses the Shrinker resource compression library. This library hacks resource functions in ntdll and kernel32 and redirect calls into their own function routines. But Ntdll code has different opcodes in Windows XP. The program failed to find the opcode signature and decided to cancel WriteProcessMemory call to write their redirection. Because of this, the necessary decompression of program code and resources were not executed and caused access violation. Shrinker compatibility fix resolves this by providing necessary opcode signature so the app could write those redirection into ntdll.”

DESCRIPTION=”Many APIs use much more stack space on NT than Win9x. This compatibility fix is command line driven and takes a list of APIs that will be hooked, making them use no stack. The format the command line is “MODULENAME!APINAME[:X]; MODULENAME!APINAME[:X] …” where X is 0 : fill old stack with zeroes 1 : fill old stack with pointers 2 : fill old stack with pointers to pointers by default, no stack filling occurs.”

DESCRIPTION=”This compatibility fix terminates an executable (.EXE) immediately upon launch.”

DESCRIPTION=”Hooks all the registry functions to allow virtual keys, redirection and expansion values.”

As you can see, the Shim Engine allows from the simplest of hacks (such as adding PEB flags) to complete ports of 9x APIs (such as the Heap Manager). Many other shims are simply extremly useful features that should be accessible easier. The ability to deal with random setup application names is something I’ve had to code on my own in the past, and the VirtualRegistry shim in XP seems to be almost as powerful as the built-in Vista feature. Yet others, dealing with delay loading DLLs, instantly killing, and redirection can be lifesavers during certain debugging scenarios.

For now, these shims have only been presented. Later series will deal with actually using this shims, but for now, we’ll have to continue exploring the system further inside the next article.

Secrets of the Application Compatilibity Database (SDB) – Part 1

For the last few days, I’ve been intimately becoming aquainted with a piece of technology in Windows XP and Vista that rarely gets the attention it deserves. It has raised my esteem and admiration towards Microsoft ten fold, and I feel it would be wise to share it, publicize it, and then of course, find (positive) ways to exploit it to turn it into a powerful backend for various purposes.

The Shim Engine, which is how I’ll call it (and is one of the official names), is a technology implemented in various DLLs (mostly shimeng.dll and apphelp.dll — which is the Application Compatibility Interface) as well as through some callbacks and hacks in the PE Loader present in ntdll.dll. It also contains various registry entries for its configuration, as well as system database files.

What does this technology do? You’ve probably seen it in action when using Windows XP/Vista’s “Compatibility Wizard”, or the dialog which gives you options such as “Disable visual themes”, “Run application in Windows 2000” mode or “run at 640×480”. The checkboxes are called “shims”, while the actual Windows 2000 or Windows 98 combo box selections are called “layers”. Although this is hidden from you, layers are usually simple large combinations of other shims, each which somehow modify the system to behave in a different way. Unfortunately, this dialog contains only 3 shims, while over 100 are present by default on a Windows installation.

However, it is enough to illustrate how the technology works. Once an application has been “shimmed” manually, registry entries are created to notify the loader. As it loads, the loader will run the Shim Engine, which will perform lookups in the system compatibility database, recovering various information. This database is called sysmain.sdb, and it is located in your AppPatch directory. On top of the default database, individual, custom databases can be created, which are registered and installed through the registry. These specify settings for programs that you’ve manually chosen to be shimmed.

The way that shims are implemented is usually through a helper DLL, which the Shim Engine will load during PE Loading, and intercept the APIs being used, much like Detours. These DLLs are prefixed “Ac” and are also in the AppPatch directory. They contain the redirected code which behaves differently then the normal system API.

The most interesting part however, is not the ability to select these options, but the fact of how much of this is being done behind your back every single time you run an application. Upon analysis, the system database contains over 5000 applications (in Windows Vista) from small Chinese publishers to the largest application vendors, including Microsoft itself.

One of the core “objects” that the database supports is the Matching File construct, which does file pattern matching to identify whether or not this entry actually applies to the program being run. These pattern matches can go from the very simple “starcraft.exe” with a timestamp and checksum entry, to the more complex entries which try to match various .bmps, .wavs and data files present in a game’s engine. Wildcards and simple boolean logic is also supported, making for powerful pattern matching abilities.

Once a matching Executable construct has been found through its children Matching Files, 4 different types of modifications can be done. The first are system shims, which are implemented typically in the acgenral.dll or aclayers.dll library, and that many products might benefit from, such as emulating an older version of an API. The second are specific shims, which are tailored to an application, and located in acspecfic.dll. The third kind is also a shim, but a Flag shim, which specifices undocumented flags which are to be sent to LUA or the Installer about this application. Finally, the fourth type of change is a binary patch, which represents actual in-memory patching on the executable, instead of a system API hook.

Sound interesting and powerful? It is. I’ll spend the next few blog entries talking more about the various parts of the system, as well as offering two applications that I’ve been writing on. The first is a complete dumper of any .SDB database, and the second will be announced at the end. Here’s an overview of the different posts that I’m expecting to make:

1 ) Introduction (You are here).
2 ) System Shims – The Most Interesting Ones.
3 ) The Private Shim Engine Interface With The PE Loader.
4 ) Built-in Shimmed Applications and Specific Shims – A Sample.
5 ) Tool 1 – CDD – Compatibility Database Dumper
6 ) Flag Shims – LUA and Installer Flags.
7 ) The Run-Time In-Memory Patching Behaviour and Analysis
8 ) The System Blocked Driver Database – The Kernel Side of SDB.
9 ) Conclusion and Tool 2.

New Object Manager Filtering APIs

The new bits of the WDK have been released, and it seems that finally, we are starting to see a glimpse of some of the new filtering technologies that were promised in Vista SP1 to help with incompatibilities due to PatchGuard. Although Vista added powerful Registry filtering support on top of the existing File filtering architecture, hooking some native calls would still have been necessary in order to filter other kinds of system behaviour. The first which seems to have been addressed are handles, and Vista SP1 now supports Object Manager Filters.

Currently, only create and duplicate can be filtered, but the ability for both pre and post notifications exist. As with the new filter model, Object Manager Filters also support Altitudes, and are fully versionned. Unfortunately, this new set of APIs seems rather disappointing to me. For starters, this functionality was already available, even behind Patchguard’s back, through native Object Manager callbacks present in the OBJECT_TYPE’s ObjectTypeInitializer structure which contains all the callbacks for the object type. This interface seems to do nothing more but expose in a more public and accessible way the same ObCreateMethod interface that has existed since NT4, except that it only works for create and duplicate (while the internal interface allows for open and inherit as well).

Nevertheless, this new filtering mechanism is clearly written to be extensible for other Object Manager actions, so hopefully we’ll see some new improvements before SP1 actually ships. For the curious, here are some of the new toys to play with:

// Registration version for Vista SP1 and Windows Server 2007
#define OB_FLT_REGISTRATION_VERSION_0100  0x0100

// This value should be used by filters for registration


#define OB_OPERATION_HANDLE_CREATE              0x00000001
#define OB_OPERATION_HANDLE_DUPLICATE           0x00000002

    __inout ACCESS_MASK         DesiredAccess;
    __in ACCESS_MASK            OriginalDesiredAccess;

    __inout ACCESS_MASK         DesiredAccess;
    __in ACCESS_MASK            OriginalDesiredAccess;
    __in PVOID                  SourceProcess;
    __in PVOID                  TargetProcess;

    __inout OB_PRE_CREATE_HANDLE_INFORMATION        CreateHandleInformation;
    __inout OB_PRE_DUPLICATE_HANDLE_INFORMATION     DuplicateHandleInformation;

    __in OB_OPERATION           Operation;
    union {
        __in ULONG Flags;
        struct {
            __in ULONG KernelHandle:1;
            __in ULONG Reserved:31;
    __in PVOID                         Object;
    __in POBJECT_TYPE                  ObjectType;
    __out PVOID                        CallContext;

    __in ACCESS_MASK            GrantedAccess;

    __in ACCESS_MASK            GrantedAccess;

    __in OB_POST_CREATE_HANDLE_INFORMATION       CreateHandleInformation;
    __in OB_POST_DUPLICATE_HANDLE_INFORMATION    DuplicateHandleInformation;

    __in OB_OPERATION  Operation;
    union {
        __in ULONG Flags;
        struct {
            __in ULONG KernelHandle:1;
            __in ULONG Reserved:31;
    __in PVOID                          Object;
    __in POBJECT_TYPE                   ObjectType;
    __in PVOID                          CallContext;
    __in NTSTATUS                       ReturnStatus;


    __in PVOID RegistrationContext,
    __inout POB_PRE_OPERATION_INFORMATION OperationInformation

typedef VOID
    __in PVOID RegistrationContext,
    __in POB_POST_OPERATION_INFORMATION OperationInformation

    __in POBJECT_TYPE                *ObjectType;
    __in OB_OPERATION                Operations;
    __in POB_PRE_OPERATION_CALLBACK  PreOperation;

    __in USHORT                     Version;
    __in USHORT                     OperationRegistrationCount;
    __in UNICODE_STRING             Altitude;
    __in PVOID                      RegistrationContext;
    __in OB_OPERATION_REGISTRATION  *OperationRegistration;

ObRegisterCallbacks (
    __in POB_CALLBACK_REGISTRATION CallbackRegistration,
    __deref_out PVOID *RegistrationHandle

ObUnRegisterCallbacks (
    __in PVOID RegistrationHandle

ObGetFilterVersion ();