Introducing Haute Secure

For the last couple of months I’ve had the chance to meet and work with some of the brightest developers and people behind what I think is a pretty revolutionary way to secure the online experience of users: the team behind Haute Secure.

In short, Haute Secure is a Malware Filter, much like a Phishing or Spam Filter in existing applications. It provides a beautiful (you really have to see it!) interface and toolbar to IE (and soon Firefox) which protects users from incoming malware on a variety of levels, starting from the site level to the execution level. If cnn.com were to be hacked tomorrow with an unreleased exploit that would attempt to download a worm or other malware on visitors’ machines, Haute would be able to detect that, and block the exploit from happening. When this happens, Haute will communicate with its servers and post a notification, so a site becomes known “bad” as users stumble upon it. But Haute doesn’t only rely on its users; it also ships with a very large database of malicious sites out there. Haute is also smart enough to avoid tagging an entire domain as “bad”. Many sites such as MySpace, Yahoo and others can host individual user content, and don’t deserve to be blacklisted due to certain sub-sites. Haute can blacklist only certain parts of a domain, such as a user’s site, and will also tag the site with a warning, to let users know that -some- pages may be dangerous.

Sandi did a pretty good review of the product on her blog, but as someone whose actually worked on the product and had intimate knowledge of its behavior (as well as having worked on similar products in the past), I’d like to give my own technical review and why I think Haute is way ahead of the pack when it comes to this market.

The first reason I love this product so much is because unlike almost all anti-virus products, firewalls and IPS software, it’s actually written to properly interface with the OS. It’s fully compatible with Vista, even 64-bits, and co-exists with PatchGuard and other integrity mechanisms. The driver behind Haute Secure (and yes, it’s a driver, not a collection of user-mode hooking DLLs!) makes use of all the filtering technology available in NT without sacrificing functionality.

The second thing that I think is exciting about Haute is the fact that it strongly relies on a community of users, and not on hard-coded rules or filters (although, like I said, it does come with a large database already). I used to work on a product called SPAMfighter ages ago, and I saw how filtering spam became much more powerful when it was driven by people’s reponses, and not by AI. Of course, Haute also must implement some smart algorithms if it thinks a site is legitimate, to perform correctly in the case where malware is being installed through an exploit. Finally, Haute also has the ability to allow users to report false positives. Because of this user input, which even includes an entire community site where users can compete against each others in terms of number of bad sites reported, Haute can respond much quicker to malicious websites, and de-blacklist fixed sites much quicker as well.

Last but not least, Haute is being worked on and designed by some very bright people with extensive experience in this area. As I said earlier, I’ve also had the chance to contribute some knowledge and code into the product, and I felt that the design was very solid and ready to be extended to other products if that path will ever open. It’s one of the reasons why Firefox support is something being worked on, and shows that Haute isn’t in any way hacked around IE.

While some of the ideas and concepts behind Haute may have been attempted by other companies and products before, I really feel that Haute has all the right stuff it needs to be user friendly, powerful, and pro-active in protecting its users. The community-centric approach will also surely pay off into making an even better product. In many ways, I see it as the iPhone of its kind (if you agree with me that the iPhone is a success story).

Secrets of the Application Compatilibity Database (SDB) – Part 4

My apologies for the long delay until this fourth part was published. I have been teaching in Seattle for the previous two weeks, and have just started to settle in Cupertino for my Apple internship, and I had very few spare moments in my hands.

In Part 3, we discussed how generic shims modify key parts of the system, usually through API hooking or undocumented flags, in order to provide compatibility with a variety of applications. We looked at shims such as the Windows 9x Heap Manager implementation in NT, and several re-direction and reflection APIs, as well as even some security bypassing shims. Today, we’ll take a look at how certain applications have specific shims implemented specifically just for them. We can find these with CDD easily, by noticing that the Shim name is usually a program name, as well as looking in the DLL which implements it. Finally, specific shims never have any descriptive text describing them. While looking through the Shim dump, I’ve chosen this one (arbitrarly):

Dumping Entry:

SHIMNAME="CorelSiteBuilder"
DLLFILE="AcSpecfc.DLL"

Any continued analysis on this shim must be done through reverse engineering, since we have no hint as to what this shim is attempting to do. By using IDA on the DLL specified, one can notice it is a series of C++ classes, each which represent a specific shim (there are of course other classes such as CString and the generic Shim Engine initialization classes). The prefix for the specific shims seems to be “NS”, so it was easy to locate our target of interest: NS_CorelSiteBuilder. Every shim class also has an initialization function that gets called, and is responsible for initializing the class and its hooks. This is usually called IniitalizeHooksMulti. In the disassembly of this function, pay special attention to loc_714F3691. This is where this class initializes the API hooks that make up this specific shim (other specific shims can also have other types of hooks, such as patches or COM hooks). The tagHOOKAPI structure contains the information required to patch an API, and one can clearly see that SetWindowTextA inside user32.dll is being hooked, and re-directed to NS_CorelSiteBuilder::APIHook_SetWindowTextA.

Now the actual hook can be looked at, and I’ve provided an analyzed and commented disassembly here. This is a pretty simple hook, and seems to check on whether the window handlw and window text that are being sent as arguments match the previous window handle and window text that the shim had saved durinng the last call. If they do match, it will simply return TRUE (success) without actually calling the original API, otherwise, the hook will save the window text that’s being set as the “old” window text (so that when the hook is called again, it will compare against this name now), and then perform a call to the original API (in tagAPIHOOK+0xC) with the unmodified arguments.

In other words, the whole point of this shim is to “absorb” SetWindowTextA calls to the Corel Site Builder window if the new text that’s being set matches the previous text, and simply return success. The reason on why such a shim would be necessary is left as an excercise to the reader.

In the next article, I will release the first version of the CDD utility which I’ve used when showing some of the Shims available, and document some of its uses.

A New Direction

It is with great excitement (and a certain amount of nostalgia) that I would like to announce two important changes in my professional life and in the direction in which I will pursue my knowledge and work on Windows Internals. The first of these changes is my debut as an instructor for David Solomon’s Expert Seminars, and the second is my departure from ReactOS, effective immediately. These plans do not change in any way my internship at Apple which will take place during the summer.

Some time ago, I had the great privilege of being approached by David Solomon, a well-known and highly regarded computer expert, teacher, consultant and co-author of Windows Internals 4th Edition (and Inside Windows 2000, 3rd Edition). For the last couple of years, David had been working with Mark Russinovich, another respected figure in the world of Windows Internals, and co-founder of Winternals and Sysinternals as well as developer of some of the most useful Windows system tools available today. Apart from working on the two books (which Mark was a co-author of), they both provided trainings and seminars on Windows internals under the “David Solomon Expert Seminars” banner. As is widely known, Microsoft realized that Mark’s experience and amazing work on the NT platform through his articles and tools could provide a highly beneficial new addition to the company. The company bought Winternals last year, and hired Mark at the highest technical level in the company, Technical Fellow. 

All this is history of course, and back to the matter at hand, Mark’s recent new employment made him unavailable for teaching new classes, which made David Solomon start the search for a new instructor which could take on the responsibility of teaching new classes. I was highly honoured to have been chosen as this person, and accepted this unique opportunity to bring my knowledge out to many more people and to work with one of my most admired Windows experts

With this new job as an added task on top of my already busy life, as well as with the imminent Apple internship, I was already planning to cut back on my involvement with ReactOS. However, since it became clear that my level of contact with Microsoft employees and resources would be in conflict with my work at ReactOS, I made the difficult choice of amicably severing my ties with the project. This decision took some time for me to finalize, but the various motivations behind it had started cropping up since early this year

When I first joined ReactOS 3 years ago, the kernel was – in my opinion – highly disorganized and hodgepodge of Linux, NT 4, Wine and Windows 9x code which was very far from its actual goal of NT Driver compatibility. In fact, the development model seemed to focus on hacking NT drivers to work on ReactOS, and not vice-versa. Coincidentally, I joined the project just as the lead kernel developer, David Welch, had just burnt out and moved to other projects and goals. For the last three years, I rewrote key subsystems such as the thread scheduler, dispatcher, locking and IRQL mechanisms, HAL, executive support, object manager, process manager, I/O manager, basic VDM and 8086 support, and much more, as well as switched the project goals from NT4 to NT 5.2. 

My ability to do this came from my extensive reverse engineering of the kernel in the past, reading internals books, access to the DDK/IFS, as well as using WinDBG and .pdb type information. In return for all the code and guidance I provided, the project gave me a lot in return as well, including a unique perspective of working on such a project, the ability to work in large and distributed teams, and using open source tools for Windows NT kernel development. With millions of lines of code, ReactOS is the kind of project that an 18 year old could’ve only dreamt work ing on. I became adept in source control repositories, regression testing, unit testing, team management, IRC administration, as well as a much better coder in C. I also made friendships of all levels with various developers, testers and users, and had a chance to mentor two students during last year’s Google Summer of Code. I was able to attend and give talks on ReactOS, exhibit it, and make connections with other people in the industry, and in the open source world. Overall, it’s been an exhilarating adventure.

After three years however, and with the many new responsibilities that had kept growing, my free time grew short. Additionally, my work in the kernel had almost reached completion. The parts that still need major work, in my opinion, require extremely skilled developers in those areas to ever be as close to NT as needed. They are also some of the most critical: the memory manager, the cache manager, the Power/PnP Manager, the configuration manager and the file system runtime library. With the current differences that exist, most modern WDM drivers as well as IFS drivers can only dream of running properly. Unfortunately, my knowledge in those areas was limited. I had never reverse engineered them as extensively as parts of the executive, and documentation on their guts is limited.  In all honesty, they’re also not parts of the system that interest me much. I could, of course, have continued working on user-mode parts of the system where my help would still bring a lot of the system forward, such as ntdll, csrss, smss, winsock and kernel32, but my interest in teaching with David Solomon and getting in touch with the developers behind NT outweighed that desire.

After three years, I learned a tremendous amount of knowledge and skills while working on ReactOS, now the time has come for me to learn even more by expanding my horizons. In many ways, I had already outgrown the project, focusing more on security research, utilities and tools, articles and non-ReactOS related talks and conferences. It was time for me to step outside and take on a new opportunity with a larger audience and which would bring me many new experiences and teachings. I wish the ReactOS Project all the luck and I know that some significant new changes are on the horizon for them. I will keep watching from a distance, and I thank them for the most fun years of my life.

This blog will continue as usual, and I am currently working on the fourth part of the SDB series. Thank you for your continued readership and support!

Secrets of the Application Compatilibity Database (SDB) – Part 3

Continuing over from where we left last time, today’s entry will look at how the loader interacts with the AppCompat/Shim Engine Interfaces to determine that a module requires shimming or not. Unfortunately, it seems like this process underwent several revisions inside Microsoft’s codebase, so it may be difficult to experiment on your own based on this information. I will however, present all the known implementations to me in a generic fashion, without going too much under the hood in terms of actual assembly code.

Like many Win32-specific features, the Shim Engine actually gets initialized by the parent process through kernel32.dll, and not by the actual PE Loader/Startup routines inside the NT System DLL (ntdll.dll), although it also plays an important role in the process. As CreateProcessInternalW executes, it eventually calls BasepCheckBadApp (which is actually an exported API). The first thing that immediately happens is a check on whether or not the Shim Engine is disabled, followed by a lookup inside the Application Compatibility Shim Cache (done through BaseCheckAppcompatCache).

This cache is implemented in 2 different ways depending on the OS. On pre-Windows 2003, kernel32 maintains a shared section which other instances can use for caching the information, and a lock/unlock is done each time the cache is accessed. On post 5.2 kernels, there is a new Native API, NtApphelpCacheControl which supports the following classes:

typedef enum _APPHELPCACHESERVICECLASS
{
    ApphelpCacheServiceLookup,
    ApphelpCacheServiceRemove,
    ApphelpCacheServiceUpdate,
    ApphelpCacheServiceFlush,
    ApphelpCacheServiceDump
} APPHELPCACHESERVICECLASS;

In both cases, if the cache lookup doesn’t find anything, a “long” lookup is performed. This is where the architectural differences are the largest. In Windows XP SP0, this is done by using CSRSS, and calling BaseSrvCheckApplicationCompatibility in basesrv.dll. In SP2, apphelp.dll is imported and ApphelpCheckExe is called directly. In Windows Server 2003, a connection to the LPC Port AELPort is made, and a lookup LPC message is sent. Finally, in Vista, we’re back to SP2’s method, albeit with a newer API, ApphelpCheckExeEx.

The end result, however, is that the Peb’s pShimData member is now filled out with Apphelp Information (we’ll see what happens with this later) if this is a “bad” application indeed, meaning that it needs to be shimmed. How are these checks actually made, however? Recall that one of the “constructs” or entries that an SDB can have is the Matching File entry. The checks first discover the “Executable” entry for the filename given, and if one is found, all Matching File entries are parsed. This can include the name of the application and its helper files, the publisher, vendor, company, version, file size, timestamp and even linker version and other obscure data. Several boolean operations are available which can be built on top of inclusion and exclusion rules. If the Matching File entries check out, then the pShimData is filled with an opaque Apphelp structure.

The next important part of the Shim Engine’s mingling with our application happens in LdrpInitializeProcess, which is part of the PE Loader inside ntdll.dll. Here, a check is made if pShimData is non-NULL. If this is the case, then this pointer is saved then cleared, and the Shim Engine DLL is loaded with a call to LdrpLoadShimEngine. A variety of callbacks are then setup through LdrpGetShimEngineInterface, which mostly consist of pre and post initialization, and DLL load/unload notifications.

A bit later during initialization, the pre-initialization hook is called, if the Shim Engine was previously loaded, and the old pShimData pointer is passed along to the Shim Engine, so that it may begin initialization. The routine is SE_InstallBeforeInit inside shimeng.dll, and most of the work is done by SeiGetShimData and SeiInit. The former unpacks the information from the PEB Shim Data pointer that it received. It also has a check to disable shimming for ntsd and windbg, as well as slsvc, on Vista (since this is a semi-protected process related to licensing). As for the latter, it will process all the compatibility layers, shims, flags and finally patches which are defined in the Apphelp entry for the executable.

Shims will usually consist of either internal flags that are saved inside shimeng.dll or inside the PEB (see +0x1d8 AppCompatFlags and +0x1e0 AppCompatFlagsUser), or by the IAT of the shimmed process to be hooked and redirected into one of the Ac***.dll files which contain an alternate, or hacked implementation. These contain two main exports, GetHookAPIs and NotifyShims which allow the Shim Engine to know which APIs should be hooked and to send notifications during loader events. The Shim Engine is smart and will also hook GetProcAddress to make sure that APIs are properly caught. Patches are done through a method that will be looked into more detail later.

During the next entry, we’ll take a look at an actual shimmed application in action, and future parts will cover patches/flags in more detail. It is my hope that this part was useful into giving some insight on how the hooking is performed. Many vendors of application/DLL hooking software risk of running into the Shim Engine during their testing and development process, so having a good handle on how and when everything happens is certainly helpful.

Secrets of the Application Compatilibity Database (SDB) – Part 2

As noted in the introductory article, Windows Vista (and XP) ship with a number of default shims which are not exposed through any control panel or dialog available to end-users. Running the CDD Utility however, one can see all the shims installed in the defalt system database (sysmain.sdb):

Compatibility Database Dumper (CDD) v1.0
Copyright (C) 2007 Alex Ionescu
http://www.alex-ionescu.com

usage: cdd.exe [-s][-e][-l][-f][-p][-d kernel-mode database file][-a user-mode database file]
  -s Show shims
  -e Show executables
  -l Show layers
  -f Show flags
  -p Show patches
  -d Use Blocked Driver Database from this path
  -a Use Application Compatibility Database from this path

NOTE: If no paths are given, the default system database is used.

Dumping Entry: SHIM

NAME=”pGetProcAddrExOverride”
DLLFILE=”ShimengInternal”
DESCRIPTION=”Our internal hook for GetProcAddress used to not check include/exclude list at all which means the GetProcAddress calls from all modules are shimmed. Then we added code to take include/exclude list into consideration and that “broke” apps that used to rely on the previous behavior. To compensate for this, you can specify this shim to get back the old behavior.”
Dumping Entry: SHIM

NAME=”3DFrogFrenzy”
DLLFILE=”AcSpecfc.DLL”

The utility continues to dump several dozen other shims. It’s still in beta for now, so the final output might not match, but it allows us to build a list of several interesting system shims, which I’ll enumerate below.  Caveat: my criteria was a mix between usefulness, interesting security implications, and completely out-of-this-world, bizare or uber-hack shims. The ones in bold are some of my favorite, but you should defintely read through them all. Once the tool is completed, you’ll be able to dump your own.

NAME=”AddProcessParametersFlags”
DLLFILE=”AcGenral.DLL”
DESCRIPTION=”Add flags to Peb-ProcessParameter-Flags. The flags are a ULONG. Specify it as a hex number (so at most 8 digits).”
NAME=”APILogger”
DLLFILE=”LogExts.dll”
DESCRIPTION=”Logs API calls made by the application to an .LGV file in %windir%\AppPatch. You must copy LogExts.dll, LogViewer.exe and the Manifest directory to %windir%\AppPatch in order for this shim to function properly.”

NAME=”ChangeAuthenticationLevel”
DLLFILE=”AcLayers.DLL”
DESCRIPTION=”Changes COM Security Level from RPC_C_AUTHN_LEVEL_NONE to RPC_C_AUTHN_LEVEL_COMMON. This enables temporary elevation of the security context for an application.

NAME=”DelayDllInit”
DLLFILE=”AcGenral.DLL”
DESCRIPTION=”Some applications may use static DLLs, which could potentially issue calls to APIs before the application is ready. This compatibility fix provides a workaround for this behavior by causing a delay in the application’s static DLLs. This compatibility fix takes a command line containing a list of the DLLs affected. They will be loaded in the reverse order of the command line listing. Note that this compatibility fix is similar to InjectDll, which works with dynamically loaded DLLs.”

NAME=”DeRandomizeExeName”
DLLFILE=”AcGenral.DLL”
DESCRIPTION=”Some installation programs will create a randomly named executable when they are launched that is responsible for performing the actual setup. This compatibility fix takes a command line that specifies what random executable name is created, and upon creation, renames it to the new name specified on the command line. The command line is given as the source name followed by the desired name. For example: *.EXE;RANDOMSETUP.EXE.”

NAME=”DisableNX”
DLLFILE=”AcGenral.DLL”
DESCRIPTION=”This compatibility fix disables execution protection (NX) for a process. This is useful for applications that decide to execute from memory region marked with NX attribute (like stack, heap etc).”

NAME=”DisableSeh”
DLLFILE=”AcGenral.DLL”
DESCRIPTION=”Disable safe exception handling.”

NAME=”EmulateEnvironmentBlock”
DLLFILE=”AcLayers.DLL”
DESCRIPTION=”This compatibility fix causes Windows XP to return a significantly reduced environment block from the environment APIs. This reduces the chance of a buffer overrun causing corruption.”

NAME=”EmulateHeap”
DLLFILE=”AcLayers.DLL”
DESCRIPTION=”This compatibility fix emulates the functionality of the Windows 9x heap manager. It is is full implementation of the Windows 9x heap manager ported to Windows XP.”

NAME=”EmulateUSER”
DLLFILE=”AcLayers.DLL”
DESCRIPTION=”Fixes for known API differences between Win9x and NT: SetWindowsHookEx, SetWindowLong, RegisterClass, ChangeDisplaySettings/ChangeDisplaySettingsEx, ToAscii/ToAsciiEx, GetMessage/PeekMessage, ShowWindow. Also persists palette state through mode changes.”

NAME=”EnableRestarts”
DLLFILE=”AcLayers.DLL”
DESCRIPTION=”In Windows 9x applications could restart the computer by calling the ExitWindowsEx API. Windows XP requires the application to run with adequate security privileges to successfully call the ExitWindowsEx API. This compatibility fix enables an application to call the ExitWindowsEx API with correct security privileges. Applies to: Windows 95, Windows 98″

NAME=”FixServiceStartupCircularDependency”
DLLFILE=”AcGenral.DLL”
DESCRIPTION=”A service startup circular dependency occurs when two or more installed services depend upon each other to start. That is, service ‘A’ cannot start until service ‘B’ starts, but service ‘B’ cannot start without service ‘A’ running. This compatibility fix attempts to counter this behavior.”

NAME=”ForceAdminAccess”
DLLFILE=”AcGenral.DLL”
DESCRIPTION=”This compatibility fix addresses issues that may be encountered when an application uses the CheckTokenInformation API call to verify if the current user is part of the Administrators group. The fix intercepts calls to CheckTokenInformation and returns a value of true.”

NAME=”HandleAPIExceptions”
DLLFILE=”AcLayers.DLL”
DESCRIPTION=”This compatibility fix addresses issues with APIs that may not gracefully handle receiving bad parameters. Currently, this works with the BackupSeek, CreateEvent, and GetFileAttributes APIs.”

NAME=”HandleWvsprintfExceptions”
DLLFILE=”AcLayers.DLL”
DESCRIPTION=”This compatibility fix provides a facility to convert the argument list from LPSTR into VA_LIST. Some native Windows 9x applications use LPSTR instead of VA_LIST. Without properly checking the return value, these applications may assume that it is safe to use Wvsprintf, but in Windows XP, this may cause an access violation. This compatibility fix takes one command line: “arglistfix” (case insensitive).”

NAME=”HeapClearAllocation”
DLLFILE=”AcGenral.DLL”
DESCRIPTION=”This compatibility fix will clear out every heap allocation for the application with zeroes, or with a DWORD value that has been supplied in the command line.”

NAME=”HeapDelayLocalFree”
DLLFILE=”AcGenral.DLL”
DESCRIPTION=”This compatibility fix will delay calls to LocalFree. This may help applications that are trying to free heap memory using LocalFree before all activities have been concluded.”

NAME=”IgnoreCRTExit”
DLLFILE=”AcGenral.DLL”
DESCRIPTION=”Prevent CRT shutdown routines from running.”

NAME=”IgnoreLoadLibrary”
DLLFILE=”AcLayers.DLL”
DESCRIPTION=”This compatibility fix will prevent specified DLLs from being loaded by the LoadLibrary API, specified on the command line. If specifying multiple DLLs on the command line, they should be seperated by spaces. This may be useful for applications that have fallback mechanisms for features that are not supported. In addition, it reduces the error mode so library problems won’t cause the system to generate an error dialog. Applies to: Windows 95, Windows 98″

NAME=”IgnoreMessageBox”
DLLFILE=”AcGenral.DLL”
DESCRIPTION=”This compatibility fix intercepts calls to the MessageBox API and, based on the supplied command line, prevents the message box from being displayed. Many applications display a message box with debugging or other extraneous content that can be confusing to users. These are normally the result of differences between Windows 9x and Windows XP.”

NAME=”IgnoreVBOverflow”
DLLFILE=”AcGenral.DLL”
DESCRIPTION=”Some VB apps try to store win32 handles in WORD size variables. On Win9x this works because most handles are 16-bit. However, on NT, the VB type checking code throws a “Runtime Error 6″. The shim intercepts the type checking code and ignores the check.”

NAME=”RedirectEXE”
DLLFILE=”AcGenral.DLL”
DESCRIPTION=”This compatibility fix calls WinExec on the passed command line, and then terminates the caller process. The command line can contain any environment variables that need to be passed to the executable.”

NAME=”Shrinker”
DLLFILE=”AcLayers.DLL”
DESCRIPTION=”This compatibility fix fixes problems with any application that uses the Shrinker resource compression library. This library hacks resource functions in ntdll and kernel32 and redirect calls into their own function routines. But Ntdll code has different opcodes in Windows XP. The program failed to find the opcode signature and decided to cancel WriteProcessMemory call to write their redirection. Because of this, the necessary decompression of program code and resources were not executed and caused access violation. Shrinker compatibility fix resolves this by providing necessary opcode signature so the app could write those redirection into ntdll.”

NAME=”StackSwap”
DLLFILE=”AcGenral.DLL”
DESCRIPTION=”Many APIs use much more stack space on NT than Win9x. This compatibility fix is command line driven and takes a list of APIs that will be hooked, making them use no stack. The format the command line is “MODULENAME!APINAME[:X]; MODULENAME!APINAME[:X] …” where X is 0 : fill old stack with zeroes 1 : fill old stack with pointers 2 : fill old stack with pointers to pointers by default, no stack filling occurs.”

NAME=”TerminateExe”
DLLFILE=”AcGenral.DLL”
DESCRIPTION=”This compatibility fix terminates an executable (.EXE) immediately upon launch.”

NAME=”VirtualRegistry”
DLLFILE=”AcLayers.DLL”
DESCRIPTION=”Hooks all the registry functions to allow virtual keys, redirection and expansion values.”

As you can see, the Shim Engine allows from the simplest of hacks (such as adding PEB flags) to complete ports of 9x APIs (such as the Heap Manager). Many other shims are simply extremly useful features that should be accessible easier. The ability to deal with random setup application names is something I’ve had to code on my own in the past, and the VirtualRegistry shim in XP seems to be almost as powerful as the built-in Vista feature. Yet others, dealing with delay loading DLLs, instantly killing, and redirection can be lifesavers during certain debugging scenarios.

For now, these shims have only been presented. Later series will deal with actually using this shims, but for now, we’ll have to continue exploring the system further inside the next article.

Secrets of the Application Compatilibity Database (SDB) – Part 1

For the last few days, I’ve been intimately becoming aquainted with a piece of technology in Windows XP and Vista that rarely gets the attention it deserves. It has raised my esteem and admiration towards Microsoft ten fold, and I feel it would be wise to share it, publicize it, and then of course, find (positive) ways to exploit it to turn it into a powerful backend for various purposes.

The Shim Engine, which is how I’ll call it (and is one of the official names), is a technology implemented in various DLLs (mostly shimeng.dll and apphelp.dll — which is the Application Compatibility Interface) as well as through some callbacks and hacks in the PE Loader present in ntdll.dll. It also contains various registry entries for its configuration, as well as system database files.

What does this technology do? You’ve probably seen it in action when using Windows XP/Vista’s “Compatibility Wizard”, or the dialog which gives you options such as “Disable visual themes”, “Run application in Windows 2000” mode or “run at 640×480”. The checkboxes are called “shims”, while the actual Windows 2000 or Windows 98 combo box selections are called “layers”. Although this is hidden from you, layers are usually simple large combinations of other shims, each which somehow modify the system to behave in a different way. Unfortunately, this dialog contains only 3 shims, while over 100 are present by default on a Windows installation.

However, it is enough to illustrate how the technology works. Once an application has been “shimmed” manually, registry entries are created to notify the loader. As it loads, the loader will run the Shim Engine, which will perform lookups in the system compatibility database, recovering various information. This database is called sysmain.sdb, and it is located in your AppPatch directory. On top of the default database, individual, custom databases can be created, which are registered and installed through the registry. These specify settings for programs that you’ve manually chosen to be shimmed.

The way that shims are implemented is usually through a helper DLL, which the Shim Engine will load during PE Loading, and intercept the APIs being used, much like Detours. These DLLs are prefixed “Ac” and are also in the AppPatch directory. They contain the redirected code which behaves differently then the normal system API.

The most interesting part however, is not the ability to select these options, but the fact of how much of this is being done behind your back every single time you run an application. Upon analysis, the system database contains over 5000 applications (in Windows Vista) from small Chinese publishers to the largest application vendors, including Microsoft itself.

One of the core “objects” that the database supports is the Matching File construct, which does file pattern matching to identify whether or not this entry actually applies to the program being run. These pattern matches can go from the very simple “starcraft.exe” with a timestamp and checksum entry, to the more complex entries which try to match various .bmps, .wavs and data files present in a game’s engine. Wildcards and simple boolean logic is also supported, making for powerful pattern matching abilities.

Once a matching Executable construct has been found through its children Matching Files, 4 different types of modifications can be done. The first are system shims, which are implemented typically in the acgenral.dll or aclayers.dll library, and that many products might benefit from, such as emulating an older version of an API. The second are specific shims, which are tailored to an application, and located in acspecfic.dll. The third kind is also a shim, but a Flag shim, which specifices undocumented flags which are to be sent to LUA or the Installer about this application. Finally, the fourth type of change is a binary patch, which represents actual in-memory patching on the executable, instead of a system API hook.

Sound interesting and powerful? It is. I’ll spend the next few blog entries talking more about the various parts of the system, as well as offering two applications that I’ve been writing on. The first is a complete dumper of any .SDB database, and the second will be announced at the end. Here’s an overview of the different posts that I’m expecting to make:

1 ) Introduction (You are here).
2 ) System Shims – The Most Interesting Ones.
3 ) The Private Shim Engine Interface With The PE Loader.
4 ) Built-in Shimmed Applications and Specific Shims – A Sample.
5 ) Tool 1 – CDD – Compatibility Database Dumper
6 ) Flag Shims – LUA and Installer Flags.
7 ) The Run-Time In-Memory Patching Behaviour and Analysis
8 ) The System Blocked Driver Database – The Kernel Side of SDB.
9 ) Conclusion and Tool 2.

New Object Manager Filtering APIs

The new bits of the WDK have been released, and it seems that finally, we are starting to see a glimpse of some of the new filtering technologies that were promised in Vista SP1 to help with incompatibilities due to PatchGuard. Although Vista added powerful Registry filtering support on top of the existing File filtering architecture, hooking some native calls would still have been necessary in order to filter other kinds of system behaviour. The first which seems to have been addressed are handles, and Vista SP1 now supports Object Manager Filters.

Currently, only create and duplicate can be filtered, but the ability for both pre and post notifications exist. As with the new filter model, Object Manager Filters also support Altitudes, and are fully versionned. Unfortunately, this new set of APIs seems rather disappointing to me. For starters, this functionality was already available, even behind Patchguard’s back, through native Object Manager callbacks present in the OBJECT_TYPE’s ObjectTypeInitializer structure which contains all the callbacks for the object type. This interface seems to do nothing more but expose in a more public and accessible way the same ObCreateMethod interface that has existed since NT4, except that it only works for create and duplicate (while the internal interface allows for open and inherit as well).

Nevertheless, this new filtering mechanism is clearly written to be extensible for other Object Manager actions, so hopefully we’ll see some new improvements before SP1 actually ships. For the curious, here are some of the new toys to play with:

//
// Registration version for Vista SP1 and Windows Server 2007
//
#define OB_FLT_REGISTRATION_VERSION_0100  0x0100

//
// This value should be used by filters for registration
//
#define OB_FLT_REGISTRATION_VERSION OB_FLT_REGISTRATION_VERSION_0100

typedef ULONG OB_OPERATION;

#define OB_OPERATION_HANDLE_CREATE              0x00000001
#define OB_OPERATION_HANDLE_DUPLICATE           0x00000002

typedef struct _OB_PRE_CREATE_HANDLE_INFORMATION {
    __inout ACCESS_MASK         DesiredAccess;
    __in ACCESS_MASK            OriginalDesiredAccess;
} OB_PRE_CREATE_HANDLE_INFORMATION, *POB_PRE_CREATE_HANDLE_INFORMATION;

typedef struct _OB_PRE_DUPLICATE_HANDLE_INFORMATION {
    __inout ACCESS_MASK         DesiredAccess;
    __in ACCESS_MASK            OriginalDesiredAccess;
    __in PVOID                  SourceProcess;
    __in PVOID                  TargetProcess;
} OB_PRE_DUPLICATE_HANDLE_INFORMATION, * POB_PRE_DUPLICATE_HANDLE_INFORMATION;

typedef union _OB_PRE_OPERATION_PARAMETERS {
    __inout OB_PRE_CREATE_HANDLE_INFORMATION        CreateHandleInformation;
    __inout OB_PRE_DUPLICATE_HANDLE_INFORMATION     DuplicateHandleInformation;
} OB_PRE_OPERATION_PARAMETERS, *POB_PRE_OPERATION_PARAMETERS;

typedef struct _OB_PRE_OPERATION_INFORMATION {
    __in OB_OPERATION           Operation;
    union {
        __in ULONG Flags;
        struct {
            __in ULONG KernelHandle:1;
            __in ULONG Reserved:31;
        };
    };
    __in PVOID                         Object;
    __in POBJECT_TYPE                  ObjectType;
    __out PVOID                        CallContext;
    __in POB_PRE_OPERATION_PARAMETERS  Parameters;
} OB_PRE_OPERATION_INFORMATION, *POB_PRE_OPERATION_INFORMATION;

typedef struct _OB_POST_CREATE_HANDLE_INFORMATION {
    __in ACCESS_MASK            GrantedAccess;
} OB_POST_CREATE_HANDLE_INFORMATION, *POB_POST_CREATE_HANDLE_INFORMATION;

typedef struct _OB_POST_DUPLICATE_HANDLE_INFORMATION {
    __in ACCESS_MASK            GrantedAccess;
} OB_POST_DUPLICATE_HANDLE_INFORMATION, * POB_POST_DUPLICATE_HANDLE_INFORMATION;

typedef union _OB_POST_OPERATION_PARAMETERS {
    __in OB_POST_CREATE_HANDLE_INFORMATION       CreateHandleInformation;
    __in OB_POST_DUPLICATE_HANDLE_INFORMATION    DuplicateHandleInformation;
} OB_POST_OPERATION_PARAMETERS, *POB_POST_OPERATION_PARAMETERS;

typedef struct _OB_POST_OPERATION_INFORMATION {
    __in OB_OPERATION  Operation;
    union {
        __in ULONG Flags;
        struct {
            __in ULONG KernelHandle:1;
            __in ULONG Reserved:31;
        };
    };
    __in PVOID                          Object;
    __in POBJECT_TYPE                   ObjectType;
    __in PVOID                          CallContext;
    __in NTSTATUS                       ReturnStatus;
    __in POB_POST_OPERATION_PARAMETERS  Parameters;
} OB_POST_OPERATION_INFORMATION,*POB_POST_OPERATION_INFORMATION;

typedef enum _OB_PREOP_CALLBACK_STATUS {
    OB_PREOP_SUCCESS
} OB_PREOP_CALLBACK_STATUS, *POB_PREOP_CALLBACK_STATUS;

typedef OB_PREOP_CALLBACK_STATUS
(*POB_PRE_OPERATION_CALLBACK) (
    __in PVOID RegistrationContext,
    __inout POB_PRE_OPERATION_INFORMATION OperationInformation
    );

typedef VOID
(*POB_POST_OPERATION_CALLBACK) (
    __in PVOID RegistrationContext,
    __in POB_POST_OPERATION_INFORMATION OperationInformation
    );

typedef struct _OB_OPERATION_REGISTRATION {
    __in POBJECT_TYPE                *ObjectType;
    __in OB_OPERATION                Operations;
    __in POB_PRE_OPERATION_CALLBACK  PreOperation;
    __in POB_POST_OPERATION_CALLBACK PostOperation;
} OB_OPERATION_REGISTRATION, *POB_OPERATION_REGISTRATION;

typedef struct _OB_CALLBACK_REGISTRATION {
    __in USHORT                     Version;
    __in USHORT                     OperationRegistrationCount;
    __in UNICODE_STRING             Altitude;
    __in PVOID                      RegistrationContext;
    __in OB_OPERATION_REGISTRATION  *OperationRegistration;
} OB_CALLBACK_REGISTRATION, *POB_CALLBACK_REGISTRATION;

#if (NTDDI_VERSION >= NTDDI_VISTASP1)
NTKERNELAPI
NTSTATUS
ObRegisterCallbacks (
    __in POB_CALLBACK_REGISTRATION CallbackRegistration,
    __deref_out PVOID *RegistrationHandle
    );

NTKERNELAPI
VOID
ObUnRegisterCallbacks (
    __in PVOID RegistrationHandle
    );

NTKERNELAPI
USHORT
ObGetFilterVersion ();
#endif

Finally Legal!

I’m not one to boast, but today marks my entry into the US “Adult” world. While I’ve been happily enjoying fine beers in Montreal for over 3 years, I can now finally do the same in the US!

I apologize as well for the lack of recent posts, I am currently finishing my semester and in exam session. I will have more exciting things to share upon my return.

Good Discussion on Protected Processes

Down at the Disparity Bit, Dan Armak has a very good discussion on why exactly he thinks protected processes are bad, and a sort of addendum to my post on the subject. Check out “Making it Clear Just Why Protected Processes are a Bad Idea.” for a more detailed explenation on the problem.

A few people have started to reverse engineer the binary I posted, and some have come up with some partial explenations and analysis. I just wanted to clear up a few things: Yes, the method uses a driver. It’s based on the Microsoft documentation which says “Please don’t use a driver to bypass this”, which led me to believe that it would be possible to do this (which wouldn’t work on 64-bit Vista, of course).

Secondly, almost everything inside the binary I provided is a low-level obfuscation to confuse any kiddies that might try to grab a hold of the expanded driver and use it for the own purposes. It was not meant for, nor is it an example of, proper techniques to obfuscate/protect a program against advanced reverse engineers.

Introducting D-Pin Purr v1.0 – 32bit Edition

As promised in my earlier blog post, I’ve finalized the utility and made it available for download here. I won’t be releasing source code for the moment because I don’t want to encourage people to start adding this kind of code into their own malware programs, nor to encourage the Symantec folks to start unprotecting every process on the system.

So until then, have fun with the tool, whether it is to explore previously protected processes, or to try out various system and application behaviour when certain processes are made protected. Here’s a screenshot of audiodg.exe after being unprotected. Try it on your own system to see the before/after difference.