Windows Internals 5th Edition, at last!

I am very pleased to announce that the 5th Edition of the Windows Internals book series is finally shipping for the past couple of weeks, and hard copies are now arriving in the hands of most customers! As my last blog post indicates, I took a hiatus from most of my typical work in the security and reverse engineering field and focused all of my energy into the book, outside of other commitments such as the Windows internals classes I teach for David Solomon Expert Seminars, so I thought it helpful to give my own perspective on the book itself, and on my work and the experience of working alongside the two legends of Windows internals knowledge. With that in mind, if you haven’t done so already, I’d invite you to read over Mark’s blog post on the book, as well as take a look at the short interview that David and Mark did on Channel 9 — it covers a lot of information on the latest release that I might not have covered in this post.

When we first set out to work on the 5th Edition, we decided early on to make three underlying changes to the existing content (and by extension, any new content as well). The first was to remove all references to previous versions other than the ones targeted by this edition (Windows Vista SP1 and Server 2008, specifically). We had realized that covering what would now be 5 different versions of the kernel (5.0 through 6.0 SP1) would generate too much redundant text, confusing explanations and questionably useful comparisons (such as let’s say, the evolution of how many buckets a given kernel component uses to store string hashes — driven probably only by the increase in average computer specifications across releases, and not some deeper mystery in the kernel). Windows mechanisms weren’t the only thing trimmed down to cover today’s reality however — references to old tools, unsupported resource kits, etc., were also removed.

When working on this edition, this was a significant challenge, because while it is relatively easy to get lots of information on major new Vista changes and improvements, it’s much harder to track down the little details that may have been valid at one time, but not anymore, and to rid the book of any archaic references, algorithms or values. Additionally, the second decision was to try minimizing giving out the values of certain variables and tuning parameters that the kernel uses. For one part, this creates the unfortunate scenario of developers copying down those values and then later depending on them in their software, which is a bad idea that only leads to more crashes for customers. For the other, it also makes it hard for us, as authors, to have to track down the exact values every single time Windows is updated — additionally, if the values changed significantly, people might expect explanations for these changes, when sometimes they are just as simple as “performance testing showed this to be a better number in today’s computers”. Because the variable name, its usage and the scale of its value are still referenced however, this still gives the reader the required understanding and, if someone really wants the value, they can use the same tools as the authors to obtain it (such as using the Windows Debugging Tools with the appropriate symbols).

As you can see, an important part of this update wasn’t even related to adding new information on Vista and Server 2008, but rather to bring the book up to even higher quality and technical standards, a lofty goal considering the already highly polished previous editions. Our editor and everyone else at Microsoft Press, as well as the dozens of reviewers (actual developers working on the features we describe!) were a big help in this area, so they deserve a very large thank you.

Of course, that’s only a small amount of the work required to create a new edition, so the bulk of the work went into creating new content that would cover the many changes and improvements that the 6.0 series of kernels added to the system, which, as you undoubtedly know, is nothing to sneeze at. However, before even discussing new content for the latest Windows release, we decided that certain older and still existing technologies and components of Windows merited some coverage in this release, especially given that many other older components had now been removed. Some of these components and mechanisms include:

  • The image loader in Ntdll.dll (the functions starting with Ldr)
  • The user-mode debugging framework (the Dbgk kernel functions and their DbgUi counterparts in Ntdll.dll)
  • 64-bit system call table and compaction
  • Kernel Patch Protection (Patchguard), introduced in 64-bit Windows Server 2003, so technically not a new Vista change
  • Hotpatch (patching at runtime) technology, also introduced in Server 2003
  • Enhanced description of the object manager component
  • Coverage of the pushlock synchronization primitive, added in XP and improved in Server 2003
  • Easier to read and updated scheduling section to cover only multiprocessor scheduling (introduced in Windows Server 2003, the older XP uniprocessor scheduler is now gone since Vista only ships multiprocessor binaries)
  • Enhancements for Non-Uniform Memory Architecture (NUMA), also introduced in Server 2003, and further improved each release
  • The crash analysis section has benefited from some more expert input thanks to seasoned reviewers, as well as certain enhanced troubleshooting scenarios (such as a stack trash)
  • The memory manager section has a new section on stacks and virtual address descriptors (VADs)
  • The crash dump analysis section now accurately describes crash dump file generation, which was improved in Server 2003
  • The Common Log File System (CLFS), introduced in Server 2003 R2, is now described in depth, as it has evolved from an optional component for servers into an essential part of the system, providing the underlying logging for the transactional registry (TxR) and file system (TxF).
  • EFI and exFAT technologies also have received better and more up to date information, as they evolved independently since the last edition

There have been smaller changes throughout the book, and you can imagine that a third pair of eyes has probably definitely helped at redefining certain terms, clarifying certain explanations, and added additional input to existing content.

Finally, we’re left with all the new content that was added specifically for this edition to cover the multiple changes in Vista and Server 2008 — I won’t list them all (because you should buy the book and discover it on your own!), but here’s a list of some of my favorite new sections and changes (this list may be long, but the total number of changes is actually more than double!)

  • User-mode locking mechanisms (run-once initialization, condition variables, and slim reader-writer (SRW) locks)
  • ALPC, advanced local procedure call
  • Hypervisor (Hyper-V)
  • Kernel Transaction Manager (KTM) as a section, as well as coverage of the built-in transactional registry (TxR) and transactional NTFS (TxF) in their respective sections
  • Code Integrity (and the Kernel Mode Code Signing policy)
  • Kernel Patch Protection, covering the latest Patchguard 3.0 features and details
  • WDI, the Windows Diagnostic Infrastructure
  • Completely revamped process and thread startup flow to cover the improvements to support protected processes and re-factor the process mechanisms, thanks to the hard work put in by Arun Kishan who owns the scheduler and process management code, which hadn’t been overhauled in a long while
  • Changes performed to the scheduler to better handle NUMA and SMP machines
  • The new worker factory kernel component which handles the user-mode and .NET thread pool
  • The re-architected storage stack (from the StorPort class driver to the volume and partition managers, as well as the new dynamic volume management and virtual disk service drivers)
  • In-depth coverage of UAC (User Account Control) and how it makes running as standard user more convenient for users, as well as information on related technologies such as integrity levels (ILs) and user interface privilege isolation (UIPI)
  • Another large section on the Windows Driver Foundation (WDF), including both KMDF (the Kernel Mode Driver Framework) and UMDF (its user-mode counterpart)
  • Updates on hardware no-execute (data execution prevention, or DEP) support, including the many flags and workarounds that are implemented
  • Complete coverage on BitLocker and TPM support — in my opinion one of the most   technical and complete descriptions of this feature and its implementation
  • Coverage of the new heap manager improvements in Vista, thanks to Adrian Marinescu once again
  • More efficient VACB (Virtual Address Control Block) array management in the cache manager
  • Completely new boot architecture, including support for UEFI/EFI, and the refactored boot process using Bootmgr and Winload (and Winresume)
  • Updates on the new error handling mechanism in Windows (WER), both for user-mode crashes and kernel-mode crashes (blue screen of death)
  • Performance: ReadyBoot and ReadyBoost are described in their appropriate sections
  • Tools: WDK, Reliability and Performance Monitor, updates to driver verifier and its Vista options and improvements, updated and new Sysinternals tools, as well as my own Winsider Seminars and Solutions tools.

One of the two chapters that I feel deserve more than just a bullet include the memory manager chapter, which covers one of the components that receives the most continuous attention and optimization even from one build to the next, thanks to the heavy work Landy Wang, its owner, puts in. These include the new dynamic virtual address space layout in kernel-mode, as well as the ASLR technology in user mode, the new NUMA optimizations, page fault clustering and other working set and PFN database optimizations and improvements, and last but not least, an entire section dedicated to the new memory prioritization and performance enhancing technology that is SuperFetch. So many people don’t understand what SuperFetch does, including myself when I first set out to document this feature, that I feel this section alone is worth getting the new edition — this is information you won’t find anywhere else at this level of accuracy (and a large part of that is thanks to the SuperFetch developers that spent entire days over the phone and lunch with me to make sure we nailed this).

The last chapter that deserves a mention is the networking chapter. I almost left this chapter as last during the book revision, thinking that there were very few things worth mention and that really needed updating. This was a mistake on my part, largely due to my inexperience with this one part of Windows (and technically, not a part of the kernel itself). I soon discovered that I was dead wrong, and that networking technologies in Vista had received among the most improvements, changes and new features, as well as a major deprecation of older technologies and services.

This chapter probably got the most updates, and almost every page has been changed, from the new user-level APIs, to the redesigned TCP/IP stack, the kernel-level deprecation of TDI and introduction of WSK (WinSock Kernel), the new NDIS 6.0, the new Windows Filtering Platform (WFP) and more. All the top services are now described, such as BITS (the Background Intelligent Transfer Service), the location and topology services such as Network Location Awareness (NLA) and Link-Layer Topology Discovery (LLTD), the quality of service services (the new policy-based QoS and qWAVE, or Quality Windows Audio Video Experience, come to mind) and let’s not forget the new peer to peer service infrastructure, as well as the Peer Name Resolution Protocol (Pnrp). More minor changes include updates to the Distributed File System (DFS) technologies, the binding infrastructure and deprecation of older networking technologies such as NetBEUI and ATM.

If you weren’t sure what’s new in this edition and if it’s really worth buying even if you own the 4th Edition, I hope this convinces you otherwise — it’s a significant and worthwhile update, and goes beyond just covering Vista. As Dave and Mark mentioned in their video, it’s also an unbeatable reference and tool for your understanding of Windows 7, since it builds upon the Vista foundation and, in most ways, works identically. And for those things that did change, you can bet we’ll have a 6th Edition out to cover the latest OS, and it’ll be a lot quicker out the door too.

Finally, on a more personal note, I’d like to publicly state that working with Dave and Mark was as much a delight as it was an honor. I have worked with, and for, many other people in the past, and could not have hoped for a smoother and more productive cooperation and work relationship than this one. As a neophyte to writing a book (especially of this magnitude) and keeping track of the dozens of things that needed to get done (from screenshots, to reviews, to writing content, to writing tools), I was probably not the most organized and timely co-author out there, but Mark and Dave understood this and made this a learning experience as well as a unique professional opportunity. I would like to thank them for bringing me on board the project in the first place, staying the course with me, and being there at every turn with suggestions, advice and help, from cross-referencing through sources to setting up meetings with Microsoft developers. I cannot wait to get started on the 6th Edition.