How Control Flow Guard Drastically Caused Windows 8.1 Address Space and Behavior Changes

Windows 8.1 radically changes the address space layout of the system by finally removing the 44-bit limitation which I described in one of the earliest blog posts on this website (and which Wikipedia even links to!). This is a little-known detail about the operating system, and an odd thing for Microsoft not to emphasize on with more aplomb, especially given that 8.1 is considered a “patch” of Windows 8.

Now, you may think that 16 TB to 256 TB is a meaningless change since no applications currently use even a fraction of that space, but the main benefit of this change are not the ability to allocate additional memory, but rather the increased entropy space available for Address Space Load Randomization (ASLR), especially given that Windows 8 introduced High Entropy ASLR (HEASLR), Top-down Randomization and Anonymous Memory Randomization.

Additionally, another key change was done in Windows 8.1 that is not mentioned anywhere. As Pavel Lebedinsky, one of the lead SDETs on the Memory Manager and an extremely helpful individual indicated on one of the blog posts from Mark Russinovich:

1. Reserved memory does contribute to commit charge, because the memory manager charges commit for pagetable space necessary to map the entire reserved range. On 64 bit this can be a significant number (reserving 1 TB of memory will consume approximately 2 GB of commit).

This means that attempting to reserve the full 8 TB of memory on Windows 7 results in 16 GB of commit, which is beyond’s most people’s commit limit, especially at the time. In Windows 8.1, this would result in 128 GB of commit being used, which only a beefy server would tolerate. While such large memory reservations are unusual, they do have usefulness in certain scenarios related to security and low-level testing. This Windows behavior prevented such reservations from reliably working, but in Windows 8.1, the limitation has been removed!

Indeed, you can easily test this by using the TestLimit tool from the Windows Internals Book, and run it with the -r option (and preferably with a large enough block size). Here’s a screenshot of hitting the 128 TB reservation:

testlimit

And here’s the resulting view in VMMap, which does not show the expected page table commit charge, but rather a much smaller size (256 MB).

memvm

So why did Microsoft change this behavior in Windows 8.1? Well, Windows 10, as well as Windows 8.1 Update 3 (November Update) make this clear. As I previously tweeted, these OS versions enable Control Flow Guard (CFG), a feature that laid dormant in the first versions of Windows 8.1. In order to function, CFG requires the use of optimized bitmaps in order to determine the validity of indirect calls, and on 64-bit Windows, this bitmap requires 2 TB of space. Not only would this cut the Windows 8 address space by 25%, it would’ve also resulted in 4 GB of per-process commit!

Here’s a screenshot of Process Hacker showing how all CFG-enabled processes now use 2 TB of virtual address space:

2tb

The final effect of this change from 8 TB to 128 TB is that the kernel address space layout has significantly changed. And sadly, the !address extension in WinDBG is broken and continues to show the Windows 8 address space layout (which I expanded on during my Blackhat 2013 talk), while the Windows Internals book is stuck on Windows 7 and doesn’t even cover Windows 8 or higher.

Therefore, I publish below what I believe to be the only public source of information on the Windows 8.1 x64 memory layout. One of the benefits of this new layout is that it now becomes extremely easy by using the first 5 or 6 nibbles of an address to determine where it’s coming from. For example, 0xFFFFD… is a kernel stack, 0xFFFFC… is paged pool, 0xFFFFF8… is a loaded image (driver or kernel), and 0xFFFFE… is nonpaged pool.

StartEndSizeDescription
FFFF0000`00000000FFFF07FF`FFFFFFFF8TBMemory Hole
FFFF0800`00000000FFFFAFFF`FFFFFFFF168TBUnused Space
FFFFB000`00000000FFFFBFFF`FFFFFFFF16TBSystem Cache
FFFFC000`00000000FFFFCFFF`FFFFFFFF16TBPaged Pool
FFFFD000`00000000FFFFDFFF`FFFFFFFF16TBSystem PTEs
FFFFE000`00000000FFFFEFFF`FFFFFFFF16TBNonpaged Pool
FFFFF000`00000000FFFFF67F`FFFFFFFF6.5TBUnused Space
FFFFF680`00000000FFFFF6FF`FFFFFFFF512GBPTE Space
FFFFF700`00000000FFFFF77F`FFFFFFFF512GBHyperSpace
FFFFF780`00000000FFFFF780`00000FFF4KShared User Data
FFFFF780`00001000FFFFF780`BFFFFFFF~3GBSystem PTE WS
FFFFF780`C0000000FFFFF780`FFFFFFFF1GBWS Hash Table
FFFFF781`00000000FFFFF791`3FFFFFFF65GBPaged Pool WS
FFFFF791`40000000FFFFF799`3FFFFFFF32GBWS Hash Table
FFFFF799`40000000FFFFF7A9`7FFFFFFF65GBSystem Cache WS
FFFFF7A9`80000000FFFFF7B1`7FFFFFFF32GBWS Hash Table
FFFFF7B1`80000000FFFFF7FF`FFFFFFFF314GBUnused Space
FFFFF800`00000000FFFFF8FF`FFFFFFFF1TBSystem View PTEs
FFFFF900`00000000FFFFF97F`FFFFFFFF512GBSession Space
FFFFF980`00000000FFFFFA70`FFFFFFFF1TBDynamic VA Space
FFFFFA80`00000000FFFFFAFF`FFFFFFFF512GBPFN Database
FFFFFFFF`FFC00000FFFFFFFF`FFFFFFFF4MBHAL Heap
Table describing the various 64-bit memory ranges in Windows 8.1

2 Responses to “How Control Flow Guard Drastically Caused Windows 8.1 Address Space and Behavior Changes”

  1. Yuhong Bao says:

    I think MS does mention CMPXCHG16B as part of the system requirements for Win8.1.

  2. aionescu says:

    They do, that’s true. They don’t quite explain why though :)

Leave a Reply