Category: Work

Posted on

Windows Internals 5th Edition, at last!

I am very pleased to announce that the 5th Edition of the Windows Internals book series is finally shipping for the past couple of weeks, and hard copies are now arriving in the hands of most customers! As my last blog post indicates, I took a hiatus from most of my typical work in the security and reverse engineering field and focused all of my energy into the book, outside of other commitments such as the Windows internals classes I teach for David Solomon Expert Seminars, so I thought it helpful to give my own perspective on the book itself, and on my work and the experience of working alongside the two legends of Windows internals knowledge. With that in mind, if you haven’t done so already, I’d invite you to read over Mark’s blog post on the book, as well as take a look at the short interview that David and Mark did on Channel 9 — it covers a lot of information on the latest release that I might not have covered in this post.

When we first set out to work on the 5th Edition, we decided early on to make three underlying changes to the existing content (and by extension, any new content as well). The first was to remove all references to previous versions other than the ones targeted by this edition (Windows Vista SP1 and Server 2008, specifically). We had realized that covering what would now be 5 different versions of the kernel (5.0 through 6.0 SP1) would generate too much redundant text, confusing explanations and questionably useful comparisons (such as let’s say, the evolution of how many buckets a given kernel component uses to store string hashes — driven probably only by the increase in average computer specifications across releases, and not some deeper mystery in the kernel). Windows mechanisms weren’t the only thing trimmed down to cover today’s reality however — references to old tools, unsupported resource kits, etc., were also removed.

When working on this edition, this was a significant challenge, because while it is relatively easy to get lots of information on major new Vista changes and improvements, it’s much harder to track down the little details that may have been valid at one time, but not anymore, and to rid the book of any archaic references, algorithms or values. Additionally, the second decision was to try minimizing giving out the values of certain variables and tuning parameters that the kernel uses. For one part, this creates the unfortunate scenario of developers copying down those values and then later depending on them in their software, which is a bad idea that only leads to more crashes for customers. For the other, it also makes it hard for us, as authors, to have to track down the exact values every single time Windows is updated — additionally, if the values changed significantly, people might expect explanations for these changes, when sometimes they are just as simple as “performance testing showed this to be a better number in today’s computers”. Because the variable name, its usage and the scale of its value are still referenced however, this still gives the reader the required understanding and, if someone really wants the value, they can use the same tools as the authors to obtain it (such as using the Windows Debugging Tools with the appropriate symbols).

As you can see, an important part of this update wasn’t even related to adding new information on Vista and Server 2008, but rather to bring the book up to even higher quality and technical standards, a lofty goal considering the already highly polished previous editions. Our editor and everyone else at Microsoft Press, as well as the dozens of reviewers (actual developers working on the features we describe!) were a big help in this area, so they deserve a very large thank you.

Of course, that’s only a small amount of the work required to create a new edition, so the bulk of the work went into creating new content that would cover the many changes and improvements that the 6.0 series of kernels added to the system, which, as you undoubtedly know, is nothing to sneeze at. However, before even discussing new content for the latest Windows release, we decided that certain older and still existing technologies and components of Windows merited some coverage in this release, especially given that many other older components had now been removed. Some of these components and mechanisms include:

  • The image loader in Ntdll.dll (the functions starting with Ldr)
  • The user-mode debugging framework (the Dbgk kernel functions and their DbgUi counterparts in Ntdll.dll)
  • 64-bit system call table and compaction
  • Kernel Patch Protection (Patchguard), introduced in 64-bit Windows Server 2003, so technically not a new Vista change
  • Hotpatch (patching at runtime) technology, also introduced in Server 2003
  • Enhanced description of the object manager component
  • Coverage of the pushlock synchronization primitive, added in XP and improved in Server 2003
  • Easier to read and updated scheduling section to cover only multiprocessor scheduling (introduced in Windows Server 2003, the older XP uniprocessor scheduler is now gone since Vista only ships multiprocessor binaries)
  • Enhancements for Non-Uniform Memory Architecture (NUMA), also introduced in Server 2003, and further improved each release
  • The crash analysis section has benefited from some more expert input thanks to seasoned reviewers, as well as certain enhanced troubleshooting scenarios (such as a stack trash)
  • The memory manager section has a new section on stacks and virtual address descriptors (VADs)
  • The crash dump analysis section now accurately describes crash dump file generation, which was improved in Server 2003
  • The Common Log File System (CLFS), introduced in Server 2003 R2, is now described in depth, as it has evolved from an optional component for servers into an essential part of the system, providing the underlying logging for the transactional registry (TxR) and file system (TxF).
  • EFI and exFAT technologies also have received better and more up to date information, as they evolved independently since the last edition

There have been smaller changes throughout the book, and you can imagine that a third pair of eyes has probably definitely helped at redefining certain terms, clarifying certain explanations, and added additional input to existing content.

Finally, we’re left with all the new content that was added specifically for this edition to cover the multiple changes in Vista and Server 2008 — I won’t list them all (because you should buy the book and discover it on your own!), but here’s a list of some of my favorite new sections and changes (this list may be long, but the total number of changes is actually more than double!)

  • User-mode locking mechanisms (run-once initialization, condition variables, and slim reader-writer (SRW) locks)
  • ALPC, advanced local procedure call
  • Hypervisor (Hyper-V)
  • Kernel Transaction Manager (KTM) as a section, as well as coverage of the built-in transactional registry (TxR) and transactional NTFS (TxF) in their respective sections
  • Code Integrity (and the Kernel Mode Code Signing policy)
  • Kernel Patch Protection, covering the latest Patchguard 3.0 features and details
  • WDI, the Windows Diagnostic Infrastructure
  • Completely revamped process and thread startup flow to cover the improvements to support protected processes and re-factor the process mechanisms, thanks to the hard work put in by Arun Kishan who owns the scheduler and process management code, which hadn’t been overhauled in a long while
  • Changes performed to the scheduler to better handle NUMA and SMP machines
  • The new worker factory kernel component which handles the user-mode and .NET thread pool
  • The re-architected storage stack (from the StorPort class driver to the volume and partition managers, as well as the new dynamic volume management and virtual disk service drivers)
  • In-depth coverage of UAC (User Account Control) and how it makes running as standard user more convenient for users, as well as information on related technologies such as integrity levels (ILs) and user interface privilege isolation (UIPI)
  • Another large section on the Windows Driver Foundation (WDF), including both KMDF (the Kernel Mode Driver Framework) and UMDF (its user-mode counterpart)
  • Updates on hardware no-execute (data execution prevention, or DEP) support, including the many flags and workarounds that are implemented
  • Complete coverage on BitLocker and TPM support — in my opinion one of the most   technical and complete descriptions of this feature and its implementation
  • Coverage of the new heap manager improvements in Vista, thanks to Adrian Marinescu once again
  • More efficient VACB (Virtual Address Control Block) array management in the cache manager
  • Completely new boot architecture, including support for UEFI/EFI, and the refactored boot process using Bootmgr and Winload (and Winresume)
  • Updates on the new error handling mechanism in Windows (WER), both for user-mode crashes and kernel-mode crashes (blue screen of death)
  • Performance: ReadyBoot and ReadyBoost are described in their appropriate sections
  • Tools: WDK, Reliability and Performance Monitor, updates to driver verifier and its Vista options and improvements, updated and new Sysinternals tools, as well as my own Winsider Seminars and Solutions tools.

One of the two chapters that I feel deserve more than just a bullet include the memory manager chapter, which covers one of the components that receives the most continuous attention and optimization even from one build to the next, thanks to the heavy work Landy Wang, its owner, puts in. These include the new dynamic virtual address space layout in kernel-mode, as well as the ASLR technology in user mode, the new NUMA optimizations, page fault clustering and other working set and PFN database optimizations and improvements, and last but not least, an entire section dedicated to the new memory prioritization and performance enhancing technology that is SuperFetch. So many people don’t understand what SuperFetch does, including myself when I first set out to document this feature, that I feel this section alone is worth getting the new edition — this is information you won’t find anywhere else at this level of accuracy (and a large part of that is thanks to the SuperFetch developers that spent entire days over the phone and lunch with me to make sure we nailed this).

The last chapter that deserves a mention is the networking chapter. I almost left this chapter as last during the book revision, thinking that there were very few things worth mention and that really needed updating. This was a mistake on my part, largely due to my inexperience with this one part of Windows (and technically, not a part of the kernel itself). I soon discovered that I was dead wrong, and that networking technologies in Vista had received among the most improvements, changes and new features, as well as a major deprecation of older technologies and services.

This chapter probably got the most updates, and almost every page has been changed, from the new user-level APIs, to the redesigned TCP/IP stack, the kernel-level deprecation of TDI and introduction of WSK (WinSock Kernel), the new NDIS 6.0, the new Windows Filtering Platform (WFP) and more. All the top services are now described, such as BITS (the Background Intelligent Transfer Service), the location and topology services such as Network Location Awareness (NLA) and Link-Layer Topology Discovery (LLTD), the quality of service services (the new policy-based QoS and qWAVE, or Quality Windows Audio Video Experience, come to mind) and let’s not forget the new peer to peer service infrastructure, as well as the Peer Name Resolution Protocol (Pnrp). More minor changes include updates to the Distributed File System (DFS) technologies, the binding infrastructure and deprecation of older networking technologies such as NetBEUI and ATM.

If you weren’t sure what’s new in this edition and if it’s really worth buying even if you own the 4th Edition, I hope this convinces you otherwise — it’s a significant and worthwhile update, and goes beyond just covering Vista. As Dave and Mark mentioned in their video, it’s also an unbeatable reference and tool for your understanding of Windows 7, since it builds upon the Vista foundation and, in most ways, works identically. And for those things that did change, you can bet we’ll have a 6th Edition out to cover the latest OS, and it’ll be a lot quicker out the door too.

Finally, on a more personal note, I’d like to publicly state that working with Dave and Mark was as much a delight as it was an honor. I have worked with, and for, many other people in the past, and could not have hoped for a smoother and more productive cooperation and work relationship than this one. As a neophyte to writing a book (especially of this magnitude) and keeping track of the dozens of things that needed to get done (from screenshots, to reviews, to writing content, to writing tools), I was probably not the most organized and timely co-author out there, but Mark and Dave understood this and made this a learning experience as well as a unique professional opportunity. I would like to thank them for bringing me on board the project in the first place, staying the course with me, and being there at every turn with suggestions, advice and help, from cross-referencing through sources to setting up meetings with Microsoft developers. I cannot wait to get started on the 6th Edition.

Posted on

My Summer At Apple

Three months ago, I posted about my experience with some of the most exciting tech companies that I had a chance to interview with and explained my decision behind joining Apple. This week, my internship comes to a close, and it’s time to review that decision and share with you my intern experience at Apple.

No amount of past experience and stories could’ve prepared me for the amazing time I had at Apple and I think I made one of the wisest decisions in my career this summer. My internship position at Apple was part of the Core OS group, which is mainly responsible for Mac OS X. However, this wasn’t just any Core OS internship — my job was at the forefront of Apple’s most anticipated product in decades — the iPhone.

When Steve Jobs announced that the iPhone would be running OS X, many people had doubts at first, since OS X had never been talked about in the embedded market. But clearly, the phone is indeed running the OS, and most of the Core frameworks and foundations that developers were used to seeing on the desktop are on the phone. Several teams at Apple were responsible for this, and none more critical then the Core OS Embedded Team, which I worked on. Although the specifics of my job and others around me is not something I can disclose, it’s needless to say that I worked with some of the brightest engineers at Apple, who took an x86 Desktop OS and turned it into a powerful embedded OS (everyone now knows that the iPhone is running on ARM) that still supported the same applications and frameworks as its counterpart.

As a Windows NT kernel expert and Intel x86 assembly guru and reverse engineer on a Dell laptop, the task of working with Darwin for ARM on a Mac Pro was new to me on all possible levels. And yet, I was able to succesfully leverage my knowledge of OS and kernel design, my attention to small details and interest in hardware-to-software relationships to surpass the goals of my internship, and pleasently surpise all levels of management as well as my co-workers. But how about the Apple internship experience?

Interns at Apple get some great benefits that other companies don’t always bother with, including a health care plan, relocation assistance (paid round-trip airfare) as well as a monthly housing stipend. My expenses this summer were minimal thanks to the care Apple makes into releaving interns of some typical moving-related stress. One of the coolest things about being an intern however, is the chance to attend Executive Speaker Series, which are weekly lunchtime events in which one of Apple’s senior executive staff comes to talk and interact with interns. The first one, was of course Steve Jobs himself. It’s not just for show either — one of the interns got up and asked Steve how he can get his idea across to an executive. Steve smiled and replied: “Go ahead.” In total, I think there must’ve been about 12 different speakers this year.

On the topic of selling ideas to executives, nothing works better then the iContest, in which teams of interns submit a feature or product design idea, improvement, or business plan for various judges to evaluate (made up of senior staff at Apple). The top 10 teams get to compete face-to-face, with a presentation to the judges (and other interns) followed by a harsh question and answer period in which the judges grill and roast the interns. What may make business sense to a 21 year old may sound like a bad idea to someone with 40 years of market experience, and they won’t be shy to let that be known. In the end, the comments are constructive, and like in any competition, the best ideas are usually scrutinized the most. The top three teams of this competition win a variety of prizes/awards, and most get their feature or idea implemented or marketed, so the contest results and presentations are some of the most confidential pieces of information an intern has to live with. My iContest idea was among the top ten chosen for the finale, and while we unfortunately did not win any of the top three prizes, I’m confident we came very close.

Talking and interacting with the senior staff doesn’t end there for interns. After about 8-10 weeks at Apple, interns in each group give a presentation to the VP of their division, as well as to the managers of other interns in that group. These presentations are usually very important to a succesfull internship program, since the quality of the intern usually reflects on their manager. This is another competitive event, as the managers get to vote on each presentation, and the top intern of each group gets to present to the Senior VP or President of the division, usually a member of the executive staff. My presentation to the VP/managers was chosen as the best and most interesting one, so I was invited to present to Bertrand Serlet, the Vice President of Software Engineering at Apple. Unfortunately, I had to attend Blackhat that week, so I gave my place to the second best intern. It’s worthwhile to mention that all interns had amazing presentations, and they all put hard work into their summer.

But Apple isn’t all about work and competing with other interns. Field trips and excursions, the food harvest, parties and welcome/goodbye dinners were some of the other fun activities offered to interns, who also got to attend WWDC 2007’s party for free. And of course, one can’t forget the corporate games, in which teams compete in a variety of games, from serious sports to water games and tug of war. Every day of the week, interns on the mailing list arranged various activities, from volleyball to movie and poker nights. Finally, interns had access to all the perks regular Apple employees have, including foosball and other game rooms, a sports court, the health and fitness center, as well as campus shuttles to the Caltrain station (with Wifi, of course!).

Even though the work environment at Apple is relaxed and flexible, I had to work hard, but the rewards and results were worth all the effort. The team I worked on (and the entire company) did a great job on the iPhone, and the future for Apple continues to be exciting and new innovations are always on the way. There are few more rewarding experiences than walking into an Apple Store and seeing people of all ages gaze in awe at our products. One day, those same people will be looking at a product, feature or device that I participated on, and I’ll smile back at them, knowing I had a hand in getting it on that shelf.

All in all, I’m defintely looking forward to returning to Cupertino once more.

To find more about the internship program at Apple, visit the official site here. Please don’t ask any questions outside the scope of what is mentionned on that page; there are even more exciting things going on at Apple, but you’ll have to come work here to find out!