Reverse Engineering Windbg Commands for Profit

In this article, I will introduce benefit of reverse engineering Windbg for understanding the Windows kernel with looking at an undocumented command, fixing an issue in it and re-implementing the same functionality on a device driver.


Windbg is a powerful resource not only because you can see thorough run-time information even if you do not know how to manually do that but also you can learn how Windbg does that with reverse engineering it. Implementation of the !timer command is, for example, where you should examine if you want to know how to enumerate all scheduled timer callbacks.

But, what if you cannot find the command that works with internals you are interested in? In my case, I was looking for a way to list items inserted into the work queues with ExQueueWorkItem(), and documents of Windbg did not tell what command I could use.

Finding an Undocumented Command

There are many undocumented commands in Windbg. By using strings.exe against DLL files under the Windbg folder, you will get hints about them and/or where to look at. Here is a result of search for "workqueue":

Debuggers\x64>strings -n 5 -s *.dll | findstr /i workqueue
...
Debuggers\x64\winxp\kdexts.dll: **** NUMA Node %i RealTime WorkQueue
Debuggers\x64\winxp\kdexts.dll: **** NUMA Node %i HyperCritical WorkQueue
Debuggers\x64\winxp\kdexts.dll: **** NUMA Node %i SuperCritical WorkQueue
Debuggers\x64\winxp\kdexts.dll: **** NUMA Node %i Critical WorkQueue
Debuggers\x64\winxp\kdexts.dll: **** NUMA Node %i Delayed WorkQueue
Debuggers\x64\winxp\kdexts.dll: **** NUMA Node %i Normal WorkQueue
Debuggers\x64\winxp\kdexts.dll: **** NUMA Node %i Background WorkQueue
Debuggers\x64\winxp\kdexts.dll: **** Critical WorkQueue
Debuggers\x64\winxp\kdexts.dll: **** Delayed WorkQueue
Debuggers\x64\winxp\kdexts.dll: **** HyperCritical WorkQueue
Debuggers\x64\winxp\kdexts.dll: ExWorkQueue
...

With firing up IDA, I easily found that there was an undocumented (*1) command named !exqueue and it was exactly what I wanted.

kd> !kdexts.help
...
exqueue [flags]              - Dump the ExWorkerQueues
...

Fixing an Issue

The issue was that this commend did not work against Windows 8.1 and 10 targets.

kd> !exqueue
GetGlobalValue: unable to get NT!KeNumberNodes type size

So, I started to debug and reverse engineer kdexts.dll a bit more, then noticed that Windbg was failing to read the nt!KeNumberNodes, which is always 1 unless you use NUMA, and a solution was just patching code to set 1 to eax.

.text:00000001801161DA   lea rcx, aNtKenumbernode ; "NT!KeNumberNodes"
.text:00000001801161E1   call read_global_variable
.text:00000001801161E1   mov eax, 1
.text:00000001801161E6   mov rbx, cs:qword_1801A8500
.text:00000001801161ED   mov rcx, rbx
.text:00000001801161F0   mov rbp, rax

Here is an output:

kd> !exqueue

**** NUMA Node 0 - ( Threads: 7/4096 ) ****
...
 -> Priority 12 - ( Concurrency: 0/2 )
...
    ExWorkItem (ffffe00084a91160) 
      Routine ListWorkItems!<lambda_bae...> (fffff800746e5290) 
      Parameter (fffff800746e7220)
    ExWorkItem (ffffe0008105ff10) 
      Routine dxgkrnl!DxgkpProcessTerminationListThread (...) 
      Parameter (ffffe0008105fbb0)
    WdfWorkItem (ffffe00082b035a0) 
      Routine cdrom!IoctlWorkItemRoutine (fffff80072c6e900)
    ExWorkItem (fffff8006b8ff0c0) 
      Routine nt!CmpDelayDerefKCBWorker (fffff8006ba5d008) 
      Parameter (0000000000000000)
...
 -> Priority 31 - ( Concurrency: 0/2 )

 -> Associated Threads
THREAD ffffe000828fe040  Cid 0004.0170  Teb: 0000000000000000 Win32Thread: 0000000000000000 WAIT
...

Re-implementing the Command

My goal was, however, not to use the command; the goal was to know how it works, so I reverse engineered !exqueue more to learn the details of the work queues and how to enumerate items in them by hand.

Analyzing Windbg commands is often a lot easier than analyzing the kernel file because it contains a lot of strings that help you know what structures are dealt with.

Image1: Strings in code of !enqueue

This time was not exception. Basically, for Windows 8 and later, the command gets a NUMA node (nt!_ENODE) structure containing a reference to the work queues with looking at nt!KeNumberNodes and nt!KeNodeBlock first. Here, I follow the same procedure as !exqueue using basic commands.

There is only one NUMA node block as I do not configure NUMA.

kd> dw nt!KeNumberNodes
fffff803`f5774008  0001

kd> dps nt!KeNodeBlock
fffff803`f576c800  fffff803`f56c3240 nt!ExNode0
fffff803`f576c808  00000000`00000000
fffff803`f576c810  00000000`00000000

Then, the command refers to the _ENODE.ExWorkQueue.WorkPriQueue.EntryListHead field, which is an array of the prioritized work queues.

kd> dt nt!_ENODE ExWorkQueue.WorkPriQueue. fffff803`f56c3240
   +0x0c0 ExWorkQueues              : [8]
   +0x100 ExWorkQueue               :
      +0x000 WorkPriQueue              :
         +0x000 Header                    : _DISPATCHER_HEADER
         +0x018 EntryListHead             : [32] _LIST_ENTRY [
                                               0xfffff803`f56c3358 -
                                               0xfffff803`f56c3358 ]
         +0x218 CurrentCount              : [32] 0n0
         +0x298 MaximumCount              : 4
         +0x2a0 ThreadListHead            : _LIST_ENTRY [
                                               0xffffe000`eb78f248 -
                                               0xffffe000`ecd89a88 ]

Each element of the EntryListHead is a list of work items (nt!_WORK_QUEUE_ITEM), and the array represents priorities of each list; in other words, there are 32 work queues and each manages items with priority 0 (the lowest) to 31 (the highest) respectively.

kd> dt nt!_ENODE -a ExWorkQueue.WorkPriQueue.EntryListHead fffff803`f56c3240
          ...
          [12] _LIST_ENTRY [ 0xffffe000`846f62f0 - 0xffffe000`82bea2c0 ]
          ...

kd> dt nt!_WORK_QUEUE_ITEM 0xffffe000`846f62f0
   +0x000 List             : _LIST_ENTRY [ 0xffffe000`8305d130 -
                                           0xfffff800`6b8ca418 ]
   +0x010 WorkerRoutine    : 0xfffff800`746dd290     
                             void  ListWorkItems!<lambda_7d382b...>+0
   +0x018 Parameter        : 0xfffff800`746df220 Void

The !exqueue command shows the contents of the lists as well as associated worker threads referred by the ThreadListHead field.

All you can do, I can do :) I wrote a driver that dumps all items in each work queue to confirm that the above analysis was correct. Note that because this driver is just PoC, it works only on the 64 bit version of Win8.1 and Win10.

https://github.com/tandasat/ListWorkItems

Conclusion

Analyzing Windbg commands often gives you good understanding of the Windows kernel with a less effort than analyzing the kernel file, and even if you do not find a helpful command at a glance, there may be an undocumented command which you can reverse engineer to unveil the Windows internals.

Side Notes

1. It seems that !exqueue used to be documented and then abandoned for some reasons. I found a description about it on a help file came with the Windbg version 6.11.001.404.

• The ExWorkQueues field in the _ENODE also holds pointers to the WorkPriQueue structures. It is likely that the structures are allocated for each processor but only one associated with the processor 0 is really used. 

• Apparently, an item does not go to the queue if any of worker threads is in a wait state (ie, waiting for an item) when the item is being queued. I guess the item is associated with a thread without being stored in the queue for performance in this case. For this reason, the command does not show the first item when multiple items are queued at once. 

Section Based Code Injection and Its Detection

Summary

I wrote a small tool to detect a possible code injection even if it is done by only section APIs.

----

A few weeks ago, I had an opportunity to analyze ransomware referred as Urausy. At a very initial stage of analysis, its behaviour seemed to be nothing surprising to me; it injected code into explorer.exe, and the injected code spawned svchost.exe hosting malicious code and initiated main ransom activities (More detaied analysis can be found on avast! blog).

Image 1: Process Tree 

I expected that the sample was injecting code using VirtualAllocEx() and CreateRemoteThread(), or relevant APIs such as NtWriteVirtualMemory() NtCreateThread/Ex(). But through analysis, I noticed that it was using none of them for the injection but using section APIs instead. Malware replaced the existing ntdll image on explorer.exe with a newly created section containing an inline hook on NtClose() and code responsible for starting svchost.exe. Output of VMmap indicates that an image of ntdll no longer exists and replaced with a shared Executable/Readable/Writable section after this injection.

Image 2: Memory Map of Explorer.exe (Before Infection)

Image 3: Memory Map of Explorer.exe (After Infection) 

Then, I started to wonder what if explorer.exe did not do any obvious activities I could easily spot and the sample did many other bad things besides that in meaningful ways (i.e., non junk operations)? I could miss the injected code.

It is also true of the case of the traditional code injection with VirtualAllocEx() and CreateRemoteThread(), but we are less likely to overlook it as we always expect to see that these APIs are used for injection and have tools or systems that tell us occurrence of typical thread injection.

So I wrote a driver, RemoteWriteMonitor, monitors inter-process memory modification by hooking NtWriteVirtualMemory() and ZwMapViewOfSection() to assist analysts to find this section based injection. This tool should report all possible code injections because if you want to execute your own code on another process from the user-mode, you need to either (1) write something onto the other process using those APIs (as far as I can think of), or (2) use a DLL file in conjunction with SetWindowsHookEx() or other type of injection mechanisms which is very easy to find due to preceding a disk write operation.

Let us see what it does in case of another Urausy sample I found on Malwr. If you installed the driver and run the sample, you see that the sample is mapping sections onto explorer.exe using ZwMapViewOfSection().

Image 4: Output on DebugView

This tool also saves the contents of memory being written as <SHA1>.bin so that you can examine what it is later. In this example, written data was code and a PE image respectively.

Image 5: File Contents (Code upside and PE downside)

This tool is more like PoC and does not have rich functionality, but I hope it helps you understand this uncommon injection method and its detection.

ARM Exception Handling and an IDAPython Script

Windows RT has differences in several points, and implementation of SEH is one of them. To sort out my understanding of ARM exception handling, I wrote an IDAPython script that interprets SEH information in an Windows RT PE file and applies it to an IDB. Here is an example of how this script helps you (I use one of PatchGuard routines uses SEH to obfuscate its code flow):

Image1: Before Use (plain output of IDA)

Image2: After Use

 In the image2, comments show that there is a __try/__except block around a call to __rt_sdiv().

Image3: Exception Filter

If you look at the location of an exception filter, you will find that the exception filter is calling another interesting looking function, which is actually authentic PatchGuard code flow. You could miss this path if you were just looking at plain output of IDA like the image1. This script will help you tell existence of SEH handlers.

About the internal of ARM exception handling, I do not explain it here as there is detailed enough explanations on MSDN[1] to understand it, but in short, it is fairly similar to one on x64. For instance, each function in a file is dictated by a RUNTIME_FUNCTION structure located in a .pdata section, and the structure points to an .xdata record consists of a SCOPE_TABLE structure and an array of its entries describing ranges of __try blocks, addresses of except filters and body blocks (or finally blocks). These are all essentially the same design as x64. 

As a note, I listed some references below which may complement your understanding of ARM exception handing[2][3][4][5]. Hope you enjoy them and my script too. 

1. ARM Exception Handling
2. Exceptional behavior: the Windows 8.1 X64 SEH Implementation
   References listed at the top of the articles are all exceptionally good, apart from this article.
3. RtlLookupFunctionEntry function
   Returns a corresponding .pdata entry for a given address.
4. .fnent (Display Function Data)
   You can dump .pdata/.xdata information with it.
5. Improving IDA Analysis of x64 Exception Handling
   An x64 version of my script. Very handy.

A List of PatchGuard v8.1 Related Functions on x64 and ARM

I was working on analyzing PatchGuard on Windows RT 8.1 (which runs on ARM) last two months and got that work done recently. Analysis tuned out to be a lot easier than I expected mostly because PatchGuard's code was written in C and had the almost same structure on both x64 which I had already analyzed and ARM.

In order to look at PatchGuard on Window RT 8.1, almost all I had to do was to identify PatchGuard related functions and map them with corresponding functions on x64.

Here is a table showing that mapping (ones have different names between platforms are highlighted).

x64	ARM
CcAdjustBcbDepth	CcUnmapBehindLazyReader
CcBcbProfiler	CcDelayedFlushTimer
CcInitializeBcbProfiler	CcPrepareDelayedFlushTimers
CmpAppendDllSection	ExpWnfAcquireNameInstanceShared
CmpEnableLazyFlushDpcRoutine	CmpEnableLazyFlushDpcRoutine
CmpLazyFlushDpcRoutine	CmpLazyFlushDpcRoutine
DeferredRoutine	<NoSymbol>
ExInitSystemPhase2	ExInitSystemPhase2
ExpCenturyDpcRoutine	ExpCenturyDpcRoutine
ExpTimerDpcRoutine	ExpTimerDpcRoutine
ExpTimeRefreshDpcRoutine	ExpTimeRefreshDpcRoutine
ExpTimeZoneDpcRoutine	ExpTimeZoneDpcRoutine
FsRtlMdlReadCompleteDevEx	RtlpExecuteHandlerForUnwind_xdata_compact
FsRtlUninitializeSmallMcb	ExpPrefetchPushLock
IopTimerDispatch	IopTimerDispatch
KeCompactServiceTable	KeCompactServiceTable
KeInitAmd64SpecificState	KeArmDiscoverCacheTopology
KiBalanceSetManagerDeferredRoutine	KiBalanceSetManagerDeferredRoutine
KiDispatchCallout	CcDelayedFlushTimer
KiDpcDispatch	<NoSymbol>
KiFastGetCallersAddress	KiFastGetCallersAddress
KiFatalExceptionFilter	KiFatalExceptionFilter
KiFilterFiberContext	KiArmDiscoverCacheTopology
KiGetGdtIdt	<NoSymbol>
KiLockExtendedServiceTable	KiLockExtendedServiceTable
KiLockServiceTable	KiLockServiceTable
KiMcaDeferredRecoveryService	KiInitializeExternalCacheController
KiScbQueueScanWorker	PopPdcSampleIdleTimeouts
KiServiceTablesLocked	KiServiceTablesLocked
KiTimerDispatch	<NoSymbol>
PopPoCoalescinCallback	PopPoCoalescinCallback
PopThermalZoneDpc	PopThermalZoneDpc
PsQueryThreadTerminationPort	PspGetReaperLink
RtlLookupFunctionEntryEx	CmpFlushLockedHives
SdbpCheckDll	PspInitDeferredResourceReservation
<NoSymbol>	CmpDelayFreeTMWorker
<NoSymbol>	FsRtlPrivateResetHighestLockOffset
<NoSymbol>	FsRtlReInitializeTunnelCache
<NoSymbol>	FsRtlRemovePerStreamContextEx
<NoSymbol>	KiCheckForDivideOverflow
<NoSymbol>	KiRundownScbQueue
<NoSymbol>	RtlInsertSmallIndex

These functions were taken from an ntoskrnl.exe version 6.3.9600.17476 and either only used by PatchGuard or have some importance from the point of view of analysis. For example, IopTimerDispatch() is not a PatchGuard dedicated function but can be used as one of its DPC routines, while KeInitAmd64SpecificState() and KeArmDiscoverCacheTopology() are dedicated and only used to initiate PatchGuard.

It seemed that some more functions were added for PatchGuard since Windows 10, but most, if not all, of these functions still remain the same name, so though this list is unlikely to be perfect, it would help you start your own analysis on both x64 and ARM.

Debugging Early Boot Stages of Windows

Recently, I have spent some time for reverse engineering bootkit. It has been a fun exercise, but I had to struggle for setting up the environment before that as I could not find a page explains these steps. So as a note for me, I wrote down how to build a bootkit debugging environment as well as how to configure Windows in order to attach a debugger at some early uncommon boot stages.

Boot Processes

Here are boot processes of BIOS based Windows XP and Windows 7 systems. I will discuss each stage except for BIOS (POST) and Ntoskrnl.exe.

Boot Process (XP)

BIOS (POST) -> MBR -> VBR -> Ntldr (Real-Mode) -> Ntldr (Protected-Mode) -> Ntoskrnl.exe

Boot Process (Windows 7)

BIOS (POST) -> MBR -> VBR -> Bootmgr (Real-Mode) -> Bootmgr (Protected-Mode) -> Winload.exe -> Ntoskrnl.exe

Debugging

MBR, VBR, Ntldr (Real-Mode) and Bootmgr (Real-Mode) 

We need Bochs as no break points are provided on the course of these steps.

Installing OS on Bochs is not hard unless you try to find the perfect configurations such as smooth mouse movement, correct clock speed and working NICs. Since we are not going to use this OS anything but boot debugging, I do not encourage you to spend time for it.

What you need to do to install OS are roughly as follows:

1. Create a flat hard disk image with 10GB size using bximage.exe.
2. Configure bochs.bxrc with boches.exe to use the created hard disk image and an OS installer ISO image.
3. Start to install OS.

Installation may take a few hours. I also strongly recommend you to use the latest version of Bochs to avoid unnecessary troubles. Even if your old IDA Pro does not work with the latest one like my case, you can switch to old one after you completed the installation process for debugging. Here are my bochsrc files for version 2.6.6 and 2.4.6 with nearly identical configurations. You may use them as samples when you are not familiar with Bochs.

Once you have finished installing the OS (no need for applying Windows updates), you can debug MBR and VBR with either running it with bochsdbg.exe or corroborating with IDA Pro. If you hope to take the former way, you can add the following settings to enable a GUI debugger.

----
display_library: win32, options="gui_debug"
----

Although the GUI debugger works perfectly fine, it is far better to use IDA Pro if you have. Here is an article about this process by HeyRays, but in a nutshell, you can follow these steps:

1. Download a file mentioned in the article.
2. Copy mbr.py to the same directory as a location of the hard disk image file.
3. Download and copy this batch file to the same directory (you will need to change paths in it).
4. Run the batch file. 

You will see IDA Pro breaks at the very beginning of MBR (0x7c00) unless there is an error in configurations. Now, you can trace the code and should be able to debug VBR as well.

Ntldr (Real-Mode) and Bootmgr (Real-Mode)

Although it is totally possible to trace the code until it reaches to Ntldr or Bootmgr, it can be time consuming. One solution is to modify their entry point code with a breakpoint. Luckily, offset 0 of these files are actually entry points, so we can change offset 0 to a breakpoint. Here is an original entry point of Ntldr.

Also, as we are using a flat hard disk image file, we can search code pattern of Ntldr or Bootmgr from the image file and change it. In my case, Ntldr was found at offset 0x133D55E00 in the image file.

Like the above image, you will probably need to install a magic_break provided by Bochs rather than regular 0xCC since the runtime environment is different from usual, the protected mode. Bochs treats an instruction 'xchg bx, bx' (0x87 0xdb) as a breakpoint when the following statement was added to the bochsrc.

---- 

magic_break: enabled=1 

----

Once you boot the virtual machine with Bochsdbg.exe, you should see the VM breaks at 0x2:0002.

0x2:0000 is the actual breakpoint which we have set. You can look an IDA to find where to go from here. In my case, the execution need to go to 0x01d8.

Then you can change EIP with a 'set $eip = 0x01d8' command and step in with a 's' command to execute a regular code sequence.

Note that my Old IDA Pro (v6.0) did not stop when execution reached at a magic_breakpoint, but this issue may have already been solved on the latest version of IDA.

Ntldr (Protected-Mode)

Steps for debugging Ntldr (Protected-mode) is relatively straightforward:

1. Download and extract Ntldr of the checked build version. 
2. Open the extracted Ntldr with a hex editor and find an 'MZ' header in it. Copy all contents after that and save it on a debugger system. We assume that you saved it as E:\osloader.exe in this article. This part contains protected mode code of Ntldr.
3. Overwrite existing C:\ntldr with the extracted Ntldr (not osloader.exe).
4. Add the following [debug] section in boot.ini.

---- 

[boot loader]
timeout=30
default=multi(0)disk(0)rdisk(0)partition(1)\WINDOWS

[debug]
/debug /debugport=COM1: /baudrate=115200 /debugstop

[operating systems]
multi(0)disk(0)rdisk(0)partition(1)\WINDOWS="Microsoft Windows XP Professional" /noexecute=optin /fastdetect
multi(0)disk(0)rdisk(0)partition(1)\WINDOWS="Microsoft Windows XP Professional (Debug)" /noexecute=optin /fastdetect /debug /debugport=COM1: /baudrate=115200

---- 

Once you reboot the debugee XP system, it waits to be connected by a kernel debugger.  In order to load symbols and suppress error messages, you can manually read an osloader.exe image.

----
kd> .readmem E:\osloader.exe 0x400000 L0x1000
Reading 1000 bytes..
kd> .imgscan /l /r 00400000
MZ at 00400000 - size 80000
  Name: osloader.EXE
  Loaded osloader.EXE module
----

Now, you are free to see how Ntoskrnl.exe and boot drivers are initialized.

Bootmgr (Protected-Mode)

Bootmgr is called the boot manager and responsible for listing a boot menu and firing up an appropriate program according to a user's selection in the list.

Unlike debugging Ntldr, you can activate the boot manager debugger with the following commands.

---- 

bcdedit /bootdebug {bootmgr} on 

----

Fore more details, you can consult MSDN, BCDEdit /bootdebug. 

Note that the executable file is located in C:\Windows\Boot\PCAT\bootmgr and contains a 32bit PE image as with Ntldr.

Winload.exe

Winload.exe is called the boot loader and used for the regular boot procedure. It basically loads Ntoskrnl.exe and some other boot drivers. Enabling the boot loader debugger can be simply done with bcdedit like the case of Bootmgr.

---- 

bcdedit /bootdebug on 
----

Further Research

• Difference in Windows 10.
• When the processor mode is changed to the long mode on the x64 system.
• QEMU may be better in terms of installing OSs in it.