Architecture of x64dbg

04 Oct 2016, by torusrxxx

x64dbg has a complex architecture. There are three basic parts, namely DBG, BRIDGE and GUI, but in fact there is a fourth part, EXE. This is the main executable, it compiles into x64dbg.exe.

Bootstrapping

When the user starts x64dbg, it will follow this initialization path to get x64dbg running:

• WinMain
• BridgeInit
  • LoadLibrary(“x64gui.dll”)
    • Initialize global variables in the GUI
  • LoadLibrary(“x64dbg.dll”)
    • Initialize global variables in the DBG
  • Load function pointers
• BridgeStart
  • _dbg_sendmessage(DBG_INITIALIZE_LOCKS)
  • _gui_guiinit -> main
    • #70: Application object created
    • #82: Set up event filter
    • #95: Load translations
    • #128: Create default bridge object
    • #134: Initialize [MainWindow](
    • #140: DbgInit -> _dbg_dbginit
      • #584: Initialize modules
      • #665: Initialize variables (varinit)
      • #667: Register commands (registercommands)
      • #670: Register expression functions (ExpressionFunctions::Init)
      • #689: Load plugins (pluginloadall)
      • #694: Handle potential command line
    • #150: Execute the application (starting the user event loop).

Debugging

To start debugging, the GUI sends an init command to the DBG. Then the following things start:

• cbDebugInit
  • Check various things
  • Run threadDebugLoop in a new thread
    • debugLoopFunction
      • #2289: Initialize various variables
      • #2323: DbSetPath
      • #2338: CreateProcess
      • #2351: Check for Wow64 mismatch
      • #2379: Set up TitanEngine handlers
      • #2392: Tell GUI to enter the initialized state
      • #2404 Call the CB_INITDEBUG plugin callback
      • #2429 Enter the DebugLoop

Message passing from GUI to DBG

There are four methods to call DBG from GUI. They are commands, directly exported functions, bridge exported functions (messages) and DbgFunctions(). Currently the directly exported functions are frozen and no new ones should be added. The message flows for each way will be described below.

Commands dispatch

DbgCmdExec is relayed by the bridge to the DBG and eventually received by the cmdloop running in the command thread. This is done asynchronously (meaning DbgCmdExec will not wait until the command is completed).

DbgCmdExecDirect is relayed by the bridge to DBG and then directly in cmddirectexec. This will only return after the command is completed.

In both cases the command is parsed and dispatched to various registered command callbacks. A command callback is similar to main() functions. It can receive number of arguments (plus one), and pointer to every argument string.

Commands are registered in the registercommands function. If you want to get a total list of supported commands, or add your own, just go to that file. Make sure to put your command in the correct category and also make sure to add it to the documentation.

Directly exported functions

There are some legacy functions still unconverted to another method, these can be found in exports.h.

Export functions dispatch

Many Dbg*** functions are exported by the bridge. It then calls _dbg_sendmessage exported by DBG to pass information. Some Dbg*** functions have exports directly in DBG.

DbgFunctions

_dbgfunctions.cpp has a function table that is accessible by anyone. The GUI can call functions in DBG through this table directly.

Message flow from DBG to GUI

There are various Gui*** functions exported by the bridge. The control flow is described below:

• Gui*** export
• Bridge calls _gui_sendmessage
• Bridge calls Bridge::processMessage
• A long list of switch statements in processMessage, basically to emit the corresponding signal. If you want to receive a system event, connect to one of the signals in Bridge::getBridge()

Important subsystems in GUI

Tables in GUI

There is three-level class architecture to support various tables. The first-level class is AbstractTableView, which only includes some basic functions. The second-level classes are Disassembly, HexDump and StdTable. They all inherit from AbstractTableView. Many basic and common functions are defined here, such as table painting, selection, content presentation and column reordering. The third-level classes inherit from the second-level classes. There are many third-level classes. The most common parent for these tables is StdTable.

Context menu management

There are two styles of context menu management. The traditional one builds actions in setupContextMenu and adds them into a menu object in contextMenuEvent. CPUStack uses this style currently. A newer way to manage context menu is to use MenuBuilder. You can see CPUDisassembly or this blog post for more details. It is the preferred way to manage context menu in newer tables, but it does not support non-table widgets out of the box. We want to convert traditional context menu systems into MenuBuilder to speed up development.

Configuration management

Configurations are stored in the Config() object which uses Utf8Ini in the bridge as its backend. When you want to add a new configuration, you have to modify the following files: Configuration.cpp and the SettingsDialog. If you are adding a color then you have to modify the AppearanceDialog as well. Config() can emit settings change signals.

Important subsystems in DBG

There are many subsystems in DBG. The following subsystems are important if you want to contribute:

threading.h

It includes various locks to prevent race condition. Without it, x64dbg will crash much more often. Don’t forget to acquire the lock when you are accessing a subsystem.

x64dbg.cpp

It registers all commands in x64dbg. The details of command processing is described above.

memory.h , module.h and thread.h, label.h and breakpoint.h, etc

They manages corresponding information of the debuggee.

scriptapi

It is intended to be used by plugins. It provides easy scripting experience for developers. x64dbg does not call any of these functions.