Weekly digest 16

This is number sixteen of the weekly digests. Last week I have been sick so this one will again account for two weeks…

Christmas

Merry Christmas everyone!

x64dbgpylib

Some effort has been made towards supporting mona.py by porting windbglib to x64dbgpy. You can help out by porting a few functions outlined in this issue.

Translations

Various people worked very hard to completely translate x64dbg in Korean, the state of the translation is as follows:

  • Korean (100%)
  • Turkish (96%)
  • Dutch (94%)
  • Chinese Simplified (89%)
  • Spanish (87%)
  • German (87%)
  • Russian (83%)

Restart as admin

If a process requires elevation on start, CreateProcess would fail with ERROR_ELEVATION_REQUIRED. This is now detected and you can allow x64dbg to restart itself as administrator.

restart as admin question

Certain operations (such as setting x64dbg as JIT debugger), also require elevation and a menu option has been added! It will automatically reload the current debuggee, but it (obviously) cannot restore the current state so think of this as the restart option.

restart as admin menu

Secure symbol servers

The default symbol servers have been switched to HTTPS. See pull request #1300 by xiaoyinl.

Microsoft symbol servers currently have issues and you might have to try to download symbols multiple times.

Fixed weird display issue on the tab bar

Issue #1339 has been fixed and the buttons to scroll in the tab bar should now appear correctly.

button bug

Various copying enhancements

There are various enhancements to copying addresses and disassembly. See pull request #1363 by ThunderCls for more details.

Fixed a bug if IMAGE_DOS_HEADERS is malformed

Executables with a malformed header, where e_lfanew points higher than 0x1000 bytes would be detected as invalid by x64dbg. This has now been corrected by jossgray in pull request #1369.

Fixed some bugs with handling big command lines

The maximum command line size has been increased to 65k to support modification of very long command lines (such as custom JVMs with many arguments).

Launcher improvements

There have been various improvements to the launcher, mostly with .NET executables and also the handling of the IMAGE_DOS_HEADER.

Load/free library in the symbols view

Pull request #1372 by ThunderCls introduced the freelib command that allows you to unload a library from the debuggee. In addition to a GUI for the loadlib command.

free library menu

String search improvements

There have been various improvements to the string search and UTF-8 strings will be escaped correctly.

Don’t change the active window when closing a tab

Previously if you detached a tab and pressed the close button it would keep that tab active, while usually the desired behaviour is to hide the tab in the background. See pull request #1375 by changeofpace for more details.

Workaround for a capstone bug

The instruction test eax, ecx is incorrectly disassembled by capstone as test ecx, eax. This has been worked around by the following ugly code that simply swaps the arguments…

//Nasty workaround for https://github.com/aquynh/capstone/issues/702
if(mSuccess && GetId() == X86_INS_TEST && x86().op_count == 2 && x86().operands[0].type == X86_OP_REG && x86().operands[1].type == X86_OP_REG)
{
    std::swap(mInstr->detail->x86.operands[0], mInstr->detail->x86.operands[1]);
    char* opstr = mInstr->op_str;
    auto commasp = strstr(opstr, ", ");
    if(commasp)
    {
        *commasp = '\0';
        char second[32] = "";
        strcpy_s(second, commasp + 2);
        auto firstsp = commasp;
        while(firstsp >= opstr && *firstsp != ' ')
            firstsp--;
        if(firstsp != opstr)
        {
            firstsp++;
            char first[32] = "";
            strcpy_s(first, firstsp);
            *firstsp = '\0';
            strcat_s(mInstr->op_str, second);
            strcat_s(mInstr->op_str, ", ");
            strcat_s(mInstr->op_str, first);
        }
    }
}

Improve autocomments

The option ‘Autocomments only on CIP’ would only show non-user comments on the CIP instruction. Issue #1386 proposed a different solution and currently only register-based comments will be hidden.

Save and restore the window position and size

Pull request #1385 by changeofpace introduced saving of the main window position and size.

Allow permanent highlighting mode

Some people prefer the way IDA handles highlighting. Clicking on a register/immediate will highlight it everywhere else, even if you want to keep the previous highlighting but want to click somewhere else. I personally think this is a bad way of handling highlighting, but an option has been introduced that has similar behaviour. Pull request #1388 had similar functionality, but I rewrote it to be optional and more intuitive.

enable permanent highlighting mode

If you don’t click on a highlightable object it will not change the highlighting so (unlike IDA) you can do your normal operations while keeping the desired highlighting.

highlighting behaviour

Copy as HTML

Pull request #1394 by torusrxxx introduces an option that copies the disassembly/dump as HTML allowing you to paste it in Word:

x64dbg word

Usual things

Thanks a lot to all the contributors!

That has been about it for this time again. If you have any questions, contact us on Telegram, Gitter or IRC. If you want to see the changes in more detail, check the commit log.

You can always get the latest release of x64dbg here. If you are interested in contributing, check out this page.

Finally, if someone is interested in hiring me to work on x64dbg more, please contact me!

Comments

Weekly digest 15

This is number fifteen of the weekly digests. This time it will highlight the things that happened in the last two weeks, since last week wasn’t so busy.

Log redirection encoding

Previously the default log redirect option was UTF-16 with BOM, but this has been changed to support UTF-8 Everywhere. You can get the old behaviour back in the settings dialog if you favor UTF-16.

UTF-16 log redirect option

Properly enforce size limits for comments and labels

The sizes of labels and comments are limited to ~256 characters and this is now properly enforced in the GUI to avoid nasty surprises. You will now also be warned if you set a duplicate label.

Large address awareness

The 32 bit version of x64dbg previously wasn’t ‘Large address aware’. It now is, which means that you can consume more than 2GB of memory if you feel like it.

Optimized logging speed

The logging should be somewhat faster now, especially when redirecting it to a file and disabling the GUI log. You can find more details here, but the numbers might be off since additional changes were not made and no benchmarks were done.

Fixed a crash when clicking out of range in the side bar

Issue #1299 described a crash and a dump was provided but I did not have debug symbols for that particular build. To figure out what was happening I used x64dbg to debug x64dbg and then some pattern searching to find the crash location in a build for which I did have symbols. The person who opened the issue and a video is available here.

Updated Scylla

Recently a tool called pe_unmapper by malware analyst hasherezade was released and I thought it would be a nice thing to have in x64dbg so I added it to Scylla since it already had a framework to do exactly that. You can find a simple video demonstration here.

Plugin API to get useful information about the current debuggee

There are some new functions available for plugins that help with querying the PROCESS_INFORMATION of the debuggee. These functions are:

BRIDGE_IMPEXP HANDLE DbgGetProcessHandle();
BRIDGE_IMPEXP HANDLE DbgGetThreadHandle();
BRIDGE_IMPEXP DWORD DbgGetProcessId();
BRIDGE_IMPEXP DWORD DbgGetThreadId();

Various improvements to the type system

Issue #1305 highlights some issues with the type system, various have been addressed and hopefully everything is a bit more stable now…

More styles

Various additional styles have been added on the wiki. Check them out below!

Color Show

Visual Studio Dark

Visual Studio Light

Case-insensitive regex search in symbol view

It is now possible to use both case sensitive and insensitive regex searching in the symbol view.

regex

GUI speed improvements

A bad locking mechanism has been replaced by Event Objects, resulting in a noticeable performance improvement, mostly when visiting types.

Intercept more functions for crashdumps

Some crash dumps were missing information and Nukem addressed this in pull request #1338. This might help on some Windows 10 installations.

Don’t change selection when the search text changes

Thanks to lynnux’ pull request #1340 the last cursor position will now be remembered when removing the search string in the search list view. This is very useful if you want to for example find string references in close proximity to one you are looking for. Below is a GIF demonstrating this new feature.

keep selection

Make x64dbg run on Wine again

There is a branch called wine that runs under Wine. The reason that x64dbg is not running under Wine is that the Concurrency::unbounded_buffer is not implemented. The branch is not very well-tested but feedback is appreciated!

Added more advanced plugin callbacks

In pull request #1314 torusrxxx added automatic detection of PEB fields as labels. This functionality has instead been moved to the LabelPEB plugin and the plugin callbacks CB_VALFROMSTRING and CB_VALTOSTRING have been added to allow plugins to add additional behavior to the expression resolver.

PEB labels

An interesting piece of documentation on access violation exceptions is now represented in x64dbg with pull request #1361 by changeofpace.

The first element of the array contains a read-write flag that indicates the type of operation that caused the access violation. If this value is zero, the thread attempted to read the inaccessible data. If this value is 1, the thread attempted to write to an inaccessible address. If this value is 8, the thread causes a user-mode data execution prevention (DEP) violation.

The second array element specifies the virtual address of the inaccessible data.

exception information

Fixed incorrect detection of unary operators

The expression (1<<5)+4 would be interpreted as incorrect because the + was treated as a unary operator. This has now been fixed!

Remove breakpoints when clearing the database

The dbclear command didn’t remove breakpoints from the process, causing some weird behavior if you hit a breakpoint anyway. This should now be fixed.

Fixed bug with searching in the memory map

A bug has been fixed in the findallmem command where the size argument was interpreted incorrectly and thus causing searching the entire process memory to fail.

Improvements to the breakpoint view

Pull requests #1359 by ThunderCls and #1346 by ner0x652 have added some improvements to the breakpoint view. You can now see if CIP is on the current breakpoint and the edit dialog will show the full symbolic address in the title.

breakpoint view

Find window in the attach dialog

You can now find a window by title in the attach dialog to attach to a process without knowing the PID. There is also a new config command that can be used by scripts to get/set configuration values. More details in pull request #1355.

attach dialog

Usual stuff

Thanks a lot to all the contributors!

That has been about it for this time again. If you have any questions, contact us on Telegram, Gitter or IRC. If you want to see the changes in more detail, check the commit log.

You can always get the latest release of x64dbg here. If you are interested in contributing, check out this page.

Finally, if someone is interested in hiring me to work on x64dbg more, please contact me!

Comments

Type system

This week there wasn’t much going on in the codebase and therefore I decided to skip the weekly digest and write a more substantial post, this time about the type system.

The goal of the type system is to provide a more powerful view of memory than just a linear stream of bytes. It can be used to visualize structures and it also supports function definitions that will be used later. Hopefully it’s an interesting read!

Internal representation

The internal representation of the types is inspired by the radare2 type profiles document by oddcoder.

Primitives

enum Primitive
{
    Void,
    Int8,
    Uint8,
    Int16,
    Uint16,
    Int32,
    Uint32,
    Int64,
    Uint64,
    Dsint,
    Duint,
    Float,
    Double,
    Pointer,
    PtrString, //char* (null-terminated)
    PtrWString //wchar_t* (null-terminated)
};

Complex types are built from primitive types (see the full list above). The Void primitive is not a real type (it cannot have a value) and it’s used as a special case. An alternative name would be Unknown but that was already taken.

All primitive types (except Void) have a fixed size, but that size is not defined as part of the primitive (abstractions love to be abstract). Notice that there is no Bit primitive, which means that bit fields or bit arrays are not supportable in the current type system. There are two primitives to represent the common null-terminated string pointer types, mostly for convenience of the user.

The generic Pointer type is equivalent to void* and can get a more specific meaning in the Type representation below.

Types

struct Type
{
    std::string owner; //Type owner
    std::string name; //Type identifier.
    std::string pointto; //Type identifier of *Type
    Primitive primitive; //Primitive type.
    int size = 0; //Size in bytes.
};

The actual type representation used to represent a primitive type, in say a struct is shown above. The comments should be pretty self-explanatory, but it is worth mentioning that the size member cannot be defined by user-types directly. You can create your own (named) types and for that you can use one of the pre-defined internal types:

p("int8_t,int8,char,byte,bool,signed char", Int8, sizeof(char));
p("uint8_t,uint8,uchar,unsigned char,ubyte", Uint8, sizeof(unsigned char));
p("int16_t,int16,wchar_t,char16_t,short", Int16, sizeof(short));
p("uint16_t,uint16,ushort,unsigned short", Int16, sizeof(unsigned short));
p("int32_t,int32,int,long", Int32, sizeof(int));
p("uint32_t,uint32,unsigned int,unsigned long", Uint32, sizeof(unsigned int));
p("int64_t,int64,long long", Int64, sizeof(long long));
p("uint64_t,uint64,unsigned long long", Uint64, sizeof(unsigned long long));
p("dsint", Dsint, sizeof(void*));
p("duint,size_t", Duint, sizeof(void*));
p("float", Float, sizeof(float));
p("double", Double, sizeof(double));
p("ptr,void*", Pointer, sizeof(void*));
p("char*,const char*", PtrString, sizeof(char*));
p("wchar_t*,const wchar_t*", PtrWString, sizeof(wchar_t*));

The p function simply binds all (comma-separated) type names to a Primitive and a size. The sizes are defined by your compiler implementation.

The owner member is used to represent what created the type. This will generally be the filename of the file it was loaded from, or cmd if the type was created with the commands.

The pointto member is used when primitive is Pointer and it’s the name of the type that the pointer points to. As an example, the type MyStruct* will have the following values:

t.owner = owner; //owner of MyStruct
t.name = "MyStruct*";
t.pointto = "MyStruct";
t.primitive = Pointer;
t.size = sizeof(void*); //predefined

The validPtr function will (recursively) create pointer type aliases if you use a construct like MyStruct* as part of checking if a type is defined.

Members

struct Member
{
    std::string name; //Member identifier
    std::string type; //Type.name
    int arrsize = 0; //Number of elements if Member is an array
    int offset = -1; //Member offset (only stored for reference)
};

If you use a definition inside a complex type (think struct) it will use the Member representation from above. A member like int arrsize; will have the following values:

m.name = "arrsize";
m.type = "int";
m.arrsize = 0; //not an array
m.offset = -1; //unused, only for reference

If the arrsize member is bigger than zero it means that the member was an array of fixed size. For instance bool threadsDone[10];.

StructUnions

struct StructUnion
{
    std::string owner; //StructUnion owner
    std::string name; //StructUnion identifier
    std::vector<Member> members; //StructUnion members
    bool isunion = false; //Is this a union?
    int size = 0;
};

The definition of a struct (or union) shouldn’t be very surprising. A struct is simply a list of Member instances. The size member is used in the Sizeof function and is the combined size of all members. This means that there is no implicit alignment. When adding a member with a defined offset it will simply put an array of padding bytes to make up for the missing space. This also means that you cannot define members out of memory order. This is to prevent overlapping members and also to prevent lots of complexity that isn’t needed for most use cases.

Functions

struct Function
{
    std::string owner; //Function owner
    std::string name; //Function identifier
    std::string rettype; //Function return type
    CallingConvention callconv; //Function calling convention
    bool noreturn; //Function does not return (ExitProcess, _exit)
    std::vector<Member> args; //Function arguments
};

Functions are similar to structs, but they also have a return type and a calling convention. You can define functions (and their arguments), but they are (currently) not used by the GUI. In the future they can be used to provide argument information.

Where is the tree?

You might have noticed that the data structures don’t have a direct tree structure. The main reason for this is that trees are annoying to both represent and manipulate in C++. They are also annoying to serialize and considering that x64dbg uses JSON as a general format I decided to store everything in dictionaries and leave the trees implicit.

There are dictionaries for the Type, StructUnion and Function structures as described above. The type field inside Member for example is a key in either of these dictionaries and that is how the tree’s edges are represented. The tree nodes are the values in the dictionary.

Visitor

struct Visitor
{
    virtual ~Visitor() { }
    virtual bool visitType(const Member & member, const Type & type) = 0;
    virtual bool visitStructUnion(const Member & member, const StructUnion & type) = 0;
    virtual bool visitArray(const Member & member) = 0;
    virtual bool visitPtr(const Member & member, const Type & type) = 0;
    virtual bool visitBack(const Member & member) = 0;
};

The tree structure returns in the Visitor. The visitMember function recursively walks a Member and it’s subtypes with depth first search and it will call one of the visitX functions to signal that a certain kind of node was visited. The visitBack function is called when a complex type subtree was left.

As an example, take the Ray structure:

struct Vec3
{
    int x;
    int y;
    int z;
};

struct Ray
{
    float speed;
    Vec3 direction;
    int lifetime;
};

The tree and the order the nodes are visited in can be visualized like this:

ray tree

The actual structure view in x64dbg will look like this:

ray struct

Conclusion

This post has mostly highlighted the internal representation of the type system, for more information on how to actually use it in x64dbg you can check out Weekly Digest 14 and if you have any questions, please leave comments and I will try to address them.

Comments