30 Oct 2016, by mrexodia
This is already number ten of the weekly digests! It will highlight the things that happened to and around x64dbg this week.
InterObfu
An ongoing effort basically sucking all my time has been on a project called InterObfu. It is basically a representation of x86 instructions that allows you to transform instructions. It also allows for extensive wildcard matching of instructions.
The obvious usage is finding a sequence of wildcard instructions in x64dbg, but another application is peephole optimization (for deobfuscation-related applications). The screenshot below shows the progress of various patterns being replaced or removed.
The implementation was particularly interesting because it involves Aho-Corasick and various tricks to make it efficient. You can check out the code on the repository if you’re interested. Shout out to Matteo for support and discussion on the topic!
Updated mnemonic database
There have been some small additions to the mnemonic database where missing alias instructions iretq and retf have been added. This allows you to more easily query information about instructions.
Replace non-printable characters with special characters
Previously null and non-printable characters were represented in the dump view by dots. This has now been changed and one unicode character (circle) represents null characters and another one (diamond) represents non-printable characters. There is a discussion going on at issue #1196 and you are welcome to come discuss what’s better (default) behavior.
Better split function for commands
Previously it was not possible to include the ; character in command strings without splitting the command in two incorrect commands. This has now been corrected and log "eax={eax};ebx={ebx}";eax++ now works as you would expect.
Fixed global notes
Quite embarrassingly the global notes were saves, but not correctly loaded because of an inverted condition. Thanks to cxj98 for the report!
Added some expression functions
The expression functions dis.iscall, func.start and func.end have been added. This can be used with conditional breakpoints or tracing.
Allow editing of the watch expression
Previously it was not possible to edit the expression of a watch. You can now do this without having to remove and re-add the watch.
Added simple logging of instructions
You can now log instructions with {i:addr} currently this is not particularly useful (except if you have multiple log breakpoints and want the instructions for some reason), but there will be use for this in the near future…
Process GUI events in the script API
An issue reported at the x64dbgpy (Python) repository has been partially worked around. Basically the GUI would freeze if the script was executed on the GUI thread and when using debug functions. This has now been resolved and you can keep using the GUI while your script is running.
Added run to selection in the graph view
It is now possible to use the run to selection (F4) option directly in the graph when debugging. This has been implemented by kkthx in pull request #1199.
Save the graph view to a file
Thanks again to kkthx you can now save the currently-visible part of the graph directly from the context menu. In the future this will be expanded to allow you to save the entire graph to a file.
Usual stuff
That has been about it for this week again. If you have any questions, contact us on Telegram, Gitter or IRC. If you want to see the changes in more detail, check the commit log.
You can always get the latest release of x64dbg here. If you are interested in contributing, check out this page.
Finally, if someone is interested in hiring me to work on x64dbg more, please contact me!
Comments
23 Oct 2016, by mrexodia
This is already number nine of the weekly digests! It will highlight the things that happened to and around x64dbg this week.
Thanks to joesavage there will now be a comment on call $0 (call the next instruction). This is useful for various packers that use this instruction to get the address of the current instruction.
The disassembly popup will now do slightly better analysis of where to stop displaying the preview. It will do some basic heuristic analysis to determine function ends and thus where to stop.
Autocomments will now combine source line information and other information. This means that you can more easily spot the context even if you have line information loaded!
Show CIP in graph overview
The current CIP block will now be highlighted in your configured color in the graph overview.
Less jumpy experience while debugging in the graph
The initial GIF that showed off graph debugging had a very “jumpy” feel, mainly because the currently selected instruction would be forced to be shown as close to the middle as possible. It will now only force the middle if the selection is out of view. You can still force the current instruction in the middle by using the “Refresh” action in the context menu.
Fine-grained dump offset control
In some cases you might want to slightly adjust the dump alignment without having to use the go to dialog. You can now do this with Ctrl+Up/Down. All it does is set the first dump address to the current address +/-1.
The _plugin_menuentrysetchecked API allows you to set your plugin menu items as checkable. This can be useful for visualizing boolean options or just for having some fun!
Codename iconic
Lots of (almost all) context menu items now have icons for a more fun and colorful experience!
Updated capstone, keystone and asmjit
The dependencies capstone, keystone and asmjit have been updated. This fixed various bugs with assembling and disassembling.
Copy as base64
The data copy dialog now allows you to copy data as Base64. Quite useful if you need to dump some private keys or something. It also supports various other formats, including C-style array (various types), string, GUID, IP addresses and Delphi arrays.
Callback for changed selection
It is now possible to register the CB_SELCHANGED callback (currently undocumented). This callback informs you of selection changes.
typedef struct
{
int hWindow;
duint VA;
} PLUG_CB_SELCHANGED;
This can be used in complement with the GuiAddInfoLine function to do context-aware analysis and display it in the GUI.
Analysis plugins
Don’t like the analysis x64dbg does? Don’t worry, you can now fully customize the graph analysis in a plugin. The (currently undocumented) CB_ANALYZE plugin callback allows you to troll your friends by adding exits to every terminal node with this simple code.
PLUG_EXPORT void CBANALYZE(CBTYPE cbType, PLUG_CB_ANALYZE* info)
{
auto graph = BridgeCFGraph(&info->graph, true);
for(auto & nodeIt : graph.nodes)
{
auto & node = nodeIt.second;
if(node.terminal)
node.exits.push_back(graph.entryPoint);
}
info->graph = graph.ToGraphList();
}
Trolls aside, this can be extremely powerful if applied in the right manner. For example deobfuscate VMProtect handlers on the fly…
Maximum trace count option
The default maximum step count for tracing is now customizable through the settings dialog.
Copy selection to file
Issue #1096 has been fixed in pull request #1177 by shamanas. You can now copy bigger selections directly to a file.
Disassembly speed improvements
There has been quite a big improvement in disassembly and overall GUI speed. The disassembly would reload itself three times, effectively disassembling every visible instruction six times. This has now been reduced to disassembling once. Additionally the GUI would be force-refreshed unnecessarily which should now also be fixed. If you encounter any issues with this, please report an issue. Scrolling in the current view will always force-refresh it.
Reports
- Issue #1188 by cxj98 fixed in 15 minutes;
- Private report, already fixed;
- Found another bug myself and double-verified it should be fully fixed now sorry for the hassle!
- Yet another report, also fixed;
Copy symbolic name
In addition to the “Help on Symbolic Name” option, that uses your favorite search engine (Google) to help you figure out what’s going on, you can now also copy the symbolic name directly if you need it for some reason. This was also implemented by shamanas!
Allow customizing of the main menus
Ever thought the menus in x64dbg are too complicated? You can now hide options you don’t use in Options | Customize menus
Fixed a bug with little/big endian when editing FPU registers
The FPU register edit dialog will now respect the configured endianness and always change the register bytes to the way you see it in the edit dialog.
The exinfo command is now executed every time an exception occurs to provide you with more information while examining the log later on.
EXCEPTION_DEBUG_INFO:
dwFirstChance: 1
ExceptionCode: C0000005 (EXCEPTION_ACCESS_VIOLATION)
ExceptionFlags: 00000000
ExceptionAddress: 00007FF7F686240F x64dbg.00007FF7F686240F
NumberParameters: 2
ExceptionInformation[00]: 0000000000000001
ExceptionInformation[01]: FFFF8F500045DF0C
First chance exception on 00007FF7F686240F (C0000005, EXCEPTION_ACCESS_VIOLATION)!
Final words
That has been about it for this week again. If you have any questions, contact us on Telegram, Gitter or IRC. If you want to see the changes in more detail, check the commit log.
You can always get the latest release of x64dbg here. If you are interested in contributing, check out this page.
Finally, if someone is interested in hiring me to work on x64dbg more, please contact me!
Comments
20 Oct 2016, by mrexodia
In a recent post about the architecture of x64dbg there was a comment to explain the “little bit of a mess” that is the threading model of x64dbg. It is indeed not particularly pretty, but everything has a purpose.
Command loop thread
The first thread of interest is the command thread. This thread is created during the initialization phase and it infinitely waits and executes commands that are given to it through the so-called CommandProvider. In the very first version of x64dbg the command thread was the main thread and since the introduction of the GUI this has been moved to a separate thread in favor of the GUI thread.
Debug thread
The debug thread runs the TitanEngine debug loop. It still works the same as two years ago:
Between WaitForDebugEvent and ContinueDebugEvent, the debuggee is in a paused state. The event handlers use event objects to communicate with the GUI. When you click the ‘Run’ button it will set an event object and continue the debug loop and in that way also continue the debuggee.
Here is a simplified version of the cbDebugRun command callback (running on the command thread):
bool cbDebugRun(int argc, char* argv[])
{
// Don't "run" twice if the program is already running.
if(dbgisrunning())
return false;
//Set the event, which makes calls to wait(WAITID_RUN) return.
unlock(WAITID_RUN);
return true;
}
On the debug loop thread we have the cbPauseBreakpoint breakpoint event handler that waits for the user to resume the debug loop (again, simplified):
void cbPauseBreakpoint()
{
//Unset (reset) the event.
lock(WAITID_RUN);
//Wait for the event to be set, a call to unlock(WAITID_RUN).
wait(WAITID_RUN);
}
Here is a simple diagram giving you an overview of what’s going on with the basic threads.
- A block represents a thread;
- A dashed arrow represents starting a new thread;
- A red arrow represents thread termination;
- A circle contains the termination condition.
Some challenging areas are properly signaling the termination of the debuggee. Issues #303, #323 and #438 were, with the great help and patience of wk-952, fixed and this signaling appears to be working now!
Script thread
When dealing with scripting, you usually want to simulate user interaction. This means that the expectation is that the following x64dbgpy (Python) script should be equivalent to:
- Setting a breakpoint on
__security_init_cookie
- Pressing the run button
- Stepping five times
- Setting
RAX to 0x2b992ddfa232
- Stepping out of the function
from x64dbgpy.pluginsdk import *
debug.SetBreakpoint(x64dbg.DbgValFromString("__security_init_cookie"))
debug.Run()
for _ in range(0,5):
debug.StepIn()
register.SetRAX(0x2b992ddfa232)
debug.StepOut()
There has to be some sort of synchronization at the end of debug.Run and debug.StepOut to make sure the debuggee is paused before the next command is executed. The implementation for this is in _plugin_waituntilpaused and looks like this:
PLUG_IMPEXP bool _plugin_waituntilpaused()
{
while(DbgIsDebugging() && dbgisrunning()) //wait until the debugger paused
Sleep(1);
return DbgIsDebugging();
}
The implementation of dbgisrunning is a check if lock(WAITID_RUN) has been called.
Worker threads
There are various threads that just do periodic background work. These include:
Other threads are triggered once to fulfill a specific purpose. These include:
TaskThread
For interaction with the GUI, performance is very important. For this purpose jdavidberger has implemented TaskThread. It’s some variadic templates that basically allow you to trigger an arbitrary function from a different thread to then quickly return to the real work.
The actual thread runs in an infinite loop, waiting for the TaskThread instance to receive a WakeUp (trigger). Once awake, the specified function is executed and after that the thread is being delayed for a configurable amount of time. This ignores all triggers (except the last one) within the delay time to avoid unnecessary work.
The relevant code:
template <typename F, typename... Args> void TaskThread_<F, Args...>::WakeUp(Args... _args)
{
wakeups++;
EnterCriticalSection(&access);
args = CompressArguments(std::forward<Args>(_args)...);
LeaveCriticalSection(&access);
// This will fail silently if it's redundant, which is what we want.
ReleaseSemaphore(wakeupSemaphore, 1, nullptr);
}
template <typename F, typename... Args> void TaskThread_<F, Args...>::Loop()
{
std::tuple<Args...> argLatch;
while(active)
{
WaitForSingleObject(wakeupSemaphore, INFINITE);
EnterCriticalSection(&access);
argLatch = args;
ResetArgs();
LeaveCriticalSection(&access);
if(active)
{
apply_from_tuple(fn, argLatch);
std::this_thread::sleep_for(std::chrono::milliseconds(minSleepTimeMs));
execs++;
}
}
}
As an example, here is the declaration and wake of the thread that updates the call stack (an expensive operation in some cases):
static DWORD WINAPI updateCallStackThread(duint csp)
{
stackupdatecallstack(csp);
GuiUpdateCallStack();
return 0;
}
void updateCallStackAsync(duint csp)
{
static TaskThread_<decltype(&updateCallStackThread), duint> updateCallStackTask(&updateCallStackThread);
updateCallStackTask.WakeUp(csp);
}
Having a different thread handle expensive operations is critial to a responsive interface. Lots of information is rarely looked at (memory map, call stack, SEH information) and can suffer a little delay (100ms) before being updated. This is the same with the current state of the disassembly. When holding F7 to quickly step a little you don’t need perfect accuracy, as long as the disassembly lands on the correct state within reasonable time after releasing the step button.
GUI Thread
The most important (and the most annoying) thread is the Qt GUI thread. If you want to know more, check out the Qt Threading Basics for a 6 page introduction on how it works.
Comments