Flare-On 11 CTF Challenge Solutions – checksum

Challange 2: checksum

Checking the challenge, we can see that it provides a 64-bit Windows executable named checksum.exe.

$ file checksum.exe
checksum.exe: PE32+ executable (console) x86-64, for MS Windows

Try to execute this file in a virtual machine and see what it does by running this exe. Upon execution, it initially behaves as a benign math quiz application.

1. Initial Examination

The program:

Presents a random number of math questions.
Accepts user input and validates the answers.
Proceeds to a “checksum” input prompt upon successful completion of the quiz.

İn this part, we don’t know the checksum what it is. So time to Reverse Engineering 🙂

strings checksum.exe > checksum.txt

Checking the strings of the executable reveals this is a 64-bit Windows executable that was originally written in golang:

2. Static Analysis

If we are opening the program in a binary analysis tool such as IDA Pro or Ghidra (used both like me), we can see that the program only contains 3 non-library functions: main, a,and b. This tells us that the main functionality of the program will be inside of these three functions, and that the program itself might not be too complicated.

_{main_main funciton}

The part of this appears in checksum.exe last part. And, let’s begin with searching “Checksum:” part.

It looks like DAT_004ca484 contains the checksum result. And, this result looks like two encryption techniques.

chacha20poly1305 might be encryption type but will see. Let’s look into it more deeply.

Package chacha20poly1305 implements the ChaCha20-Poly1305 AEAD and its extended nonce variant XChaCha20-Poly1305, as specified in RFC 8439 and draft-irtf-cfrg-xchacha-01.

const (
	// KeySize is the size of the key used by this AEAD, in bytes.
	KeySize = 32

	// NonceSize is the size of the nonce used with the standard variant of this
	// AEAD, in bytes.
	//
	// Note that this is too short to be safely generated at random if the same
	// key is reused more than 2³² times.
	NonceSize = 12

	// NonceSizeX is the size of the nonce used with the XChaCha20-Poly1305
	// variant of this AEAD, in bytes.
	NonceSizeX = 24

	// Overhead is the size of the Poly1305 authentication tag, and the
	// difference between a ciphertext length and its plaintext.
	Overhead = 16
)

We see that the crypto algorithm’s key size must be 32-bytes with 2 different nonce sizes. The 12-byte nonce is used for the standard ChaCha20-Poly1305 variant, while the 24-byte nonce is used for the XChaCha20-Poly1305 algorithm.

The program checks if the length of the decoded input is 32 bytes and throws the error “chacha20poly1305: bad key length” if the check fails. It indicates that the “checksum” input is a 32-byte hex string that will be used as a ChaCha20-Poly1305 key.

We can guess that the program calls a function to decrypt the flag data encrypted using XChaCha20-Poly1305 with the key and nonce in the checksum input.

We see a call to main_a, which determines whether the program should continue executing. Examining this function reveals more information about the checksum result.

lea:Load Effective Address calculates its src operand in the same way as the mov instruction does, but rather than loading the contents of that address into the dest operand, it loads the address itself.

eax – accumulator: Very general purpose. Used for math, data, and function return values. This tells us that there is a hidden X0R. That is, the Base64 encoded data is also X0R. The program also compares the SHA256 checksum to ensure that the data is properly decrypted. First, it converts the SHA256 checksum string into a byte slice and XOR-encodes it with the XOR key “FlareOn2024“.

To be honest, at this stage I was so sure that the key was ‘FlareOn2024bad’ that I wasted a lot of time looking for the error elsewhere 🙂

The output is checksum result. “Checksum:7fd7dd1d0e959f74c133c13abb740b9faa61ab06bd0ecd177645e93b1e3825dd”. But that doesn’t the end. Keep going anyway.

At this time I had to review the code again because the code had to work correctly. The exe must have been corrupted because even if I entered the checksum it didn’t say “Noice!” and just closed. But, no giving up.

If it worked correctly it should generate “REAL_FLAREON_FLAG.JPG“. So I tried to find it.

AppData is a hidden folder located in C:\Users\<username>\AppData. The AppData folder contains custom settings and other information needed by applications. For example, you might find the following in your AppData folder: Web browser bookmarks and cache.