So of course first we check whether it is packed or unpacked with PEiD.
So it shows nothing found? Means nothing to worry? Haha. So second step we do strings the program to check any weird/interesting string. Nothing interesting for me except this.
Then we try open the program with OllyDbg. And try to run the program with dummy input.
For sure got wrong. Then try to search for reference strings. But got no clue since the ouput "Wrong" and "Correct!!" did not display. So I search in Memory Map (alt + M), then double click .text.
And then searching and searching, I found this "Correct" and "Wrong" strings!
So I mark a breakpoint at the JBE instruction at 0x0119292C. So wen run the program again, and using dummy input to check whether it stops at our breakpoint marked or not?
No! It still display "Wrong" status, it means that it didn't stop at our breakpoint marked. So when we click at "Wrong" section, we can see in Hint Pane that the jump to "Wrong" section come from two address which were 0x01192915 and 0x0119292C.
So 0x0119292C is our breakpoint marked just now. And the other one is not, so we need to mark breakpoint at 0x01192915 too which is JNZ instruction. After that just dummy input to see whether it stop at our new breakpoint.
Yeah it stop at 0x01192915! From the instruction there, it compare register AX with BX, if it same then continue to the next instruction, if not go to "Wrong" section.
So we just change the zero flag so that we can continue, then we reach to our second breakpoint where it compare the EAX register with constant value 0F(in decimal 15), JBE(Jump if below or equal), so I assume that it compare the length of the strings. If the string length below or equal to 15 then it will go to the "Wrong" section, else continue to "Correct" section. So from there we just to reverse the step to get the string.
When we reverse to our first breakpoint, we know that it compare AX and BX register, so we need to know how AX and BX get the value. From the cpu window, there are 2 functions that have been called before the comparing section happen.
So we set a breakpoint at both functions and then analyse what happen when the function is called with our dummy input.
Then we reach to the first function, and then press F7 to step into the function called.
So here are the function 1 procedure.
So we just step into the function procedure, and then return back. From my understanding, the algorithm is just encode the string input into 4 bytes. Below is C code that I translate.
Then we go to the second function. Try to understand the algorithm. Based on my understanding, the EAX register will change to 7EEF. And 60 bytes are reserve in the stack. So we check that they input the value in the stack in unordered. After input all the value in the stack, then they do XOR instruction around 0x28 (40) times.
Then return back, and compare the AX register with BX register. So that's how the program work.
1) Encode the strings to 4 bytes, and then return it. EAX hold the values of 4 bytes then move the value EBX register.
2) Then in the second function will place some value in stack and do some XOR then return back.
3) Compare AX and BX.
I assume that our strings maybe 40 characters, why? Because it is weird to reserve bytes in stack and do XOR for 40 times.
And also at step 3, AX value always 7EEF, while BX value depends on our input. So if we can make a string that will generate a value 7EEF after it encoded, it would be nice!
So I try to reverse the encoded function, but no luck! Haha. Around 2 days I spend to reverse the algorithm. And I also put a breakpoint at first instruction in both function.
First function
Second function
Then my luck comes, when I restart the program, I try to run at each breakpoint,
then I found interesting string at first function procedure before the program start!
"This message is encrypted with blowfish"I just copy the strings and input it to our program.
Yeah! Finally got the answer.
So I assume all this algorithm are blowfish cipher algorithm, that is why it was hard to reverse. Haha.
But but.. my technique was not so efficient, since I was just lucky to see the strings before the program start.. But but it was fun challenge!