Tutorial 2: Detecting a Valid PE File

In this tutorial, we will learn how to check if a given file is a valid PE file.

Theory:

How can you verify if a given file is a PE file? That question is difficult to answer. That depends on the length that you want to go to do that. You can verify every data structure defined in the PE file format or you are satisfied with verifying only the crucial ones. Most of the time, it's pretty pointless to verify every single structure in the files. If the crucial structures are valid, we can assume that the file is a valid PE. And we will use that assumption.

The essential structure we will verify is the PE header itself. So we need to know a little about it, programmatically. The PE header is actually a structure called IMAGE_NT_HEADERS. It has the following definition:

IMAGE_NT_HEADERS STRUCT

   Signature dd ?

   FileHeader IMAGE_FILE_HEADER <>

   OptionalHeader IMAGE_OPTIONAL_HEADER32 <>

IMAGE_NT_HEADERS ENDS

Signature is a dword that contains the value 50h, 45h, 00h, 00h. In more human term, it contains the text "PE" followed by two terminating zeroes. This member is the PE signature so we will use it in verifying if a given file is a valid PE one.

FileHeader is a structure that contains information about the physical layout of the PE file such as the number of sections, the machine the file is targeted and so on.

OptionalHeader is a structure that contains information about the logical layout of the PE file. Despite the "Optional" in its name, it's always present.

Our goal is now clear. If value of the signature member of the IMAGE_NT_HEADERS is equal to "PE" followed by two zeroes, then the file is a valid PE. In fact, for comparison purpose, Microsoft has defined a constant named IMAGE_NT_SIGNATURE which we can readily use.

IMAGE_DOS_SIGNATURE equ 5A4Dh

IMAGE_OS2_SIGNATURE equ 454Eh

IMAGE_OS2_SIGNATURE_LE equ 454Ch

IMAGE_VXD_SIGNATURE equ 454Ch

IMAGE_NT_SIGNATURE equ 4550h

The next question: how can we know where the PE header is? The answer is simple: the DOS MZ header contains the file offset of the PE header. The DOS MZ header is defined as IMAGE_DOS_HEADER structure. You can check it out in windows.inc. The e_lfanew member of the IMAGE_DOS_HEADER structure contains the file offset of the PE header.

The steps are now as follows:

1. Verify if the given file has a valid DOS MZ header by comparing the first word of the file with the value IMAGE_DOS_SIGNATURE.
2. If the file has a valid DOS header, use the value in e_lfanew member to find the PE header
3. Comparing the first word of the PE header with the value IMAGE_NT_HEADER. If both values match, then we can assume that the file is a valid PE.

Example:

.386

.model flat,stdcall

option casemap:none

include \masm32\include\windows.inc

include \masm32\include\kernel32.inc

include \masm32\include\comdlg32.inc

include \masm32\include\user32.inc

includelib \masm32\lib\user32.lib

includelib \masm32\lib\kernel32.lib

includelib \masm32\lib\comdlg32.lib

SEH struct

PrevLink dd ?    ; the address of the previous seh structure

CurrentHandler dd ?    ; the address of the exception handler

SafeOffset dd ?    ; The offset where it's safe to continue execution

PrevEsp dd ?      ; the old value in esp

PrevEbp dd ?     ; The old value in ebp

SEH ends

.data

AppName db "PE tutorial no.2",0

ofn OPENFILENAME <>

FilterString db "Executable Files (*.exe, *.dll)",0,"*.exe;*.dll",0

                 db "All Files",0,"*.*",0,0

FileOpenError db "Cannot open the file for reading",0

FileOpenMappingError db "Cannot open the file for memory mapping",0

FileMappingError db "Cannot map the file into memory",0

FileValidPE db "This file is a valid PE",0

FileInValidPE db "This file is not a valid PE",0

.data?

buffer db 512 dup(?)

hFile dd ?

hMapping dd ?

pMapping dd ?

ValidPE dd ?

.code

start proc

LOCAL seh:SEH

mov ofn.lStructSize,SIZEOF ofn

mov ofn.lpstrFilter, OFFSET FilterString

mov ofn.lpstrFile, OFFSET buffer

mov ofn.nMaxFile,512

mov ofn.Flags, OFN_FILEMUSTEXIST or OFN_PATHMUSTEXIST or OFN_LONGNAMES or OFN_EXPLORER or OFN_HIDEREADONLY

invoke GetOpenFileName, ADDR ofn

.if eax==TRUE

    invoke CreateFile, addr buffer, GENERIC_READ, FILE_SHARE_READ, NULL, OPEN_EXISTING, FILE_ATTRIBUTE_NORMAL, NULL

    .if eax!=INVALID_HANDLE_VALUE

       mov hFile, eax

       invoke CreateFileMapping, hFile, NULL, PAGE_READONLY,0,0,0

       .if eax!=NULL

          mov hMapping, eax

          invoke MapViewOfFile,hMapping,FILE_MAP_READ,0,0,0

          .if eax!=NULL

             mov pMapping,eax

             assume fs:nothing

             push fs:[0]

             pop seh.PrevLink

              mov seh.CurrentHandler,offset SEHHandler

             mov seh.SafeOffset,offset FinalExit

             lea eax,seh

              mov fs:[0], eax

             mov seh.PrevEsp,esp

             mov seh.PrevEbp,ebp

             mov edi, pMapping

             assume edi:ptr IMAGE_DOS_HEADER

             .if [edi].e_magic==IMAGE_DOS_SIGNATURE

                add edi, [edi].e_lfanew

                assume edi:ptr IMAGE_NT_HEADERS

                .if [edi].Signature==IMAGE_NT_SIGNATURE

                   mov ValidPE, TRUE

                .else

                   mov ValidPE, FALSE

                .endif

             .else

                 mov ValidPE,FALSE

              .endif

FinalExit:

             .if ValidPE==TRUE

                  invoke MessageBox, 0, addr FileValidPE, addr AppName, MB_OK+MB_ICONINFORMATION

             .else

                invoke MessageBox, 0, addr FileInValidPE, addr AppName, MB_OK+MB_ICONINFORMATION

             .endif

             push seh.PrevLink

             pop fs:[0]

              invoke UnmapViewOfFile, pMapping

          .else

             invoke MessageBox, 0, addr FileMappingError, addr AppName, MB_OK+MB_ICONERROR

          .endif

          invoke CloseHandle,hMapping

       .else

          invoke MessageBox, 0, addr FileOpenMappingError, addr AppName, MB_OK+MB_ICONERROR

       .endif

       invoke CloseHandle, hFile

    .else

       invoke MessageBox, 0, addr FileOpenError, addr AppName, MB_OK+MB_ICONERROR

    .endif

.endif

invoke ExitProcess, 0

start endp

SEHHandler proc C uses edx pExcept:DWORD, pFrame:DWORD, pContext:DWORD, pDispatch:DWORD

    mov edx,pFrame

    assume edx:ptr SEH

    mov eax,pContext

    assume eax:ptr CONTEXT

    push [edx].SafeOffset

    pop [eax].regEip

    push [edx].PrevEsp

    pop [eax].regEsp

    push [edx].PrevEbp

    pop [eax].regEbp

    mov ValidPE, FALSE

    mov eax,ExceptionContinueExecution

    ret

SEHHandler endp

end start

Analysis:

The program opens a file and checks if the DOS header is valid, if it is, it checks the PE header if it's valid. If it is, then it assumes the file is a valid PE. In this example, I use structured exception handling (SEH) so that we don't have to check for every possible error: if a fault occurs, we assume that it's because the file is not a valid PE thus giving our program wrong information. Windows itself uses SEH heavily in its parameter validation routines. If you're interested in SEH, read  http://www.jorgon.freeserve.co.uk/Except/Except.htm  by Jeremy Gordon.

The program displays an open file common dialog to the user and when the user chooses an executable file, it opens the file and maps it into memory. Before it goes on with the verification, it sets up a SEH:

   assume fs:nothing

   push fs:[0]

   pop seh.PrevLink

   mov seh.CurrentHandler,offset SEHHandler

   mov seh.SafeOffset,offset FinalExit

   lea eax,seh

   mov fs:[0], eax

   mov seh.PrevEsp,esp

   mov seh.PrevEbp,ebp

We start by assuming the use of fs register as nothing. This must be done because MASM assumes the use of fs register to ERROR. Next we store the address of the previous SEH handler in our structure for use by Windows. We store the address of our SEH handler, the address where the execution can safely resume if a fault occurs, the current values of esp and ebp so that our SEH handler can get the state of the stack back to normal before it resumes the execution of our program.

   mov edi, pMapping

   assume edi:ptr IMAGE_DOS_HEADER

   .if [edi].e_magic==IMAGE_DOS_SIGNATURE

After we are done with setting up SEH, we continue with the verification. We put the address of the first byte of the target file in edi, which is the first byte of the DOS header. For ease of comparison, we tell the assembler that it can assume edi as pointing to the IMAGE_DOS_HEADER structure (which is the truth). We then compare the first word of the DOS header with the string "MZ" which is defined as a constant in windows.inc named IMAGE_DOS_SIGNATURE. If the comparison is ok, we continue to the PE header. If not, we set the value in ValidPE to FALSE, meaning that the file is not a valid PE.

      add edi, [edi].e_lfanew

      assume edi:ptr IMAGE_NT_HEADERS

      .if [edi].Signature==IMAGE_NT_SIGNATURE

         mov ValidPE, TRUE

      .else

         mov ValidPE, FALSE

      .endif

To get to the PE header, we need the value in e_lfanew of the DOS header. This field contains the file offset of the PE header, relative to the file beginning. Thus we add this value to edi and we get to the first byte of the PE header. It's this place that a fault may occur. If the file is really not a PE file, the value in e_lfanew will be incorrect and thus using it amounts to using a wild pointer. If we don't use SEH, we must check the value of the e_lfanew against the file size which is ugly. If all goes well, we compare the first dword of the PE header with the string "PE". Again there is a handy constant named IMAGE_NT_SIGNATURE which we can use. If the result of comparison is true, we assume the file is a valid PE.

If the value in e_lfanew is incorrect, a fault may occur and our SEH handler will get control. It simply restores the stack pointer, bsae pointer and resumes the execution at the safe offset which is at the FinalExit label.

FinalExit:

   .if ValidPE==TRUE

      invoke MessageBox, 0, addr FileValidPE, addr AppName, MB_OK+MB_ICONINFORMATION

   .else

      invoke MessageBox, 0, addr FileInValidPE, addr AppName, MB_OK+MB_ICONINFORMATION

   .endif

The above code is simplicity itself. It checks the value in ValidPE and displays a message to the user accordingly.

   push seh.PrevLink

   pop fs:[0]

When the SEH is no longer used, we dissociate it from the SEH chain.

+

+

+

https://vahidmy.blog.ir/post/Tutorial-1-Overview-of-PE-file-format

https://vahidmy.blog.ir/post/pe-txt

+

---