Tutorial 2: Detecting a Valid PE
File
In this tutorial, we will learn how to
check if a given file is a valid PE file.
Theory:
How can you verify if a given file is a PE
file? That question is difficult to answer. That depends on the length that you
want to go to do that. You can verify every data structure defined in the PE
file format or you are satisfied with verifying only the crucial ones. Most of
the time, it's pretty pointless to verify every single structure in the files.
If the crucial structures are valid, we can assume that the file is a valid PE.
And we will use that assumption.
The essential structure we will verify is
the PE header itself. So we need to know a little about it, programmatically.
The PE header is actually a structure called IMAGE_NT_HEADERS. It has the
following definition:
IMAGE_NT_HEADERS STRUCT
Signature dd ?
FileHeader IMAGE_FILE_HEADER <>
OptionalHeader IMAGE_OPTIONAL_HEADER32 <>
IMAGE_NT_HEADERS ENDS
Signature is a dword that contains the value 50h, 45h,
00h, 00h. In more human term, it contains the text "PE" followed by two
terminating zeroes. This member is the PE signature so we will use it in
verifying if a given file is a valid PE one.
FileHeader is a structure
that contains information about the physical layout of the PE file such as the
number of sections, the machine the file is targeted and so on.
OptionalHeader is a structure that contains
information about the logical layout of the PE file. Despite the "Optional" in
its name, it's always present.
Our goal is now clear. If value of the
signature member of the IMAGE_NT_HEADERS is
equal to "PE" followed by two zeroes, then the file is a valid PE. In fact, for
comparison purpose, Microsoft has defined a constant named IMAGE_NT_SIGNATURE which we can readily
use.
IMAGE_DOS_SIGNATURE
equ 5A4Dh
IMAGE_OS2_SIGNATURE equ 454Eh
IMAGE_OS2_SIGNATURE_LE
equ 454Ch
IMAGE_VXD_SIGNATURE equ 454Ch
IMAGE_NT_SIGNATURE equ 4550h
The next question: how can we know where
the PE header is? The answer is simple: the DOS MZ header contains the file
offset of the PE header. The DOS MZ header is defined as
IMAGE_DOS_HEADER structure. You can check it out in windows.inc. The
e_lfanew member of the IMAGE_DOS_HEADER structure contains the file offset
of the PE header.
The steps are now as follows:
-
Verify if the given file has a valid DOS
MZ header by comparing the first word of the file with the value IMAGE_DOS_SIGNATURE.
-
If the file has a valid DOS header, use
the value in e_lfanew member to find the PE header
-
Comparing the first word of the PE header
with the value IMAGE_NT_HEADER. If both values
match, then we can assume that the file is a valid PE.
Example:
.386
.model flat,stdcall
option casemap:none
include \masm32\include\windows.inc
include \masm32\include\kernel32.inc
include \masm32\include\comdlg32.inc
include \masm32\include\user32.inc
includelib \masm32\lib\user32.lib
includelib \masm32\lib\kernel32.lib
includelib \masm32\lib\comdlg32.lib
SEH struct
PrevLink dd ?
; the address of the previous seh structure
CurrentHandler dd ? ; the
address of the exception handler
SafeOffset dd ? ; The offset where it's
safe to continue execution
PrevEsp dd ? ; the old value in esp
PrevEbp dd ? ; The old value in ebp
SEH ends
.data
AppName db "PE tutorial no.2",0
ofn OPENFILENAME <>
FilterString db "Executable Files (*.exe, *.dll)",0,"*.exe;*.dll",0
db "All Files",0,"*.*",0,0
FileOpenError db "Cannot
open the file for reading",0
FileOpenMappingError db "Cannot open the file
for memory mapping",0
FileMappingError db "Cannot map the file into
memory",0
FileValidPE db "This file is a valid PE",0
FileInValidPE db
"This file is not a valid PE",0
.data?
buffer db 512 dup(?)
hFile dd ?
hMapping dd ?
pMapping dd ?
ValidPE dd ?
.code
start proc
LOCAL seh:SEH
mov ofn.lStructSize,SIZEOF
ofn
mov ofn.lpstrFilter, OFFSET FilterString
mov ofn.lpstrFile, OFFSET
buffer
mov ofn.nMaxFile,512
mov ofn.Flags, OFN_FILEMUSTEXIST or
OFN_PATHMUSTEXIST or OFN_LONGNAMES or OFN_EXPLORER or OFN_HIDEREADONLY
invoke GetOpenFileName, ADDR ofn
.if eax==TRUE
invoke
CreateFile, addr buffer, GENERIC_READ, FILE_SHARE_READ, NULL, OPEN_EXISTING,
FILE_ATTRIBUTE_NORMAL, NULL
.if eax!=INVALID_HANDLE_VALUE
mov
hFile, eax
invoke CreateFileMapping, hFile, NULL, PAGE_READONLY,0,0,0
.if eax!=NULL
mov hMapping, eax
invoke
MapViewOfFile,hMapping,FILE_MAP_READ,0,0,0
.if eax!=NULL
mov pMapping,eax
assume fs:nothing
push fs:[0]
pop seh.PrevLink
mov seh.CurrentHandler,offset SEHHandler
mov
seh.SafeOffset,offset FinalExit
lea eax,seh
mov fs:[0], eax
mov seh.PrevEsp,esp
mov
seh.PrevEbp,ebp
mov edi, pMapping
assume
edi:ptr IMAGE_DOS_HEADER
.if [edi].e_magic==IMAGE_DOS_SIGNATURE
add edi, [edi].e_lfanew
assume edi:ptr
IMAGE_NT_HEADERS
.if [edi].Signature==IMAGE_NT_SIGNATURE
mov ValidPE, TRUE
.else
mov ValidPE, FALSE
.endif
.else
mov ValidPE,FALSE
.endif
FinalExit:
.if ValidPE==TRUE
invoke MessageBox, 0, addr FileValidPE, addr AppName, MB_OK+MB_ICONINFORMATION
.else
invoke MessageBox, 0, addr
FileInValidPE, addr AppName, MB_OK+MB_ICONINFORMATION
.endif
push seh.PrevLink
pop fs:[0]
invoke UnmapViewOfFile, pMapping
.else
invoke
MessageBox, 0, addr FileMappingError, addr AppName, MB_OK+MB_ICONERROR
.endif
invoke CloseHandle,hMapping
.else
invoke MessageBox, 0, addr FileOpenMappingError, addr AppName,
MB_OK+MB_ICONERROR
.endif
invoke CloseHandle, hFile
.else
invoke MessageBox, 0, addr FileOpenError, addr AppName,
MB_OK+MB_ICONERROR
.endif
.endif
invoke ExitProcess, 0
start
endp
SEHHandler proc C uses edx pExcept:DWORD, pFrame:DWORD,
pContext:DWORD, pDispatch:DWORD
mov edx,pFrame
assume edx:ptr
SEH
mov eax,pContext
assume eax:ptr CONTEXT
push
[edx].SafeOffset
pop [eax].regEip
push [edx].PrevEsp
pop
[eax].regEsp
push [edx].PrevEbp
pop [eax].regEbp
mov
ValidPE, FALSE
mov eax,ExceptionContinueExecution
ret
SEHHandler endp
end start
Analysis:
The program opens a file and checks if the
DOS header is valid, if it is, it checks the PE header if it's valid. If it is,
then it assumes the file is a valid PE. In this example, I use structured
exception handling (SEH) so that we don't have to check for every possible
error: if a fault occurs, we assume that it's because the file is not a valid PE
thus giving our program wrong information. Windows itself uses SEH heavily in
its parameter validation routines. If you're interested in SEH, read
http://www.jorgon.freeserve.co.uk/Except/Except.htm by Jeremy Gordon.
The program displays an open file common
dialog to the user and when the user chooses an executable file, it opens the
file and maps it into memory. Before it goes on with the verification, it sets
up a SEH:
assume fs:nothing
push fs:[0]
pop
seh.PrevLink
mov seh.CurrentHandler,offset SEHHandler
mov
seh.SafeOffset,offset FinalExit
lea eax,seh
mov fs:[0], eax
mov seh.PrevEsp,esp
mov seh.PrevEbp,ebp
We start by assuming the use of fs
register as nothing. This must be done because MASM assumes the use of fs
register to ERROR. Next we store the address of the previous SEH handler in our
structure for use by Windows. We store the address of our SEH handler, the
address where the execution can safely resume if a fault occurs, the current
values of esp and ebp so that our SEH handler can get the state of the stack
back to normal before it resumes the execution of our program.
mov edi, pMapping
assume edi:ptr
IMAGE_DOS_HEADER
.if [edi].e_magic==IMAGE_DOS_SIGNATURE
After we are done with setting up SEH, we
continue with the verification. We put the address of the first byte of the
target file in edi, which is the first byte of the DOS header. For ease of
comparison, we tell the assembler that it can assume edi as pointing to the IMAGE_DOS_HEADER structure (which is the truth). We
then compare the first word of the DOS header with the string "MZ" which is
defined as a constant in windows.inc named IMAGE_DOS_SIGNATURE. If the comparison is ok, we
continue to the PE header. If not, we set the value in ValidPE to FALSE, meaning that the file is not a
valid PE.
add edi, [edi].e_lfanew
assume edi:ptr
IMAGE_NT_HEADERS
.if [edi].Signature==IMAGE_NT_SIGNATURE
mov ValidPE, TRUE
.else
mov ValidPE, FALSE
.endif
To get to the PE header, we need the value
in e_lfanew of the DOS header. This field
contains the file offset of the PE header, relative to the file beginning. Thus
we add this value to edi and we get to the first byte of the PE header. It's
this place that a fault may occur. If the file is really not a PE file, the
value in e_lfanew will be incorrect and thus using it amounts to using a wild
pointer. If we don't use SEH, we must check the value of the e_lfanew against the file size which is ugly. If all
goes well, we compare the first dword of the PE header with the string "PE".
Again there is a handy constant named IMAGE_NT_SIGNATURE which we can use. If the result
of comparison is true, we assume the file is a valid PE.
If the value in e_lfanew is incorrect, a fault may occur and our
SEH handler will get control. It simply restores the stack pointer, bsae pointer
and resumes the execution at the safe offset which is at the FinalExit
label.
FinalExit:
.if ValidPE==TRUE
invoke
MessageBox, 0, addr FileValidPE, addr AppName, MB_OK+MB_ICONINFORMATION
.else
invoke MessageBox, 0, addr FileInValidPE, addr AppName,
MB_OK+MB_ICONINFORMATION
.endif
The above code is simplicity itself. It
checks the value in ValidPE and displays a message to the user
accordingly.
push seh.PrevLink
pop fs:[0]
When the SEH is no longer used, we
dissociate it from the SEH chain.
+
+
+
https://vahidmy.blog.ir/post/Tutorial-1-Overview-of-PE-file-format
https://vahidmy.blog.ir/post/pe-txt
+