How IoCancelFileOpen works

WDK says

"IoCancelFileOpen sets the FO_FILE_OPEN_CANCELLED flag in the Flags member of the file object that FileObject points to. This flag indicates that the IRP_MJ_CREATE request has been canceled, and an IRP_MJ_CLOSE request will be issued for this file object."

But this does not tell the full story. First of all IoCancelFileOpen issues IRP_MJ_CLEANUP , then sets the FO_FILE_OPEN_CANCELLED  flag. Also, IoCancelFileOpen checks that no handles have been created for the file object, if this check fails the system will crash itself with KeBugCheck. Here you should say 

  ... Wait a minute! What about IRP_MJ_CLEANUP being sent? Should it be sent only for object with handles?

 The answer is NO. The system always sends IRP_MJ_CLEANUP for all file objects, if there were no handles created for a file object the IRP_MJ_CLEANUP  request is sent by IopDeleteFile  ( called by ObDereferenceObject ) before issuing  IRP_MJ_CLOSE. Here you must understand why IoCancelFileOpen does not send IRP_MJ_CLOSE , because it is sent by IopDeleteFile called by ObDereferenceObject .

Lets now change our focus on FO_FILE_OPEN_CANCELLED . What is this flag for? This flag is used by IoCreateFile when it decides how to reclaim  file object resources when an error is returned by IoCallDevice, if the flag is set then ObDereferenceObject or IopDeleteFile is called for the file object so the file system and attached filters will receive close request. If the flag is not set then the DeviceObject member of the file object is set to NULL so the close and cleanup request will not be sent when ObDereferenceObject  or  IopDeleteFile is called to reclaim the memory occupied by the file object, the latter is a case of an error returned by the lowest driver in the stack which is a file system driver.

Below is a call stack for create request processing when an attached filter called IoCancellFileOpen that resulted in sending close request from ObfDereferenceObject 

nt!IofCallDriver+0x3f

nt!IopDeleteFile+0xef

nt!ObpRemoveObjectRoutine+0x43

nt!ObfDereferenceObjectWithTag+0x5c

nt!ObfDereferenceObject+0xd

nt!IopParseDevice+0x167a

nt!ObpLookupObjectName+0x251

nt!ObOpenObjectByName+0xfe

nt!IopCreateFile+0x2a5

nt!IoCreateFileEx+0x88

nt!IoCreateFileSpecifyDeviceObjectHint+0x59