Windows Desktop Search Index

Microsoft Extensible Storage Engine (ESE) database is used by a variety of Microsoft services including Exchange, Windows Mail, Active Directory, and Windows Desktop Search. I recently began wondering what forensic artifacts might be indexed by Windows Desktop Search (WDS) and available to an analyst. By default, user documents and IE internet history are indexed, but Outlook 2007/2010 also integrates with WDS. Consequently, this might be an additional source of email artifacts. While there can be a wealth of information available to a responder in an enterprise that utilizes Microsoft Exchange and any of a variety of email archiving solutions, the WDS ESE database may still be useful in non-enterprise settings.

After some searching, I came across Joachim Metz research on the ESE format and WDS as part of the libesedb project. Metz documents the ESE database structure, data obfuscation, and compression thoroughly. Consequently, I am not going to summarize all of his research but fully recommend you read it if interested.

The libesedb project contains two tools; esedbinfo and esedbexport. Esedbinfo provides detail about the structure of the ESE file and Esedbexport allows you to extract the tables for analysis. The following is an example of running Esedbexport on the WDS database (the default location is C:\ProgramData\Microsoft\Search\Data\Applications\Windows\Windows.edb). It should be noted that the Windows Search (WSearch) service needs to be stopped to access this file on a live system.

The SystemIndex_0A table contains the bulk of useful information. The following is an example of the Outlook Welcome email obtained from the parsed table.

To the best of my knowledge, it is unknown how long indexed data is kept but I was able to obtain previously deleted emails from several days prior without issue. This included the full body of the email (see update below). Again, I am unsure how often a forensicator would need to utilize these artifacts. In addition to the aforementioned resources available in an enterprise, Microsoft Outlook also utilizes the Personal Folder File (PFF) format for Personal Storage Table (PST) and Outlook Storage Table (OST) files. These are both commonly known as Outook Data Files. The former (PST) is used in a non enterprise setting when configuring outlook with email services such as pop/smtp and the later is created in enterprises with Outlook cached mode is enabled.

In addition to the libesedb project, Joachim Metz also runs the libpff project. His research there provides a tremendous amount of insight into the PFF file structure and usefulness.

So what do you say? Is the Microsoft ESE file format a useful artifact for file forensics?

Happy Hunting!

Updated: September 09, 2011

Dave Hull was kind enough to post a comment and share some of his experiences with WDS and deleted files. This consequently got me to revisit my testing with a larger poking stick. After several hours I determined a few things about deleted emails and the affects on the WDS index.

First and foremost, I could not duplicate finding deleted emails in the index. I am unsure if my initial testing was flawed or if there is internal workings unknown to me. I did however note the following behavior when deleting emails.

When an email is sent to the Deleted Items folder in Outlook the "System_IsDeleted" is marked as True and the "System_ItemFolderPathDisplay" value is changed to reflect this new location. This comes as no surprise. This was the case with my initial testing and the example I gave of the Outlook Welcome Email.

Once the email is removed from the Deleted Items folder, the Index Record is removed very quickly. I confirmed this multiple times. This leaves a missing DocID in the table which is eventually re-used for another index record. This is very similar to the behavior of the NTFS Master File Table when a files/folders are deleted.

I re-read Joachim Metz's initial research and he does mention that the WDS index can contain deleted file information and content but was unsure how long this is kept. He also mentions a table called "SystemIndex_DeletedDocIds" which contains the deleted DocId's in Windows Vista and above. Unfortunately, the Esedbexport tool does not seem to extract this table as of yet.

All things considered, a very interesting experiment.