After some searching, I came across Joachim Metz research on the ESE format and WDS as part of the libesedb project. Metz documents the ESE database structure, data obfuscation, and compression thoroughly. Consequently, I am not going to summarize all of his research but fully recommend you read it if interested.
The libesedb project contains two tools; esedbinfo and esedbexport. Esedbinfo provides detail about the structure of the ESE file and Esedbexport allows you to extract the tables for analysis. The following is an example of running Esedbexport on the WDS database (the default location is C:\ProgramData\Microsoft\Search\Data\Applications\Windows\Windows.edb). It should be noted that the Windows Search (WSearch) service needs to be stopped to access this file on a live system.
The SystemIndex_0A table contains the bulk of useful information. The following is an example of the Outlook Welcome email obtained from the parsed table.
To the best of my knowledge, it is unknown how long indexed data is kept
In addition to the libesedb project, Joachim Metz also runs the libpff project. His research there provides a tremendous amount of insight into the PFF file structure and usefulness.
So what do you say? Is the Microsoft ESE file format a useful artifact for file forensics?
Updated: September 09, 2011
Dave Hull was kind enough to post a comment and share some of his experiences with WDS and deleted files. This consequently got me to revisit my testing with a larger poking stick. After several hours I determined a few things about deleted emails and the affects on the WDS index.
First and foremost, I could not duplicate finding deleted emails in the index. I am unsure if my initial testing was flawed or if there is internal workings unknown to me. I did however note the following behavior when deleting emails.
When an email is sent to the Deleted Items folder in Outlook the "System_IsDeleted" is marked as True and the "System_ItemFolderPathDisplay" value is changed to reflect this new location. This comes as no surprise. This was the case with my initial testing and the example I gave of the Outlook Welcome Email.
Once the email is removed from the Deleted Items folder, the Index Record is removed very quickly. I confirmed this multiple times. This leaves a missing DocID in the table which is eventually re-used for another index record. This is very similar to the behavior of the NTFS Master File Table when a files/folders are deleted.
I re-read Joachim Metz's initial research and he does mention that the WDS index can contain deleted file information and content but was unsure how long this is kept. He also mentions a table called "SystemIndex_DeletedDocIds" which contains the deleted DocId's in Windows Vista and above. Unfortunately, the Esedbexport tool does not seem to extract this table as of yet.
All things considered, a very interesting experiment.