Tuesday, May 26, 2009

Conclusive way of testing if content has changed since last FTIndex

The standard way of testing, db.LastModified > db.LastFTIndexed, works great in a production environment. When you're doing performance tuning of an application that updates it's FT-index, it's not so great.

NotesDatabase.LastModified also includes changes to design elements. If you use the above test to fire a NotesDatabase.UpdateFTIndex, then the check, db.LastModified > db.LastFTIndexed, will return true until a document is modified in the db/the FTIndex is updated. Changes to design elements is ignored by the FT-indexing service on the server.

NotesDatabase.UpdateFTIndex is relatively expensive (I would think the more documents in the database -> The more time to check if the FTIndex should be updated). In an application of mine, I use NotesView.FTSearch to find certain content in views (to create HTML menu, etc). I use a class that updates the FT-index before it does the search. On regular pages, there is created two instances of this class. In an application with 200 documents, it takes the server ~130ms to do two calls to NotesDatabase.UpdateFTIndex, when the application is indexed. When I do performance tuning, I do changes to design elements, not content. Therefore the aforementioned test to determine if content has changed is flawed.

This is my so far best alternative to the "standard check". It takes 16ms to do twice in the aforementioned application. A savings of ~115ms.

Function contentChangedSinceLastFTIndex( db As NotesDatabase ) As Boolean
Dim session As New NotesSession

Dim lastFTIndexed As Variant
lastFTIndexed = db.LastFTIndexed

'First test (least resource-hungry)
If Not ( db.LastModified > lastFTIndexed ) Then
contentChangedSinceLastFTIndex = False
Exit Function
End If

'Conclusive test - test if actual documents (not design elements)
'have been changed since last FTIndex
Dim dateFTIndexed As New NotesDateTime ( lastFTIndexed )

Dim col As NotesDocumentCollection
Set col = db.GetModifiedDocuments( dateFTIndexed )
If col.Count > 0 Then contentChangedSinceLastFTIndex = True
End Function

2 comments:

nick wall said...

Tommy,

I need to verify this in a test, but wondering whether you know the answer...

Let's say you have an agent that wants to check for any new or modified content in database, just like you have done, but what if there is a local replica of the database, some documents in it were created or modifed (extreme case to illustrate point) 3 months ago, but have only just been replicated to the server replica. The modified\ created date is much older than the NotesDateTime in db.GetModifiedDocuments( NotesDateTime, DBMOD_DOC_DATA), so they will not be picked up.

I'm trying to fugure out a way to "get all new or modified documents in a database since my agent last looked at it, including documents that might be replicated to it".

Thanks,

Nick

Tommy Valand said...

I don't know how to do it, but a little search in the Notes/Domino forums lead me to this:
http://www-10.lotus.com/ldd/46dom.nsf/c21908baf7e06eb085256a39006eae9f/3184fa2e5a813af580256b61003b2e12?OpenDocument