activepivot

Handlers for File Removal


After loading our initial facts into the cube, we then load a second file that adds measures to the existing facts (so no new facts are created by the second file). We use a Handler to do this.

When the second file is removed from the filesystem, we would like to remove just the relevant measures from the facts.

Is there a way for us to plug into the Directory/File Watcher mechanism to accomplish this?


Solution

  • You could extend

    .CSVSource.onFileAction(IFileWatcher watcher, Collection<String> added, Collection<String> modified, Collection<String> deleted)
    

    by calling super.onFileAction(...) which will process the added and modified files, and add more logic to handle deleted files.

    This can be done by updating the facts which has contributed a deleted file in their deletedFile field. Such a field could be filled automatically by adding the FILEPATH metadata in your LoadInstructions.csv file:

    Format,FilePattern,FilePath,MetaData
    FormatName,formatRegex.csv,someFolder,FILEPATH=N/A
    

    and having a field like:

    <field name="FILEPATH" type="string" indexation="dictionary" nullable="true" defaultValue="N/A" />