To improve the performance of the system when running computations (correlations, mappings, etc), we need to pre-cache the datasets in text files.
The triggers for pre-caching could be:
I propose an external job be triggered whenever any of the triggers above happen. The job could, among other things:
Maybe, if possible, we could have the pre-cache service generate the cache files based on dates for latest changes without deleting older cache files -- we could look into whether this is possible.