Currently, we unconditionally rebuild the xapian index once every day regardless of whether the database has actually changed over the last day. Not only is this computationally wasteful, but it also means that we cannot respond to database updates quickly enough.
We need to be rebuilding the xapian index only when the database changes. Here's how:
We now rebuild the xapian index up to once an hour. A cron job runs every hour and checks if the database checksums have changed. If they have indeed changed, the cron job triggers the xapian index rebuild CI job.