This patch is replacing patch 763, which had serious performance problems.

Revision 3 contains a fix for bugs in the previous revisions. It has not
the recovery / cleanup which revision 1 had. Instead it prevents the same
issue, by preventing duplication of 'add' and 'check_and_split_data' processes.
This will avoid corruptions of the database in some cases.

The cleanup of a corrupt database will be made available as a script.

Revision 4 is for better performance when processing data-*-*-*.temp files
in propagate data types.

This patch has added functionality for cleaning up in the database.

Sometimes the database has been inconsistent due to various error situations.

* Link which points to a directory where there is no file
* Data file missing where the back-end would expect to find a file during aggregation

These two issues have been fixed.

Revision 2: Recover old data when temp file is present. "add" will also run in only one process. If
another process is started, it will trigger another loop of "add" and exit.