Cleanup of database after crash or a similar situation.

The cleanup script is found under /opt/openit/contrib/bin/db_clean_structure.pl.

It can be activated to run regularly by making a file:

/var/opt/openit/etc/database_test_active

It will then run whenever the database is being updated by aggregation.

It can be run manually, and then it has to run as the "openit" user.

This patch is replacing patch 763, which had serious performance problems.

Revision 3 contains a fix for bugs in the previous revisions. It has not
the recovery / cleanup which revision 1 had. Instead it prevents the same
issue, by preventing duplication of 'add' and 'check_and_split_data' processes.
This will avoid corruptions of the database in some cases.

The cleanup of a corrupt database will be made available as a script.

Revision 4 is for better performance when processing data-*-*-*.temp files
in propagate data types.

Supports removal of massive lists of classification values
in the WebGUI, to speed up processing and to make the
list easier to handle.

To activate it, set, in /etc/opt/openit/openit.cfg:

CLASSVAR_REGEXP_FILTER   $CONFIG/classvar_regexp_filter

Make the file, and enter into it:

<data type name>:<classification name>:<regexp>

for each data type / classification wanted.

When the regexp matches, the entry will be removed from
the classification list in the WebGUI.

Rev 2: Added requirement for patch 723.

check_and_split_data used to abort processing when encountering an olap file with illegal time
stamp. This has been fixed so that it will rename file to *.inconsistent and proceed with next file.

Revision 3: Added Perl modules for compatibility with older versions of Open iT.

Revision 4: Supersedes 561.

This patch has added functionality for cleaning up in the database.

Sometimes the database has been inconsistent due to various error situations.

* Link which points to a directory where there is no file
* Data file missing where the back-end would expect to find a file during aggregation

These two issues have been fixed.

Revision 2: Recover old data when temp file is present. "add" will also run in only one process. If
another process is started, it will trigger another loop of "add" and exit.

* Processing in chunks, avoiding multiple check_and_split_data processes.
* Handling of start/end time of days with DST changes.