[Dev] Auto cleanup for Hive ORC compaction

Hi all,

Since, openlookeng is now supporting compaction/vacuum operation on Hive transactional tables which merges all base and delta files together (according to the compaction type). So that upcoming read queries can be redirected to use newly created, compacted directories which improves performance. This makes old, smaller delta directories eligible for cleanup. Therefore, a background process is required which will be responsible for removing any redundant directory from table path.

Check attached document for more details. Kindly let me know if there are any comments/suggestions.

I have created a slack channel (#auto-cleanup). Kindly join it for discussion.

Thanks, Aman Omer