Storage Layout Optimization

Compaction is a background process that optimizes the storage layout of your filesystem. An optimized storage layout provides faster performance and can reduce memory usage.

This article describes how to speed up the compaction process to quickly optimize the layout, for example after an initial import to the filesystem.

Steps

1. Install ObjectiveFS 6.8 or newer on an EC2 instance in the same region as your S3 bucket. Check that the instance type does not have limited network bandwidth for best performance.
   Note: High compaction can use a lot of bandwidth and data transfer cost is free for EC2 in the same region as the S3 bucket.

2. Mount your filesystem with the compact=5,noratelimit,freebw,clean=2 mount options.

# mount.objectivefs -ocompact=5,noratelimit,freebw,clean=2 <filesystem> <dir>

The corresponding /etc/fstab entry for mounting on boot is:

<filesystem> <dir> objectivefs auto,_netdev,compact=5,noratelimit,freebw,clean=2 0 0

Note: noratelimit is needed for compaction levels 4 and 5 to take effect.

1. Depending on the size of your filesystem, it could take a few days to achieve the optimal layout. You can monitor the compaction progress by checking the rate of object deletion (DELETE) in the log file or by checking the IUsed value from df -i (which shows the number of objects in the bucket).

Tips

1. Use a server in the same region as the S3 bucket to avoid AWS data transfer charges.
2. Use a spot instance for cost savings.
3. Using a dedicated high compaction mount on a spot instance (instead of running it on a server with other mounts or applications) is recommended.
4. Running the high compaction mount continuously is more efficient than restarting it frequently.

Last updated by ObjectiveFS staff, May 4, 2021
ObjectiveFS is a shared file system for macOS and Linux that automatically scales and gives you scalable cloud storage. If you have questions or article idea suggestions, please email us at support@objectivefs.com