Storage Layout Performance Optimization

The new compaction optimizations and storage cleaner available in the 5.2 release and newer will optimize the storage layout of your ObjectiveFS filesystems. An optimized storage layout can improve your existing filesystem’s performance, such as the mount time, directory list time, file access time, etc. The storage layout optimization will occur automatically in the background when your filesystem is mounted with compaction (default on).

This document describes how to speed up the storage layout optimization, so you can achieve an optimized layout on your filesystem faster.

The 5.2 release is backwards compatible with older releases. You can continue to run your existing mounts with older releases if you choose to, while optimizing the storage layout on another server running the 5.2 release using the following steps.

Before you begin: We recommend having a current backup of your filesystem before proceeding with the steps below.

Steps

  1. Start an EC2 spot instance (r4.* or i3.* instance) in the same region as your S3 bucket.
    Using a server in the same region as the S3 bucket is highly recommended to avoid AWS data transfer charges. Using a server with sufficient memory and network performance will help with the storage optimization.

  2. On the spot instance, mount your filesystem with the 5.2 release maximum compaction rate and storage cleaner enabled. For best performance, this spot instance should be dedicated for the storage optimization only.
    Note: the noratelimit option is needed for compaction levels 4 and 5.

    $ sudo mount.objectivefs -o compact=5,clean,noratelimit <filesystem> <directory>

  3. (Optional, useful for faster compaction) On all your regular mounts, upgrade to the 5.2 release and mount your filesystem with the default compaction level and storage cleaner.

    $ sudo mount.objectivefs -o clean[,<regular mount options>] <filesystem> <directory>

Note

  1. We recommend running compact=5 on only one server. The rest of your servers (if upgrading to release 5.2 or newer) can run with the default compaction level (or at most compaction level 3).
  2. The rate of storage optimization will depend on your filesystem size, current storage layout, EC2 instance bandwidth and other factors. We recommend running the spot instance with maximum compaction (compact=5) for several days and monitor the compaction progress. Larger filesystems may require longer to achieve the optimal layout.
  3. You can monitor the compaction progress by checking the rate of object deletion (DELETE) and data in/out (IN GB, OUT GB) in the log file.

by ObjectiveFS staff, July 26, 2017
ObjectiveFS is a shared file system for OS X and Linux that automatically scales and gives you scalable cloud storage. If you have questions or article idea suggestions, please email us at support@objectivefs.com

Get updates about ObjectiveFS