Sign Up Free Log In
Why OFS? Features Pricing Docs FAQ

Storage Cleaner Guide

The storage cleaner works together with compaction to reclaim unused storage from deleted snapshots.

ObjectiveFS 6.8 and newer versions have a new architecture for the cleaner+ with improved heuristics and more efficient compaction. The new cleaner+ is also multithreaded and tier-aware.

Two new mount options mtplus and freebw were introduced in version 6.8 and they work together with the new storage cleaner architecture for more efficient storage reclamation.

This document describes how to best use the cleaner+ and mount options to reclaim storage from your filesystem.

Background Information

The storage cleaner reclaims unused storage no longer needed by any snapshots. It works together with compaction and the speed of cleaning is determined by the compaction level. Higher compaction levels will result in faster storage reclamation.

The storage cleaner has two modes: standard cleaner (clean=1) and cleaner+ (clean=2). The cleaner+ is better at identifying unused storage but can use more resources. To enable cleaner+, use the clean=2 mount option with compaction enabled. Verify that all other mounts of this filesystem are running version 6.4 or newer.

Using Cleaner+ For Storage Reclamation

This section outlines the steps to speed up the storage reclamation process using a dedicated cleaner+ mount.

We recommend having a current backup of your filesystem before proceeding.

Steps To Run Cleaner+

  1. Use an EC2 instance in the same region as your S3 bucket.

    • To avoid AWS data transfer charges, run in the same region and connect directly to your S3 bucket.
    • For best performance, use a dedicated server for storage reclamation with 8+ GB of RAM and sufficient network performance. For larger filesystems, use a larger instance.
    • [Optional] Use a spot instance for cost savings.


  2. Mount your filesystem using version 6.8 or newer with cleaner+ enabled (clean=2), highest compaction level (compact=5,noratelimit) and freebw, nosnapshots, noatime on this instance. For plans with multithreading, enabling multithreading will speed up storage reclamation.

    A. Corporate and Enterprise plans (use the new mtplus option): 

$ sudo mount.objectivefs -o clean=2,compact=5,noratelimit,mtplus,freebw,nosnapshots,noatime <filesystem> <directory>

B. Team and Business plans:

$ sudo mount.objectivefs -o clean=2,compact=5,noratelimit,mt,freebw,nosnapshots,noatime <filesystem> <directory>

C. Other plans:

$ sudo mount.objectivefs -o clean=2,compact=5,noratelimit,freebw,nosnapshots,noatime <filesystem> <directory>

* `noratelimit` is required to enable compaction levels 4 and 5.
* `freebw` will use as much bandwidth as needed for fastest cleaning. Only use this option when connecting directly to S3 from EC2 instances in the same region to avoid AWS data transfer charges.
* You can check the starting line of the [objectivefs log](/userguide#logging) to verify the mount options have taken effect. The starting line should contain `clean+`, `freebw` and `compact 5`.

<br>
  1. For most efficient cleaning, verify all other mounts of this filesystem are using version 6.8 or newer.

Monitoring Progress

  1. The CLEAN field in the objectivefs log shows the cumulative storage reclaimed by this mount since starting.

  2. Version 6.8 or newer will also print out the current cleaning statistics when the USR1 signal is sent to the objectivefs process id. Statistics generation can slow down the mount and therefore is only recommended for dedicated cleaner mounts.
    IMPORTANT: do not send the USR1 signal to versions before 6.8.

    To get the objectivefs process id:

$ ps -ef | grep mount.objectivefs

To print the cleaning statistics one time:

$ kill -USR1 <PID>

To print the cleaning statistics every 10 minutes: 

$ while sleep 600; do kill -USR1 <PID> ; done
  1. During active storage reclamation, df -h output of the mount may report slightly higher storage than the actual amount used. To get the current storage size, you can run df -h on a fresh read-only mount with noocache.
$ sudo mount.objectivefs -oro,noocache <filesystem> /ofs_readonly
$ df -h /ofs_readonly
$ umount /ofs_readonly

  1. The rate of storage reclamation depends on the filesystem size, filesystem age, amount of reclaimable storage, EC2 instance bandwidth and other factors. We recommend running the spot instance with cleaner+ and maximum compaction for several days and monitor the progress. Larger filesystems may require longer to reclaim the storage.

Note

  1. For filesystems that are 10TB or smaller, we suggest running only one mount with compact=5,noratelimit,clean=2,freebw. All other mounts can run with default compaction level (or at most level 3).

  2. For larger filesystems, you can run multiple high compaction mounts (on the same server or different servers). When running multiple high compaction mounts on the same server, check that there is sufficient memory and network bandwidth so that the cleaner+ can be effective.

  3. For fastest cleaning, we suggest keeping the mount with cleaner+ running continuously and avoid restarting this mount.

  4. ObjectiveFS is backwards and forward compatible. You can run mixed versions of ObjectiveFS and do a rolling upgrade.

  5. For most efficient cleaning, we recommend running version 6.8 or newer for all mounts of the filesystem.

Reference

by ObjectiveFS staff, March 4, 2021
ObjectiveFS is a shared file system for OS X and Linux that automatically scales and gives you scalable cloud storage. If you have questions or article idea suggestions, please email us at support@objectivefs.com