The storage cleaner works together with compaction to reclaim unused storage from deleted snapshots.
The 6.8 release has a new architecture for the cleaner+ with improved heuristics and more efficient compaction. The new cleaner+ is also multithreaded and tier-aware.
Two new mount options
freebw are introduced in the 6.8 release and they work together with the new storage cleaner architecture for more efficient storage reclamation.
This document describes how to best use the cleaner+ and mount options to reclaim storage from your filesystem.
The storage cleaner reclaims unused storage no longer needed by any snapshots. It works together with compaction and the speed of cleaning is determined by the compaction level. Higher compaction levels will result in faster storage reclamation.
The storage cleaner has two modes: standard cleaner (
clean=1) and cleaner+ (
clean=2). The cleaner+ is better at identifying unused storage but can use more resources. To enable cleaner+, use the
clean=2 mount option with compaction enabled. Verify that all other mounts of this filesystem are running version 6.4 or newer.
This section outlines the steps to speed up the storage reclamation process using a dedicated cleaner+ mount.
We recommend having a current backup of your filesystem before proceeding.
Steps To Run Cleaner+
Use an EC2 instance in the same region as your S3 bucket.
Mount your filesystem using the 6.8 release with cleaner+ enabled (
clean=2), highest compaction level (
noatime on this instance.
For plans with multithreading, enabling multithreading will speed up storage reclamation.
A. Corporate and Enterprise plans (use the new
B. Team and Business plans:
$ sudo mount.objectivefs -o clean=2,compact=5,noratelimit,mtplus,freebw,nosnapshots,noatime <filesystem> <directory>
C. Other plans:
$ sudo mount.objectivefs -o clean=2,compact=5,noratelimit,mt,freebw,nosnapshots,noatime <filesystem> <directory>
$ sudo mount.objectivefs -o clean=2,compact=5,noratelimit,freebw,nosnapshots,noatime <filesystem> <directory>
noratelimitis required to enable compaction levels 4 and 5.
freebwwill use as much bandwidth as needed for fastest cleaning. Only use this option when connecting directly to S3 from EC2 instances in the same region to avoid AWS data transfer charges.
For most efficient cleaning, verify all other mounts of this filesystem are using version 6.8 or newer.
CLEAN field in the objectivefs log shows the cumulative storage reclaimed by this mount since starting.
The 6.8 release will also print out the current cleaning statistics when the
USR1 signal is sent to the objectivefs process id. Statistics generation can slow down the mount and therefore is only recommended for dedicated cleaner mounts.
IMPORTANT: do not send the
USR1 signal to versions before 6.8.
To get the objectivefs process id:
To print the cleaning statistics one time:
$ ps -ef | grep mount.objectivefs
To print the cleaning statistics every 10 minutes:
$ kill -USR1 <PID>
$ while sleep 600; do kill -USR1 <PID> ; done
During active storage reclamation,
df -h output of the mount may report slightly higher storage than the actual amount used. To get the current storage size, you can run
df -h on a fresh read-only mount with
$ sudo mount.objectivefs -oro,noocache <filesystem> /ofs_readonly $ df -h /ofs_readonly $ umount /ofs_readonly
The rate of storage reclamation depends on the filesystem size, filesystem age, amount of reclaimable storage, EC2 instance bandwidth and other factors. We recommend running the spot instance with cleaner+ and maximum compaction for several days and monitor the progress. Larger filesystems may require longer to reclaim the storage.
For filesystems that are 10TB or smaller, we suggest running only one mount with
compact=5,noratelimit,clean=2,freebw. All other mounts can run with default compaction level (or at most level 3).
For larger filesystems, you can run multiple high compaction mounts (on the same server or different servers). When running multiple high compaction mounts on the same server, check that there is sufficient memory and network bandwidth so that the cleaner+ can be effective.
For fastest cleaning, we suggest keeping the mount with cleaner+ running continuously and avoid restarting this mount.
ObjectiveFS is backwards and forward compatible. You can run mixed versions of ObjectiveFS and do a rolling upgrade.
For most efficient cleaning, we recommend running the 6.8 release for all mounts of the filesystem.
by ObjectiveFS staff, March 4, 2021
ObjectiveFS is a shared file system for OS X and Linux that automatically scales and gives you scalable cloud storage. If you have questions or article idea suggestions, please email us at email@example.com