Performance Optimization Guide

For best performance, we recommend running on ObjectiveFS on servers with at least 2GB of RAM and 2 cpus.

Performance Recommendations

1. Use a larger memory cache

• Use 2G+ memory cache for best performance. Larger memory cache gives better performance.
• Memory cache can be adjusted using the CACHESIZE environment variable.
• Reference: Memory Cache User Guide

2. Enable disk cache

• Use the local instance store or NVMe SSD as disk cache.
• The disk cache maintains good performance even with a smaller memory cache and reduces the number of S3 operations.
• Reference: Disk Cache User Guide, Instance Store Disk Cache Setup Guide

3. Run your servers in the same region as your object store

• Running your servers in the same region as your object store (e.g. S3 bucket) gives you lower latency and higher bandwidth.

4. Enable multithreading

• Multithreading can lower latency and improve throughput for your workload.
• Multithreading is a mount option and can be enabled with -o mt.

  $ mount.objectivefs -o mt <filesystem> <directory> 

• You can also configure the number of CPU and IO threads explicitly by using the cputhreads and iothreads options (see user guide).
• Reference: Multithreading User Guide, Mount options

5. Use kernel cache

• Kernel cache can improve re-read performance, especially for web server and simulation workload. (available in 6.0 and newer)
• Kernel cache is activated when multithreading is enabled and your Linux kernel supports this option.
• Reference: Kernel Cache User Guide

6. Consider noatime and nodiratime

• If your application doesn’t need atime or diratime (timestamp when a file or directory was last read), using the noatime and nodiratime mount options will reduce the number of writes to your filesystem.
• Example mount command with noatime and nodiratime:

  $ mount.objectivefs -o noatime,nodiratime <filesystem> <directory> 

• Example /etc/fstab entry with noatime and nodiratime:

  <filesystem> <directory> objectivefs auto,_netdev,noatime,nodiratime  0 0

7. Try hpc mode for batch workload

• The high performance computing (hpc) mode is a mount option that can be enabled with
  -o hpc.
• Enabling hpc will tradeoff throughput over latency when possible, e.g. by sending larger writes to the object store and doing larger read ahead steps.

  $ mount.objectivefs -o hpc <filesystem> <directory> 

8. Read and write data in multiples of 128KB

• Using multiples of 128KB for read and write sizes is good for performance and reduces system overhead.

9. Use compaction to optimize the storage layout

• Compaction improves performance and reduces the number of requests to the object store.
• You can use a temporary high compaction mount to quickly optimize the layout, e.g. after an initial import to the filesystem. See the Storage layout optimization guide.

10. Exclude your filesystem from programs that regularly scan directories

• If your server runs programs that scan the filesystem, such as mlocate, we recommend excluding your ObjectiveFS directories from the scan for best performance.
• Reference: How to exclude your filesystem from mlocate scanning

11. S3 Bucket Versioning and Lifecycle Rules

• Check that versioning is not enabled on your S3 bucket. Versioning is not required for ObjectiveFS and may result in slower performance.
• If your S3 bucket has versioning enabled, make sure that there is a lifecycle rule to clean up expired object delete markers
• Reference: How to check if versioning is enabled, Lifecycle rule to clean up delete markers

12. Performance testing considerations

• /dev/random or /dev/urandom can sometimes become the bottleneck during the filesystem performance testing. We recommend pre-generating the random data in /tmp and copy them to your filesystem for an accurate measurement.
• /dev/zero will be compressed.

Reference

• Memory Cache User Guide
• Disk Cache User Guide
• Multithreading User Guide

If you have questions, please email us at support@objectivefs.com.

Last updated by ObjectiveFS staff, October 3, 2022

ObjectiveFS is a shared filesystem for Linux and macOS that automatically scales up and out with high performance. In production use by Fortune 500 companies since 2013.