Disk Cache Warming Guide

Using a disk cache on a local SSD can improve performance and reduce the number of requests to the object store. The disk cache is filled on demand, as your filesystem is used by your applications. Using one of the following techniques, you can warm (prepopulate) the disk cache with commonly used data.

1. Prefetch metadata into the disk cache

You can warm the disk cache by prefetching file metadata in commonly used directories after your filesystem is mounted.

Example command to prefetch metadata in a specific directory tree:

# find <path to specific directory> -type d -print0 | xargs -0 -P128 -n1 ls

2. Prefetch both metadata and data into the disk cache

For directories containing commonly used small files, it can sometimes be useful to prefetch both the metadata and data into the disk cache.

Example command to prefetch both metadata and file data in a specific directory tree:

# find <path to specific directory> -type f -print0 | xargs -0 -P128 -n1 md5sum

3. Copy from an already warmed disk cache

Another way to prepopulate the disk cache with data is to copy or rsync the data from the disk cache of another machine that has been running for a while. The copy or rsync can be done before or during your filesystem’s mount process or after your filesystem is already mounted.

4. Copy from a stored disk cache image

If your filesystem metadata or data doesn’t change often, you can also store the warmed disk cache on a persistent storage (e.g. EBS). To warm the disk cache, you can copy or rsync the stored warmed data from persistent storage to the disk cache. This can also be done before or during the filesystem’s mount or after your filesystem is already mounted.

Disk Cache Tips

You can start populating the disk cache directory (specified by DISKCACHE_PATH) as soon as it is available. The disk cache is robust and can be written to or deleted while the filesystem is mounted and running. ObjectiveFS will automatically use what is available and refresh the data as needed.

The disk cache will stay persistent on the same EC2 instance, and the disk cache data will be reused when you unmount and remount your filesystem.

You can also check if the disk cache hit rate by looking at the last column of the log file (i.e. DISK [95.0% HIT]).

Reference

by ObjectiveFS staff, January 4, 2018
ObjectiveFS is a shared file system for OS X and Linux that automatically scales and gives you scalable cloud storage. If you have questions or article idea suggestions, please email us at support@objectivefs.com