Performance: Amazon EFS vs ObjectiveFS

Small file performance is important for many workloads such as e-commerce platforms, home directories, content management systems, etc. We compared the performance of two shared filesystems commonly used to scale these workloads on AWS EC2 instances: Amazon Elastic File System (EFS) and ObjectiveFS. For details on how Amazon EFS differs from ObjectiveFS, see here.

Summary

ObjectiveFS can be over 100X faster than Amazon EFS for small file reads and writes.

Setup

We compared the performance of an EFS filesystem with bursting throughput, an EFS filesystem with provisioned throughput and an ObjectiveFS filesystem running on a z1d.6xlarge EC2 instance with Ubuntu 20.04 LTS in the us-east-2 region. All filesystems are also in the us-east-2 region.

For EFS, we used the General Purpose performance mode since AWS states that this mode is better than the Max I/O performance mode for latency-sensitive workloads such as small file workloads. For the EFS filesystem with provisioned throughput, we selected 50 MiB/s for throughput based on cost (US$300/month) which is similar to an ObjectiveFS Team plan.

For ObjectiveFS, we used the default memory cache setting and enabled multithreading. For the disk cache test, we used the local NVMe SSD as the disk cache.

Setup Summary

Test

1. Small File Writes

The write test writes ~54K small files and 3657 directories with a total size of 762MB to the filesystem by unpacking the linux 4.7.2 kernel source. We measured the time it takes to untar these files on each of the target filesystem: EFS (Burst), EFS (Provisioned) and ObjectiveFS.

# cd /tmp
# wget https://cdn.kernel.org/pub/linux/kernel/v4.x/linux-4.7.2.tar.xz
# cd $TESTDIR
# time tar xvfJ /tmp/linux-4.7.2.tar.xz > /dev/null

2. Small File Reads

The read test performs random reads of small files similar to a web server workload. This test uses a program called fsperf and performs 16 parallel reads to 10K small files on the filesystem with an 80/20 access pattern. Each filesystem contains the linux 4.7.2 kernel source tree with ~54K files and 3657 directories. The total size of the filesystem is 726MB and all files are smaller than 2MB.

For ObjectiveFS, we measured the performance of three settings:
a. When the caches are not warmed (e.g. on initial boot)
b. When the disk cache is warmed (e.g. after system reboots)
c. When the memory cache is warmed (during use)

Kernel caches are dropped using echo 3 > /proc/sys/vm/drop_caches before each run.

# find $TESTDIR -type f > /tmp/filelist
# echo 3 > /proc/sys/vm/drop_caches
# /sbin/fsperf -P16 -n 10000 /tmp/filelist

Results

1. Small File Write Performance

The table below shows the time it takes to untar the linux 4.7.2 kernel source tree which writes ~54K small files with a total size of 726MB. The EFS filesystem with bursting throughput took 1444s to complete the untar, while the EFS filesystem with provisioned throughput performed slightly better at 1357s. ObjectiveFS completed the untar in 11s and is 125X faster than the baseline EFS filesystem with bursting throughput.

2. Small File Read Performance

The table below shows the throughput for doing 16 parallel reads of 10K small files, each smaller than 2MB, with an 80/20 access pattern. For ObjectiveFS, we measured the scenarios with cold cache (i.e. reading directly from S3), wtih a warmed disk cache and with a warmed memory cache.

Conclusion

For small file workloads, ObjectiveFS has higher performance compared to Amazon EFS. The write and read throughput are both over 100X faster than EFS.

To see how large file workloads compare, see Large File Performance: Amazon EFS vs ObjectiveFS. To see how ObjectiveFS performs for your workload, you can download a copy and try it with our free trial.

Last updated by ObjectiveFS staff, August 11, 2020
ObjectiveFS is a high performance scalable shared file system for Linux and macOS that scales automatically and gives you unlimited storage. If you have questions or article idea suggestions, please email us at support@objectivefs.com