Large File Performance: Amazon EFS vs ObjectiveFS

For processing large files such as videos, images and life science data, sequential read and write performance is a key factor when selecting a filesystem. We compared the performance of two shared filesystems commonly used to scale these workloads on AWS EC2 instances: Amazon Elastic File System (EFS) and ObjectiveFS. For details on how Amazon EFS differs from ObjectiveFS, see here.

Summary

ObjectiveFS has 400 MB/s to 460 MB/s throughput per node for large sequential reads and writes from S3 and is around 4X faster than Amazon EFS.

Setup

We selected an EC2 instance type with 10 Gbit network so the performance test is not limited by network bandwidth. We compared the performance of an EFS filesystem with bursting throughput, an EFS filesystem with provisioned throughput and an ObjectiveFS filesystem, all running on a z1d.6xlarge EC2 instance with Ubuntu 20.04 LTS in the us-east-2 region.

For both EFS with bursting and provisioned throughput, we used the default General Purpose performance mode since this test was done from a single machine. For the EFS filesystem with provisioned throughput, we selected 50 MiB/s for throughput based on cost (US$300/month) which is similar to an ObjectiveFS Team plan.

For ObjectiveFS, we enabled multithreading and hpc mode for best throughput and used the default memory cache setting.

Note: For EFS, we kept the test within the burst time limit to get the best possible EFS performance. For a 100 GiB filesystem, the EFS burst limit is 72 minutes per day. After that, Amazon EFS performance is limited to 5 MiB/s. For more details, see the Throughput Scaling section of Amazon EFS Performance. ObjectiveFS does not limit performance and can sustain the full throughput consistently 24x7.

Setup Summary

Test

1. Large Sequential Write

The write test consists of writing a large non-compressible 10 GB file to the filesystem. The source file was pregenerated using shred on the local NVMe SSD and is cached in the kernel cache to ensure reading the source file is not the bottleneck during the write test.

# touch /nvme/10G_File
# shred -n1 --size=10G /nvme/10G_File

# # Warm source file in kernel cache for faster reading
# dd if=/nvme/10G_File of=/dev/null bs=1M &> /dev/null
# dd if=/nvme/10G_File of=/dev/null bs=1M &> /dev/null
# dd if=/nvme/10G_File of=/dev/null bs=1M &> /dev/null

# time dd if=/nvme/10G_File of=$TESTDIR/10G_File bs=1M

2. Large Sequential Read

The read test consists of reading a large non-compressible 10 GB file from the filesystem. For ObjectiveFS, we measured the throughput for the initial read from S3 (cold cache) and subsequent read when the data is cached in memory. Kernel caches are dropped using echo 3 > /proc/sys/vm/drop_caches before each run for all cases to avoid any effects from kernel caching.

# echo 3 > /proc/sys/vm/drop_caches
# time dd if=$TESTDIR/10G_File of=/dev/null bs=1M

Results

1. Large Sequential Write Performance

The table below shows the throughput for writing a non-compressible 10 GB file to EFS and ObjectiveFS. The EFS filesystem with bursting throughput reached 102 MB/s (see note a), while the EFS filesystem with provisioned throughput stayed at the provisioned 52 MB/s. ObjectiveFS achieved 464 MB/s and is over 4.5X faster than the faster of the two EFS filesystems.

a: The EFS burst performance is limited to 72 minutes per day for a 100GiB filesystem. Subsequent throughput is limited to 5.25 MB/s (see “Throughput Scaling” in Amazon EFS Documentation).

2. Large Sequential Read Performance

The table below shows the throughput of reading a non-compressible 10 GB file from EFS and ObjectiveFS. The EFS filesystem with bursting throughput was 108 MB/s (see note a), while the EFS filesystem with provisioned throughput stayed at the provisioned amount of 53 MB/s. ObjectiveFS achieved 414 MB/s for large sequential reads directly from S3 and 2.4 GB/s when the data is cached in the memory cache.

a: The EFS burst performance is limited to 72 minutes per day for a 100GiB filesystem. Subsequent throughput is limited to 5.25 MB/s (see “Throughput Scaling” in Amazon EFS Documentation).

Conclusion

For large files workloads, ObjectiveFS has 400–460 MB/s for reading and writing directly to S3 and is 3.8X to 4.5X faster than EFS in the fastest burst mode. For large reads that can be cached in the memory cache, we see a throughput of 2.4 GB/s which is 22X faster than EFS.

To see how small file workloads compare, see how ObjectiveFS compared to EFS on small file workloads. To see how ObjectiveFS performs for your workload, you can download a copy and try it with our free trial.

Last updated by ObjectiveFS staff, August 11, 2020
ObjectiveFS is a high performance scalable shared file system for Linux and macOS that scales automatically and gives you unlimited storage. If you have questions or article idea suggestions, please email us at support@objectivefs.com