Last month, Amazon launched its EBS Provisioned IOPS (PIOPS) for running high
performance databases in the cloud, and introduced EBS-Optimized instances to deliver
dedicated throughput between EC2 and EBS. As the provider of the Redis Cloud service, we
wanted to know whether the PIOPS EBS with EBS-Optimized instances would really perform
better than Standard EBS with non-optimized instances for Redis.
Running Redis on AWS has been a challenge for Redis users, mainly due to EBS’ slow and
unpredictable network and I/O throughput. Though Redis is a pure in-memory datastore
system and serves everything from RAM, it also has two persistent-storage mechanisms for
better durability: AOF (Append Only File) and Snapshotting. These mechanisms are very
efficient when interfacing with persistent storage, as they only use sequential writes with
no seek time. As a result, Redis can run pretty well with data-persistence even on non-SSD
hardware.
That being said, we decided to conduct some benchmarks for Amazon’s EBS PIOPS because
slow storage devices can still be very painful for Redis. We see this when the AOF “fsync
every second” policy blocks an entire Redis operation due to slow disk I/O. In addition,
background saving for point-in-time snapshots or AOF rewrites takes longer, and therefore
more memory is used for copy-on-write — leaving less memory available for your app.
You may have seen the detailed post from Strattalux comparing the performance of PIOPS
EBS vs. Standard EBS using EBS-Optimized and non-Optimized instances.
While their tests were interesting, we thought we’d dig a little deeper and examine whether the performance of a simple Redis application is influenced by different storage architecture. Therefore, we were very specific and only tested the performance of sequential writes, ignoring other scenarios like random read/write and sequential read. For our benchmark, we also reviewed cost implications, and used what we believe is a more noptimal performance configuration of Standard EBS (1 x 1TB EBS volume vs. 10 x 100GB RAID0 EBS, per a very educational blog about Standard EBS performance by Adrian Cockcroft from Netflix). So, without further ado, here are the details of our tests…
Benchmark Setup
We used the following setup: PIOPS EBS with Optimized-Instance (i.e. m2.4xlarge with
EBS-Optimized attribute set to true) vs. Standard EBS with non-Optimized Instance
(m2.4xlarge with EBS-Optimized attribute set to false).
We also tested another setup in which we compared PIOPS EBS vs. Standard EBS on a non
Optimized-Instance (m2.2xlarge). Our intention was to see whether AWS users who cannot
use the EBS Optimized-Instance for various reasons (such as cost) can get benefit from the
new PIOPS EBS. Those results were very similar to the setup described above.
For all tests, we used 2 x m1.xlarge instances that run our memtier_benchmark load
generation tool (memtier_benchmark is an advanced load generator tool we developed for
testing Redis and Memcached datastores, which we will soon share in our github account).
We ran 8 tests on each configuration to simulate a Redis application, then ran each of these
tests 3 times and calculated the average results.
Our tests combinations included:
- 100 and 500 connections on each load-generator machine (200 and 1000 connections total).
- 100B and 1KB object size
- 50%/50% read/write scenario, and 100% write scenario (for imposing load on EBS)
Benchmark Results
Note – we have chosen to show the results from the 1KB object size tests, which impose
higher load on the EBS.
The RPS and Latency results aggregated from our two memteir_benchmark tools are shown
above. Our latency tests take into account the network round-trip, the Redis processing
time, and the time it takes for the memtier_benchmark tool to parse the result. As you
can see, RPS and latency are approximately the same on all tests, though there are some
differences which might be due to the fluctuations in the AWS infrastructure during the test
time. Needless to say, Redis runs better with fewer disk accesses.
The results above are taken from iostats commands executed while the tests were running
(here’s a useful AWS forum post that explains how to interpret Linux iostats). We ran
iostats every 2.5 seconds and aggregated results from both EBS volumes. These charts
only present relevant performance parameters for Redis — “read” parameters were not
presented as they were constantly zero during these tests. As you can see, except from
few glitches here and there, iostats show approximately the same results for both EBS
configurations.
Conclusion
After 32 intensive tests with Redis on AWS (each run in 3 iterations for a total of 96 test
iterations), we found that neither the non-optimized EBS instances nor the optimized-
EBS instances worked better with Amazon’s PIOPS EBS for Redis. According to our results,
using the right standard EBS configuration can provide equal if not better performance
than PIOPS EBS, and should actually save you money. Keep in mind that these findings
represent a specific storage usage pattern (sequential write only), so if you have different
storage access patterns, you may want to review the Stratalux test results (while taking
into account the different Standard EBS configuration we suggest).
Given these results, we plan to continue running Garantia Data’s clusters on a Standard EBS
configuration. We welcome comments from the Amazon EBS Team on our benchmark, and
will be sure to share any new information we discover.
