EBS backup volumes. Ephemeral storage is ideally used for any temporary data such as cache, buffers, session data, swap volume etc. Apart from this, multiple ephemeral storage can be used together as RAID volumes and for specific Hadoop jobs where high performance and multiple nodes sharing the same data is desired. Ephemeral storage is a non-billable resource that is included in the cost of the instance. However, ephemeral storage is only limited to certain instance types and the number and sizes of ephemeral drives differ for different instance types. In addition, it is available via two options, magnetic disk, or SSDs (which again depends upon the instance type you are using). It is not a part of recent instance type families such as M4 or C4. Learn more here. There is a variety of use cases where multiple ephemeral storages in RAID configurations are used to store crucial processing data which is shared across nodes. These ephemeral storages are attached to the same instance, and an instance failure can lead to total collapse. In order to achieve persistence, it is advisable to replicate your ephemeral storage to EBS volumes, so that data loss can be avoided in case of an instance failure. This article will focus on how you can mirror your Ephemeral storage to an EBS volume, ensuring that the EBS volume is used solely for write operations and the ephemeral storage is solely for read operations. This way you can avoid data loss in the case that the instance is lost/stopped/terminated due to a hardware failure or an accidental event.
Mirror Ephemeral Storage to EBS VolumeFor mirroring ephemeral storage to an EBS volume, RAID 1 can be setup across both volumes. To achieve this, follow the steps below:
- Launch an EC2 Instance with Ephemeral Storage and an EBS Volume.It is important to note three points while selecting an instance type :
- Ephemeral storage is supported only for selected instance types.
- The size of Ephemeral storage supported by each instance type is different. Kindly consider this for your ephemeral storage requirements.
- The number of Ephemeral storage volumes supported by each instance type varies.
- Command to Format EBS Volume : # mkfs.ext4 <device-path>
- Command to Mount EBS Volume : # mount <device-path> <mount-point>
- Configure Software RAID between Ephemeral Storage and EBS Volume with –write-mostly option. Linux: As typical RAID configurations will distribute read and write operations across different volumes, setting up the –write-mostly option on the EBS volume will ensure it is used solely for write purposes, while read operations are performed by the ephemeral storage.Note : Umount volumes in the case that they are already mounted, and make sure entries are removed from fstab too.# sudo mdadm –create –verbose /dev/md0 –level=1 –name=<RAID_NAME> –raid-devices=<number_of_devices> <device-1> –write-mostly <device-2> Once a RAID volume has been created, format the volume. # sudo mkfs.ext4 /dev/md0 Once the volume is formatted, mount it using a mount command. # sudo mount /dev/md0 <mount-point> To make sure mounts are persistent across the reboot operation, a fstab entry can be made. echo “/dev/md0 /backup ext4 defaults 0 0” >> /etc/fstab That’s it. Now, the RAID volume is ready to use. Any writes will automatically go to the EBS volume while read operations will be done by Ephemeral operations. This will ensure high read performance and the ability to minimize data loss.