Ephemeral storage is the volatile temporary storage attached to your instances which is only present during the running lifetime of the instance. In the case that the instance is stopped or terminated or underlying hardware faces an issue, any data stored on ephemeral storage would be lost. This storage is part of the disk attached to the instance. It turns out to be a fast performing storage solution, but a non-persistent one, when compared to EBS backup volumes.
Ephemeral storage is ideally used for any temporary data such as cache, buffers, session data, swap volume etc. Apart from this, multiple ephemeral storage can be used together as RAID volumes and for specific Hadoop jobs where high performance and multiple nodes sharing the same data is desired.
Ephemeral storage is a non-billable resource that is included in the cost of the instance. However, ephemeral storage is only limited to certain instance types and the number and sizes of ephemeral drives differ for different instance types. In addition, it is available via two options, magnetic disk, or SSDs (which again depends upon the instance type you are using). It is not a part of recent instance type families such as M4 or C4. Learn more here.
There are a variety of use cases where multiple ephemeral storages in RAID configurations are used to store crucial processing data which is shared across nodes. These ephemeral storages are attached to the same instance, and an instance failure can lead to total collapse. In order to achieve persistence, it is advisable to replicate your ephemeral storage to EBS volumes, so that data loss can be avoided in case of an instance failure.
This article will focus on how you can mirror your Ephemeral storage to an EBS volume, ensuring that the EBS volume is used solely for write operations and the ephemeral storage is solely for read operations. This way you can avoid data loss in the case that the instance is lost/stopped/terminated due to a hardware failure or an accidental event.
Mirror Ephemeral Storage to EBS Volume
For mirroring ephemeral storage to an EBS volume, RAID 1 can be setup across both volumes. To achieve this, follow the steps below:
Launch an EC2 Instance with Ephemeral Storage and an EBS Volume.It is important to note three points while selecting an instance type :
Ephemeral storage is supported only for selected instance types.
The size of Ephemeral storage supported by each instance type is different. Kindly consider this for your ephemeral storage requirements.
The number of Ephemeral storage volumes supported by each instance type varies. For more details on above points, you can refer to this AWS documentation link. For demonstration purposes, we have chosen a m3.large instance which offers an ephemeral storage volume of 32GB and an attached 32GB of EBS volume.
2. Access your Ephemeral Storage and EBS Volume. In most of the cases, ephemeral storage is already formatted and mounted. On Windows, EBS volumes are also formatted to NTFS if they are attached during launch time. Linux :
To access your EBS volume, it can be formatted and mounted by using the below commands.
Command to Format EBS Volume : # mkfs.ext4
Command to Mount EBS Volume : # mount
Windows :
Configure Software RAID between Ephemeral Storage and EBS Volume with –write-mostly option.
Linux: As typical RAID configurations will distribute read and write operations across different volumes, setting up the –write-mostly option on the EBS volume will ensure it is used solely for write purposes, while read operations are performed by the ephemeral storage.
Note : Unmount volumes in the case that they are already mounted, and make sure entries are removed from fstab too.
# sudo mdadm --create --verbose /dev/md0 --level=1 --name=<RAID_NAME> --raid-devices=<number_of_devices> <device-1> --write-mostly <device-2>
Once a RAID volume has been created, format the volume.
# sudo mkfs.ext4 /dev/md0
Once the volume is formatted, mount it using a mount command.
# sudo mount /dev/md0 <mount-point>
To make sure mounts are persistent across the reboot operation, a fstab entry can be made.
echo “/dev/md0/backup ext4 defaults 0 0” >> /etc/fstab
That’s it. Now, the RAID volume is ready to use. Any writes will automatically go to the EBS volume while read operations will be done by Ephemeral operations. This will ensure high read performance and the ability to minimize data loss.
Windows:
To create a software RAID1 in Windows, navigate to Disk Management under Computer Management, select any of your volumes and right click on it to create a mirrored volume. When selecting New Mirrored Volume, select the volumes that should be a part of the Mirrored volume.
Clicking on next will ask you to assign a drive letter, and later on you will be prompted to format the volumes. Once done, the volume will be formatted and automatically mounted.
Now you have established mirroring between the ephemeral storage and the EBS volume, which means that all data will be in mirrored between both disks.
Ephemeral storage volumes are a great option for holding data where data persistence is not an issue, hence allowing you to achieve high-performance levels. A right combination of ephemeral storage and EBS volume storage can provide an ideal combination of performance and data persistence.
N2WS Backup & Recovery is an enterprise-class backup-recovery and disaster recovery solution for AWS EC2. N2WS Backup & Recovery is available as a service model that allows users to manage multiple AWS accounts and configure policies and schedules to take automated snapshot backups. It also features a Windows agent to consistently back up Windows applications.
N2WS allows you to recover a volume from a snapshot, increase its size and switch it with an existing attached volume in a single step. Try N2WS Backup & Recovery for FREE!
Read Also
