How-to Copy Data from S3 to EBS

n2ws

Let’s look at the case of a media company that is migrating from their on-premises datacenter to AWS.
They began their migration by moving their regular backup data into Amazon S3.

As a part of their recovery procedures, they need to copy/move the files from S3 to an EBS volume and extract the data. In this post, we will show how-to copy data from Amazon S3 to an EBS volume.

Assumptions

  1. You are familiar with Amazon S3, Amazon EC2 and Amazon EBS volumes.
  2. You know how to provision an EC2 instance, Amazon S3 bucket and EBS volumes.
  3. You know how to log into a Linux EC2 instance.
  4. You are aware of the concepts of Access Key and Secret Access Key for IAM users.
  5. You are aware of the concept of IAM roles.

Environment Details

  • Region: US-East (Northern Virginia)
  • Linux Operating System: Amazon Linux

Initial Configuration Procedure

Provision an S3 Bucket

To provision an S3 bucket, you need to log into the AWS Console and navigate to S3 services. Click on ‘Create Bucket’, provide a name for your bucket and choose a region. Next, click on ‘Create’ to create your S3 bucket.
n2ws

At first, the S3 bucket will be empty.
n2ws

Upload Files to an Amazon S3 Bucket

When you click on ‘Upload’, you will receive file uploading options (e.g., drag and drop or click on ‘Add Files’).
n2ws

Once the files are uploaded, you can browse through them directly.
n2ws

Launch Your Instances with or without an IAM Role

There are two ways in which your instances can access S3 buckets: by assigning IAM roles during your instance launch or by configuring access keys and secret access keys on your instances.

1. Assigning IAM Roles During Instance Launches

This process requires you to ensure that IAM roles with the appropriate permissions are already created. If not, they need to be provisioned. For the sake of this article, we have created an IAM role with full S3 permissions.

To create an IAM role, please navigate to IAM in the AWS Management Console and follow the steps outlined below:

A. Click on ‘Create Role’
N2WS
B. Provide a name for your IAM role.
n2ws
C. Select a role type
n2ws
D. Attach a policy
n2ws
E. Review and create
n2ws
Once done, you can assign the role to your instance and launch. During the instance launch, enter the IAM role that you created under the “Configure Instance” section.
n2ws
That’s it. These simple steps will launch your instance using IAM role.

2. Configuring Access Keys and Secret Access Keys on Your Instance If you plan on accessing your S3 bucket and S3 objects using Access Keys and Secret Access Keys, nothing additional needs to be done while launching the instance.

Configure Your Instance to Access the S3 Bucket

In order for your instance to access your S3 bucket, configure the AWS CLI on your instances. S3 buckets are already pre-bundled on Amazon Linux instances. However, S3 bucket access needs to be manually configured for other Linux instances. Please refer to the links below:

To install using PIP:
http://docs.aws.amazon.com/cli/latest/userguide/installing.html#install-with-pip
To install using Bundled Installer: http://docs.aws.amazon.com/cli/latest/userguide/installing.html#install-bundle-other-os
For Windows Instances: http://docs.aws.amazon.com/cli/latest/userguide/installing.html#install-msi-on-windows

After you’ve installed the AWS CLI, you need to ensure that it is properly configured. If you are using an IAM role, nothing additional needs to be done. But for Access Key and Secret Access Key users, you need to follow the additional step of configuring the AWS CLI to use your Access Key and Secret Access Key.

Log into your EC2 instance and execute the command below:
# aws configure
This command will prompt you to enter your Access Key, Secret Access Key, region and output format.
n2ws

Verify Access to Your S3 Buckets

Before you start copying data, it is good to verify access to your S3 buckets. In order to do so, we will use the AWS CLI commands.
To view a list of S3 buckets, type the following command:
# aws s3 ls
n2ws

To view the content of your S3 bucket (e.g., s3-to-ebs-data-transfer-example), type the following command:
# aws s3 ls s3://s3-to-ebs-data-transfer-example
n2ws

Copy Data to Your S3 Bucket

In order to copy data to your S3 bucket, please ensure that you’ve already created a destination directory. If not, please create a one:
# mkdir data
If you want to copy a specific file from your S3 bucket to your destination directory on your EBS volume:
# aws s3 cp s3://s3-to-ebs-data-transfer-example/sample.log data/
n2ws

If you want to copy all files recursively from your S3 bucket to your destination directory on your EBS volume:
# aws s3 cp s3://s3-to-ebs-data-transfer-example/ data/ –recursive
n2ws

To verify whether the files were properly copied, please browse your destination directory:
n2ws

Apart from using the AWS CLI commands, Windows users can copy files from S3 to EBS volumes by using RDP into a Windows instance. After you connect to the AWS Management Console, you can directly copy files from the S3 Console to your EBS volumes. For this, you need to ensure that the correct login credentials (username, password, IAM URL) are readily available.

N2Ws offers a solution that enables you to further simplify the process of using snapshots. Cloud Protection Manager (CPM) is an enterprise-class backup-recovery and disaster recovery solution for the EC2 compute cloud. CPM is available as a service model that allows users to manage multiple AWS accounts and configure policies and schedules to take automated snapshot backups. It also has a Windows agent to consistently back up Windows applications. CPM allows you to recover a volume from a snapshot, increase its size and switch it with an existing attached volume, in a single step. To see the different editions with pricing and details, refer to N2Ws’ pricing and purchases page on our website.

Share this post →

You might also like: