Amazon Web Services Tips

  


Build a Custom Amazon EC2 Machine Image - (CentOS 6.2)

by Jeff Hunter, Sr. Database Administrator

Contents

Introduction

There is no shortage when it comes to finding an available Amazon Elastic Compute Cloud (EC2) Machine Image (AMI). Often times, however, finding an image from the community AMIs that meets your particular needs can be a challenge. In many cases the image is bloated, provides too much customization, performs poorly, or lacks any type of reasonable documentation. Not to mention the inherent security concerns associated with some 3rd party AMIs.

In this article, I will demonstrate how to create your own instance store-backed (a.k.a. S3-backed) and EBS-backed Amazon EC2 image of CentOS 6.2 (64-bit) with its own kernel. Creating your own AMI allows you to make the most of Amazon EC2 and provides better control over performance, security, and reproducibility. Your AMIs become the basic unit of deployment which allow you to rapidly boot new custom instances as you need them.

 

This guide has been successfully tested and verified to work with the Linux distributions listed below. Substitute any references to CentOS 6.2 with the appropriate distribution name and version number when completing the tasks in this guide.

  • CentOS 6.2 (64-bit)
  • CentOS 6.3 (64-bit)

There are two methods to prepare your own custom Amazon EC2 instances for Linux/UNIX systems:

Although preparing a new AMI from an existing one is often the easiest method, this guide will document the procedures to create a new AMI from scratch using a fresh OS install of CentOS 6.2 (64-bit) on an empty file system mounted by loopback.

Creating AMIs through a loopback involves performing a full operating system installation on a clean root file system. Using an empty file system mounted by loopback avoids having to create a new root disk partition and file system on a separate physical disk. After installing the operating system, the resulting image gets bundled as an AMI with the ec2-bundle-image command, which should be noted is part of the Amazon EC2 AMI Tools (and not the Amazon EC2 API Tools). Finally, the new AMI will be uploaded as an instance store-backed image and registered with Amazon EC2 using the command-line tools. Instructions will also be provided at the end of this guide to convert the instance store-backed AMI to an EBS-backed AMI.

Requirements

Ensure the following prerequisites have been met before creating your new Amazon EC2 image. Although you may have already fulfilled some of the requirements discussed below, please do not skip over this section as it contains important configuration information needed to follow along with the examples presented in this guide. For example, which Amazon AWS services to sign up for and how to prepare the machine and set environment variables used to create the new AMI.

  1. Amazon AWS Account

    Obviously the first requirement is to create an AWS Account from the Amazon Web Services site if you don't already have one. Creating an AWS account is free; however, you will need to provide a credit card when signing up for the Amazon Elastic Compute Cloud (EC2), Amazon Simple Storage Service (S3), and Elastic Block Store (EBS) services (discussed below).

  2. AWS Account Number

    The account number (sometimes called the account id) shows up when you go to the Account Activity area of the AWS web site. The account number is a 12 digit number that appears in the top-right of the Account Activity page and is in the form:

    9999-9999-9999

    When you use the account number in the context of the APIs, you should leave out the hyphens and just enter the 12 digits.

    In this guide, your AWS account number will be assigned to the environment variable AWS_ACCOUNT_NUMBER on the build machine.


    AWS_ACCOUNT_NUMBER=999999999999

  3. AWS EC2 Service

    Sign up for the Amazon Elastic Compute Cloud (Amazon EC2) service if you haven't already done so.

  4. AWS S3 Service

    The AMI created in this guide will be instance store-backed (a.k.a. S3-backed) and requires you to be signed up for the Amazon Simple Storage Service (S3) service.

  5. AWS EBS Service

    If you intend to convert the instance store-backed AMI to an EBS-backed AMI then sign up for the Elastic Block Store (EBS) service.

  6. AWS Access Key ID and Secret Access Key

    The AWS Access Key ID and Secret Access Key serve the purpose of ID and Password to access Amazon S3. Navigate to Security Credentials, click on the Access Keys tab under Access Credentials to create or view your Access Key ID and Secret Access Key.

    The Access Key and Secret Key will be assigned to the environment variables AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY respectively on the build machine.


    AWS_ACCESS_KEY_ID=your_access_key_id AWS_SECRET_ACCESS_KEY=your_secret_access_key

  7. EC2 Private Key File and EC2 Certificate File

    If you have not already created an X.509 Certificate, you need to create or upload one from the AWS Management Console. Navigate to Security Credentials, click on the X.509 Certificates tab under Access Credentials, and click "Create a new Certificate".

    Important: After creating an X-509 Certificate, make sure to download the Private Key file before navigating away from the "X509 Certificate Created" window. AWS does not store your private key information and you will not be able to download the private key file at any other time. If you do not have access to your private key file, you will have to create a new certificate and private key.

    For the purpose of this guide, I will be renaming my private key file from pk-2L7LZYRTNEAC4KGZMPPZWAOZ4KYCTCA4.pem to ec2-pk.pem.

    The private key file name and path will be assigned to the environment variable EC2_PRIVATE_KEY on the build machine.


    EC2_PRIVATE_KEY=/opt/ec2/certificates/ec2-pk.pem

    Once you have a registered X.509 Certificate, it can always be downloaded from the AWS Management Console be navigating to Security Credentials and clicking on the X.509 Certificates tab under Access Credentials.

    As mentioned already, you will need to have the private key file associated with the X.509 certificate. Amazon does not store your private key information. If you do not have access to your private key file, you will have to create a new certificate and private key.

    For the purpose of this guide, I will be renaming my certificate file from cert-2L7LZYRTNEAC4KGZMPPZWAOZ4KYCTCA4.pem to ec2-cert.pem.

    The certificate file name and path will be assigned to the environment variable EC2_CERT on the build machine.


    EC2_CERT=/opt/ec2/certificates/ec2-cert.pem

  8. Build CentOS Machine

    Build a CentOS 6.2 machine that will be used to create a new image on. The CentOS install only needs to include the base packages through a Minimal package installation. In this guide, I am using a physical machine (not imaged) that will be used to build the new image on. The environment must have network access in order to download all necessary Linux packages using yum.

    The remaining prerequisites in this section deal with preparing a basic environment on the CentOS 6.2 build machine that will be used to create the new image.

  9. Set Environment Variables

    After creating the CentOS Build Machine, add the following environment variables to your shell login script (i.e. /root/.bashrc).


    # cp /root/.bashrc /root/.bashrc.backup # vi /root/.bashrc export EC2_HOME=/opt/ec2/tools export EC2_PRIVATE_KEY=/opt/ec2/certificates/ec2-pk.pem export EC2_CERT=/opt/ec2/certificates/ec2-cert.pem export EC2_URL=https://ec2.amazonaws.com export AWS_ACCOUNT_NUMBER=<999999999999> export AWS_ACCESS_KEY_ID=<your_access_key_id> export AWS_SECRET_ACCESS_KEY=<your_secret_access_key> export AWS_AMI_BUCKET=idevelopment-amis/x86-64/Linux/CentOS/6.2 export PATH=$PATH:/usr/local/bin:/usr/local/sbin:/usr/bin:/usr/sbin:/bin:/sbin:$EC2_HOME/bin export JAVA_HOME=/usr # source ~/.bashrc

  10. Install Required Linux Packages

    Install the following Linux packages required to build images.


    # yum -y install e2fsprogs ruby java-1.6.0-openjdk unzip MAKEDEV

  11. Copy EC2 Private Key and Certificate

    Copy your EC2 private key and X.509 certificate to the following directories on the build machine:


    # mkdir -p /opt/ec2/certificates # cp pk-2L7LZYRTNEAC4KGZMPPZWAOZ4KYCTCA4.pem /opt/ec2/certificates/ec2-pk.pem # cp cert-2L7LZYRTNEAC4KGZMPPZWAOZ4KYCTCA4.pem /opt/ec2/certificates/ec2-cert.pem

  12. Download the Amazon EC2 Tools

    Download the Amazon EC2 API Tools.


    # mkdir -p /opt/ec2/tools # curl -o /tmp/ec2-api-tools.zip http://s3.amazonaws.com/ec2-downloads/ec2-api-tools.zip # unzip /tmp/ec2-api-tools.zip -d /tmp # cp -r /tmp/ec2-api-tools-*/* /opt/ec2/tools

    Download the Amazon EC2 AMI Tools.


    # curl -o /tmp/ec2-ami-tools.zip http://s3.amazonaws.com/ec2-downloads/ec2-ami-tools.zip # unzip /tmp/ec2-ami-tools.zip -d /tmp # cp -rf /tmp/ec2-ami-tools-*/* /opt/ec2/tools

    The Amazon EC2 AMI Tools and EC2 API Tools are Java based, so verify the JAVA_HOME environment variable is set and confirm that Java is installed correctly.


    # echo $JAVA_HOME /usr # java -version java version "1.6.0_24" OpenJDK Runtime Environment (IcedTea6 1.11.3) (rhel-1.48.1.11.3.el6_2-x86_64) OpenJDK 64-Bit Server VM (build 20.0-b12, mixed mode)

  13. Get to Know Your Region and Availability Zone

    AWS infrastructure services like EC2, S3, EBS, etc. are hosted in multiple locations around the world which include the United States, South America, Europe, and Asia Pacific. These Regions are logically isolated from one another, so for example, you will not be able to access US East EC2 resources when communicating with a South America endpoint. When creating AWS services, users should choose a Region which optimizes latency, minimizes costs, or addresses particular regulatory requirements.

    Figure 1: AWS Regions

    Nearly every command or call you make to EC2 must target a specific Region. Amazon chooses a default Region based on the value of the environment variable EC2_URL (or the --url command-line flag) which you use to specify your Region's endpoint. For example:


    EC2_URL=https://ec2.amazonaws.com

    The default Region for the above endpoint is us-east-1 and will be the one I use throughout this guide based on my geographic location near the east coast.

    Most of the commands in the Amazon EC2 API and AMI Tools will allow users to override the default Region if a URL endpoint is specified by the EC2_URL environment variable or --url flag by using the --region command-line parameter. For example:


    <ec2-command> --region us-east-1

    When running the Amazon EC2 API and AMI Tools in this guide, I will not be manually specifying a Region using the --region command-line parameter and will allow the default Region be chosen based off the the endpoint specified by the EC2_URL environment variable.

    To obtain a list of regions you have access to, use:


    # ec2-describe-regions | sort REGION ap-northeast-1 ec2.ap-northeast-1.amazonaws.com REGION ap-southeast-1 ec2.ap-southeast-1.amazonaws.com REGION eu-west-1 ec2.eu-west-1.amazonaws.com REGION sa-east-1 ec2.sa-east-1.amazonaws.com REGION us-east-1 ec2.us-east-1.amazonaws.com REGION us-west-1 ec2.us-west-1.amazonaws.com REGION us-west-2 ec2.us-west-2.amazonaws.com

     

    If you receive the following error message running any of the EC2 tools, make certain that the date and time are set correctly on the build machine.

    # ec2-describe-regions
    Client.InvalidSecurity: Request has expired
          
    # date
    Thu Jun  7 02:53:27 EDT 2012
    
    /* The above date is 17 days off */
    
    # date -s "24 JUN 2012 13:36:00"
    Sun Jun 24 13:36:00 EDT 2012

    When you launch an EC2 instance, you can optionally specify an Availability Zone within your Region. If you do not specify an Availability Zone, Amazon EC2 selects one for you in the Region that you are using. When launching your initial instances, Amazon recommends accepting the default Availability Zone, which allows Amazon EC2 to select the best Availability Zone for you based on system health and available capacity.

    When launching EC2 instances in this guide, I will not manually specify the Availability Zone and will allow EC2 to select the recommended one. When converting the instance store-backed AMI to an EBS-backed AMI, it is important to know which Availability Zone your EC2 instance is running in when creating the EBS volume. When creating an EBS volume, you must specify an Availability Zone and that Availability Zone for the volume must be the same as the instance to which it attaches.

    To obtain a list of availability zones for your region, use:


    # ec2-describe-availability-zones --region us-east-1 AVAILABILITYZONE us-east-1a available us-east-1 AVAILABILITYZONE us-east-1b available us-east-1 AVAILABILITYZONE us-east-1c available us-east-1 AVAILABILITYZONE us-east-1d available us-east-1

    Click here to learn more about Amazon Regions and Availability Zones.

Build the AMI

With the prerequisites out of the way, it's time to start preparing the new instance store-backed AMI which involves building an operating system installation from scratch to a clean root file system. This will be performed on a stand-alone physical machine (known in this guide as the build machine) installed with CentOS 6.2 (64-bit). Later in this guide, we will convert the instance store-backed AMI to an EBS-backed AMI.

Prepare New Image

In this section we will perform the tasks necessary to prepare an Amazon EC2 instance store-backed AMI to an empty file system mounted by loopback. Creating the AMI through a loopback avoids having to create a new root disk partition and file system on a separate physical disk. The AMI will be created through a loopback which involves doing a full operating system installation on a clean root file system.

Start by creating a disk image with an empty ext4 file system mounted by loopback. The loopback module enables you to use a normal file as if it were a raw device. Think of it as a file system within a file. Mounting a file system image file through loopback presents it as part of the normal file system. You can then modify it using your favorite file management tools and utilities.

Make sure to create a disk image file large enough to host the operating system, tools, and applications that you will install. For example, a baseline Linux/UNIX installation requires about 700 MB, so your file should be at least 1 GB.

 

Amazon EC2 instance store-backed AMIs are limited to 10 GiB storage for the root device, whereas Amazon EBS-backed AMIs are limited to 1 TiB.

The following example creates an empty 10 GiB file system mounted by loopback.


# mkdir -p /opt/ec2/images # dd if=/dev/zero of=/opt/ec2/images/centos-6.2-x86_64-base.img bs=1M count=10240 # mkfs.ext4 -F -j /opt/ec2/images/centos-6.2-x86_64-base.img # mkdir -p /mnt/ec2-image # mount -o loop /opt/ec2/images/centos-6.2-x86_64-base.img /mnt/ec2-image/ # mount /dev/mapper/vg_awsnode-lv_root on / type ext4 (rw) proc on /proc type proc (rw) sysfs on /sys type sysfs (rw) devpts on /dev/pts type devpts (rw,gid=5,mode=620) tmpfs on /dev/shm type tmpfs (rw,rootcontext="system_u:object_r:tmpfs_t:s0") /dev/sda1 on /boot type ext4 (rw) none on /proc/sys/fs/binfmt_misc type binfmt_misc (rw) /opt/ec2/images/centos-6.2-x86_64-base.img on /mnt/ec2-image type ext4 (rw,loop=/dev/loop0)

Before installing the operating system, prepare the image by creating directories in the root file system to hold system files and devices.


# mkdir -p /mnt/ec2-image/{dev,etc,proc,sys} # mkdir -p /mnt/ec2-image/var/{cache,log,lock,lib/rpm}

Populate the /dev directory with a minimal set of devices. Ignore any MAKEDEV: mkdir: File exists warnings.


# /sbin/MAKEDEV -d /mnt/ec2-image/dev -x console # /sbin/MAKEDEV -d /mnt/ec2-image/dev -x null # /sbin/MAKEDEV -d /mnt/ec2-image/dev -x zero # /sbin/MAKEDEV -d /mnt/ec2-image/dev -x urandom

Mount dev, pts, shm, proc, and sys in the new root file system.


# mount -o bind /dev /mnt/ec2-image/dev # mount -o bind /dev/pts /mnt/ec2-image/dev/pts # mount -o bind /dev/shm /mnt/ec2-image/dev/shm # mount -o bind /proc /mnt/ec2-image/proc # mount -o bind /sys /mnt/ec2-image/sys # mount /dev/mapper/vg_awsnode-lv_root on / type ext4 (rw) proc on /proc type proc (rw) sysfs on /sys type sysfs (rw) devpts on /dev/pts type devpts (rw,gid=5,mode=620) tmpfs on /dev/shm type tmpfs (rw,rootcontext="system_u:object_r:tmpfs_t:s0") /dev/sda1 on /boot type ext4 (rw) none on /proc/sys/fs/binfmt_misc type binfmt_misc (rw) /opt/ec2/images/centos-6.2-x86_64-base.img on /mnt/ec2-image type ext4 (rw,loop=/dev/loop0) /dev on /mnt/ec2-image/dev type none (rw,bind) /dev/pts on /mnt/ec2-image/dev/pts type none (rw,bind) /dev/shm on /mnt/ec2-image/dev/shm type none (rw,bind) /proc on /mnt/ec2-image/proc type none (rw,bind) /sys on /mnt/ec2-image/sys type none (rw,bind)

Install the Operating System

With the system directories and device files in place, you are ready to install the CentOS operating system in the image file. Depending on the speed of the host and network link to the repository, this process might take a while.

Create a yum configuration file (e.g. yum-xen.conf) that you will use to install the base OS. The configuration file ensures that all the required basic packages and utilities are installed. You can locate this file anywhere on your main file system (not on your loopback file system) and is used only during installation.


# mkdir -p /opt/ec2/yum # vi /opt/ec2/yum/yum-xen.conf [base] name=CentOS-6.2 - Base mirrorlist=http://mirrorlist.centos.org/?release=6.2&arch=x86_64&repo=os #baseurl=http://mirror.centos.org/centos/6.2/os/x86_64/ gpgcheck=1 gpgkey=http://mirror.centos.org/centos/RPM-GPG-KEY-CentOS-6 [updates] name=CentOS-6.2 - Updates mirrorlist=http://mirrorlist.centos.org/?release=6.2&arch=x86_64&repo=updates #baseurl=http://mirror.centos.org/centos/6.2/updates/x86_64/ gpgcheck=1 gpgkey=http://mirror.centos.org/centos/RPM-GPG-KEY-CentOS-6 [extras] name=CentOS-6.2 - Extras mirrorlist=http://mirrorlist.centos.org/?release=6.2&arch=x86_64&repo=extras #baseurl=http://mirror.centos.org/centos/6.2/extras/x86_64/ gpgcheck=1 gpgkey=http://mirror.centos.org/centos/RPM-GPG-KEY-6 [centosplus] name=CentOS-6.2 - Plus mirrorlist=http://mirrorlist.centos.org/?release=6.2&arch=x86_64&repo=centosplus #baseurl=http://mirror.centos.org/centos/6.2/centosplus/x86_64/ gpgcheck=1 enabled=0 gpgkey=http://mirror.centos.org/centos/RPM-GPG-KEY-6 [contrib] name=CentOS-6.2 - Contrib mirrorlist=http://mirrorlist.centos.org/?release=6.2&arch=x86_64&repo=contrib #baseurl=http://mirror.centos.org/centos/6.2/contrib/x86_64/ gpgcheck=1 enabled=0 gpgkey=http://mirror.centos.org/centos/RPM-GPG-KEY-6

Install the Base package and supporting utilities to the image using the yum configuration file. We need e2fsprogs for the fsck.ext4 command. We install yum-plugin-fastestmirror.noarch so that yum tests for a faster repository mirror rather than connecting to one at random.


# yum -c /opt/ec2/yum/yum-xen.conf --installroot=/mnt/ec2-image -y groupinstall Base # yum -c /opt/ec2/yum/yum-xen.conf --installroot=/mnt/ec2-image -y install *openssh* dhclient # yum -c /opt/ec2/yum/yum-xen.conf --installroot=/mnt/ec2-image -y install grub # yum -c /opt/ec2/yum/yum-xen.conf --installroot=/mnt/ec2-image -y install e2fsprogs # yum -c /opt/ec2/yum/yum-xen.conf --installroot=/mnt/ec2-image -y install yum-plugin-fastestmirror.noarch # yum -c /opt/ec2/yum/yum-xen.conf --installroot=/mnt/ec2-image -y install selinux-policy selinux-policy-targeted

At this point you can add any other yum or rpm installs to the image.


# yum -c /opt/ec2/yum/yum-xen.conf --installroot=/mnt/ec2-image -y install vi

Configure the Operating System

After successfully installing the base operating system, the next step is to configure your networking, hard drives, and security settings to work in the Amazon EC2 environment.

Shell Login Script - (optional)

Create a shell login script on the new image for the root account.


# vi /mnt/ec2-image/root/.bashrc # .bashrc # User specific aliases and functions alias rm='rm -i' alias cp='cp -i' alias mv='mv -i' # Source global definitions if [ -f /etc/bashrc ]; then . /etc/bashrc fi ------------------------------------------------- # vi /mnt/ec2-image/root/.bash_profile # .bash_profile # Get the aliases and functions if [ -f ~/.bashrc ]; then . ~/.bashrc fi # User specific environment and startup programs PATH=$PATH:$HOME/bin export PATH

Network

Configure networking options for the image.


# vi /mnt/ec2-image/etc/sysconfig/network NETWORKING=yes HOSTNAME=localhost.localdomain


# vi /mnt/ec2-image/etc/sysconfig/network-scripts/ifcfg-eth0 DEVICE="eth0" NM_CONTROLLED="yes" ONBOOT=yes TYPE=Ethernet BOOTPROTO=dhcp DEFROUTE=yes PEERDNS=yes PEERROUTES=yes IPV4_FAILURE_FATAL=yes IPV6INIT=no

 

Note that the Amazon EC2 DHCP server ignores hostname requests. If you attempt to set DHCP_HOSTNAME, that hostname will be used for the local hostname and set on the instance but not externally. Additionally, the local hostname will be the same for all instances of the AMI, which might be confusing having all images with that same hostname.

Ensure that the network will be started on boot.


# /usr/sbin/chroot /mnt/ec2-image /sbin/chkconfig --level 2345 network on

SELinux

CentOS comes with SELinux set to enforcing by default; however, in some cases it doesn't get labelled correctly depending on the instance being created. It is best to assume that for the first start of the instance that it is not properly labelled. Run the following to ensure labelling is executed on the first start of the instance.


# touch /mnt/ec2-image/.autorelabel

If you decide not to enable SELinux, this can be set by modifying the file /mnt/ec2-image/etc/sysconfig/selinux as follows:


SELINUX=disabled

Disks

Create an fstab file and any optional mount points for the image that supports the target Amazon EC2 instance type. Although the example fstab file demonstrated in this guide will allow an image to successfully boot from the volume attached for the root device for all Amazon EC2 instance types, it will only correctly mount the local instance storage (a.k.a. ephemeral storage) for the following instance types:

The following example fstab file will mount a single instance storage volume to /mnt/vol1 as well as swap (/dev/xvde3). For a Small instance type, the local instance storage is 160 GiB while for a Medium instance type, the local instance storage is 410 GiB. The amount of local instance storage included with all other Amazon EC2 instance types as well as several Amazon EC2 Instance Storage Usage Scenarios can be found at Amazon's AWS documentation website.

First, create a mount point for each local instance storage volume necessary for your configuration according to EC2 instance type. This example will only include a single instance storage volume which is suitable for Small and Medium size instance types.


# mkdir -p /mnt/ec2-image/mnt/vol1

Next, create the fstab file for the image. Since Amazon uses Xen drivers, the first drive starts at /dev/xvde.

Small and Medium Instance Types


# vi /mnt/ec2-image/etc/fstab /dev/xvde1 / ext4 defaults 1 1 none /dev/pts devpts gid=5,mode=620 0 0 none /dev/shm tmpfs defaults 0 0 none /proc proc defaults 0 0 none /sys sysfs defaults 0 0 /dev/xvde2 /mnt/vol1 ext4 defaults 0 0 /dev/xvde3 swap swap defaults 0 0

You can learn more about block device mapping at Amazon's AWS documentation website.

Create a grub configuration file for the image and boot settings so the Amazon Kernel Image (AKI) can boot into the new kernel.


# vi /mnt/ec2-image/boot/grub/grub.conf default=0 timeout=0 title CentOS 6.2 (Custom AMI) root (hd0) kernel /boot/vmlinuz ro root=/dev/xvde1 rd_NO_PLYMOUTH initrd /boot/initramfs # unalias ls # ln -s /boot/grub/grub.conf /mnt/ec2-image/boot/grub/menu.lst # kern=`ls /mnt/ec2-image/boot/vmlin*|awk -F/ '{print $NF}'` # ird=`ls /mnt/ec2-image/boot/initramfs*.img|awk -F/ '{print $NF}'` # sed -ie "s/vmlinuz/$kern/" /mnt/ec2-image/boot/grub/grub.conf # sed -ie "s/initramfs/$ird/" /mnt/ec2-image/boot/grub/grub.conf # cat /mnt/ec2-image/boot/grub/grub.conf default=0 timeout=0 title CentOS 6.2 (Custom AMI) root (hd0) kernel /boot/vmlinuz-2.6.32-220.23.1.el6.x86_64 ro root=/dev/xvde1 rd_NO_PLYMOUTH initrd /boot/initramfs-2.6.32-220.23.1.el6.x86_64.img

Security

Add the following entries to the image's sshd_config file in order to allow root login without password. This is helpful since I intend to use a private key to log in to the instance.


# vi /mnt/ec2-image/etc/ssh/sshd_config ... UseDNS no PermitRootLogin without-password ...

Create a script that captures the public key credentials for your root login from instance metadata. In this example, public key 0 (in the OpenSSH key format) is fetched from instance metadata using HTTP and written to /root/.ssh/authorized_keys in order to allow root to log in without a password using his private key.


# vi /mnt/ec2-image/etc/init.d/ec2-get-ssh #!/bin/bash # chkconfig: 2345 95 20 # processname: ec2-get-ssh # description: Capture AWS public key credentials for EC2 user # Source function library . /etc/rc.d/init.d/functions # Source networking configuration [ -r /etc/sysconfig/network ] && . /etc/sysconfig/network # Replace the following environment variables for your system export PATH=:/usr/local/bin:/usr/local/sbin:/usr/bin:/usr/sbin:/bin:/sbin # Check that networking is configured if [ "${NETWORKING}" = "no" ]; then echo "Networking is not configured." exit 1 fi start() { if [ ! -d /root/.ssh ]; then mkdir -p /root/.ssh chmod 700 /root/.ssh fi # Retrieve public key from metadata server using HTTP curl -f http://169.254.169.254/latest/meta-data/public-keys/0/openssh-key > /tmp/my-public-key if [ $? -eq 0 ]; then echo "EC2: Retrieve public key from metadata server using HTTP." cat /tmp/my-public-key >> /root/.ssh/authorized_keys chmod 600 /root/.ssh/authorized_keys rm /tmp/my-public-key fi } stop() { echo "Nothing to do here" } restart() { stop start } # See how we were called. case "$1" in start) start ;; stop) stop ;; restart) restart ;; *) echo $"Usage: $0 {start|stop|restart}" exit 1 esac exit $?

Update the runlevel information for the new system service on the image.


# /bin/chmod +x /mnt/ec2-image/etc/init.d/ec2-get-ssh # /usr/sbin/chroot /mnt/ec2-image /sbin/chkconfig --level 34 ec2-get-ssh on

Clean Up

Clean up the image.


# yum -c /opt/ec2/yum/yum-xen.conf --installroot=/mnt/ec2-image -y clean packages # rm -rf /mnt/ec2-image/root/.bash_history # rm -rf /mnt/ec2-image/var/cache/yum # rm -rf /mnt/ec2-image/var/lib/yum # sync; sync; sync; sync # umount /mnt/ec2-image/dev/shm # umount /mnt/ec2-image/dev/pts # umount /mnt/ec2-image/dev # umount /mnt/ec2-image/sys # umount /mnt/ec2-image/proc # umount /mnt/ec2-image # mount /dev/mapper/vg_awsnode-lv_root on / type ext4 (rw) proc on /proc type proc (rw) sysfs on /sys type sysfs (rw) devpts on /dev/pts type devpts (rw,gid=5,mode=620) tmpfs on /dev/shm type tmpfs (rw,rootcontext="system_u:object_r:tmpfs_t:s0") /dev/sda1 on /boot type ext4 (rw) none on /proc/sys/fs/binfmt_misc type binfmt_misc (rw)

Bundle the AMI

Now that the image is created, it must be bundled using ec2-bundle-image which compresses, encrypts, and then spits it to prepare for upload to S3.

Understanding the Amazon Kernel Image (AKI)

When we go to register the AMI with Amazon EC2 later in this guide, we must set the default kernel as one which supports the GRUB boot loader. To enable user-provided kernels, Amazon has published Amazon Kernel Images (AKIs) that use a system called PV-GRUB. PV-GRUB is a paravirtual "mini-OS" that runs a version of GNU GRUB, the standard Linux boot loader. PV-GRUB selects the kernel to boot by reading /boot/grub/menu.lst from your image which we configured earlier in this guide. It will load the kernel specified by your image (the CentOS 6.2 kernel) and then shut down the "mini-OS", so that it no longer consumes any resources. One of the advantages of this solution is that PV-GRUB understands standard grub.conf or menu.lst commands, which allows it to work with most existing Linux distributions. I'm going to bundle a default AKI with the image so that it is included in the manifest. I will then specify the AKI again when registering the AMI with Amazon EC2 later in this guide.

 

Some documentation on AWS refers to AKI as the Kernel ID.

Several PV-GRUB AKIs are available depending on the type of the instance and the Region where it is located. There are AKIs for 32-bit and 64-bit architecture types, with each having one AKI for partitioned images and another AKI for partitionless images. You must choose an AKI with "hd0" in the name if you want a raw or unpartitioned disk image (most images). Choose an AKI with "hd00" in the name if you want an image that has a partition table.

Most vendors, such as Fedora, Red Hat, Ubuntu, and Novell, use unpartitioned disk images (a.k.a. partitionless images). This means that they use the hd0 variants of PV-GRUB; almost without exception most users will want to use the hd0 variants. The AMI created in this guide is an unpartitioned disk image and therefore will use an hd0 AKI.

 

You cannot use the 64-bit version of PV-GRUB to start a 32-bit kernel or vice versa.

 

You must not specify an Amazon ramdisk image (ARI) when using a PV-GRUB AKI.

Use the ec2-describe-images command (which is part of the EC2 AMI Tools) to check for the latest available published AKI for your instance type, disk layout, architecture, and Region.


# ec2-describe-images \ --owner amazon \ --region us-east-1 | grep "amazon\/pv-grub-hd0" | awk '{ print $1, $2, $3, $5, $7 }' IMAGE aki-b2aa75db amazon/pv-grub-hd00_1.03-i386.gz available i386 IMAGE aki-b4aa75dd amazon/pv-grub-hd00_1.03-x86_64.gz available x86_64 IMAGE aki-b6aa75df amazon/pv-grub-hd0_1.03-i386.gz available i386 IMAGE aki-88aa75e1 amazon/pv-grub-hd0_1.03-x86_64.gz available x86_64

From the above list, I am looking for a published AKI for an "unpartitioned disk image", "64-bit" architecture, in the "us-east-1" Region so I chose aki-88aa75e1.

 

If you receive the following error message running any of the EC2 tools, make certain that the date and time are set correctly on the build machine.

# ec2-describe-regions
Client.InvalidSecurity: Request has expired

# date
Thu Jun  7 02:53:27 EDT 2012

/* The above date is 17 days off */

# date -s "24 JUN 2012 13:36:00"
Sun Jun 24 13:36:00 EDT 2012

 

A full list of User Provided Kernels can also be found at Amazon's AWS documentation website.

Bundle the AMI

We now have the information needed to bundle the image on the build machine to prepare for upload to S3.

Use ec2-bundle-image which is part of the EC2 AMI Tools.


# ec2-bundle-image \ --cert $EC2_CERT \ --privatekey $EC2_PRIVATE_KEY \ --image /opt/ec2/images/centos-6.2-x86_64-base.img \ --prefix centos-6.2-x86_64-base \ --user $AWS_ACCOUNT_NUMBER \ --destination /opt/ec2/images \ --arch x86_64 \ --kernel aki-88aa75e1 Bundling image file... Splitting /opt/ec2/images/centos-6.2-x86_64-base.tar.gz.enc... Created centos-6.2-x86_64-base.part.00 Created centos-6.2-x86_64-base.part.01 Created centos-6.2-x86_64-base.part.02 Created centos-6.2-x86_64-base.part.03 Created centos-6.2-x86_64-base.part.04 Created centos-6.2-x86_64-base.part.05 Created centos-6.2-x86_64-base.part.06 Created centos-6.2-x86_64-base.part.07 Created centos-6.2-x86_64-base.part.08 Created centos-6.2-x86_64-base.part.09 Created centos-6.2-x86_64-base.part.10 Created centos-6.2-x86_64-base.part.11 Created centos-6.2-x86_64-base.part.12 Created centos-6.2-x86_64-base.part.13 Created centos-6.2-x86_64-base.part.14 Created centos-6.2-x86_64-base.part.15 Created centos-6.2-x86_64-base.part.16 Created centos-6.2-x86_64-base.part.17 Created centos-6.2-x86_64-base.part.18 Created centos-6.2-x86_64-base.part.19 Created centos-6.2-x86_64-base.part.20 Created centos-6.2-x86_64-base.part.21 Created centos-6.2-x86_64-base.part.22 Created centos-6.2-x86_64-base.part.23 Created centos-6.2-x86_64-base.part.24 Created centos-6.2-x86_64-base.part.25 Created centos-6.2-x86_64-base.part.26 Created centos-6.2-x86_64-base.part.27 Created centos-6.2-x86_64-base.part.28 Created centos-6.2-x86_64-base.part.29 Created centos-6.2-x86_64-base.part.30 Created centos-6.2-x86_64-base.part.31 Created centos-6.2-x86_64-base.part.32 Created centos-6.2-x86_64-base.part.33 Created centos-6.2-x86_64-base.part.34 Created centos-6.2-x86_64-base.part.35 Created centos-6.2-x86_64-base.part.36 Created centos-6.2-x86_64-base.part.37 Created centos-6.2-x86_64-base.part.38 Created centos-6.2-x86_64-base.part.39 Created centos-6.2-x86_64-base.part.40 Created centos-6.2-x86_64-base.part.41 Created centos-6.2-x86_64-base.part.42 Created centos-6.2-x86_64-base.part.43 Generating digests for each part... Digests generated. Creating bundle manifest... ec2-bundle-image complete.

Upload the AMI to Amazon S3

The bundled AMI now needs to be uploaded to Amazon S3 before Amazon EC2 can access it. Use ec2-upload-bundle which is part of the EC2 AMI Tools.

The bundled AMI will be uploaded to Amazon S3 in a bucket specified using the --bucket parameter (i.e. idevelopment-amis/x86-64/Linux/CentOS/6.2). Amazon S3 stores data objects in buckets, which are similar to directories. Buckets must have globally unique names. The ec2-upload-bundle utility uploads the bundled AMI to the specified bucket. If the specified bucket does not exist, it is created. If the specified bucket exists and belongs to another AWS account, the ec2-upload-bundle command will fail.

Use the --manifest parameter to specify the full path to the manifest file created in the previous section. Note that the manifest file must be in a bucket in the same region where the AMI is to be created. The AMI manifest file and all image parts will be uploaded to Amazon S3. The manifest file is encrypted with the Amazon EC2 public key before being uploaded.

Include your AWS Access Key and your AWS Secret Key for S3 authentication using the --access-key and --secret-key parameters respectively.


# ec2-upload-bundle \ --manifest /opt/ec2/images/centos-6.2-x86_64-base.manifest.xml \ --bucket $AWS_AMI_BUCKET \ --access-key $AWS_ACCESS_KEY_ID \ --secret-key $AWS_SECRET_ACCESS_KEY Uploading bundled image parts to the S3 bucket idevelopment-amis ... Uploaded centos-6.2-x86_64-base.part.00 Uploaded centos-6.2-x86_64-base.part.01 Uploaded centos-6.2-x86_64-base.part.02 Uploaded centos-6.2-x86_64-base.part.03 Uploaded centos-6.2-x86_64-base.part.04 Uploaded centos-6.2-x86_64-base.part.05 Uploaded centos-6.2-x86_64-base.part.06 Uploaded centos-6.2-x86_64-base.part.07 Uploaded centos-6.2-x86_64-base.part.08 Uploaded centos-6.2-x86_64-base.part.09 Uploaded centos-6.2-x86_64-base.part.10 Uploaded centos-6.2-x86_64-base.part.11 Uploaded centos-6.2-x86_64-base.part.12 Uploaded centos-6.2-x86_64-base.part.13 Uploaded centos-6.2-x86_64-base.part.14 Uploaded centos-6.2-x86_64-base.part.15 Uploaded centos-6.2-x86_64-base.part.16 Uploaded centos-6.2-x86_64-base.part.17 Uploaded centos-6.2-x86_64-base.part.18 Uploaded centos-6.2-x86_64-base.part.19 Uploaded centos-6.2-x86_64-base.part.20 Uploaded centos-6.2-x86_64-base.part.21 Uploaded centos-6.2-x86_64-base.part.22 Uploaded centos-6.2-x86_64-base.part.23 Uploaded centos-6.2-x86_64-base.part.24 Uploaded centos-6.2-x86_64-base.part.25 Uploaded centos-6.2-x86_64-base.part.26 Uploaded centos-6.2-x86_64-base.part.27 Uploaded centos-6.2-x86_64-base.part.28 Uploaded centos-6.2-x86_64-base.part.29 Uploaded centos-6.2-x86_64-base.part.30 Uploaded centos-6.2-x86_64-base.part.31 Uploaded centos-6.2-x86_64-base.part.32 Uploaded centos-6.2-x86_64-base.part.33 Uploaded centos-6.2-x86_64-base.part.34 Uploaded centos-6.2-x86_64-base.part.35 Uploaded centos-6.2-x86_64-base.part.36 Uploaded centos-6.2-x86_64-base.part.37 Uploaded centos-6.2-x86_64-base.part.38 Uploaded centos-6.2-x86_64-base.part.39 Uploaded centos-6.2-x86_64-base.part.40 Uploaded centos-6.2-x86_64-base.part.41 Uploaded centos-6.2-x86_64-base.part.42 Uploaded centos-6.2-x86_64-base.part.43 Uploading manifest ... Uploaded manifest. Bundle upload completed.

Register the AMI

The uploaded image now needs to be registered so that Amazon EC2 can locate it and run instances based on it.

Make certain to include the same AKI ID when using the ec2-register command that was used when bundling the image.

 

If you make any changes to the source image stored in Amazon S3, you must re-register the image.

 

Like ec2-bundle-image, the ec2-register command below requires you to authenticate to AWS using your EC2 Private Key File and EC2 Certificate File.

ec2-bundle-image is part of the EC2 AMI Tools where it is mandatory to specify the --privatekey and --cert parameters on the command-line. ec2-register, on the other hand, is part of the EC2 API Tools where it is possible to assign the private key file and certificate file to the EC2_PRIVATE_KEY and EC2_CERT environment variables respectively and therefore we do not need to be specify the --privatekey and --cert parameters on the command-line. These two environment variables are included in the shell login script and should be set for the current session.


# ec2-register \ $AWS_AMI_BUCKET/centos-6.2-x86_64-base.manifest.xml \ --name "CentOS 6.2 (x86_64)" \ --description "CentOS 6.2 (x86_64) Base AMI" \ --architecture x86_64 \ --kernel aki-88aa75e1 IMAGE ami-6418ba0d

The ec2-register command returns an AMI Identifier (AMI ID), the value next to the IMAGE tag (ami-6418ba0d in the example). An AMI ID is a unique identifier for an individual image which is assigned by EC2 and used to run and manage instances.

Launch the Instance

You can now launch an instance of the new AMI using ec2-run-instances and specifying the image identifier (AMI ID) you received when you registered the image in the previous step.

Create Key Pair

Before launching an instance (and connecting to it), you will need an RSA Key Pair. Note that this is the not the same as your X.509 private key and certificate used earlier when authenticating to AWS. If you haven't already created a Key Pair for Amazon EC2, use ec2-create-keypair from the build machine. The public key will get stored by Amazon EC2 and the private key will be displayed on the console. Remember that Amazon does not store your private key information. If you loose your private key, you will not be able to access any EC2 instance for which its public key partner was used.

The following creates a new Key Pair named idevelopment-ec2-key. Note that the private key displayed below is not my actual private key. It is nothing more than a set of random characters and is only being shown to display example output. It is not valid. However, if you have a lot of time on your hands, give it a try!


# ec2-create-keypair idevelopment-ec2-key KEYPAIR idevelopment-ec2-key 02:68:4a:86:41:c9:56:8a:a6:07:d1:2c:67:61:85:a8:05:18:09:5f -----BEGIN RSA PRIVATE KEY----- MIIEpAIBAAKCAQEA0DJ9RpBvstjiBqtlQhrGC8KP74ZDuUliNIbQN/6G+pankPwyYxTv4mmbzNZX sYa6B5PWQaHDMtWBgGh03UJ2pLFUfg/na9E6KBwheRH6tZTIrKRzKEC4A+a9pSZ3G9wq2VCLaMDd Tk75gAeWVH0z1MSJA+xXH71avn7y9P1omReMpDcMLnqJjC7i0DOiodOpn7nVpt4Omss08igsIsBl X8tcF26Nk8m/ndPMIkAXefy895DlpzDCNJzk3p78y0G0G4cQp57E9EmK4tnbh9/KFdgA8W4KpVQk MA+H8DlxUnR9ogSDY5YnqhKI2SNHQ4tepuEJPLgk33/LKMsrY8NiqwIDA5ABAoIBAQDBAX1XdEDI iP0H43qMjKt6bXTn9Q1FcL1EupG+xwu0tQNEijff2KIFwCe5/h6vqZk9uxSOJXsC3iE+vBdSovG/ /dC5SN8d+Ymzc1iolz1XJkXenllPQKgWncrDlmQnBLUIXv3OL7U5hs0YXnBDhQoOvd/TLj8aVm/Z V3C+I0nbWOduro9G6ZwtLo0SjVfmvqVv7J/0qJ/okEtZLACLONImRTKraByWKxSg50/ah5OKrYbE aobqEwXcLijYDpw2RIkzmlVLWMRI7oV1klA7xKI0WzmscC5Pk/MfquEWPVJPrle2NkjpfPmPWBK/ iBMsKAo0lFSaBEn06YMzBLHBwr8pAoGBAPdPC3XrBAaHs72cnwOb0u9yAJ1uzGypEehPnlGwbs+/ XzzBv1PXJof6fR7QUw7E9KAikXatYnZ2YXp/rNa/j5Ftoyqw0ufAbic0Imiy2mFXBvd0CjhyvUq1 qlrq9zMPO2sW0LHVhcvAtnWpjQWacx3ycWxgJd2GoeRSwkkQ55mvAoGBANeDkioDU9iDY4Sx5Cfy CmYIP0GTilr4A5HGT5RsnvxYtGQQrKAwwjN8180NvBRfqKNSyVdvk9YGkD2EBih4E3kQBymoWLCa NzYkGe0t1sv7IPyUoyDA7NJgca2zVu0+Gxp0Y8cbJlzLio86lMH3DPrJmDtEle+ucJLxwsyp4pHF AoGAJtGae+zi2d+NIBMohpl79mZXoF6W6etz6yIsOKhYDHX+jfvfe1qHKsrvvtl2YBgcSdKDQLyg evXEAUVvLNouv+YUuYu93zC+WZr+V67TRnoxlr/5Jrkj60hW03yURsh6TCntoixg7wWV3kw53YpL zYSTty40Y1xRpxQyeOOVZvMCgYEAvrCIL+xpz/XRpKPSgRXE5ar+bYOitAMnBWE3INlLg3TpiHDI jaUW8H/SYnl4pX5OtNSXmmn7oTCFiCi9WimE8WWhMNboQrrxh1JCLn6UxEAcW+EYxeounWdAY9V/ YMPHL400jAQV2Z2vqDPpVAbI0U53CFoTYNQ0f5CzEGxxMRECgYBYT3pqHTLgQGGkpaV5yWnenoo4 hxQlA8iRO/vGUgMdnkWs3XlG1RwXMpi45RNGIDETbJxTyt/eI6Za86w7m7UbYhKN/LYYOi0VCz7W nlrbGgfkaiYePjFltwA3i5zaMvtonD5r2HX7ge4g2HdRQAHeCeAgHmtQezxUr09UJwnmqW== -----END RSA PRIVATE KEY-----

The private key is returned to the console as an unencrypted PEM encoded PKCS#8 private key. Save the private key to any machine you will be using to connect SSH to the instance from. Copy and paste the contents of the private key from -----BEGIN RSA PRIVATE KEY----- to -----END RSA PRIVATE KEY----- including those two lines and save it to a keypair file. For example:


# vi ~/.ssh/idevelopment-ec2-key.pem -----BEGIN RSA PRIVATE KEY----- MIIEpAIBAAKCAQEA0DJ9RpBvstjiBqtlQhrGC8KP74ZDuUliNIbQN/6G+pankPwyYxTv4mmbzNZX sYa6B5PWQaHDMtWBgGh03UJ2pLFUfg/na9E6KBwheRH6tZTIrKRzKEC4A+a9pSZ3G9wq2VCLaMDd Tk75gAeWVH0z1MSJA+xXH71avn7y9P1omReMpDcMLnqJjC7i0DOiodOpn7nVpt4Omss08igsIsBl X8tcF26Nk8m/ndPMIkAXefy895DlpzDCNJzk3p78y0G0G4cQp57E9EmK4tnbh9/KFdgA8W4KpVQk MA+H8DlxUnR9ogSDY5YnqhKI2SNHQ4tepuEJPLgk33/LKMsrY8NiqwIDA5ABAoIBAQDBAX1XdEDI iP0H43qMjKt6bXTn9Q1FcL1EupG+xwu0tQNEijff2KIFwCe5/h6vqZk9uxSOJXsC3iE+vBdSovG/ /dC5SN8d+Ymzc1iolz1XJkXenllPQKgWncrDlmQnBLUIXv3OL7U5hs0YXnBDhQoOvd/TLj8aVm/Z V3C+I0nbWOduro9G6ZwtLo0SjVfmvqVv7J/0qJ/okEtZLACLONImRTKraByWKxSg50/ah5OKrYbE aobqEwXcLijYDpw2RIkzmlVLWMRI7oV1klA7xKI0WzmscC5Pk/MfquEWPVJPrle2NkjpfPmPWBK/ iBMsKAo0lFSaBEn06YMzBLHBwr8pAoGBAPdPC3XrBAaHs72cnwOb0u9yAJ1uzGypEehPnlGwbs+/ XzzBv1PXJof6fR7QUw7E9KAikXatYnZ2YXp/rNa/j5Ftoyqw0ufAbic0Imiy2mFXBvd0CjhyvUq1 qlrq9zMPO2sW0LHVhcvAtnWpjQWacx3ycWxgJd2GoeRSwkkQ55mvAoGBANeDkioDU9iDY4Sx5Cfy CmYIP0GTilr4A5HGT5RsnvxYtGQQrKAwwjN8180NvBRfqKNSyVdvk9YGkD2EBih4E3kQBymoWLCa NzYkGe0t1sv7IPyUoyDA7NJgca2zVu0+Gxp0Y8cbJlzLio86lMH3DPrJmDtEle+ucJLxwsyp4pHF AoGAJtGae+zi2d+NIBMohpl79mZXoF6W6etz6yIsOKhYDHX+jfvfe1qHKsrvvtl2YBgcSdKDQLyg evXEAUVvLNouv+YUuYu93zC+WZr+V67TRnoxlr/5Jrkj60hW03yURsh6TCntoixg7wWV3kw53YpL zYSTty40Y1xRpxQyeOOVZvMCgYEAvrCIL+xpz/XRpKPSgRXE5ar+bYOitAMnBWE3INlLg3TpiHDI jaUW8H/SYnl4pX5OtNSXmmn7oTCFiCi9WimE8WWhMNboQrrxh1JCLn6UxEAcW+EYxeounWdAY9V/ YMPHL400jAQV2Z2vqDPpVAbI0U53CFoTYNQ0f5CzEGxxMRECgYBYT3pqHTLgQGGkpaV5yWnenoo4 hxQlA8iRO/vGUgMdnkWs3XlG1RwXMpi45RNGIDETbJxTyt/eI6Za86w7m7UbYhKN/LYYOi0VCz7W nlrbGgfkaiYePjFltwA3i5zaMvtonD5r2HX7ge4g2HdRQAHeCeAgHmtQezxUr09UJwnmqW== -----END RSA PRIVATE KEY----- # chmod 600 ~/.ssh/idevelopment-ec2-key.pem

Create Security Group

Create an AWS Security Group (if one doesn't exist) that will be used by the instance to authorize access via SSH (port 22) from the public internet 0.0.0.0/0 along with any optional services like ICMP (ping), HTTP (port 80), or HTTPS (port 443). You could further restrict access to these ports by specifying specific machines or networks (24.144.233.0/24 for example) that can access the instance instead of the public internet. At a minimum, authorize access for SSH.


# ec2addgrp idevelopment-security-group --description "iDevelopment Security Group" GROUP sg-e5ff738d idevelopment-security-group iDevelopment Security Group /* Optional */ # ec2auth idevelopment-security-group --protocol tcp --port-range 22 --cidr 0.0.0.0/0 GROUP idevelopment-security-group PERMISSION idevelopment-security-group ALLOWS tcp 22 22 FROM CIDR 0.0.0.0/0 ingress # ec2auth idevelopment-security-group --protocol icmp --icmp-type-code -1:-1 --cidr 0.0.0.0/0 GROUP idevelopment-security-group PERMISSION idevelopment-security-group ALLOWS icmp -1 -1 FROM CIDR 0.0.0.0/0 ingress # ec2auth idevelopment-security-group --protocol tcp --port-range 80 --cidr 0.0.0.0/0 GROUP idevelopment-security-group PERMISSION idevelopment-security-group ALLOWS tcp 80 80 FROM CIDR 0.0.0.0/0 ingress # ec2auth idevelopment-security-group --protocol tcp --port-range 443 --cidr 0.0.0.0/0 GROUP idevelopment-security-group PERMISSION idevelopment-security-group ALLOWS tcp 443 443 FROM CIDR 0.0.0.0/0 ingress

Launch the Instance

Execute the ec2-run-instances command with the image identifier (AMI ID) that was returned by ec2-register in the previous section along with the instance type, key pair, and security group to launch the instance.

Select the Amazon EC2 instance type compatible with your configuration.


# ec2-run-instances ami-6418ba0d \ --instance-type m1.small \ --key idevelopment-ec2-key \ --group idevelopment-security-group RESERVATION r-cc53cba9 395393972166 idevelopment-security-group INSTANCE i-e0c4f599 ami-6418ba0d pending idevelopment-ec2-key 0 m1.small 2012-06-24T22:23:37+0000 us-east-1a aki-88aa75e1 monitoring-disabled

This will start a single instance (instance store-backed) based on your newly created AMI and provide you with an instance identifier, the value immediately to the right of the INSTANCE tag. The instance identifier can be used to monitor the status of the running instance and to confirm when the instance is available for access.

Check the status of the instance with ec2-describe-instances using the instance identifier.


# ec2-describe-instances i-e0c4f599 RESERVATION r-cc53cba9 395393972166 idevelopment-security-group INSTANCE i-e0c4f599 ami-6418ba0d pending idevelopment-ec2-key 0 m1.small 2012-06-24T22:23:37+0000 us-east-1a aki-88aa75e1 monitoring-disabled

It will take a while to start the instance, so don't panic when you can't ping or log into the new instance immediately; even after the status shows running. This is especially true if you decided to use SELinux as shown in this guide. The re-labelling will take approximately 2-3 minutes to finish and that happens long before networking and SSH are started.

Keep checking the instance description until the status returns running.


# ec2-describe-instances i-e0c4f599 RESERVATION r-cc53cba9 395393972166 idevelopment-security-group INSTANCE i-e0c4f599 ami-6418ba0d ec2-23-20-121-112.compute-1.amazonaws.com ip-10-28-203-15.ec2.internal running idevelopment-ec2-key 0 m1.small 2012-06-24T22:23:37+0000 us-east-1a aki-88aa75e1 monitoring-disabled 23.20.121.112 10.28.203.15 instance-store paravirtual xen sg-e5ff738d default

Finally, connect to the instance using an SSH client and the private key.


# ping -c 3 ec2-23-20-121-112.compute-1.amazonaws.com PING ec2-23-20-121-112.compute-1.amazonaws.com (23.20.121.112) 56(84) bytes of data. 64 bytes from ec2-23-20-121-112.compute-1.amazonaws.com (23.20.121.112): icmp_seq=1 ttl=47 time=25.9 ms 64 bytes from ec2-23-20-121-112.compute-1.amazonaws.com (23.20.121.112): icmp_seq=2 ttl=47 time=29.0 ms 64 bytes from ec2-23-20-121-112.compute-1.amazonaws.com (23.20.121.112): icmp_seq=3 ttl=47 time=27.4 ms --- ec2-23-20-121-112.compute-1.amazonaws.com ping statistics --- 3 packets transmitted, 3 received, 0% packet loss, time 1999ms rtt min/avg/max/mdev = 25.961/27.487/29.069/1.276 ms # ssh -i ~/.ssh/idevelopment-ec2-key.pem root@ec2-23-20-121-112.compute-1.amazonaws.com The authenticity of host 'ec2-23-20-121-112.compute-1.amazonaws.com (23.20.121.112)' can't be established. RSA key fingerprint is c6:60:43:5a:36:37:c5:c3:77:2b:01:fc:de:7a:cf:ef. Are you sure you want to continue connecting (yes/no)? yes Warning: Permanently added 'ec2-23-20-121-112.compute-1.amazonaws.com,23.20.121.112' (RSA) to the list of known hosts. -bash-4.1# -bash-4.1# hostname ip-10-28-203-15 -bash-4.1# uname -r 2.6.32-220.23.1.el6.x86_64

Verify local instance storage (ephemeral storage) and swap.


-bash-4.1# mount /dev/xvde1 on / type ext4 (rw) none on /proc type proc (rw) none on /sys type sysfs (rw) none on /dev/pts type devpts (rw,gid=5,mode=620) none on /dev/shm type tmpfs (rw) /dev/xvde2 on /mnt/vol1 type ext4 (rw) -bash-4.1# df -k Filesystem 1K-blocks Used Available Use% Mounted on /dev/xvde1 10321208 860480 8936440 9% / none 847740 0 847740 0% /dev/shm /dev/xvde2 153899044 192068 145889352 1% /mnt/vol1 -bash-4.1# cat /proc/meminfo | grep SwapTotal SwapTotal: 917496 kB

 

If during the disk image configuration you elected not to create a mount point directory and forgo configuring local instance storage and swap in the image's /etc/fstab file, you can still manually mount both devices for the current instance.

-bash-4.1# /bin/egrep '[xvsh]d[a-z].*$' /proc/partitions
 202       65   10485760 xvde1
 202       66  156352512 xvde2
 202       67     917504 xvde3

-bash-4.1# mount -t ext4 /dev/xvde2 /mnt

-bash-4.1# swapon /dev/xvde3

-bash-4.1# mount
/dev/xvde1 on / type ext4 (rw)
none on /proc type proc (rw)
none on /sys type sysfs (rw)
none on /dev/pts type devpts (rw,gid=5,mode=620)
none on /dev/shm type tmpfs (rw)
none on /proc/sys/fs/binfmt_misc type binfmt_misc (rw)
/dev/xvde2 on /mnt type ext4 (rw)

-bash-4.1# cat /proc/meminfo | grep SwapTotal
SwapTotal:        917496 kB

Terminate the Instance

When you are done using the instance, it can be shutdown. Given this is an instance store-backed AMI, we can only terminate the instance which discards any changes made.


# ec2-terminate-instances i-e0c4f599 INSTANCE i-e0c4f599 running shutting-down

Terminated instances will remain visible in the AWS Console after termination for approximately one hour.

Congratulations!

You have successfully built and deployed your very own custom AMI for CentOS 6.2, and launched an instance based on it. This custom AMI is private to your account. You can build as many custom AMIs as required and use them to launch as many instances as you need.

In the next section, I will provide instructions for converting the current instance store-backed AMI to an EBS-backed AMI.

Convert Instance Store-Backed AMI to an EBS-Backed AMI

As of yet, there is no simple API or button in the AWS Management Console that allows users to convert an existing Amazon EC2 instance store-backed AMI to an Amazon EBS-backed AMI. The process, however, is not terribly difficult and will be fully explained in this section.

The process for converting a Linux/UNIX Amazon EC2 instance store-backed AMI to an EBS-Backed AMI can be broken down into the following steps.

  1. Launch the instance store-backed AMI.
  2. Create a new EBS volume and attach it to the instance.
  3. Mount the EBS volume and copy the AMI's root device and dev file system information to it.
  4. Unmount and detach the EBS volume.
  5. Create a snapshot of the EBS volume.
  6. Register a new image with a block device mapping that maps the root device name to the previously created snapshot.

Now lets convert the instance store-backed AMI created in this guide to an EBS-backed AMI. Please note that the steps in this section can be performed using the AWS Management Console or the EC2 API Tools from the build machine. I will be demonstrating all of the steps using the EC2 API tools.

Launch the Instance Store-Backed AMI

Use the same methods described in the previous section to launch the instance store-backed AMI.


# ec2-run-instances ami-6418ba0d \ --instance-type m1.small \ --key idevelopment-ec2-key \ --group idevelopment-security-group RESERVATION r-1862fa7d 395393972166 idevelopment-security-group INSTANCE i-2280b15b ami-6418ba0d pending idevelopment-ec2-key 0 m1.small 2012-06-24T23:21:13+0000 us-east-1b aki-88aa75e1 monitoring-disabled

Verify that the instance is running and take note of the Availability Zone assigned to the instance. In this example, the instance was assigned to the us-east-1b Availability Zone. You'll need this when creating the EBS volume in the next section.


# ec2-describe-instances i-2280b15b RESERVATION r-1862fa7d 395393972166 idevelopment-security-group INSTANCE i-2280b15b ami-6418ba0d ec2-23-20-244-127.compute-1.amazonaws.com ip-10-190-129-80.ec2.internal running idevelopment-ec2-key 0 m1.small 2012-06-24T23:21:13+0000 us-east-1b aki-88aa75e1 monitoring-disabled 23.20.244.127 10.190.129.80 instance-store paravirtual xen sg-e5ff738d default

Create a New EBS Volume

Create a new Amazon EBS volume. For the purpose of this example, I will create a 10 GiB volume that will be used as the root disk for any new EBS-backed instance. The volume size of 10 GiB is being used because that is the largest size for an instance store-backed AMI. At the end of this section I will demonstrate how to increase the volume size at run time.

 

You must specify an Availability Zone when creating a volume. The volume and the instance to which it attaches must be in the same Availability Zone. Use the Availability Zone that was returned from instance metadata when launching the instance.


# ec2-create-volume --size 10 --availability-zone us-east-1b VOLUME vol-61e2b80f 10 us-east-1b creating 2012-06-24T23:50:49+0000

The creation process may take awhile depending on the size of the volume. The creation process for the volume will be finished when the status returns 'available'.


# ec2-describe-volumes vol-61e2b80f VOLUME vol-61e2b80f 10 us-east-1b available 2012-06-24T23:50:49+0000

Attach the EBS Volume

Attach the new EBS volume to the running EC2 instance store-backed AMI created earlier in this guide. The following will attach the volume and expose it as the specified device.


# ec2-attach-volume vol-61e2b80f --instance i-2280b15b --device /dev/sdf ATTACHMENT vol-61e2b80f i-2280b15b /dev/sdf attaching 2012-06-24T23:55:09+0000

Verify the volume was successfully attached to the instance.


# ec2-describe-volumes vol-61e2b80f VOLUME vol-61e2b80f 10 us-east-1b in-use 2012-06-24T23:50:49+0000 ATTACHMENT vol-61e2b80f i-2280b15b /dev/sdf attached 2012-06-24T23:55:18+0000

Prepare the EBS Volume

Log in to the EC2 instance store-backed AMI and create an ext4 filesystem type on the partitionless EBS volume.


-bash-4.1# /bin/egrep '[xvsh]d[a-z].*$' /proc/partitions 202 65 10485760 xvde1 202 66 156352512 xvde2 202 67 917504 xvde3 202 144 10485760 xvdj -bash-4.1# mkfs.ext4 /dev/xvdj

 

Note that while we specified to attach the EBS volume as device /dev/sdf in the previous step, Amazon Linux uses the Xen virtual disk notation /dev/xvdn. The volume in my case was attached as /dev/xvdj.

Create a mount point directory and mount the EBS volume.


-bash-4.1# mkdir -p /opt/ec2/mnt -bash-4.1# mount -t ext4 /dev/xvdj /opt/ec2/mnt -bash-4.1# mount /dev/xvde1 on / type ext4 (rw) none on /proc type proc (rw) none on /sys type sysfs (rw) none on /dev/pts type devpts (rw,gid=5,mode=620) none on /dev/shm type tmpfs (rw) /dev/xvde2 on /mnt/vol1 type ext4 (rw) none on /proc/sys/fs/binfmt_misc type binfmt_misc (rw) /dev/xvdj on /opt/ec2/mnt type ext4 (rw)

Remove any local instance storage entries from /etc/fstab if they exist. Booting from an EBS volume does not use local instance storage by default. If you followed this guide to create the instance store-backed AMI, remove the local instance storage entry in /etc/fstab which mounts on /mnt/vol1.


-bash-4.1# cat /etc/fstab | grep -v mnt > /tmp/fstab -bash-4.1# mv /etc/fstab /etc/fstab.bak -bash-4.1# mv /tmp/fstab /etc/fstab

Sync the root and dev file systems to the EBS volume.


-bash-4.1# rsync -avHx / /opt/ec2/mnt -bash-4.1# rsync -avHx /dev /opt/ec2/mnt

Label the disk.


-bash-4.1# tune2fs -L '/' /dev/xvdj

Flush all writes and unmount the volume.


-bash-4.1# sync;sync;sync;sync -bash-4.1# umount /opt/ec2/mnt

Detach the EBS Volume

From the build machine, detach the EBS volume from the instance.


# ec2-detach-volume vol-61e2b80f --instance i-2280b15b ATTACHMENT vol-61e2b80f i-2280b15b /dev/sdf detaching 2012-06-24T23:55:18+0000

Verify that the volume was successfully detached and has a status of 'available'.


# ec2-describe-volumes vol-61e2b80f VOLUME vol-61e2b80f 10 us-east-1b available 2012-06-24T23:50:49+0000

Create a Snapshot of the EBS Volume

Create a snapshot of the EBS volume and provide an optional description for the snapshot.


# ec2-create-snapshot vol-61e2b80f --description "EBS-Backed AMI for CentOS 6.2 (x86_64)" SNAPSHOT snap-d9f97aa7 vol-61e2b80f pending 2012-06-25T00:42:26+0000 395393972166 10 EBS-Backed AMI for CentOS 6.2 (x86_64)

The snapshot may take awhile to create. Check the progress of the snapshot creation until the status returns 'completed'.


# ec2-describe-snapshots snap-d9f97aa7 SNAPSHOT snap-d9f97aa7 vol-61e2b80f completed 2012-06-25T00:42:26+0000 100% 395393972166 10 EBS-Backed AMI for CentOS 6.2 (x86_64)

Register Snapshot as an AMI

Finally, delete the original EBS volume and register the new EBS-backed image with a block device mapping that maps the root device name to the previously created snapshot. Since this is a partitionless EBS volume with the same architecture and located in the same Region, I am able to use the same kernel AKI.


# ec2-delete-volume vol-61e2b80f VOLUME vol-61e2b80f # ec2-register \ --block-device-mapping /dev/sda1=snap-d9f97aa7::true \ --name "CentOS 6.2 (x86_64) EBS-Backed" \ --description "CentOS 6.2 (x86_64) Base EBS-Backed AMI" \ --architecture x86_64 \ --kernel aki-88aa75e1 IMAGE ami-421fbd2b

 

It is also possible to register the EBS-backed image with a root partition larger than the default 10 GiB we currently have. For example, to register the root partition as 160 GiB, run the following:

ec2-register \
--block-device-mapping /dev/sda1=snap-d9f97aa7:160:true \
--name "CentOS 6.2 (x86_64) EBS 160 GiB" \
--description "CentOS 6.2 (x86_64) Base EBS-Backed AMI (160 GiB)" \
--architecture x86_64 \
--kernel aki-88aa75e1

Note: After launching and logging in to an instance based on this AMI, you will need manually resize the filesystem.

-bash-4.1# df -k
Filesystem           1K-blocks      Used Available Use% Mounted on
/dev/xvde1            10321208    908296   8888624  10% /
none                    847740         0    847740   0% /dev/shm

-bash-4.1# resize2fs /dev/xvde1
resize2fs 1.41.12 (17-May-2010)
Filesystem at /dev/xvde1 is mounted on /; on-line resizing required
old desc_blocks = 1, new_desc_blocks = 10
Performing an on-line resize of /dev/xvde1 to 41943040 (4k) blocks.
The filesystem on /dev/xvde1 is now 41943040 blocks long.

-bash-4.1# df -k
Filesystem           1K-blocks      Used Available Use% Mounted on
/dev/xvde1           165139820    923024 155830108   1% /
none                    847740         0    847740   0% /dev/shm

Launch Instance

Use the image identifier (AMI ID) that was returned by ec2-register in the previous section along with the instance type, key pair, and security group to launch the instance.


# ec2-run-instances ami-421fbd2b \ --instance-type m1.small \ --key idevelopment-ec2-key \ --group idevelopment-security-group RESERVATION r-2ccd5649 395393972166 idevelopment-security-group INSTANCE i-b21c2ccb ami-421fbd2b pending idevelopment-ec2-key 0 m1.small 2012-06-25T01:05:35+0000 us-east-1c aki-88aa75e1 monitoring-disabled ebs paravirtual xen sg-e5ff738d default

Check the status of the new EBS-backed instance.


# ec2-describe-instances i-b21c2ccb RESERVATION r-2ccd5649 395393972166 idevelopment-security-group INSTANCE i-b21c2ccb ami-421fbd2b ec2-23-21-26-185.compute-1.amazonaws.com ip-10-116-71-95.ec2.internal running idevelopment-ec2-key 0 m1.small 2012-06-25T01:05:35+0000 us-east-1c aki-88aa75e1 monitoring-disabled 23.21.26.185 10.116.71.95 ebs paravirtualxen sg-e5ff738d default BLOCKDEVICE /dev/sda1 vol-3b517455 2012-06-25T01:05:59.000Z true

 

It is also possible to launch the EBS-backed image with a root partition larger than the default 10 GiB we currently have. For example, to launch the instance with a root partition of 160 GiB, run the following:

ec2-run-instances ami-421fbd2b \
--block-device-mapping /dev/sda1=:160:true \
--instance-type m1.small \
--key idevelopment-ec2-key \
--group idevelopment-security-group

Note: You will need to manually resize the filesystem after logging on to the instance.

-bash-4.1# df -k
Filesystem           1K-blocks      Used Available Use% Mounted on
/dev/xvde1            10321208    908296   8888624  10% /
none                    847740         0    847740   0% /dev/shm

-bash-4.1# resize2fs /dev/xvde1
resize2fs 1.41.12 (17-May-2010)
Filesystem at /dev/xvde1 is mounted on /; on-line resizing required
old desc_blocks = 1, new_desc_blocks = 10
Performing an on-line resize of /dev/xvde1 to 41943040 (4k) blocks.
The filesystem on /dev/xvde1 is now 41943040 blocks long.

-bash-4.1# df -k
Filesystem           1K-blocks      Used Available Use% Mounted on
/dev/xvde1           165139820    923024 155830108   1% /
none                    847740         0    847740   0% /dev/shm

Just as before, connect to the instance using an SSH client and the private key.


# ping -c 3 ec2-23-21-26-185.compute-1.amazonaws.com PING ec2-23-21-26-185.compute-1.amazonaws.com (23.21.26.185) 56(84) bytes of data. 64 bytes from ec2-23-21-26-185.compute-1.amazonaws.com (23.21.26.185): icmp_seq=1 ttl=49 time=25.7 ms 64 bytes from ec2-23-21-26-185.compute-1.amazonaws.com (23.21.26.185): icmp_seq=2 ttl=49 time=26.2 ms 64 bytes from ec2-23-21-26-185.compute-1.amazonaws.com (23.21.26.185): icmp_seq=3 ttl=49 time=26.2 ms --- ec2-23-21-26-185.compute-1.amazonaws.com ping statistics --- 3 packets transmitted, 3 received, 0% packet loss, time 2000ms rtt min/avg/max/mdev = 25.738/26.075/26.285/0.304 ms # ssh -i ~/.ssh/idevelopment-ec2-key.pem root@ec2-23-21-26-185.compute-1.amazonaws.com The authenticity of host 'ec2-23-21-26-185.compute-1.amazonaws.com (23.21.26.185)' can't be established. RSA key fingerprint is 22:18:7e:39:13:3a:5b:b7:9f:35:ba:ca:3e:c8:6d:ed. Are you sure you want to continue connecting (yes/no)? yes Warning: Permanently added 'ec2-23-21-26-185.compute-1.amazonaws.com,23.21.26.185' (RSA) to the list of known hosts. Last login: Sun Jun 24 19:59:30 2012 from 24.144.233.98 -bash-4.1# -bash-4.1# hostname domU-12-31-39-05-40-F5 -bash-4.1# /bin/egrep '[xvsh]d[a-z].*$' /proc/partitions 202 65 10485760 xvde1 202 67 917504 xvde3 -bash-4.1# mount /dev/xvde1 on / type ext4 (rw) none on /proc type proc (rw) none on /sys type sysfs (rw) none on /dev/pts type devpts (rw,gid=5,mode=620) none on /dev/shm type tmpfs (rw) none on /proc/sys/fs/binfmt_misc type binfmt_misc (rw) -bash-4.1# df -k Filesystem 1K-blocks Used Available Use% Mounted on /dev/xvde1 10321208 914388 8882532 10% / none 847740 0 847740 0% /dev/shm -bash-4.1# cat /proc/meminfo | grep SwapTotal SwapTotal: 917496 kB

About the Author

Jeffrey Hunter is an Oracle Certified Professional, Java Development Certified Professional, Author, and an Oracle ACE. Jeff currently works as a Senior Database Administrator for The DBA Zone, Inc. located in Pittsburgh, Pennsylvania. His work includes advanced performance tuning, Java and PL/SQL programming, developing high availability solutions, capacity planning, database security, and physical / logical database design in a UNIX / Linux server environment. Jeff's other interests include mathematical encryption theory, tutoring advanced mathematics, programming language processors (compilers and interpreters) in Java and C, LDAP, writing web-based database administration tools, and of course Linux. He has been a Sr. Database Administrator and Software Engineer for over 20 years and maintains his own website site at: http://www.iDevelopment.info. Jeff graduated from Stanislaus State University in Turlock, California, with a Bachelor's degree in Computer Science and Mathematics.



Copyright (c) 1998-2014 Jeffrey M. Hunter. All rights reserved.

All articles, scripts and material located at the Internet address of http://www.idevelopment.info is the copyright of Jeffrey M. Hunter and is protected under copyright laws of the United States. This document may not be hosted on any other site without my express, prior, written permission. Application to host any of the material elsewhere can be made by contacting me at jhunter@idevelopment.info.

I have made every effort and taken great care in making sure that the material included on my web site is technically accurate, but I disclaim any and all responsibility for any loss, damage or destruction of data or any other property which may arise from relying on it. I will in no case be liable for any monetary damages arising from such loss, damage or destruction.

Last modified on
Monday, 28-Apr-2014 04:27:13 EDT
Page Count: 68853