Amazon has released an updated version of the Elastic Compute Cloud (EC2) API with the version name 2008-02-01. This release includes a number of new features that were not available in the 2007-08-29 version of the API that I discussed in the book Programming Amazon Web Services (PAWS). To help keep my readers up-to-date with the capabilities of the EC2 service, this article contains a description of the new features and demonstrates how to use them in code.
The 2008-02-01 EC2 API provides three important new features:
User Selectable Kernels – Choose to run an alternative Linux kernel on your instance, in order to benefit from newer or differently-configured kernel versions. Availability Zones – Choose the locations in which your instances will run. You can distribute your instances across different locations for better fault tolerance, or concentrate them in a single location to minimize network latency and data transfer costs. Elastic IP Addresses – Reserve your own public IP addresses in the EC2 environment, and assign these addresses to your instances on demand. Elastic IPs allow you to make an instance accessible at a known public IP address, without the need to use a dynamic DNS service to simulate a static IP.Getting Started
The code examples in this article are based on an updated EC2 client implementation that has been added to the sample code archive on the PAWS Examples web site. In this article I will use the Ruby implementation, but the archive also includes equivalent implementations in Java and Python.
Download the latest PAWS_examples.zip archive file, unzip it, and change to thePAWS_examples/ruby directory. This directory should contain two EC2 clients:
$ cd PAWS_examples/ruby$ ls EC2*EC2.rb EC2_2008-02-01.rb
The first file, EC2.rb, contains the original EC2 API implementation from the PAWS book. The second file, EC2_2008-02-01.rb, contains new and updated methods that work with the latest EC2 API version 2008-02-01. This is the EC2 client we will use in this article.
Start an interactive Ruby session with the irb command, and create a client object from the latest EC2 API implementation:
irb> require 'EC2_2008-02-01' # => trueirb> ec2 = EC2.new('YourAwsAccessKey', 'YourAwsSecretKey')
To confirm that you are using the correct API version, use thedescribe_images method to list the Amazon Machine Images (AMIs) provided to the public by Amazon. The listing should include different image types identified by the :type attribute. The image types should include machine,kernel and ramdisk images. Some machine images will also have :kernel_idand :ramdisk_id attributes.
Here is a portion of the results from a listing of Amazon’s images. The first item is a kernel image, and the second is a machine image that includes kernel and ramdisk identifier attributes:
irb> require 'pp' # => trueirb> pp ec2.describe_images(:owners => 'amazon')[{:type=>"kernel", :is_public=>true, :architecture=>"x86_64", :owner_id=>"amazon", :state=>"available", :location=> "ec2-public-images/vmlinuz-2.6.18-xenU-ec2-v1.0.x86_64.aki.manifest.xml", :id=>"aki-9800e5f1"}, . . . {:type=>"machine", :is_public=>true, :architecture=>"i386", :kernel_id=>"aki-a71cf9ce", :owner_id=>"amazon", :ramdisk_id=>"ari-a51cf9cc", :state=>"available", :location=>"ec2-public-images/fedora-8-i386-base-v1.06.manifest.xml", :id=>"ami-f51aff9c"},
The kernel and ramdisk image types are new additions to the EC2 API. They make it possible to select an alternative Linux kernel.
User Selectable Kernels
The User Selectable Kernels feature allows you to choose to run an alternative Linux kernel on your instances, in place of the default kernel provided by Amazon. In prior versions of the EC2 API there was only one kernel available for each instance type; a variation of the Linux 2.6.16 Xen kernel. You can now choose from a collection of kernels that have been made available by Amazon or Amazon-endorsed vendors.
With a range of kernels now available in EC2, it is easier to take advantage of kernel improvements or alternative kernel configuration settings that may benefit your application. Unfortunately, the range of kernels is still limited to what Amazon and other vendors provide. You cannot create your own customized kernel, though hopefully it will become possible to do so in the future.
New Image Types: AKIs and ARIs join AMIs
Alternative kernels are made available in the EC2 environment as a new type of image, known as an Amazon Kernel Image (AKI). Because some kernels require additional drivers when they launch, there is another new image type called an Amazon RAM disk Image (ARI) which stores driver files. These two image types join the venerable Amazon Machine Image (AMI) which has always been used in EC2 to store an instance’s root volume.
On the Amazon Machine Images page, you can see a listing of the publicly available kernel, ramdisk, and machine images. This listing contains links to pages with further information about the images. For example, the page for a kernel image may describe the kernel’s configuration, indicate whether it requires a ramdisk image, and include links to module files that are compatible with the kernel. This information can help you decide which kernel will work best with your application.
You can also use the DescribeImages API operation to programmatically list all the images that are available to you. The listing returned by the service will include an attribute that describes the type of each image, and you can also tell the type of an image from the beginning of its identifier string which will be ami-, aki- or ari-.
Amazon machine images may be configured with default values for the kernel and ramdisk that instances will use. If you list the attributes for Amazon’s Fedora 8 public AMI (ami-f51aff9c), you can see that it is associated with the kernel image aki-a71cf9ce and ramdisk image ari-a51cf9cc:
irb> pp ec2.describe_images(:image_ids => 'ami-f51aff9c')[{:type=>"machine", :location=>"ec2-public-images/fedora-8-i386-base-v1.06.manifest.xml", :is_public=>true, :owner_id=>"amazon", :architecture=>"i386", :kernel_id=>"aki-a71cf9ce", :state=>"available", :ramdisk_id=>"ari-a51cf9cc", :id=>"ami-f51aff9c"}]Selecting Your Kernel
To run your instance with an alternative kernel, you specify the identifiers of your chosen kernel and ramdisk images when you perform the RunInstances operation. Not all kernels require extra drivers, so you will only need to include the ramdisk identifier when the kernel’s description page indicates that the ramdisk is necessary.
You can select any of the available kernel images when you launch an instance, however it is your responsibility to check that the kernel will actually work with the AMI you are launching. EC2 will not prevent you from using a kernel that is incompatible with the machine’s image or the instance’s architecture, nor will it force you to use the correct ramdisk image. Before you launch an instance with a non-standard kernel, you should always double-check that it is compatible with your AMI.
To demonstrate how to select an alternative kernel, we will launch an instance based on the Fedora 4 Getting Started machine image (ami-2bb65342) that has been available for some time. However, we will launch it with the new 2.6.18 Xen 3.1.0 kernel (aki-9b00e5f2) instead of the default 2.6.16 version.
To launch the instance, invoke the run_instances method and provide a:kernel_id option to specify the alternative kernel’s identifier. Because this newer kernel does not require a ramdisk image, you will not need to include the:ramdisk_id option.
irb> reservation = ec2.run_instances('ami-2bb65342', 1, 1, {:key_name => 'ec2-private-key', :kernel_id => 'aki-9b00e5f2'})
Once the instance has entered the running state and has been assigned a public DNS name…
irb> pp ec2.describe_instances. . . :instances=> [{:public_dns=>"ec2-72-44-52-218.compute-1.amazonaws.com", :image_id=>"ami-2bb65342", :kernel_id=>"aki-9b00e5f2", :state=>"running", :id=>"i-fb1ad992", . . .
…log in to the instance using ssh, and confirm that it is indeed running with the 2.6.18 Linux kernel.
$ ssh -i ec2-private-key.pem root@ec2-72-44-52-218.compute-1.amazonaws.comec2# uname --kernel-release2.6.18-xenU-ec2-v1.0
On an instance, you can find out which kernel and ramdisk images it is using by referring to the kernel-id and ramdisk-id metadata items:
ec2# curl -f http://169.254.169.254/2008-02-01/meta-data/kernel-idaki-9b00e5f2
If your instance is running with a non-standard kernel and ramdisk, these metadata items will return the identifier values. If you did not specify an alternative kernel or ramdisk, or if the instance was launched using an earlier API version, these metadata items may return 404 errors. This is the case for the instance we just launched, which does not have an associated ramdisk image:
ec2# curl -f http://169.254.169.254/2008-02-01/meta-data/ramdisk-idcurl: (22) The requested URL returned error: 404Modules for User Selected Kernels
If you launch an instance with a non-standard kernel, it is likely that the machine image will not include compatible kernel modules. If you check the module files available by default on the Getting Started AMI, you will see that they are not compatible with the new kernel you are running:
ec2# ls /lib/modules2.6.16-1.2069_FC4 2.6.16-xenU 2.6.17-1.2142_FC4
To obtain kernel modules that are compatible with your chosen kernel, you must either obtain pre-compiled module files, or compile them yourself. Fortunately, the information page for the 2.6.18 kernel includes a download link for compatible modules. To install these modules, you can simply download an archive file and extract its contents to the root of your instance’s file system:
ec2# wget http://ec2-downloads.s3.amazonaws.com/ec2-modules-2.6.18-xenU-ec2-v1.0-i686.tgzec2# tar xzf ec2-modules-2.6.18-xenU-ec2-v1.0-i686.tgz -C /ec2# ls /lib/modules2.6.16-1.2069_FC4 2.6.16-xenU 2.6.17-1.2142_FC4 2.6.18-xenU-ec2-v1.0ec2# modprobe -l/lib/modules/2.6.18-xenU-ec2-v1.0/kernel/security/seclvl.ko/lib/modules/2.6.18-xenU-ec2-v1.0/kernel/security/commoncap.ko. . .
Pre-prepared module files are only available for some of the alternative kernels. For the kernels without readily available modules, you should check the EC2 developer forums to see if anyone has posted details on where to find modules, or how to compile them.
Bundling AMIs to Use a Specific Kernel
When you use the latest version of the ec2-bundle-vol tool to create your own AMI from a running instance, the tool will refer to the instance’s metadata to discover which kernel and ramdisk images it is using. The tool will then automatically apply these settings to the bundled image it creates, which means that you can easily bundle a new AMI from an instance without having to manually specify the kernel or ramdisk identifiers.
If you wish to explicitly set the kernel and ramdisk identifiers, the ec2-bundle-voland ec2-bundle-image tools allow you to do so by providing the --kernel and --ramdisk options. There are a few situations where you may need to explicitly set these options:
If you are bundling an image created outside EC2, and you want the AMI to use an alternative kernel or ramdisk. If you are bundling a running instance, but you want the resultant AMI to use a different kernel or ramdisk. If you do not trust the ec2-bundle-vol tool to retrieve the instance’s settings from the metadata service.Availability Zones
The Availability Zones feature allows you to specify the location, or locations, where your instances will be deployed and run by the EC2 environment. By controlling the placement of your instances, you can easily disperse them across multiple EC2 locations for better fault tolerance, or concentrate them in a single location to minimize network latency and data transfer fees.
You can specify the location for your instances when you launch them, or you can skip this step and allow EC2 to decide where to place your instances based on the availability and health of each location. In prior API versions, EC2 always chose the location for instances and there was no way you could control, or query, their location.
EC2 Locations: Regions and Zones
EC2 data center locations are described in terms of “regions” and “availability zones”. A region is a broad expanse such as a country or geographic area. At present there is only one EC2 region available, the U.S. E