NAME

awsapi - Low-level Bourne shell access to Amazon EC2 etc.


SYNOPSIS

awsapi [options] <action> [parameters]


DESCRIPTION

awsapi makes it easy for Bourne shell scripts to call the low-level API of Amazon EC2 and other services that follow the same call conventions. It knows little about the API, but sends given actions and parameters exactly as given, in the hope that users will know all the details.

Requests

A typical shell command would look something like this:

    $(awsapi ec2.RunInstances ImageId=$imageId MinCount=1 MaxCount=1 \
        instanceId:instancesSet.1.instanceId)

The action ec2.RunInstances consists of a prefix (ec2), which is used to select API version and endpoint for the call, and an action name (RunInstances) which is passed directly to the underlying API.

If the action prefix is xyz, an XYZ_API_VERSION environment variable must be set to a corresponding API version. The XYZ_ENDPOINT may be set to a specific URL. If not set explicitly, it will usually default to https://xyz.amazonaws.com/, which is often what you want anyway.

Request parameters are simply given as name=value pairs, as expected. Common request parameters (Action, Version, AWSAccessKeyId, Timestamp, Signature, SignatureMethod, and SignatureVersion) are automatically added to each query.

Responses

The surrounding $(...) is because awsapi will print commands on stdout that it expects the calling shell to execute. Values are returned to the caller by setting variables that way. The second line in the above example says that our local shell variable instanceId should be set to the value returned in instancesSet.1.instanceId. The instanceId: part is optional in this case, since the name of the response element is used by default, but sometimes you may have a better name in mind.

Use the .n. notation to collect multiple values into a single list:

    $(awsapi ec2.DescribeInstances Filter.1.Name=instance-state-name \
        Filter.1.Value.1=running ipAddressList:reservationSet.n. \
            instancesSet.n.ipAddress)

As seen here, you may break long response elements after any dot and continue on the next line. Collected values are put in space-separated lists of items, suitable for use in a shell for loop.

"Objects"

The + notation gives you a separate "object" for each combination of values matched by one or more .n. specs:

    $(awsapi ec2.DescribeInstances \
        instance+state: \
            reservationSet.n.instancesSet.n.instanceState.name \
        instance+ \
            reservationSet.n.instancesSet.n.ipAddress)

This returns an instanceList shell variable, which contains one value for each instance: instanceTable[1.1], instanceTable[1.2], etc. You can then loop over this list and expand the value of each individual field:

    for instance in $instanceList; do
        printf "%-10s %-16s\n" $(instance.state) $(instance.ipAddress)
    done

There is a catch, though. This convenient notation didn't come out of nowhere, and you may need to understand some details. In this example, instance.state is the name of a temporary shell script, which reads the exported instance variable before printing a suitable value on stdout. Therefore, the name of your loop variable is not really negotiable.

Brace Expansion

The previous DescribeInstances example still looks way too complicated. In practice, it would probably be written like this instead:

    INSTANCES=instance+reservationSet.n.instancesSet.n
    $(awsapi ec2.DescribeInstances $INSTANCES.{ \
        state:instanceState.name, ipAddress \
    })

The {...} is similar to a bash-style brace expansion. If you run this command in bash(1) and skip some whitespace, bash does the expansion. Otherwise, awsapi will do the same thing. In any case, a.{b,c} will end up being the same as a.b a.c. In this example, the state: part will end up in the middle of the output spec, like this:

    instance+reservationSet.n.instancesSet.n.state:instanceState.name

This is OK. The colon is really just a renaming operator, so in this case it means that reservationSet.n.instancesSet.n.instanceState.name works as if it had been named reservationSet.n.instancesSet.n.state.

Filtering

You can add a display filter to your queries like this:

    $(awsapi ec2.DescribeInstances $INSTANCES.{ \
        state:instanceState.name eq running, ipAddress \
    })

In this case, the instanceList will only contain instances that were in fact running when the query was made. The six filtering operators are what you may expect: eq, ne, lt, gt, le, and ge correspond to C operators ==, !=, <, >, <=, and >=, but may be used unquoted in the shell. Comparison is numeric if both arguments look like numbers. Otherwise these operators do a stringwise comparison. Fortunately, timestamps in the EC2 API will have alphabetic order.

If you want to filter on a response element without returning it as a shell variable in the results, you may simply rename it to nothing:

    $(awsapi ec2.DescribeInstances $INSTANCES.{ \
        :instanceState.name eq running, ipAddress \
    })

It is possible to filter for one of many values by using slashes to separate the individual values like this:

    $(awsapi ec2.DescribeInstances $INSTANCES.{ \
        state:instanceState.name, ipAddress eq 192.168.0.1/192.168.0.2 \
    })

This builds that from the space-separated values of $ipAddressList:

    $(awsapi ec2.DescribeInstances $INSTANCES.{ \
        state:instanceState.name, ipAddress eq @ipAddressList \
    })

For clarity, you may also use in instead of eq in this case.

Default Values

The or filter provides a default value when a result is missing:

    $(awsapi ec2.DescribeSnapshots SnapshotId.1=$snapshotId \
        snapshotSet.1.progress or "0%")

A newly created EC2 snapshot does not necessarily have any kind of progress value, but it's convenient to pretend that it does, since awsapi would otherwise complain:

    awsapi: no snapshotSet.1.progress returned

Empty strings will also count as missing results, so you can use the default value to replace a response element that exists but is blank.

Verification

It is often necessary to verify the returned values. For convenience, awsapi will do this for you if an expected value is given like this:

    $(awsapi ec2.AssociateAddress PublicIp=$ipAddress \
        InstanceId=$instanceId status:return := true)

In this case, if the value of response element return is in fact not true, awsapi will complain on stderr and print "eval false" to stdout. This is the standard behavior for other types of error as well.

Waiting for something to happen is also a common activity. If you give multiple expected values, awsapi will repeat the call regularly until the final value has been returned. Each time, the returned value must match one of the expected values. Otherwise an error is signalled:

    $(awsapi ec2.DescribeVolumes VolumeId.1=$volumeId \
        volumeSet.1.status := attaching/attached)

There may be a delay before newly created resources are visible for further API calls. To avoid annoying complaints about these missing resources, you may use a '-' to represent the missing value:

    $(awsapi ec2.DescribeInstances \
        Filter.1.{ Name="instance-id", Value.1="$instanceId" } \
        reservationSet.1.instancesSet.1.instanceState.name \
            := -/pending/running)

Joining Tables

This is a low-level tool. Each call to awsapi corresponds to a single call to the underlying API. However, it is convenient to see the name of each returned object, and Amazon EC2 stores names as separate tags.

The solution to this problem would look something like this:

    # Grab the "Name" tags for all "instance" resources
    $(awsapi ec2.DescribeTags tag@resourceId+tagSet.n.{ \
        resourceId, resourceType eq instance, key eq Name, name:value \
    })
    # Include the tag.name result of the first query here
    $(awsapi --table ec2.DescribeInstances $INSTANCES.{ \
        instanceId, state:instanceState.name, \
        ~tag.name@instanceId, ipAddress \
    })

In the first query, the @resourceId part says that the result should be indexed by the value of resourceId (which should be unique), instead of using the number matched by .n., as usual.

The second query uses the tag.name table column that was generated by the first query. A value in this column is selected by instanceId, which should match some resourceId in the previous query.

The --table option displays this combined result in a pretty way.

Settings

Your AWS "Secret Access Key" and "Access Key ID" must be stored in a secret ~/.awsapirc file. Remember to use chmod 600 ~/.awsapirc to keep your secrets secret. A sample ~/.awsapirc would look like this:

    secretAccessKey: eW91dHViZS5jb20vd2F0Y2g/dj1SU3NKMTlzeTNKSQ==
    accessKeyId: AKIADQKE4SARGYLE

Furthermore, each script you write should start with one or more lines that set the API versions of all relevant services:

    export EC2_API_VERSION="2010-11-15"
    export SQS_API_VERSION="2009-02-01"

See the EXAMPLES section for a complete example of a shell script.


OPTIONS

--debug

Enables debugging, which prints extra information to stderr.

--help

Prints a brief help message and exits.

--table

Prints your query results in a nice table layout.

--man

Displays the complete awsapi man page.

--version

Displays the date of this awsapi version.


EXAMPLES

 #!/bin/sh
 set -e
 ### Script initialization ##################################
 # These are usually needed
 export EC2_API_VERSION="2010-11-15"
 METADATA=http://169.254.169.254/latest/meta-data
 PATH="$(dirname $0):$PATH"
 ### IP address grabbing ####################################
 # Describe the instance we're running on
 instanceId=$(curl -s "$METADATA/instance-id")
 $(awsapi ec2.DescribeInstances InstanceId.1=$instanceId \
     reservationSet.1.instancesSet.1.ipAddress)
 # Print the IP address
 echo "IP address: $ipAddress"
 ### Volume creation ########################################
 # Create an empty volume and get its ID
 $(awsapi ec2.CreateVolume AvailabilityZone=us-east-1d \
     Size=8 volumeId)
 # Wait for the volume to become available
 $(awsapi ec2.DescribeVolumes VolumeId.1=$volumeId \
     volumeSet.1.status := creating/available)
 ############################################################


ENVIRONMENT

The following environment variables should be set when awsapi is used. XYZ is a fake prefix that should be replaced by EC2, SQS, etc.

XYZ_API_VERSION

Used verbatim as the "Version" parameter in API calls. The script doesn't really care about versions, but Amazon XYZ may reject your requests if you are programming against the wrong version.

XYZ_ENDPOINT

Used to set an endpoint URL if https://xyz.amazonaws.com/ is not good enough for you. For example, Amazon EC2 will typically need https://ec2.eu-west-1.amazonaws.com/ to use the Irish region.

AWSAPI_FAILURE_COMMAND

The command executed when awsapi fails. The default, "eval false", is handled like any other failing command. However, it's sometimes useful to treat AWS failures differently. For example, you may use "return 1" to exit a function, or simply call "handle_aws_failure".

AWSAPI_FILE_DIR

The name of a directory containing automatically generated scripts. Your first awsapi call will create this directory and also set the environment variable. The directory is automatically deleted when the current shell exits.

AWSAPI_USER_AGENT

The "User-Agent" header that identifies a client in each request to the AWS servers. For more advanced scripts, you may want to change the default "awsapi" identification to some more specific name.

HOME

Used to determine the user's home directory, which is where an .awsapirc file with the AWS "Secret Access Key" and "Access Key ID" should be placed.


FILES

~/.awsapirc -- contains your AWS secrets.

This file contains secret data. It should only be accessible by the awsapi user, so remember to "chmod 600 ~/.awsapirc" before adding your keys. All settings are given as lines of "name: value" pairs. The secretAccessKey and accessKeyId settings are required.


BUGS

Amazon S3 does not have a query API. This version of awsapi has some rudimentary code that tries to use SOAP instead, but that support is very limited and not expected to help you in any way.


SEE ALSO

http://aws.amazon.com/documentation/


AUTHOR

Henrik Gulbrandsen <henrik@gulbra.net>