awsapi - Low-level Bourne shell access to Amazon EC2 etc.
awsapi [options] <action> [parameters]
awsapi makes it easy for Bourne shell scripts to call the low-level API of Amazon EC2 and other services that follow the same call conventions. It knows little about the API, but sends given actions and parameters exactly as given, in the hope that users will know all the details.
A typical shell command would look something like this:
$(awsapi ec2.RunInstances ImageId=$imageId MinCount=1 MaxCount=1 \ instanceId:instancesSet.1.instanceId)
The action ec2.RunInstances consists of a prefix (ec2), which is used to select API version and endpoint for the call, and an action name (RunInstances) which is passed directly to the underlying API.
If the action prefix is xyz, an XYZ_API_VERSION environment variable must be set to a corresponding API version. The XYZ_ENDPOINT may be set to a specific URL. If not set explicitly, it will usually default to https://xyz.amazonaws.com/, which is often what you want anyway.
Request parameters are simply given as name=value pairs, as expected. Common request parameters (Action, Version, AWSAccessKeyId, Timestamp, Signature, SignatureMethod, and SignatureVersion) are automatically added to each query.
The surrounding $(...) is because awsapi will print commands on stdout that it expects the calling shell to execute. Values are returned to the caller by setting variables that way. The second line in the above example says that our local shell variable instanceId should be set to the value returned in instancesSet.1.instanceId. The instanceId: part is optional in this case, since the name of the response element is used by default, but sometimes you may have a better name in mind.
Use the .n.
notation to collect multiple values into a single list:
$(awsapi ec2.DescribeInstances Filter.1.Name=instance-state-name \ Filter.1.Value.1=running ipAddressList:reservationSet.n. \ instancesSet.n.ipAddress)
As seen here, you may break long response elements after any dot and
continue on the next line. Collected values are put in space-separated
lists of items, suitable for use in a shell for
loop.
The +
notation gives you a separate "object" for each combination of
values matched by one or more .n.
specs:
$(awsapi ec2.DescribeInstances \ instance+state: \ reservationSet.n.instancesSet.n.instanceState.name \ instance+ \ reservationSet.n.instancesSet.n.ipAddress)
This returns an instanceList shell variable, which contains one value for each instance: instanceTable[1.1], instanceTable[1.2], etc. You can then loop over this list and expand the value of each individual field:
for instance in $instanceList; do printf "%-10s %-16s\n" $(instance.state) $(instance.ipAddress) done
There is a catch, though. This convenient notation didn't come out of nowhere, and you may need to understand some details. In this example, instance.state is the name of a temporary shell script, which reads the exported instance variable before printing a suitable value on stdout. Therefore, the name of your loop variable is not really negotiable.
The previous DescribeInstances example still looks way too complicated. In practice, it would probably be written like this instead:
INSTANCES=instance+reservationSet.n.instancesSet.n $(awsapi ec2.DescribeInstances $INSTANCES.{ \ state:instanceState.name, ipAddress \ })
The {...}
is similar to a bash-style brace expansion. If you run this
command in bash(1)
and skip some whitespace, bash does the expansion.
Otherwise, awsapi will do the same thing. In any case, a.{b,c}
will
end up being the same as a.b a.c
. In this example, the state:
part
will end up in the middle of the output spec, like this:
instance+reservationSet.n.instancesSet.n.state:instanceState.name
This is OK. The colon is really just a renaming operator, so in this
case it means that reservationSet.n.instancesSet.n.instanceState.name
works as if it had been named reservationSet.n.instancesSet.n.state
.
You can add a display filter to your queries like this:
$(awsapi ec2.DescribeInstances $INSTANCES.{ \ state:instanceState.name eq running, ipAddress \ })
In this case, the instanceList will only contain instances that were in
fact running when the query was made. The six filtering operators are what
you may expect: eq
, ne
, lt
, gt
, le
, and ge
correspond
to C operators ==
, !=
, <
, >
, <=
, and >=
,
but may be used
unquoted in the shell. Comparison is numeric if both arguments look
like numbers. Otherwise these operators do a stringwise comparison.
Fortunately, timestamps in the EC2 API will have alphabetic order.
If you want to filter on a response element without returning it as a shell variable in the results, you may simply rename it to nothing:
$(awsapi ec2.DescribeInstances $INSTANCES.{ \ :instanceState.name eq running, ipAddress \ })
It is possible to filter for one of many values by using slashes to separate the individual values like this:
$(awsapi ec2.DescribeInstances $INSTANCES.{ \ state:instanceState.name, ipAddress eq 192.168.0.1/192.168.0.2 \ })
This builds that from the space-separated values of $ipAddressList
:
$(awsapi ec2.DescribeInstances $INSTANCES.{ \ state:instanceState.name, ipAddress eq @ipAddressList \ })
For clarity, you may also use in
instead of eq
in this case.
The or
filter provides a default value when a result is missing:
$(awsapi ec2.DescribeSnapshots SnapshotId.1=$snapshotId \ snapshotSet.1.progress or "0%")
A newly created EC2 snapshot does not necessarily have any kind of progress value, but it's convenient to pretend that it does, since awsapi would otherwise complain:
awsapi: no snapshotSet.1.progress returned
Empty strings will also count as missing results, so you can use the default value to replace a response element that exists but is blank.
It is often necessary to verify the returned values. For convenience, awsapi will do this for you if an expected value is given like this:
$(awsapi ec2.AssociateAddress PublicIp=$ipAddress \ InstanceId=$instanceId status:return := true)
In this case, if the value of response element return is in fact not true, awsapi will complain on stderr and print "eval false" to stdout. This is the standard behavior for other types of error as well.
Waiting for something to happen is also a common activity. If you give multiple expected values, awsapi will repeat the call regularly until the final value has been returned. Each time, the returned value must match one of the expected values. Otherwise an error is signalled:
$(awsapi ec2.DescribeVolumes VolumeId.1=$volumeId \ volumeSet.1.status := attaching/attached)
There may be a delay before newly created resources are visible for further API calls. To avoid annoying complaints about these missing resources, you may use a '-' to represent the missing value:
$(awsapi ec2.DescribeInstances \ Filter.1.{ Name="instance-id", Value.1="$instanceId" } \ reservationSet.1.instancesSet.1.instanceState.name \ := -/pending/running)
This is a low-level tool. Each call to awsapi corresponds to a single call to the underlying API. However, it is convenient to see the name of each returned object, and Amazon EC2 stores names as separate tags.
The solution to this problem would look something like this:
# Grab the "Name" tags for all "instance" resources $(awsapi ec2.DescribeTags tag@resourceId+tagSet.n.{ \ resourceId, resourceType eq instance, key eq Name, name:value \ })
# Include the tag.name result of the first query here $(awsapi --table ec2.DescribeInstances $INSTANCES.{ \ instanceId, state:instanceState.name, \ ~tag.name@instanceId, ipAddress \ })
In the first query, the @resourceId
part says that the result should
be indexed by the value of resourceId
(which should be unique), instead
of using the number matched by .n.
, as usual.
The second query uses the tag.name
table column that was generated
by the first query. A value in this column is selected by instanceId
,
which should match some resourceId
in the previous query.
The --table option displays this combined result in a pretty way.
Your AWS "Secret Access Key" and "Access Key ID" must be stored in a
secret ~/.awsapirc file. Remember to use chmod 600 ~/.awsapirc
to
keep your secrets secret. A sample ~/.awsapirc would look like this:
secretAccessKey: eW91dHViZS5jb20vd2F0Y2g/dj1SU3NKMTlzeTNKSQ== accessKeyId: AKIADQKE4SARGYLE
Furthermore, each script you write should start with one or more lines that set the API versions of all relevant services:
export EC2_API_VERSION="2010-11-15" export SQS_API_VERSION="2009-02-01"
See the EXAMPLES section for a complete example of a shell script.
Enables debugging, which prints extra information to stderr.
Prints a brief help message and exits.
Prints your query results in a nice table layout.
Displays the complete awsapi man page.
Displays the date of this awsapi version.
#!/bin/sh set -e
### Script initialization ##################################
# These are usually needed export EC2_API_VERSION="2010-11-15" METADATA=http://169.254.169.254/latest/meta-data PATH="$(dirname $0):$PATH"
### IP address grabbing ####################################
# Describe the instance we're running on instanceId=$(curl -s "$METADATA/instance-id") $(awsapi ec2.DescribeInstances InstanceId.1=$instanceId \ reservationSet.1.instancesSet.1.ipAddress)
# Print the IP address echo "IP address: $ipAddress"
### Volume creation ########################################
# Create an empty volume and get its ID $(awsapi ec2.CreateVolume AvailabilityZone=us-east-1d \ Size=8 volumeId)
# Wait for the volume to become available $(awsapi ec2.DescribeVolumes VolumeId.1=$volumeId \ volumeSet.1.status := creating/available)
############################################################
The following environment variables should be set when awsapi is used. XYZ is a fake prefix that should be replaced by EC2, SQS, etc.
Used verbatim as the "Version" parameter in API calls. The script doesn't really care about versions, but Amazon XYZ may reject your requests if you are programming against the wrong version.
Used to set an endpoint URL if https://xyz.amazonaws.com/ is not good enough for you. For example, Amazon EC2 will typically need https://ec2.eu-west-1.amazonaws.com/ to use the Irish region.
The command executed when awsapi fails. The default, "eval false", is handled like any other failing command. However, it's sometimes useful to treat AWS failures differently. For example, you may use "return 1" to exit a function, or simply call "handle_aws_failure".
The name of a directory containing automatically generated scripts. Your first awsapi call will create this directory and also set the environment variable. The directory is automatically deleted when the current shell exits.
The "User-Agent" header that identifies a client in each request to the AWS servers. For more advanced scripts, you may want to change the default "awsapi" identification to some more specific name.
Used to determine the user's home directory, which is where an .awsapirc file with the AWS "Secret Access Key" and "Access Key ID" should be placed.
This file contains secret data. It should only be accessible by the awsapi user, so remember to "chmod 600 ~/.awsapirc" before adding your keys. All settings are given as lines of "name: value" pairs. The secretAccessKey and accessKeyId settings are required.
Amazon S3 does not have a query API. This version of awsapi has some rudimentary code that tries to use SOAP instead, but that support is very limited and not expected to help you in any way.
http://aws.amazon.com/documentation/
Henrik Gulbrandsen <henrik@gulbra.net>