NAME

Amazon::API - A generic base class for AWS Services

SYNOPSIS

use Amazon::API;

my $service = Amazon::API->new( service => 'events', api => 'AWSEvents');
my $rules = $service->invoke_api('ListRules');

DESCRIPTION

https://github.com/rlauer6/perl-Amazon-API/actions/workflows/build.yml/badge.svg

Generic class for constructing AWS API interfaces. Typically used as a parent class, but can be used directly. This package can also generates stubs for Amazon APIs using the Botocore project metadata. (See "BOTOCORE SUPPORT").

The typical use of this API is through the classes you build with the included tool (amzn-api). The tool leverages the Botocore project's metadata to build classes that are specific to each API (and are documented via the separate Amazon::API::Help tool). Using Amazon::API directly may not work in all circumstances unless you are very familiar with the API you are calling. If you decide to take the Luddite approaches, read the documentation carefully before using Amazon::API.

BACKGROUND AND MOTIVATION

A comprehensive Perl interface to AWS services similar to the Botocore library for Python has been a long time in coming. The Paws project has been creating an always up-to-date AWS interface with community support. If you are looking for an extensible method of installing and invoking a subset of services you might want to consider Amazon::API.

Think of this class as a DIY kit for installing only the APIs and methods you need for your AWS project. Using the included amzn-api utility you can also roll your own complete Amazon API classes that include support for serializing requests and responses based on metadata provided by the Botocore project. The classes you create with amzn-api include full documentation as pod. (See "BOTOCORE SUPPORT" for more details).

NOTE: The original Amazon::API was written in 2017 as a very lightweight way to call a handfull of APIs. The evolution of the module was based on discovering, without much documentation or help, the nature of Amazon APIs. In retrospect, even back then, it would have been easier to consult the Botocore project and decipher how that project managed to create a library from the metadata. Fast forward to 2022 and Amazon::API began using the Botocore metadata in order to, in most cases, correctly call any AWS service. The Amazon::API module can still be used without the assistance of Botocore metadata, but it works a heckuva lot better with it.

You can use Amazon::API in several different ways:

Take the Luddite approach

my $queues = Amazon::API->new(
 {
  service     => 'sqs',
  http_method => 'GET'
 })->invoke_api('ListQueues');

Build your own API classes with just what you need

package Amazon::API::SQS;

use strict;
use warnings;

use parent qw( Amazon::API );

our @API_METHODS = qw(
  ListQueues
  PurgeQueue
  ReceiveMessage
  SendMessage
);

sub new {
  my ( $class, @options ) = @_;
  $class = ref($class) || $class;

  my %options = ref( $options[0] ) ? %{ $options[0] } : @options;

  return $class->SUPER::new(
    { service       => 'sqs',
      http_method   => 'GET',
      api_methods   => \@API_METHODS,
      decode_always => 1,
      %options
    }
  );
}

1;

use Amazon::API::SQS;
use Data::Dumper;

my $sqs = Amazon::API::SQS->new;

print {*STDERR} Dumper($sqs->ListQueues);

Amazon::API DarkPAN

Many of the Amazon API classes have already been pre-built and are available in the Amazon::API DarkPAN:

https://cpan.openbedrock.net/orepan2

Each module that is generated and installed in the DarkPAN is signed. You can find instructions on how to verify the signatures by visiting https://cpan.openbedrock.net/signature/index.html.

Modules not currently available can be requested on the same site.

Build from the cloned https://github.com/rlauer6/Amazon-API.git project

You can build a complete CPAN distribution by cloning the project from GitHub. You'll need to install the build time dependencies before you can build a distribution.

cat build-requires| cpm install --resolver 02packages,https://cpan.openbedrock.net/orepan2 -

Once you have installed all of the required modules:

PERL5LIB=$(pwd)/local/lib/perl5:$(pwd)/lib make cpan-dist SERVICE=sts

This will pull the latest master Botocore branch and create a CPAN tarball you can install with your preferred installer:

cpanm -v -l $HOME Amazon-API-STS-2011.06.15.tar.gz

Use the Botocore metadata to build classes locally

In order to use Botocore metadata you must clone the Botocore repository yourself and point the utility at the repository directory.

This method is recommended solely for experimenting with the Amazon::API.

THE APPROACH

Essentially, most AWS APIs are RESTful services that adhere to a common protocol, but differences in services make a single solution difficult. All services more or less adhere to this framework:

Specific details of the more recent AWS services are well documented, however early services were usually implemented as simple HTTP services that accepted a query string. This module attempts to account for most of the nuances involved in invoking AWS services and provide a fairly generic way of invoking these APIs in the most lightweight way possible.

Using Amazon::API as a generic, lightweight module, naturally does not provide nuanced support for individual AWS services. To use this class in that manner for invoking the AWS APIs, you need to be very familiar with the specific API requirements and responses and be willng to invest time reading the documentation on Amazon's website. The payoff is that you can probably use this class to call any AWS API without installing a large number of dependencies.

If you don't mind a few extra dependencies and overhead, you should generate the stub APIs using the amzn-api utility or download the classes from the "Amazon::API". The stubs produced by the utility will serialize and deserialize requests and responses correctly by using the Botocore metadata. Botocore metadata provides the necessary information to create classes that can successfully invoke all of the Amazon APIs.

Invoking some of the APIs can potentially be as easy as:

Amazon::API->new(
  service     => 'sqs',
  http_method => 'GET'
}
)->invoke_api('ListQueues');

...again using this approach requires that you understand the requirements for each API.

BOTOCORE SUPPORT

Using Botocore metadata and the utilities in this project, you can create Perl classes that simplify calling AWS services. After creating service classes from the Botocore metadata calling AWS APIs will look something like this:

use Amazon::API::SQS;

my $sqs = Amazon::API::SQS->new;
my $rsp = $sqs->ListQueues();

The Amazon::API::Role::Botocore module augments Amazon::API by using Botocore metadata for determining how to call individual services and serialize parameters passed to its API methods. amzn-api is provided to help you generate Perl classes for all AWS services using the Botocore metadata.

Perl classes that represent AWS data structures (aka shapes) that are passed to or returned from services are generated dynamically from the Botocore metadata. These classes allow you to call all of the API methods for a given service using simple Perl objects that are serialized correctly for a specific method.

Service classes are subclassed from Amazon::API so their new() constructor takes the same arguments as Amazon::API::new().

my $credentials = Amazon::Credential->new();

my $sqs = Amazon::API::SQS->new( credentials => $credentials );

Response Serialization

With little documentation to go on, interpretting the Botocore metadata and deducing how to serialize Botocore shapes (using a single serializer) from Perl objects has been a difficult task. It's likely that there are still some edge cases and bugs lurking in the serialization methods. Accordingly, starting with version 1.4.5, serialization exceptions or exceptions that occur while attempting to decode a response, will result in the raw response being returned to the caller. The idea being that getting something back that allows you figure out what to do with the response might be better than receiving an error.

OTOH, you might want to see the error, report it, or possibly contribute to its resolution. You can prevent errors from being surpressed by setting the raise_serializtion_errors to a true value. The default is false.

Throughout the rest of this documentation a request made using one of the classes created by the Botocore support scripts will be referred to as a Botocore request or Botocore API.

Starting with version 2.0.12 serialization has become much more reliable, but there are still some differences in the way the Python Botocore library serialize responses. For example, some serializers may include or exclude members that are not present in the response payload. If you are testing a response element, the best approach is to first test the truthiness and then test the presence of content.

if ( $result->{$key} && @{$result->{$key}} ) 

if ( $result->{$key} && %{result->{$key}} )

The above concerns the response side. For validation of the values you send, see "Request Validation" below.

Request Validation

Botocore shape definitions carry more than structure -- many members declare value constraints: an enum of permitted strings, a min and max (length for strings, magnitude for numbers, element count for lists), and a regular-expression pattern. Starting with version 2.4.6, a Botocore request validates the values you supply against these constraints before the request is serialized and sent, so a bad value is caught locally -- with the offending parameter named -- rather than surfacing as an opaque InvalidParameterValue from AWS several layers away.

Validation is controlled by the package variable $Amazon::API::VALIDATE_MODE, which takes one of three values:

Because the constraint data is only as current as the Botocore metadata this client was generated from, strict is deliberately safe against staleness: the exception it raises tells you how to relax the mode. If AWS has added an enum value (or loosened a bound) since your metadata was built, strict will reject a value that is in fact now valid -- so the message points you at warn and off as the escape hatch. This is the same posture the scanner takes toward unrecognized members: act on the metadata you have, but never let a point-in-time snapshot become a hard wall.

The mode is a package global, so set it once to change the default for every request:

$Amazon::API::VALIDATE_MODE = 'warn';

or localize it to relax validation for a single call while leaving the global default intact:

{
  local $Amazon::API::VALIDATE_MODE = 'off';
  $ec2->RunInstances( \%params );
}

Validation applies only to Botocore requests -- requests built from generated service classes, where the shape constraints are available. A request you construct by hand (the Luddite approach) is sent as-is.

ERRORS

When an error is returned from an API request, an exception class (Amazon::API::Error) will be raised if raise_error has been set to a true value (the default). If you set print_error to true AND raise_error is false, then errors will be printed to STDERR.

See Amazon::API::Error for more details.

METHODS AND SUBROUTINES

Reminder: You can mostly ignore this part of the documentation when you are leveraging Botocore to generate your API classes.

new

new(options)

All options are described below. options can be a list of key/values or hash reference.

invoke_api

invoke_api(action, [parameters], [content-type], [headers]);

or using named parameters...

invoke_api({ action => args, ... } )

Invokes the API with the provided parameters.

Note: This method consults the raise_error and print_error options to determine how errors are handled.

decode_response

Boolean that indicates whether or not to deserialize the most recent response from an invoked API based on the Content-Type header returned. If there is no Content-Type header, then the method will try to decode it first as a JSON string and then as an XML string. If both of those fail, the raw content is returned.

You can enable or disable deserializing responses globally by setting the decode_always attribute when you call the new constructor.

default: true

By default, Amazon::API will retrieve all results for Botocore based API calls that require pagination. To turn this behavior off, set use_paginator to a false value when you instantiate the API service.

my $ec2 = Amazon::API->new(use_paginator => 0);

print_error

Prints a formatted version of the last error encountered to STDERR.

submit

submit(options)

This method is used internally by invoke_api and normally should not be called by your applications.

options is a reference to a hash of options:

generate_xml

generate_xml(object)

Generates XML from a Perl object (uses XML::Twig).

EXPORTED METHODS

get_api_service

get_api_service(api, options)

Convenience routine that will return an API instance.

my $sqs = get_api_service 'sqs';

Equivalent to:

require Amazon::API::SQS;

my $sqs = Amazon::API::SQS->new(%options);

create_url_encoded_content

create_urlencoded_content(parameters, action, version)

Returns a URL encoded query string. parameters can be any of SCALAR, ARRAY, or HASH. See below.

param_n

param_n(parameters)

Format parameters in the "param.n" notation.

parameters should be a hash or array reference.

A good example of a service that uses this notation is the SendMessageBatch SQS API call.

The sample request can be found here:

SendMessageBatch

https://sqs.us-east-2.amazonaws.com/123456789012/MyQueue/
?Action=SendMessageBatch
&SendMessageBatchRequestEntry.1.Id=test_msg_001
&SendMessageBatchRequestEntry.1.MessageBody=test%20message%20body%201
&SendMessageBatchRequestEntry.2.Id=test_msg_002
&SendMessageBatchRequestEntry.2.MessageBody=test%20message%20body%202
&SendMessageBatchRequestEntry.2.DelaySeconds=60
&SendMessageBatchRequestEntry.2.MessageAttribute.1.Name=test_attribute_name_1
&SendMessageBatchRequestEntry.2.MessageAttribute.1.Value.StringValue=test_attribute_value_1
&SendMessageBatchRequestEntry.2.MessageAttribute.1.Value.DataType=String
&Expires=2020-05-05T22%3A52%3A43PST
&Version=2012-11-05
&AUTHPARAMS

To produce this message you would pass the Perl object below to param_n():

my $message = {
  SendMessageBatchRequestEntry => [
    { Id          => 'test_msg_001',
      MessageBody => 'test message body 1'
    },
    { Id               => 'test_msg_002',
      MessageBody      => 'test message body 2',
      DelaySeconds     => 60,
      MessageAttribute => [
        { Name  => 'test_attribute_name_1',
          Value =>
            { StringValue => 'test_attribute_value_1', DataType => 'String' }
        }
      ]
    }
  ]
};

CAVEATS

Request-side value validation (enum, min/max, pattern) is enforced against the Botocore metadata this client was built from, which is a point-in-time snapshot. See "Request Validation" for how $Amazon::API::VALIDATE_MODE governs this and how to relax it if the metadata has fallen behind the live API.

IMPLEMENTATION NOTES

If you have taken the advice above and created classes using the amzn-api script you can probably ignore this section. This section is intended to help those trying to create the lightest weight possible AWS API class.

Just a reminder for those wanting to go lite...

Headers

X-Amz-Target

Most of the newer AWS APIs are invoked as HTTP POST operations and accept a header X-Amz-Target in lieu of the CGI parameter Action to specify the specific API action. Some APIs also want the version in the target, some don't. There is sparse documentation about the nuances of using the REST interface directly to call AWS APIs, but you kinda sorta figure it out by parsing the Botocore data for a particular API.

When invoking an API, the class uses the api value to indicate that the action should be set in the X-Amz-Target header. We also check to see if the version needs to be attached to the action value as required by some APIs.

if ( $self->get_api ) {
  if ( $self->get_version) {
    $self->set_target(sprintf('%s_%s.%s', $self->get_api, $self->get_version, $self->get_action));
  }
  else {
    $self->set_target(sprintf('%s.%s', $self->get_api, $self->get_action));
  }

  $request->header('X-Amz-Target', $self->get_target());
}

DynamoDB and KMS seem to be able to use this in lieu of query variables Action and Version, although again, there seems to be a lot of inconsistency (and sometimes flexibility) in the APIs. DynamoDB uses DynamoDB_YYYYMMDD.Action while KMS does not require the version that way and prefers TrentService.Action (with no version). There is no explanation in any of the documentations I have been able to find as to what "TrentService" might actually mean. Again, your best approach is to read Amazon's documentation and look at their sample requests for guidance. You can also look to the Botocore project for information regarding the service. Checkout the service-2.json file within the sub-directory botocore/botocore/data/{api-version}/{service-name} which contains details for each service.

In general, the AWS API ecosystem is very organic. Each service seems to have its own rules and protocol regarding what the content of the headers should be.

As noted, this generic API interface tries to make it possible to use one class Amazon::API as a sort of gateway to the APIs. The most generic interface is simply sending query variables and not much else in the header. Services like EC2 conform to that protocol and can be invoked with relatively little fanfare.

use Amazon::API;
use Data::Dumper;

print Dumper(
  Amazon::API->new(
    service => 'ec2',
    version => '2016-11-15'
  )->invoke_api('DescribeInstances')
);

Note that invoking the API in this fashion, version is required.

For more hints regarding how to call a particular service, you can use the AWS CLI with the --debug option. Invoke the service using the CLI and examine the payloads sent by the Botocore library.

Rolling a New API

Once again, your best bet is to use the amzn-api script to roll a class from the Botocore metadata, but if you really want to create your own class the lite way read on.

The Amazon::API class will stub out methods for the API if you pass an array of API method names. The stub is equivalent to:

sub some_api {
  my $self = shift;

  $self->invoke_api('SomeApi', @_);
}

Some will also be happy to know that the class will create an equivalent CamelCase version of the method.

As an example, here is a possible implementation of Amazon::CloudWatchEvents that implements one of the API calls.

package Amazon::CloudWatchEvents;

use strict;
use warnings;

use parent qw(Amazon::API);

sub new {
  my ($class, $options) = @_;

  my $self = $class->SUPER::new(
    { %{$options},
      api         => 'AWSEvents',
      service     => 'events',
      api_methods => [qw( ListRules )],
    }
  );

  return $self;
}

Then...

use Data::Dumper;

print Dumper(Amazon::CloudWatchEvents->new->ListRules({}));

Of course, creating a class for the service is optional. It may be desirable however to create higher level and more convenient methods that aid the developer in utilizing a particular API.

Overriding Methods

Because the class does some symbol table munging, you cannot easily override the methods in the usual way.

sub ListRules {
  my $self = shift;
  ...
  $self->SUPER::ListRules(@_)
}

Instead, you should re-implement the method as implemented by this class.

sub ListRules {
  my $self = shift;
  ...
  $self->invoke_api('ListRules', @_);
}

Content-Type

Yet another piece of evidence that suggests the organic nature of the Amazon API ecosystem is their use of different Content-Type headers. Some of the variations include:

application/json
application/x-amz-json-1.0
application/x-amz-json-1.1
application/x-www-form-urlencoded

Accordingly, the invoke_api() method can be passed the Content-Type or will try to make its best guess based on the service protocol or the type of object being passed as parameters. There is a hash of service names and service types that this module uses to determine the content type required by the service. If services are added that hash needs to be updated.

You can also set the default content type used for the calling service by passing the content_type option to the constructor.

$class->SUPER::new(
  content_type => 'application/x-amz-json-1.1',
  api          => 'AWSEvents',
  service      => 'events'
);

ADDITIONAL HINTS

VERSION

This documentation refers to version 2.6.0 of Amazon::API.

DIAGNOSTICS

To enable diagnostic output, set debug to a true value when calling the constructor. You can also set the DEBUG environment variable to a true value to enable diagnostics.

Logging

By default Amazon::API creates a Log::Log4perl logger to log at the DEBUG and TRACE levels. Setting the environment variable DEBUG to some value or passing a true value for debug in the constructor will trigger extremely verbose logging. This is to help debug edge cases especially around serialization which is particularly prone to exceptions and API specific scenarios.

If you pass a logger to the constructor, Amazon::API will attempt to use that if it has the appropriate logging level methods (error, warn, info, debug, trace, level). If Log::Log4perl is unavailable and you do not pass a logger, logging is essentially disabled at any level.

If, for some reason you set the enviroment variable DEBUG to a true value or have your own Log4perl logger set at the debug level but do not want Amazon::API to log messages at that level you can turn off logging as shown below:

my $ec2 = Amazon::API::EC2->new(log_level => 'info');

In other words, do not send a logger but send a log level. The constructor will recognize that you have a Log4perl logger initialized and just set its log level to your desired level.

BUGS AND LIMITATIONS

This module has not been tested on Windows OS. Please report any issues found by opening an issue here:

https://github.com/rlauer6/perl-Amazon-API/issues

FAQs

Why should I use this module instead of Paws?

Maybe you shouldn't. Paws is a community supported project and may be a better choice for most people. The programmers who created Paws are luminaries in the pantheon of Perl programming (alliteration intended). If you don't want to install of the AWS services but only need to use a single service, Amazon::API may be the right choice for you. Paws may also have some edge cases for some of the seldom used services and you might find this module easier to use and debug.

Does it perform better than Paws?

It depends on how you define better. Amazon::API has fewer dependencies and will load faster than Paws. It will consume less memory as well. API calls to AWS services will be about the same, with an edge to Amazon::API which returns plain Perl objects instead of Moose classes. The overhead introduced by this module and Paws may be insignificant compared to the API performance.

Does this work for all APIs?

I don't know. I have not tested every API. However I have tested an API for each of the protocols that are defined by the Botocore metadata (with the exception of smithy-rpc-v2-cbor) and serialization for every shape. Feedback is appreciated.

Some AWS services are difficult if not impossible to create an API class that implements them (talkin' to you S3!). If you want to use this to invoke S3 APIs, don't. I haven't tried it and I'm pretty sure it would'nt work anyway. There are modules designed specifically for S3; Amazon::S3::Lite, Amazon::S3, Net::Amazon::S3. Use them instead.

This code does not use "Modern Perl". Why?

This code has evolved over the years from being a simple way to make RESTful calls to a few Amazon APIs, to an API that now incorporates the use of the Botocore metadata to support nearly every API service.

The code did not start out as a well designed attempt to interpret the Botocore data like the Paws project...and that has turned out to be a happy accident. By avoiding frameworks like Moose the resulting interface is about as lightweight as it can possibly be. The code does however embrace Perl Best Practices to create a more maintainable and familiar distribution.

How do I pass AWS credentials to the API?

There is a bit of magic here as Amazon::API will use Amazon::Credentials transparently if you do not explicitly pass the credentials object. I've taken great pains to try to make the aforementioned module somewhat useful and secure.

See Amazon::Credentials.

Can I use more than one set of credentials to invoke different APIs?

Yes. See Amazon::Credentials.

How stable is the interface?

As of version 2.1.0 the interface is quite stable. There are currently (as of 2.6.0) no outstanding bugs logged against the project. I now consider this project "production ready".

Why are you using XML::Simple when it clearly says "DO NOT"?

It's simple. And it seems easier to build than other modules that almost do the same thing.

How do I stop the client from rejecting a parameter value?

If a request dies with a validation error naming an enum, length, or pattern constraint, the value you passed does not match this client's Botocore metadata. Most often that is a typo. If instead AWS has changed the API since your metadata was generated, relax validation with $Amazon::API::VALIDATE_MODE -- set it to warn to downgrade the error to a warning, or off to skip the check. See "Request Validation".

I tried to use this with XYZ service and it didn't work. What should do I do?

There are several reasons why your call might not have worked. The most likely place for API calls to fail is when serializing requests or serializing results. Enable debugging and see how far the API gets. Report whether the serialization on the request or response failed. If the serialization of the results failed, you can set decode_always to false which will prevent serialization of the result and return the raw content sent from the API. Other reasons your call may have failed include:

BETTER TOGETHER

The motivation behind Amazon::API has been to provide a lightweight implementation of Amazon APIs for Perl. Several companion projects have been developed with the same philosophy.

LICENSE AND COPYRIGHT

This module is free software. It may be used, redistributed and/or modified under the same terms as Perl itself.

SEE ALSO

Amazon::API::Help, Amazon::Credentials, Amazon::API::Error Amazon::Lambda::Runtime, Amazon::Lambda::Runtime::Builder, Paws

AUTHOR

Rob Lauer - rlauer@tresurersbriefcase.com