The service categories were extracted from Botocore project:

grep -ri '"protocol":"' botocore/botocore/data/* | grep 'xml' | \ cut -f 8 -d '/' | sort -u > xml.services

grep -ri '"protocol":"' botocore/botocore/data/* | grep 'json' | \ cut -f 8 -d '/' | sort -u > json.services

grep -ri '"protocol":"' botocore/botocore/data/* | grep 'query' | \ cut -f 8 -d '/' | sort -u > query.services

These service categories are used to deduce content type for parameters sent to methods for these APIs when Botocore metadata was not used to create an API class. Content-Type can however be overridden when invoking APIs if we guess wrong.

rest-json => application/x-amz-1.1
rest-xml  => application/xml
query     => application/x-www-form-urlencoded

our %SERVICE_CONTENT_TYPES = ( ec2 => 'application/x-www-form-urlencoded', query => 'application/x-www-form-urlencoded', 'rest-json' => 'application/json', json => 'application/x-amz-json', 'rest-xml' => 'application/xml', );

our %API_TYPES = ( query => [ qw( autoscaling cloudformation cloudsearch cloudwatch docdb ec2 elasticache elasticbeanstalk elb elbv2 iam importexport neptune rds redshift sdb ses sns sts ) ], xml => [ qw( cloudfront route53 s3 s3control ) ], json => [ qw( accessanalyzer account acm acm-pca amp amplify amplifybackend amplifyuibuilder apigateway apigatewaymanagementapi apigatewayv2 appconfig appconfigdata appfabric appflow appintegrations application-autoscaling application-insights application-signals applicationcostprofiler appmesh apprunner appstream appsync apptest arc-zonal-shift artifact athena auditmanager autoscaling-plans b2bi backup backup-gateway backupsearch batch bcm-data-exports bcm-pricing-calculator bedrock bedrock-agent bedrock-agent-runtime bedrock-data-automation bedrock-data-automation-runtime bedrock-runtime billing billingconductor braket budgets ce chatbot chime chime-sdk-identity chime-sdk-media-pipelines chime-sdk-meetings chime-sdk-messaging chime-sdk-voice cleanrooms cleanroomsml cloud9 cloudcontrol clouddirectory cloudfront-keyvaluestore cloudhsm cloudhsmv2 cloudsearchdomain cloudtrail cloudtrail-data codeartifact codebuild codecatalyst codecommit codeconnections codedeploy codeguru-reviewer codeguru-security codeguruprofiler codepipeline codestar-connections codestar-notifications cognito-identity cognito-idp cognito-sync comprehend comprehendmedical compute-optimizer config connect connect-contact-lens connectcampaigns connectcampaignsv2 connectcases connectparticipant controlcatalog controltower cost-optimization-hub cur customer-profiles databrew dataexchange datapipeline datasync datazone dax deadline detective devicefarm devops-guru directconnect discovery dlm dms docdb-elastic drs ds ds-data dsql dynamodb dynamodbstreams ebs ec2-instance-connect ecr ecr-public ecs efs eks eks-auth elastictranscoder emr emr-containers emr-serverless entityresolution es events evidently finspace finspace-data firehose fis fms forecast forecastquery frauddetector freetier fsx gamelift gameliftstreams geo-maps geo-places geo-routes glacier globalaccelerator glue grafana greengrass greengrassv2 groundstation guardduty health healthlake identitystore imagebuilder inspector inspector-scan inspector2 internetmonitor invoicing iot iot-data iot-jobs-data iot-managed-integrations iotanalytics iotdeviceadvisor iotevents iotevents-data iotfleethub iotfleetwise iotsecuretunneling iotsitewise iotthingsgraph iottwinmaker iotwireless ivs ivs-realtime ivschat kafka kafkaconnect kendra kendra-ranking keyspaces kinesis kinesis-video-archived-media kinesis-video-media kinesis-video-signaling kinesis-video-webrtc-storage kinesisanalytics kinesisanalyticsv2 kinesisvideo kms lakeformation lambda launch-wizard lex-models lex-runtime lexv2-models lexv2-runtime license-manager license-manager-linux-subscriptions license-manager-user-subscriptions lightsail location logs lookoutequipment lookoutmetrics lookoutvision m2 machinelearning macie2 mailmanager managedblockchain managedblockchain-query marketplace-agreement marketplace-catalog marketplace-deployment marketplace-entitlement marketplace-reporting marketplacecommerceanalytics mediaconnect mediaconvert medialive mediapackage mediapackage-vod mediapackagev2 mediastore mediastore-data mediatailor medical-imaging memorydb meteringmarketplace mgh mgn migration-hub-refactor-spaces migrationhub-config migrationhuborchestrator migrationhubstrategy mq mturk mwaa neptune-graph neptunedata network-firewall networkflowmonitor networkmanager networkmonitor notifications notificationscontacts oam observabilityadmin omics opensearch opensearchserverless opsworks opsworkscm organizations osis outposts panorama partnercentral-selling payment-cryptography payment-cryptography-data pca-connector-ad pca-connector-scep pcs personalize personalize-events personalize-runtime pi pinpoint pinpoint-email pinpoint-sms-voice pinpoint-sms-voice-v2 pipes polly pricing privatenetworks proton qapps qbusiness qconnect qldb qldb-session quicksight ram rbin rds-data redshift-data redshift-serverless rekognition repostspace resiliencehub resource-explorer-2 resource-groups resourcegroupstaggingapi robomaker rolesanywhere route53-recovery-cluster route53-recovery-control-config route53-recovery-readiness route53domains route53profiles route53resolver rum s3outposts s3tables sagemaker sagemaker-a2i-runtime sagemaker-edge sagemaker-featurestore-runtime sagemaker-geospatial sagemaker-metrics sagemaker-runtime savingsplans scheduler schemas secretsmanager security-ir securityhub securitylake serverlessrepo service-quotas servicecatalog servicecatalog-appregistry servicediscovery sesv2 shield signer simspaceweaver sms sms-voice snow-device-management snowball socialmessaging sqs ssm ssm-contacts ssm-incidents ssm-quicksetup ssm-sap sso sso-admin sso-oidc stepfunctions storagegateway supplychain support support-app swf synthetics taxsettings textract timestream-influxdb timestream-query timestream-write tnb transcribe transfer translate trustedadvisor verifiedpermissions voice-id vpc-lattice waf waf-regional wafv2 wellarchitected wisdom workdocs workmail workmailmessageflow workspaces workspaces-thin-client workspaces-web xray ) ], );

our @GLOBAL_SERVICES = qw( cloudfront iam importexport route53 s3 savingsplans sts );

our @REQUIRED_KEYS = qw( aws_access_key_id aws_secret_access_key );

######################################################################## sub choose(&) { return $_[0]->(); } ## no critic ########################################################################

######################################################################## sub paginator { ######################################################################## load 'Amazon::API::Botocore';

return Amazon::API::Botocore::paginator(@_);

}

######################################################################## sub new { ######################################################################## my ( $class, @options ) = @_;

$class = ref $class || $class;

my %options = ref $options[0] ? %{ $options[0] } : @options;

my $log_level = delete $options{log_level};
$log_level //= $options{debug} || $ENV{DEBUG} ? 'debug' : 'error';

my $self = $class->SUPER::new( \%options );

if ( $self->get_service_url_base ) {
  $self->set_service( $self->get_service_url_base );
}

croak 'service is required'
  if !$self->get_service;

$self->_set_defaults( %options, log_level => $log_level );

$self->_set_default_logger($log_level);

$self->set_log_level( log_level => $log_level );

$self->_create_methods;

if ( $self->is_botocore_api ) {
  my $serializer = Amazon::API::Botocore::Shape::Serializer->new(
    logger  => $self->get_logger,
    service => get_service_from_class( ref $self ),
  );
  $self->set_serializer($serializer);
}

return $self;

}

######################################################################## sub set_log_level { ######################################################################## my ( $self, $log_level ) = @_;

$self->set__log_level($log_level);

my $logger = $self->get_logger;

return
  if !$logger;

my $log4perl_level = $LOG4PERL_LOG_LEVELS{$log_level};

$logger->level($log4perl_level);

if ( ref($logger) =~ /Log4perl/xsm ) {
  Log::Log4perl->get_logger('Amazon::API::Botocore')->level($log4perl_level);
}

return $self;

}

######################################################################## sub get_log_level { ######################################################################## my ($self) = @_;

return $self->get__log_level();

}

######################################################################## sub service { goto &get_api_service; } ########################################################################

######################################################################## sub get_api_service { ######################################################################## my ( $service_name, @args ) = @_;

$service_name = create_module_name $service_name;

my $class = sprintf 'Amazon::API::%s', $service_name;

require_class $class;

return $class->new(@args);

}

######################################################################## sub decode_response { ######################################################################## my ( $self, $response, $serialize ) = @_;

$serialize //= $TRUE;

$response = $response || $self->get_response;

# could be Furl or HTTP?
if ( !ref $response && $response->can('content') ) {
  croak q{can't decode response - not a response object: } . ref $response;
}

my $content = $response->content;

my $content_type = $response->content_type;

# this is expected to fail if !$self->is_botocore_api
my ( $protocol, $botocore_action )
  = eval { return ( $self->get_botocore_metadata->{protocol}, $self->get_botocore_action ); };

$self->get_logger->trace(
  sub {
    return Dumper(
      [ protocol         => $protocol,
        response_content => $content
      ]
    );
  }
);

my $decoded_content;

if ($content) {

  $decoded_content = eval {
    if ( $content_type =~ /json/xmsi ) {
      decode_json($content);
    }
    elsif ( $content_type =~ /xml/xmsi ) {
      load 'XML::Simple';

      ################################################################
      # Maddening interpretation of XML output from these ass hats (-:|3
      #
      # A little more explanation: some rest-xml and ec2 protocol
      # API responses have wrapper elements that are not part of the
      # shape description...some do. It probably would have been
      # best not to try to serialize this XML object here, but
      # rather downstream when trying to create shapes from
      # it. However, the serializer now tries to figure out whether
      # to disgard the wrapper or keep it. The botocore metadata
      # could have offered some clues, however I believe the
      # serializers in Botocore have been tuned to specific APIs in
      # some cases making it difficult to fix legacy mistakes in
      # these APIs.
      ################################################################
      XML::Simple::XMLin(
        $content,
        KeepRoot      => $TRUE,
        SuppressEmpty => $FALSE,
        ForceContent  => $self->is_botocore_api && $botocore_action->{output},
        ForceArray    => ['item'],
        #          $self->get_force_array ? ( ForceArray => ['item'] ) : ()
      );
    }
  };

  # disregard content_type (it might be misleading?)
  # this is almost certainly going to be a problem somewhere...
  if ( !$decoded_content || $EVAL_ERROR ) {

    $self->get_logger->warn("unable to decode content: $EVAL_ERROR");
    $self->get_logger->warn('Trying again using JSON and XML decoders.');

    $decoded_content = eval { return decode_json($content); };

    if ( !$decoded_content || $EVAL_ERROR ) {
      load 'XML::Simple';
      $decoded_content = eval {
        return XML::Simple::XMLin(
          $content,
          SuppressEmpty => 0,  # $protocol eq 'ec2' ? $FALSE : $TRUE,
          $self->get_force_array ? ( ForceArray => ['item'] ) : ()
        );
      };
    }
  }
}

$content = $decoded_content || $content;

$self->get_logger->debug( sub { return Dumper( [ 'content' => $content ] ) } );

# we'll only have a "serializer" if this is a Botocore generated API
my $serializer = $self->get_serializer;

return $content
  if !ref $content || !$serializer || !$serialize;

my $output = $botocore_action->{output};

return $decoded_content
  if !$output;

my $orig_content = $content;

if ( $protocol && $protocol eq 'query' && ref $content eq 'HASH' ) {
  my ($root_key) = grep {/Response$/xsm} keys %{$content};
  $content = $content->{$root_key} if $root_key;
}

if ( $output->{resultWrapper} ) {
  $content = $content->{ $output->{resultWrapper} };
}

$self->get_logger->debug(
  sub {
    return Dumper(
      [ content                => $content,
        botocore_action_output => $output
      ]
    );
  }
);

$serializer->set_logger( $self->get_logger );

$content = eval {
  $serializer->serialize(
    service => get_service_from_class( ref $self ),
    shape   => $output->{shape},
    data    => $content
  );
};

# ...but this isn't necessarily where things STB
if ( !$content || $EVAL_ERROR ) {
  if ( $self->get_raise_serialization_errors ) {
    die $EVAL_ERROR;
  }
  elsif ($EVAL_ERROR) {
    carp "error serializing content: please report this error\n$EVAL_ERROR";
    $content = $orig_content;
  }
}

return $content;

}

######################################################################## sub get_botocore_action { ######################################################################## my ($self) = @_;

return $self->get_botocore_operations->{ $self->get_action };

}

######################################################################## sub is_botocore_shape { ######################################################################## my ($request) = @_;

my $shape_name = ref $request;

if ( $shape_name =~ /Botocore::Shape::([^:]+)::([^:]+)$/xsm ) {
  $shape_name = $2;
}
else {
  $shape_name = undef;
}

return $shape_name;

}

# returns the 'locationName' of the element or '' locationName for a # given parameter type (uri, querystring, etc) # # determines where to find the parameter in the input payload

######################################################################## sub is_param_type { ######################################################################## my ( $self, $shape_name, $param, $type ) = @_;

my $members = $self->get_botocore_shapes->{$shape_name}->{members};
my $member  = $members->{$param};

my $location = $member->{location};

$self->get_logger->trace(
  sub {
    return Dumper(
      [ members    => $members,
        member     => $member,
        location   => $location,
        param      => $param,
        type       => $type,
        shape_name => $shape_name
      ]
    );
  }
);

return ( $location && $location eq $type ) ? $member->{locationName} : $EMPTY;

}

######################################################################## sub is_query_param { ######################################################################## my ( $self, $shape_name, $param ) = @_;

return $self->is_param_type( $shape_name, $param, 'querystring' );

}

######################################################################## sub is_uri_param { ######################################################################## my ( $self, $shape_name, $param ) = @_;

return $self->is_param_type( $shape_name, $param, 'uri' );

}

######################################################################## sub create_botocore_request { ######################################################################## my ( $self, %args ) = @_;

my ( $parameters, $action ) = @args{qw(parameters action)};

$action //= $self->get_action;

croak 'no action'
  if !$action;

croak 'no parameters'
  if !$parameters;

croak 'not a botocore API'
  if !$self->is_botocore_api($action);

my $botocore_operations = $self->get_botocore_operations->{$action};

my $input = $botocore_operations->{input};
my $shape = $input->{shape};

my $class = require_shape( $shape, get_service_from_class($self) );

croak "could not create request shape: $shape\n$EVAL_ERROR"
  if !$class;

local $Amazon::API::Botocore::Shape::LOGGER_SOURCE = sub { $self->get_logger };

my $request = $class->new($parameters);

return $request;

}

######################################################################## sub find_content_type { ######################################################################## my ($self) = @_;

my $metadata  = $self->get_botocore_metadata;
my $protocols = $metadata->{protocols} // [ $metadata->{protocol} ];

foreach my $protocol ( @{$protocols} ) {
  next if !exists $SERVICE_CONTENT_TYPES{$protocol};

  my $content_type = $SERVICE_CONTENT_TYPES{$protocol};

  if ( $protocol eq 'json' ) {
    $content_type = sprintf '%s-%s', $content_type, $metadata->{jsonVersion};
  }

  $self->set_protocol($protocol);

  return $content_type;
}

return $EMPTY;

}

######################################################################## # init_botocore_request( $self, $request) ########################################################################

# This function will accept either an object which is a sub-class of # Amazon::API::Botocore::Shape, or a hash if the parameters have been # constructed "by-hand", # # The parameters are used to populate both the URL if some parameters # are passed in the URL and either a JSON or XML payload depending on # the API type (rest-json, rest-xml).

######################################################################## sub init_botocore_request { ######################################################################## my ( $self, $request ) = @_;

my $metadata = $self->get_botocore_metadata;

my $protocol = $metadata->{protocol};

my $content_type = $self->find_content_type // $EMPTY;
$self->set_content_type($content_type);

$request //= {};

my $action = $self->get_action;

my $botocore_operations = $self->get_botocore_operations->{$action};

my $http = $botocore_operations->{http};

my $method = $http->{method};

$self->get_logger->trace(
  sub {
    return Dumper(
      [ request             => $request,
        protocol            => $protocol,
        action              => $action,
        method              => $method,
        content_type        => $content_type,
        botocore_metadata   => $metadata,
        botocore_operations => $botocore_operations,
      ]
    );
  }
);

my $input = $botocore_operations->{input};

my $shape = $input->{shape};

my $request_shape_name = is_botocore_shape($request);

# if a shape object is passed, it must be the correct type
croak "$action requires a $shape object, not a $request_shape_name object"
  if $request_shape_name && $request_shape_name ne $shape;

# try to create a Botocore request shape
my $boto_request;

if ( !$request_shape_name && $self->is_botocore_api ) {

  $boto_request = $self->create_botocore_request( parameters => $request );

  if ( !$boto_request ) {
    croak "could not create a botocore request object\n$EVAL_ERROR\n";
  }
  else {
    $request_shape_name = is_botocore_shape($boto_request);

    $request = $boto_request;
  }
}

# some XML requests require an xmlns attribute
$self->find_namespace( $request, $input );

my %parameters;

# is the request a Botocore::Shape object? if so we can use metadata
# to create URI and payload, otherwise it's up to the caller to make
# sure the URI and the payload are correct...good luck!

if ( !$request_shape_name ) {
  %parameters = %{$request};
}
else {
  my $finalized_request = $request->finalize($protocol);

  $self->get_logger->debug(
    sub {
      return Dumper(
        [ request           => $request,
          finalized_request => $finalized_request,
        ]
      );
    }
  );

  if ( $protocol =~ /rest\-(xml|json)/xsm ) {
    $finalized_request = { $request_shape_name => $finalized_request };
  }
  elsif ( $protocol eq 'ec2' ) {
    $finalized_request = ref $finalized_request ? [ param_n($finalized_request) ] : [];
    $self->set_http_method($method);
    return $finalized_request;
  }
  elsif ( $protocol eq 'query' ) {
    $finalized_request = ref $finalized_request ? [ query_param_n($finalized_request) ] : [];
    $self->set_http_method($method);
    return $finalized_request;
  }

  $self->get_logger->debug(
    sub {
      return Dumper( [ finalized_request => $finalized_request, ] );
    }
  );

  if ( ref($finalized_request) eq 'HASH' ) {
    %parameters = %{$finalized_request};
  }
}

$self->set_http_method($method);

my $uri;

if ( $protocol =~ /^rest\-(json|xml)/xsm ) {
  my @args = @{ $http->{parsed_request_uri}->{parameters} // [] };

  my $request_uri_tpl = $http->{parsed_request_uri}->{request_uri_tpl};

  $self->get_logger->debug(
    sub {
      return Dumper(
        [ args               => \@args,
          request_uri_tpl    => $request_uri_tpl,
          request_shape_name => $request_shape_name,
          input              => $input,
        ]
      );
    }
  );

  # if the request is a shape, we've already checked for required
  # parameters but some may be buried in the payload!
  if ( !$request_shape_name ) {
    foreach my $p (@args) {
      croak 'required parameter ' . $p . ' not found.'
        if !exists $parameters{$p};
    }

    $uri = sprintf $request_uri_tpl, @parameters{@args};
    $self->set_request_uri($uri);

    delete @parameters{@args};
  }
  else {
    $uri = $http->{requestUri};  # use the Botocore template

    my $shape_parameters = $parameters{$shape};

    $self->get_logger->debug(
      sub {
        return Dumper(
          [ requestUri       => $uri,
            shape_parameters => $shape_parameters,
            shape            => $shape,
            parameters       => \%parameters,
          ]
        );
      }
    );

    foreach my $p ( keys %{$shape_parameters} ) {
      $self->get_logger->trace( Dumper( [ parameter => $p ] ) );

      if ( my $var = $self->is_uri_param( $request_shape_name, $p ) ) {

        my $val = $shape_parameters->{$p};

        $self->get_logger->trace(
          Dumper(
            [ var => $var,
              val => $val,
            ]
          )
        );

        $uri =~ s/[{]$var[}]/$val/xsm;

        delete $shape_parameters->{$p};
      }
    }

    # we're not done yet...just to make things interesting, some
    # APIs embed request parameters in the uri, payload AND query
    # string!

    if ($request_shape_name) {
      my %query_parameters;

      my $shape_parameters = $parameters{$shape};

      foreach my $p ( keys %{$shape_parameters} ) {
        if ( my $var = $self->is_query_param( $request_shape_name, $p ) ) {
          $query_parameters{$var} = $shape_parameters->{$p};
          delete $shape_parameters->{$p};
        }
      }

      if ( keys %query_parameters ) {
        # $self->set_content_type(undef);

        $uri = sprintf '%s?%s', $uri, create_urlencoded_content( \%query_parameters );
      }

      if ( !keys %{$shape_parameters} ) {
        %parameters = ();
      }
    }
  }

  $self->set_request_uri($uri);

  # payload (I think) tells us where the actual object to pass will
  # be found in the request object. Since we may be formatting an
  # XML request we need to insert the namespace so our formatter can
  # properly serialize the request
  if ( $request->{payload} && $parameters{$request_shape_name}->{ $request->{payload} } ) {
    %parameters = %{ $parameters{$request_shape_name} };
    if ( $self->get_namespace ) {
      $parameters{ $request->{payload} }->{_attr} = { xmlns => $self->get_namespace };
    }
  }
  elsif ( $self->get_namespace ) {
    my $locationName = $input->{locationName};
    $parameters{$locationName}->{_attr} = { xmlns => $self->get_namespace };
  }

}

my $content = \%parameters;

if ( $method ne 'POST' && !keys %parameters ) {
  $content = undef;
}

$self->get_logger->debug(
  sub {
    return Dumper [
      namespace   => $self->get_namespace,
      parameters  => \%parameters,
      request_uri => $uri,
      content     => $content,
    ];
  }
);

return $content;

}

######################################################################## # namespaces may be buried a bit in the members specfication of # request objects or can be part of the specification for the request # method itself (cloudfront.CreateInvalidationBatch) ######################################################################## sub find_namespace { ######################################################################## my ( $self, $request, $input ) = @_;

my ($has_namespace) = grep { defined $_ }
  map { $request->{members}->{$_}->{xmlNamespace}->{uri} } keys %{ $request->{members} // {} };

if ( !$has_namespace && $input->{xmlNamespace} ) {
  $has_namespace = $input->{xmlNamespace}->{uri};
}

$self->set_namespace($has_namespace);

return;

}

######################################################################## sub is_botocore_api { ######################################################################## my ($self) = @_;

return defined $self->get_botocore_metadata;

}

######################################################################## sub serialize_content { ######################################################################## my ( $self, $parameters ) = @_;

my $content      = $parameters;
my $action       = $self->get_action;
my $version      = $self->get_version;
my $content_type = $self->get_content_type;

$self->get_logger->trace(
  sub {
    return Dumper(
      [ content_type => $content_type,
        parameters   => $parameters,
        service      => $self->get_service,
      ]
    );
  }
);

# if the API is a query API, url encode parameters
if ( any { $_ eq lc $self->get_service } @{ $API_TYPES{query} } ) {
  $parameters //= [];
  $content = create_urlencoded_content( $parameters, $action, $version );
}
elsif ( $parameters && ref $parameters && reftype($parameters) eq 'HASH' ) {
  if ( $content_type =~ /json/xsm ) {
    delete $parameters->{_attr};
    $content = encode_json($parameters);
  }
  elsif ( $content_type =~ /xml/xms ) {
    return
      if !ref $content || !keys %{$content};

    $content = $self->generate_xml($parameters);
  }
}

return $content;

}

######################################################################## # invoke_api( action, parameters, content-type, headers) ######################################################################## sub invoke_api { ######################################################################## my ( $self, @args ) = @_;

my ( $action, $parameters, $content_type, $headers )
  = $self->_unpack_args( \@args, qw(action parameters content_type headers) );

# use_botocore overrides the creation of a Botocore request
# object. In that case the arguments are expected to conform to the
# format specified for this particular AWS API.
my $use_botocore = choose {
  return $FALSE
    if $parameters && !ref $parameters;

  return $FALSE
    if $parameters && reftype($parameters) ne 'HASH';

  return $self->is_botocore_api;
};

my $protocol = eval { $self->get_botocore_metadata->{protocol}; };

$self->get_logger->debug(
  sub {
    return Dumper(
      [ parameters      => $parameters,
        'content-type'  => $content_type,
        protocol        => $protocol,
        use_botocore    => $use_botocore,
        is_botocore_api => $self->is_botocore_api,
      ]
    );
  }
);

$self->set_action($action);
$self->set_last_action($action);
$self->set_error(undef);

# Check to see if caller is sending a blessed object, possibly a
# Botocore request object using a class that does not have the
# Botocore metadata
croak sprintf qq{"%s" was not generated with Botocore support.\nParameters should be simple objects, not blessed.\n},
  ref $self
  if blessed($parameters) && !$self->is_botocore_api;

# auto pagination is only available when using the Botocore metadata
my $paginator = $self->_init_paginator( $use_botocore, $action );

my $limit_key = $paginator ? $paginator->{limit_key}   : undef;
my $limit     = $limit_key ? $parameters->{$limit_key} : undef;

# save this for pagination
my $original_content = $parameters;

# set content type and parameters
if ($use_botocore) {
  $parameters   = $self->init_botocore_request($parameters);
  $content_type = $self->get_content_type;
}
else {
  # try to automatically set content type
  $content_type ||= $self->set_content_type( $self->_set_content_type );

  if ( !$parameters && $content_type =~ /json/xsm ) {
    $parameters = {};
  }
}

my $serialized_content = $self->serialize_content($parameters);

$self->get_logger->debug(
  sub {
    Dumper [
      content_type       => $content_type,
      parameters         => $parameters,
      serialized_content => $serialized_content
    ];
  }
);

my $page_count = 0;
my @paged_results;

while ($TRUE) {

  ++$page_count;

  $self->get_logger->debug( sub { return Dumper( [ page => $page_count ] ) } );

  my $rsp = $self->submit(
    content      => $serialized_content,
    content_type => $content_type,
    headers      => $headers,
  );

  $self->get_logger->debug(
    sub {
      return Dumper(
        [ decode_always => $self->get_decode_always,
          content       => $rsp->content
        ]
      );
    }
  );

  $self->set_response($rsp);

  last
    if !$self->_check_response($rsp);

  # non-botocore or raw content path
  if ( !$paginator ) {
    return $rsp->content
      if !$self->get_decode_always;

    return $self->decode_response;
  }

  my $result = $self->decode_response;

  $self->get_logger->debug(
    sub {
      return Dumper(
        [ paginator     => $paginator,
          paged_results => @paged_results,
          result        => $result
        ]
      );
    }
  );

  my $actual_result = dig( $result, $paginator, 'result_key' );

  last if !$actual_result;

  push @paged_results, @{$actual_result};

  $self->get_logger->debug(
    sub {
      return Dumper(
        [ result        => $actual_result,
          paged_results => \@paged_results,
          more_results  => $paginator->{more_results},
          paginator     => $paginator,
        ]
      );
    }
  );

  last if !dig( $result, $paginator, 'more_results' );

  $limit //= $result->{$limit_key};

  $serialized_content = $self->serialize_content(
    $self->init_botocore_request(
      { %{$original_content},
        $limit ? ( $limit_key => $limit ) : (),
        $paginator->{input_token} => dig( $result, $paginator, 'output_token' ),
      }
    )
  );

}

return bury( \@paged_results, $paginator, 'result_key' );

}

######################################################################## # the analog to dig, we need to set the result to a hash element specified # by a dot encoded string. Example: DistributionList.Item ######################################################################## sub bury { ######################################################################## my ( $result, $paginator, $key ) = @_;

return $result
  if !$paginator;

$key = $paginator->{$key};

return { $key => $result }
  if $key !~ /[.]/xsm;

my $parent = {};
my $child  = $parent;

my @keys = split /[.]/xsm, $key;

my $result_key = pop @keys;  # remove last element

foreach (@keys) {
  $child->{$_} = {};
  $child = $child->{$_};
}

$child->{$result_key} = $result;

return $parent;

}

######################################################################## # follow the . encoded key to find the hash element # example: DistributionList.Items ######################################################################## sub dig { ######################################################################## my ( $result, $paginator, $key, $delete ) = @_;

return $result
  if !$paginator || !$paginator->{$key};

$key = $paginator->{$key};

return $result->{$key}
  if $key !~ /[.]/xsm;

foreach ( split /[.]/xsm, $key ) {
  $result = $result->{$_};
  $key    = $_;
}

if ($delete) {
  delete $result->{$key};
}

return $result;

}

######################################################################## sub print_error { ######################################################################## my ($self) = @_;

my $error = eval {
  my $last_error = $self->get_error;

  return "$last_error"
    if $last_error && ref($last_error) =~ /Amazon::API::Error/xms;

  return $last_error;
};

$error //= $EVAL_ERROR;

if ($error) {
  print {*STDERR} $error;
}

return $error;

}

######################################################################## sub get_valid_token { ######################################################################## my ($self) = @_;

my $credentials = $self->get_credentials;

return
  if !$credentials->get_token;

if ( $credentials->can('is_token_expired') ) {
  if ( $credentials->is_token_expired ) {
    croak 'token expired'
      if !$credentials->can('refresh_token') || !$credentials->refresh_token;
  }
}

$self->get_logger->trace( sub { return Dumper( [ 'valid token:', $credentials->get_token ] ) } );

return $credentials->get_token;

}

######################################################################## sub submit { ######################################################################## my ( $self, %options ) = @_;

$self->get_logger->debug( sub { return Dumper [ submit => \%options ] } );

my $method  = $self->get_http_method || 'POST';
my $headers = $options{headers}      || [];

my $url = $self->get_url;

my $botocore_protocol = eval { $self->get_botocore_metadata->{protocol} } // $EMPTY;

if ( $botocore_protocol =~ /^rest\-(json|xml)/xsm ) {
  croak 'no request URI provided for rest-json call'
    if !$self->get_request_uri;

  $url .= $self->get_request_uri;
}

$self->get_logger->debug(
  sub {
    return Dumper [
      method  => $method,
      url     => $url,
      headers => $headers
    ];
  }
);

my $request = HTTP::Request->new( $method, $url, $headers );

# 1. set the header
# 2. set the content
# 3. sign the request
# 4. send the request & return result

# see IMPLEMENTATION NOTES for an explanation
if ( $self->get_api || $self->get_target_prefix ) {
  $request = $self->_set_x_amz_target($request);
}

$self->_set_request_content( request => $request, %options );
my $credentials = $self->get_credentials;

if ( my $token = $self->get_valid_token ) {
  $request->header( 'X-Amz-Security-Token' => $token );
}

# TODO: global end-points
my $region = $self->get_region;

# sign the request
Amazon::API::Signature4->new(
  -access_key     => $credentials->get_aws_access_key_id,
  -secret_key     => $credentials->get_aws_secret_access_key,
  -security_token => $credentials->get_token || undef,
  service         => $self->get_service,
  region          => $region,
)->sign( $request, $self->get_region );

$self->get_logger->debug( sub { return Dumper( [ request => $request ] ) } );

# make the request, return response object
my $ua  = $self->get_user_agent;
my $rsp = $ua->request($request);

$self->get_logger->debug(
  sub {
    return Dumper [ response => $rsp ];
  }
);

return $rsp;

}

# +------------------+ # | EXPORTED METHODS | # +------------------+

######################################################################## sub generate_xml { ######################################################################## my ( $self, $data ) = @_;

load 'XML::LibXML';

my $doc = XML::LibXML::Document->new( '1.0', 'UTF-8' );

my $fragment = XML::LibXML::DocumentFragment->new();

for my $key ( keys %{$data} ) {
  _to_xml( $doc, $fragment, $key, $data->{$key} );
}

my $xml = $doc->toString(1) . $fragment->toString(1);

$self->get_logger->trace(
  sub {
    return Dumper(
      [ data => $data,
        xml  => $xml
      ]
    );
  }
);

return $xml;

}

######################################################################## sub query_param_n { ######################################################################## my (@args) = @_;

return Amazon::API::Botocore::Shape::Utils::query_param_n(@args);

}

######################################################################## sub param_n { ######################################################################## my (@args) = @_;

return Amazon::API::Botocore::Shape::Utils::param_n(@args);

}

######################################################################## # create_urlencoded_content(parameters, action, version) # input: # parameters: # SCALAR - query string to encode (x=y&w=z...) # ARRAY - either an array of hashes or... # key/value pairs of the form x=y or... # key/values pairs # HASH - key/value pairs, if the value is an array then # it is assumed to be a list of hashes # action: API method # version: wsdl version for API # # output: # URL encodode query string # ######################################################################## sub create_urlencoded_content { ######################################################################## my ( $parameters, $action, $version ) = @_;

my @args;

if ( $parameters && !ref $parameters ) {
  @args = map { split /=/xsm } split /&/xsm, $parameters;
}
elsif ( $parameters && reftype($parameters) eq 'HASH' ) {
  foreach my $key ( keys %{$parameters} ) {
    if ( ref $parameters->{$key}
      && reftype( $parameters->{$key} ) eq 'ARRAY' ) {
      push @args, map { %{$_} } @{ $parameters->{$key} };
    }
    else {
      push @args, $key, $parameters->{$key};
    }
  }
}
elsif ( $parameters && reftype($parameters) eq 'ARRAY' ) {

  # if any are refs then they should be hashes...
  if ( any {ref} @{$parameters} ) {

    @args = map { %{$_} } @{$parameters};  # list of hashes
  }
  elsif ( any {/=/xsm} @{$parameters} ) {
    @args = map { split /=/xsm } @{$parameters};  # formatted list
  }
  else {
    @args = @{$parameters};  # simple list
  }
}

my $content;

if ( $action && !any { $_ eq 'Action' } @args ) {
  push @args, 'Action', $action;
}

if ( $version && !any {/Version/xsm} @args ) {
  push @args, 'Version', $version;
}

return join $AMPERSAND, map { sprintf '%s=%s', $_->[0], url_encode( $_->[1] ) } pairs @args;

}

######################################################################## sub has_keys { ######################################################################## my ( $self, %options ) = @_;

# note that self should NOT really have keys! Not sure why I added this test
my %creds = keys %options ? %options : map { $_ => $self->{$_} } @REQUIRED_KEYS;

return $creds{aws_secret_access_key} && $creds{aws_access_key_id};

}

# +-----------------+ # | PRIVATE METHODS | # +-----------------+

######################################################################## sub _unpack_args { ######################################################################## my ( $self, $args, @var_names ) = @_;

$self->get_logger->debug(
  sub {
    return Dumper( [ args => $args ] );
  }
);

return @{$args}
  if @{$args} > 1 || !ref $args->[0];

return @{ $args->[0] }{@var_names};

}

######################################################################## sub _check_response { ######################################################################## my ( $self, $rsp ) = @_;

return $TRUE
  if $rsp->is_success;

$self->set_error(
  Amazon::API::Error->new(
    { error        => $rsp->code,
      message_raw  => $rsp->content,
      content_type => scalar $rsp->content_type,
      api          => ref $self,
      response     => $rsp,
      action       => $self->get_last_action,
    }
  )
);

die $self->get_error
  if $self->get_raise_error;

if ( $self->get_print_error ) {
  print {*STDERR} $self->get_error;
}

return;

}

######################################################################## sub _init_paginator { ######################################################################## my ( $self, $use_botocore, $action ) = @_;

return if !$use_botocore || !$self->get_use_paginator || !$self->get_decode_always;

my $paginator = $self->get_paginators && $self->get_paginators->{$action};

return
  if !$paginator;

$paginator->{more_results} //= $paginator->{output_token};

return $paginator;

}

# should not be called if we have a Botocore definition ######################################################################## sub _set_content_type { ######################################################################## my ($self) = @_;

my $service = $self->get_service;

# default content-type
my $content_type = $self->get_content_type;

return 'application/x-www-form-urlencoded'
  if any { $_ eq $service } @{ $API_TYPES{query} };

return 'application/x-amz-json-1.1'
  if any { $_ eq $service } @{ $API_TYPES{json} };

return 'application/xml'
  if any { $_ eq $service } @{ $API_TYPES{xml} };

return $content_type;

}

######################################################################## sub _create_methods { ######################################################################## my ($self) = @_;

my $class = ref $self || $self;

if ( $self->get_api_methods ) {
  no strict 'refs'; ## no critic (TestingAndDebugging::ProhibitNoStrict)
  no warnings 'redefine'; ## no critic (TestingAndDebugging::ProhibitNoWarnings)

  my $stash = \%{ __PACKAGE__ . $DOUBLE_COLON };

  foreach my $api ( @{ $self->get_api_methods } ) {

    my $method = lcfirst $api;

    $method =~ s/([[:lower:]])([[:upper:]])/$1_$2/xmsg;
    $method = lc $method;

    my $snake_case_method = $class . $DOUBLE_COLON . $method;
    my $camel_case_method = $class . $DOUBLE_COLON . $api;

    # snake case rules the day

    if ( !$stash->{$method} ) {
      *{$snake_case_method} = sub {
        my $self = shift;

        $self->invoke_api( $api, @_ );
      };
    }

    # ...but some prefer camels
    if ( !$stash->{$api} ) {
      *{$camel_case_method} = sub {
        my $self = shift;

        $self->$method(@_);
      };
    }
  }

}

return $self;

}

######################################################################## sub _set_default_logger { ######################################################################## my ( $self, $log_level_string ) = @_;

if ( $self->get_no_logger ) {
  $self->set_logger( Amazon::API::NullLogger->new );
  return $self;
}

load 'Log::Log4perl';

my $log_level_numeric = $LOG4PERL_LOG_LEVELS{ lc $log_level_string } // $LOG4PERL_LOG_LEVELS{info};
my $log_level_name    = uc $log_level_string;

my $pattern_layout = $self->get_log4perl_layout // $DEFAULT_LOG4PERL_LAYOUT;

if ( !Log::Log4perl->initialized ) {

  my $appender_config_str;

  # 2. Determine Appender Configuration as a string
  my $file = $self->get_log_file;

  if ( defined $file && $file ne q{} ) {
    $appender_config_str = qq{
              log4perl.appender.ROOT_APPENDER = Log::Log4perl::Appender::File
              log4perl.appender.ROOT_APPENDER.filename = $file
              log4perl.appender.ROOT_APPENDER.mode = append
          };
  }
  else {
    $appender_config_str = qq{
              log4perl.appender.ROOT_APPENDER = Log::Log4perl::Appender::Screen
              log4perl.appender.ROOT_APPENDER.stderr = 1
          };
  }

  # 3. Construct the full text blob configuration
  my $config_blob = qq{
          # Root Logger
          log4perl.rootLogger = $log_level_name, ROOT_APPENDER

          # Appender Definition (dynamic)
          $appender_config_str

          # Layout Definition
          log4perl.appender.ROOT_APPENDER.layout = Log::Log4perl::Layout::PatternLayout
          log4perl.appender.ROOT_APPENDER.layout.ConversionPattern = $pattern_layout
      };

  Log::Log4perl::init( \$config_blob );  # Pass a reference to the scalar string
}

my $logger = Log::Log4perl->get_logger( ref $self );
$logger->level($log_level_numeric);

$self->set_logger($logger);

return $self;

}

######################################################################## sub _set_defaults { ######################################################################## my ( $self, %options ) = @_;

$self->set_raise_error( $self->get_raise_error     // $TRUE );
$self->set_print_error( $self->get_print_error     // $TRUE );
$self->set_use_paginator( $self->get_use_paginator // $TRUE );
$self->set_decode_always( $self->get_decode_always // $TRUE );
$self->set_user_agent( $self->get_user_agent       // Amazon::API::HTTP::UserAgent->new );
$self->set_protocol( $self->get_protocol()         // 'https' );
$self->set_http_method( $self->get_http_method     // 'POST' );  # most API services are POST

my $region = $self->get_region;

# note some APIs are global, hence an API may send '' to indicate global
if ( !$region ) {
  $region = $ENV{AWS_REGION} || $ENV{AWS_DEFAULT_REGION} || $DEFAULT_REGION;
  $self->set_region($region);
}

my $log_level = $options{log_level};

if ( !$log_level || $log_level !~ /\A(?:debug|trace)\z/xsm ) {
  $self->set_no_logger($TRUE);
}

else {
  $self->set_log_level($log_level);
}

if ( !$self->get_credentials ) {
  if ( $self->has_keys(%options) ) {
    $self->set_credentials(
      Amazon::Credentials->new(
        { aws_secret_access_key => $options{aws_secret_access_key},
          aws_access_key_id     => $options{aws_access_key_id},
          token                 => $options{token},
          region                => $region,
          no_passkey_warning    => $options{no_passkey_warning},
        }
      )
    );
  }
  else {
    $self->set_credentials(
      Amazon::Credentials->new(
        order              => $options{order},
        region             => $region,
        no_passkey_warning => $options{no_passkey_warning},
      )
    );
  }
}

# set URL last since it contains region
$self->_set_url;

return $self;

}

######################################################################## sub _create_service_url { ######################################################################## my ($self) = @_;

my $url;

my $botocore_metadata = $self->get_botocore_metadata;
my $service           = $self->get_service;

if ( $botocore_metadata && $botocore_metadata->{globalEndpoint} ) {
  $url = sprintf '%s://%s', $self->get_protocol, $botocore_metadata->{globalEndpoint};
}
else {
  my $endpoint = $self->get_endpoint_prefix || $service;

  if ( any { $_ eq $service } @GLOBAL_SERVICES ) {
    $url = sprintf $GLOBAL_URL_FMT, $self->get_protocol, $endpoint;
  }
  # this could probably be an else since we always set region to a default
  elsif ( $self->get_region ) {
    $url = sprintf $REGIONAL_URL_FMT, $self->get_protocol, $endpoint, $self->get_region;
  }
  else {
    $url = sprintf $REGIONAL_URL_FMT, $self->get_protocol, $endpoint, 'us-east-1';
  }
}

return $url;

}

######################################################################## sub _set_url { ######################################################################## my ($self) = @_;

my $url = $self->get_url;

if ( !$url ) {
  $url = $self->_create_service_url;
}
else {
  if ( $url !~ /^https?/xmsi ) {
    $url =~ s/^\///xms;  # just remove leading slash...
    $url = $self->get_protocol . '://' . $url;
  }

}

$self->set_url($url);

return $self;

}

######################################################################## sub _set_x_amz_target { ######################################################################## my ( $self, $request ) = @_;

my $target  = $self->get_target_prefix;
my $version = $self->get_version;
my $api     = $self->get_api;
my $action  = $self->get_action;

if ( !$target ) {
  $target = $version ? $api . $UNDERSCORE . $version : $api;
}

$target = $target . $DOT . $action;

$self->set_target($target);

$request->header( 'X-Amz-Target' => $target );

return $request;

}

######################################################################## sub _set_request_content { ######################################################################## my ( $self, %args ) = @_;

my $request      = $args{request};
my $content      = $args{content};
my $content_type = $args{content_type} || $self->get_content_type;

$self->get_logger->trace(
  sub {
    return Dumper(
      [ method       => $self->get_http_method,
        args         => \%args,
        request      => $request,
        content      => $content,
        content_type => $content_type
      ]
    );
  }
);

if ( $self->get_http_method ne 'GET' || !defined $content ) {
  if ($content_type) {
    $request->content_type( $content_type . '; charset=utf-8' );
  }
  $request->content($content);
}
else {
  $request->uri( $request->uri . $QUESTION_MARK . $content );
}

$self->get_logger->debug( sub { return Dumper( [ request => $request ] ); } );

return $request;

}

######################################################################## # Convert a Perl data object to XML ######################################################################## sub _to_xml { ######################################################################## my ( $doc, $parent, $key, $value ) = @_;

# Handle array references (multiple child elements)
if ( ref $value eq 'ARRAY' ) {
  for my $item ( @{$value} ) {
    _to_xml( $doc, $parent, $key, $item );
  }
}
# Handle hash references (nested structures)
elsif ( ref $value eq 'HASH' ) {
  my $element = $doc->createElement($key);

  # Handle attributes (if _attr key exists)
  if ( exists $value->{_attr} ) {
    my $attrs = delete $value->{_attr};
    for ( keys %{$attrs} ) {
      $element->setAttribute( $_, $attrs->{$_} );
    }
  }

  # Process nested elements
  for my $subkey ( keys %{$value} ) {
    _to_xml( $doc, $element, $subkey, $value->{$subkey} );
  }

  $parent->appendChild($element);
}
# Handle scalar values (text nodes)
else {
  my $element = $doc->createElement($key);
  $element->appendTextNode($value);
  $parent->appendChild($element);
}

return;

}

1;

__END__

[![amazon-api](https://github.com/rlauer6/perl-Amazon-API/actions/workflows/build.yml/badge.svg)](https://github.com/rlauer6/perl-Amazon-API/actions/workflows/build.yml)

Generic class for constructing AWS API interfaces. Typically used as a parent class, but can be used directly. This package can also generates stubs for Amazon APIs using the Botocore project metadata. (See "BOTOCORE SUPPORT").

The typical use of this is API is through the classes you build with the included tool (amazon-api). The tool leverages the Botocore project's metadata to build classes that are specific to each API (and are documented in the perlish way). Using Amazon::API directly may not work in all circumstances unless you are very familiar with the API you are calling. If you decide to take the Luddite approaches, read the documentation carefully before using Amazon::API.

BACKGROUND AND MOTIVATION

A comprehensive Perl interface to AWS services similar to the Botocore library for Python has been a long time in coming. The Paws project has been creating an always up-to-date AWS interface with community support. If you are looking for an extensible method of installing and invoking a subset of services you might want to consider Amazon::API.

Think of this class as a DIY kit for installing only the APIs and methods you need for your AWS project. Using the included amazon-api utility you can also roll your own complete Amazon API classes that include support for serializing requests and responses based on metadata provided by the Botocore project. The classes you create with amazon-api include full documentation as pod. (See "BOTOCORE SUPPORT" for more details).

NOTE: The original Amazon::API was written in 2017 as a very lightweight way to call a handfull of APIs. The evolution of the module was based on discovering, without much documentation or help, the nature of Amazon APIs. In retrospect, even back then, it would have been easier to consult the Botocore project and decipher how that project managed to create a library from the metadata. Fast forward to 2022 and Amazon::API began using the Botocore metadata in order to, in most cases, correctly call any AWS service. The Amazon::API module can still be used without the assistance of Botocore metadata, but it works a heckuva lot better with it.

You can use Amazon::API in 3 different ways:

Take the Luddite approach

my $queues = Amazon::API->new(
 {
  service     => 'sqs',
  http_method => 'GET'
 })->invoke_api('ListQueues');

Build your own API classes with just what you need

package Amazon::API::SQS;

use strict;
use warnings;

use parent qw( Amazon::API );

our @API_METHODS = qw(
  ListQueues
  PurgeQueue
  ReceiveMessage
  SendMessage
);

sub new {
  my ( $class, @options ) = @_;
  $class = ref($class) || $class;

  my %options = ref( $options[0] ) ? %{ $options[0] } : @options;

  return $class->SUPER::new(
    { service       => 'sqs',
      http_method   => 'GET',
      api_methods   => \@API_METHODS,
      decode_always => 1,
      %options
    }
  );
}

1;

use Amazon::API::SQS;
use Data::Dumper;

my $sqs = Amazon::API::SQS->new;

print {*STDERR} Dumper($sqs->ListQueues);

Use the Botocore metadata to build classes for you

amazon-api -s sqs create-stubs
amazon-api -s sqs create-shapes

perl -I . -MData::Dumper -MAmazon::API:SQS -e 'print Dumper(Amazon::API::SQS->new->ListQueues);'

NOTE: In order to use Botocore metadata you must clone the Botocore repository and point the utility to the repo.

Clone the Botocore project from GitHub:

mkdir ~/git
cd git
git clone https://github.com/boto/botocore.git

Generate stub classes for the API and shapes:

amazon-api -b ~/git/botocore -s sqs -o ~/lib/perl5 create-stubs
amazon-api -b ~/git/botocore -s sqs -o ~/lib/perl5 create-shapes

perldoc Amazon::API::SQS

See Amazon::API::Botocore::Pod for more details regarding building stubs and shapes.

THE APPROACH

Essentially, most AWS APIs are RESTful services that adhere to a common protocol, but differences in services make a single solution difficult. All services more or less adhere to this framework:

Specific details of the more recent AWS services are well documented, however early services were usually implemented as simple HTTP services that accepted a query string. This module attempts to account for most of the nuances involved in invoking AWS services and provide a fairly generic way of invoking these APIs in the most lightweight way possible.

Using Amazon::API as a generic, lightweight module, naturally does not provide nuanced support for individual AWS services. To use this class in that manner for invoking the AWS APIs, you need to be very familiar with the specific API requirements and responses and be willng to invest time reading the documentation on Amazon's website. The payoff is that you can probably use this class to call any AWS API without installing a large number of dependencies.

If you don't mind a few extra dependencies and overhead, you should generate the stub APIs and support classes using the amazon-api utility. The stubs and shapes produced by the utility will serialize and deserialize requests and responses correctly by using the Botocore metadata. Botocore metadata provides the necessary information to create classes that can successfully invoke all of the Amazon APIs.

A good example of creating a quick and dirty interface to CloudWatch Events can be found here:

Amazon::CloudWatchEvents

And invoking some of the APIs can be as easy as:

Amazon::API->new(
  service     => 'sqs',
  http_method => 'GET'
}
)->invoke_api('ListQueues');

BOTOCORE SUPPORT

Using Botocore metadata and the utilities in this project, you can create Perl classes that simplify calling AWS services. After creating service classes and shape objects from the Botocore metadata calling AWS APIs will look something like this:

use Amazon::API::SQS;

my $sqs = Amazon::API::SQS->new;
my $rsp = $sqs->ListQueues();

The Amazon::API::Botocore module augments Amazon::API by using Botocore metadata for determining how to call individual services and serialize parameters passed to its API methods. A utility (amazon-api) is provided that can generate Perl classes for all AWS services using the Botocore metadata.

Perl classes that represent AWS data structures (aka shapes) that are passed to or returned from services can also be generated. These classes allow you to call all of the API methods for a given service using simple Perl objects that are serialized correctly for a specific method.

Service classes are subclassed from Amazon::API so their new() constructor takes the same arguments as Amazon::API::new().

my $credentials = Amazon::Credential->new();

my $sqs = Amazon::API::SQS->new( credentials => $credentials );

If you are going to use the Botocore support and automatically generate API classes you must also create the data structure classes that are used by each service. The Botocore based APIs will use these classes to serialize requests and responses.

For more information on generating API classes, see Amazon::API::Botocore::Pod.

Response Serialization

With little documentation to go on, interpretting the Botocore metadata and deducing how to serialize Botocore shapes (using a single serializer) from Perl objects has been a difficult task. It's likely that there are still some edge cases and bugs lurking in the serialization methods. Accordingly, starting with version 1.4.5, serialization exceptions or exceptions that occur while attempting to decode a response, will result in the raw response being returned to the caller. The idea being that getting something back that allows you figure out what to do with the response might be better than receiving an error.

OTOH, you might want to see the error, report it, or possibly contribute to its resolution. You can prevent errors from being surpressed by setting the raise_serializtion_errors to a true value. The default is false.

Throughout the rest of this documentation a request made using one of the classes created by the Botocore support scripts will be referred to as a Botocore request or Botocore API.

Starting with version 2.0.12 serialization has become much more reliable, but there are still some differences in the way the Python Botocore library serialize responses. For example, some serializers may include or exclude members that are not present in the response payload. If you are testing a response element, the best approach is to first test the truthiness and then test the presence of content.

if ( $result->{$key} && @{$result->{$key}} ) 

if ( $result->{$key} && %{result->{$key}} )

ERRORS

When an error is returned from an API request, an exception class (Amazon::API::Error) will be raised if raise_error has been set to a true value (the default). If you set print_error to true AND raise_error is false, then errors will be printed to STDERR.

See Amazon::API::Error for more details.

METHODS AND SUBROUTINES

Reminder: You can mostly ignore this part of the documentation when you are leveraging Botocore to generate your API classes.

new

new(options)

All options are described below. options can be a list of key/values or hash reference.

invoke_api

invoke_api(action, [parameters], [content-type], [headers]);

or using named parameters...

invoke_api({ action => args, ... } )

Invokes the API with the provided parameters.

Note: This method consults the raise_error and print_error options to determine how errors are handled.

decode_response

Boolean that indicates whether or not to deserialize the most recent response from an invoked API based on the Content-Type header returned. If there is no Content-Type header, then the method will try to decode it first as a JSON string and then as an XML string. If both of those fail, the raw content is returned.

You can enable or disable deserializing responses globally by setting the decode_always attribute when you call the new constructor.

default: true

By default, Amazon::API will retrieve all results for Botocore based API calls that require pagination. To turn this behavior off, set use_paginator to a false value when you instantiate the API service.

my $ec2 = Amazon::API->new(use_paginator => 0);

print_error

Prints a formatted version of the last error encountered to STDERR.

submit

submit(options)

This method is used internally by invoke_api and normally should not be called by your applications.

options is a reference to a hash of options:

generate_xml

generate_xml(object)

Generates XML from a Perl object (uses XML::LibXML). This seems to do a much better job than XMLout() in allowing a mix of attributes and nested objects. With XMLout() you need to choose between allowing attributes (which we need to add the namespace for certain requests) and nested elements (NoAttr => 1).

EXPORTED METHODS

get_api_service

get_api_service(api, options)

Convenience routine that will return an API instance.

my $sqs = get_api_service 'sqs';

Equivalent to:

require Amazon::API::SQS;

my $sqs = Amazon::API::SQS->new(%options);

create_url_encoded_content

create_urlencoded_content(parameters, action, version)

Returns a URL encoded query string. parameters can be any of SCALAR, ARRAY, or HASH. See below.

paginator

paginator(service, api, request)

Returns an array containing the results of an API call that requires pagination,

my $result = paginator($ec2, 'DescribeInstances', { MaxResults => 10 });

param_n

param_n(parameters)

Format parameters in the "param.n" notation.

parameters should be a hash or array reference.

A good example of a service that uses this notation is the SendMessageBatch SQS API call.

The sample request can be found here:

SendMessageBatch

https://sqs.us-east-2.amazonaws.com/123456789012/MyQueue/
?Action=SendMessageBatch
&SendMessageBatchRequestEntry.1.Id=test_msg_001
&SendMessageBatchRequestEntry.1.MessageBody=test%20message%20body%201
&SendMessageBatchRequestEntry.2.Id=test_msg_002
&SendMessageBatchRequestEntry.2.MessageBody=test%20message%20body%202
&SendMessageBatchRequestEntry.2.DelaySeconds=60
&SendMessageBatchRequestEntry.2.MessageAttribute.1.Name=test_attribute_name_1
&SendMessageBatchRequestEntry.2.MessageAttribute.1.Value.StringValue=test_attribute_value_1
&SendMessageBatchRequestEntry.2.MessageAttribute.1.Value.DataType=String
&Expires=2020-05-05T22%3A52%3A43PST
&Version=2012-11-05
&AUTHPARAMS

To produce this message you would pass the Perl object below to param_n():

my $message = {
  SendMessageBatchRequestEntry => [
    { Id          => 'test_msg_001',
      MessageBody => 'test message body 1'
    },
    { Id               => 'test_msg_002',
      MessageBody      => 'test message body 2',
      DelaySeconds     => 60,
      MessageAttribute => [
        { Name  => 'test_attribute_name_1',
          Value =>
            { StringValue => 'test_attribute_value_1', DataType => 'String' }
        }
      ]
    }
  ]
};

CAVEATS

IMPLEMENTATION NOTES

If you have taken the advice above and created classes using the amazon-api script you can probably ignore this section. This section is intended to help those trying to create the lightest weight possible AWS API class.

Just a reminder for those wanting to go lite...

Headers

X-Amz-Target

Most of the newer AWS APIs are invoked as HTTP POST operations and accept a header X-Amz-Target in lieu of the CGI parameter Action to specify the specific API action. Some APIs also want the version in the target, some don't. There is sparse documentation about the nuances of using the REST interface directly to call AWS APIs, but you kinda sorta figure it out by parsing the Botocore data for a particular API.

When invoking an API, the class uses the api value to indicate that the action should be set in the X-Amz-Target header. We also check to see if the version needs to be attached to the action value as required by some APIs.

if ( $self->get_api ) {
  if ( $self->get_version) {
    $self->set_target(sprintf('%s_%s.%s', $self->get_api, $self->get_version, $self->get_action));
  }
  else {
    $self->set_target(sprintf('%s.%s', $self->get_api, $self->get_action));
  }

  $request->header('X-Amz-Target', $self->get_target());
}

DynamoDB and KMS seem to be able to use this in lieu of query variables Action and Version, although again, there seems to be a lot of inconsistency (and sometimes flexibility) in the APIs. DynamoDB uses DynamoDB_YYYYMMDD.Action while KMS does not require the version that way and prefers TrentService.Action (with no version). There is no explanation in any of the documentations I have been able to find as to what "TrentService" might actually mean. Again, your best approach is to read Amazon's documentation and look at their sample requests for guidance. You can also look to the Botocore project for information regarding the service. Checkout the service-2.json file within the sub-directory botocore/botocore/data/{api-version}/{service-name} which contains details for each service.

In general, the AWS API ecosystem is very organic. Each service seems to have its own rules and protocol regarding what the content of the headers should be.

As noted, this generic API interface tries to make it possible to use one class Amazon::API as a sort of gateway to the APIs. The most generic interface is simply sending query variables and not much else in the header. Services like EC2 conform to that protocol and can be invoked with relatively little fanfare.

use Amazon::API;
use Data::Dumper;

print Dumper(
  Amazon::API->new(
    service => 'ec2',
    version => '2016-11-15'
  )->invoke_api('DescribeInstances')
);

Note that invoking the API in this fashion, version is required.

For more hints regarding how to call a particular service, you can use the AWS CLI with the --debug option. Invoke the service using the CLI and examine the payloads sent by the Botocore library.

Rolling a New API

Once again, your best bet is to use the amazon-api script to roll a class from the Botocore metadata, but if you really want to create your own class the lite way read on.

The Amazon::API class will stub out methods for the API if you pass an array of API method names. The stub is equivalent to:

sub some_api {
  my $self = shift;

  $self->invoke_api('SomeApi', @_);
}

Some will also be happy to know that the class will create an equivalent CamelCase version of the method.

As an example, here is a possible implementation of Amazon::CloudWatchEvents that implements one of the API calls.

package Amazon::CloudWatchEvents;

use strict;
use warnings;

use parent qw(Amazon::API);

sub new {
  my ($class, $options) = @_;

  my $self = $class->SUPER::new(
    { %{$options},
      api         => 'AWSEvents',
      service     => 'events',
      api_methods => [qw( ListRules )],
    }
  );

  return $self;
}

Then...

use Data::Dumper;

print Dumper(Amazon::CloudWatchEvents->new->ListRules({}));

Of course, creating a class for the service is optional. It may be desirable however to create higher level and more convenient methods that aid the developer in utilizing a particular API.

Overriding Methods

Because the class does some symbol table munging, you cannot easily override the methods in the usual way.

sub ListRules {
  my $self = shift;
  ...
  $self->SUPER::ListRules(@_)
}

Instead, you should re-implement the method as implemented by this class.

sub ListRules {
  my $self = shift;
  ...
  $self->invoke_api('ListRules', @_);
}

Content-Type

Yet another piece of evidence that suggests the organic nature of the Amazon API ecosystem is their use of different Content-Type headers. Some of the variations include:

application/json
application/x-amz-json-1.0
application/x-amz-json-1.1
application/x-www-form-urlencoded

Accordingly, the invoke_api() method can be passed the Content-Type or will try to make its best guess based on the service protocol or the type of object being passed as parameters. There is a hash of service names and service types that this module uses to determine the content type required by the service. If services are added that hash needs to be updated.

You can also set the default content type used for the calling service by passing the content_type option to the constructor.

$class->SUPER::new(
  content_type => 'application/x-amz-json-1.1',
  api          => 'AWSEvents',
  service      => 'events'
);

ADDITIONAL HINTS

VERSION

This documentation refers to version 2.2.1 of Amazon::API.

DIAGNOSTICS

To enable diagnostic output, set debug to a true value when calling the constructor. You can also set the DEBUG environment variable to a true value to enable diagnostics.

Logging

By default Amazon::API creates a Log::Log4perl logger to log at the DEBUG and TRACE levels. Setting the environment variable DEBUG to some value or passing a true value for debug in the constructor will trigger extremely verbose logging. This is to help debug edge cases especially around serialization which is particularly prone to exceptions and API specific scenarios.

If you pass a logger to the constructor, Amazon::API will attempt to use that if it has the appropriate logging level methods (error, warn, info, debug, trace, level). If Log::Log4perl is unavailable and you do not pass a logger, logging is essentially disabled at any level.

If, for some reason you set the enviroment variable DEBUG to a true value or have your own Log4perl logger set at the debug level but do not want Amazon::API to log messages at that level you can turn off logging as shown below:

my $ec2 = Amazon::API::EC2->new(log_level => 'info');

In other words, do not send a logger but send a log level. The constructor will recognize that you have a Log4perl logger initialized and just set its log level to your desired level.

BUGS AND LIMITATIONS

This module has not been tested on Windows OS. Please report any issues found by opening an issue here:

https://github.com/rlauer6/perl-Amazon-API/issues

FAQs

Why should I use this module instead of Paws?

Maybe you shouldn't. Paws is a community supported project and may be a better choice for most people. The programmers who created Paws are luminaries in the pantheon of Perl programming (alliteration intended). If you don't want to install of the AWS services but only need to use a single service, Amazon::API may be the right choice for you. Paws may also have some edge cases for some of the seldom used services and you might find this module easier to use and debug.

Does it perform better than Paws?

Probably not. But individual API calls to Amazon services have their own performance characteristics and idiosyncracies. The overhead introduced by this module and Paws may be insignificant compared to the API performance itself, however Paws is implemented using Moose and the startup time for a Moose script can longer than the startup script when using this module. YMMV.

Does this work for all APIs?

I don't know. Probably not? Feedback is appreciated. Amazon::API has been developed based on my needs and used accordingly. Although I have tested it on many APIs, there may still be some cases that are not handled properly and I am still deciphering the nuances of flattening, boxing and serializing objects to send to Amazon APIs. The newer versions of this module using Botocore metadata have become increasingly reliable over time and I'm somewhat confident that my interpretation of the Botocore data produces working classes.

However, keep in mind that Amazon APIs are not created equal, homogenous or invoked in the the same way for all services. Some accept parameters as a query strings, some parameters are embedded in the URI, some are sent as JSON payloads and others as XML. Content types for payloads are all over the map. Likewise with return values.

Luckily, the Botocore metadata describes the protocols, parameters and return values for all APIs. The Botocore metadata is quite amazing actually. It is used to provide information to the Botocore library for calling any of the AWS services and even for creating documentation!

Amazon::API can use that information for creating the Perl classes that invoke each API but may not interpret the metadata correctly in all circumstances, so it is likely bugs may still exist.

If you want to use this to invoke S3 APIs, don't. I haven't tried it and I'm pretty sure it would not work anyway. There are modules designed specifically for S3; Amazon::S3, Net::Amazon::S3. Use them instead.

Do I have to create the shape classes when I generate stubs for a service?

Probably. If you create stubs manually, then you do not need the shape classes. If you use the scripts provided to create the API stubs using Botocore metadata, then yes, you must create the shapes so that the Botocore API methods know how to serialize requests. Note that you can create the shape stubs using the Botocore metadata while not creating the API services. You might want to do that if you want a lean stub but want the benefits of using the shape stubs for serialization of the parameters (or you want the pod that comes with those classes).

If you produce your stubs manually and do not create the shape stubs, then you must pass parameters to your API methods that are ready to be serialized by Amazon::API. Creating data structures that will be serialized correctly however is done for you if you use the shape classes. For example, to create an SQS queue using the shape stubs, you can call the CreateQueue API method as describe in the Botocore documentation.

$sqs->CreateQueue(
  { QueueName => $queue_name,
    tags      => { Name => 'my-new-queue' },
    { Env => 'dev' },
    Attributes => { VisibilityTimeout => 40 },
    { DelaySeconds => 60 }
  }
);

If you do not use the shape classes, then you must pass the arguments in the form that will eventually be serialized in the correct manner as a query string.

$sqs->CreateQueue([
 'QueueName=foo',
 'Attributes.1.Value=100',
 'Attributes.1.Name=VisibilityTimeout',
 'Tag.1.Key=Name',
 'Tag.1.Value=foo',
 'Tag.2.Key=Env',
 'Tag.2.Value=dev'
]);

This code does not use "Modern Perl". Why?

This code has evolved over the years from being ONLY a way to make RESTful calls to a few Amazon APIs, to incorporating the use of the Botocore metadata. It was one person's effort to create a somewhat lightweight interface to selected AWS APIs.

The code did not start out as well designed attempt to interpret the Botocore data by creating a monolithic framework to call ANY AWS API. Perhaps if it were designed today it might use more of Modern Perl, like Moose as does Paws. The code does however embrace Perl Best Practices. Running perlcritic with the Perl Best Practices theme should show no or very few findings.

How do I pass AWS credentials to the API?

There is a bit of magic here as Amazon::API will use Amazon::Credentials transparently if you do not explicitly pass the credentials object. I've taken great pains to try to make the aforementioned module somewhat useful and secure.

See Amazon::Credentials.

Can I use more than one set of credentials to invoke different APIs?

Yes. See Amazon::Credentials.

How stable is the interface?

As of version 2.1.0 the interface is quite stable. I'm not aware of any current bugs and now consider this project "production ready".

Why are you using XML::Simple when it clearly says "DO NOT"?

It's simple. And it seems easier to build than other modules that almost do the same thing.

I tried to use this with XYZ service and it didn't work. What should do I do?

There are several reasons why your call might not have worked. The most likely place for API calls to fail is when serializing requests or serializing results. Enable debugging and see how far the API gets. Report whether the serialization on the request or response failed. If the serialization of the results failed, you can set decode_always to false which will prevent serialization of the result and return the raw content sent from the API. Other reasons your call may have failed include:

BETTER TOGETHER

The motivation behind Amazon::API has been to provide a lightweight implementation of Amazon APIs for Perl. Several companion projects have been developed with the same philosophy.

LICENSE AND COPYRIGHT

This module is free software. It may be used, redistributed and/or modified under the same terms as Perl itself.

TBD

SEE OTHER

Amazon::Credentials, Amazon::API::Error, AWS::Signature4, Amazon::API::Botocore, Paws

AUTHOR

Rob Lauer - rlauer@tresurersbriefcase.com