NAME

OrePAN2::S3 - Manage a DarkPAN CPAN mirror on Amazon S3

SYNOPSIS

# Upload a distribution to DarkPAN without indexing
orepan2-s3 upload My-Dist-1.0.tar.gz
# or using the alias
orepan2-s3 add My-Dist-1.0.tar.gz

# Upload AND index a new distribution
orepan2-s3 inject My-App-1.0.0.tar.gz

# Regenerate the DarkPAN home page and upload it
orepan2-s3 --upload create-site-index

# ...or upload an already-generated index.html on its own
orepan2-s3 upload-index

# Upload custom artifacts specified in config
orepan2-s3 upload-artifacts

DESCRIPTION

OrePAN2::S3 provides a command-line interface for creating and maintaining an S3-backed DarkPAN repository, including distribution publishing, incremental package-index maintenance, documentation and site generation, deletion, and optional CloudFront integration.

FEATURES

• Upload and inject Perl distributions into an S3-backed DarkPAN.
• Maintain 02packages.details.txt.gz incrementally when distributions are added or removed.
• Delete individual distribution versions, or multiple matching versions, without rebuilding the entire repository index.
• Generate and publish a customizable HTML index for the repository.
• Extract POD, README, and changelog documentation from distributions and publish generated documentation to S3.
• Publish additional static assets used by the DarkPAN site.
• Optionally use CloudFront and automatically invalidate cached repository content after updates.
• Support multiple repository profiles, AWS profiles, configurable author paths, and custom index templates.
• Inspect and download the current package index and list distributions stored in the repository.
• Protect uploads from dirty builds, with explicit override and dry-run support for administrative operations.

USAGE

orepan2-s3 [options] command [args]

Options

Both commands and options may be abbreviated to any unique prefix. Boolean options marked [negatable] accept a --no- form (for example --no-invalidate-index).

• -h, --help

  Display this help message.

• -b, --bucket-name name

  S3 bucket name. Overrides the AWS.bucket config value.

• -c, --config-file path

  Path to the configuration file. Default: ~/.orepan2-s3.json.

• -d, --distribution path

  Path to the target distribution tarball when adding a new distribution.

  Tarball name or tarball prefix when deleting distributions. Examples:

    orepan2-s3 --distribution workdir/Foo-Bar-1.2.3.tar.gz add
  
    orepan2-s3 --distribution Foo-Bar-1.2.3.tar.gz delete
  
    orepan2-s3 --distribution Foo-Bar delete
  

• -n, --profile-name name

  Configuration profile section name inside the config file. Default: default.

• -p, --profile name

  AWS/IAM profile name. Default: $AWS_PROFILE.

• -t, --template path

  Path to a custom Template::Toolkit template for index.html.

• -o, --output path

  Output path for commands that write a file locally.

• -U, --url url

  Base URL of the DarkPAN, used by create-docs when retrieving a distribution remotely. May also be set as the url key in the configuration profile.

• --format format

  Output format for informational commands (e.g. list-packages). Default: json.

• --dirty-check [negatable]

  Check the distribution's $GIT_DIRTY global before uploading and abort if it is dirty. Enabled by default; use --no-dirty-check (or --force) to override.

• --force

  Force upload of an uncommitted (dirty) distribution.

• --invalidate-index [negatable]

  Invalidate CloudFront paths after creating the site index. Enabled by default; use --no-invalidate-index to skip invalidation.

• --update-site-index [negatable]

  Update the site index after commands that modify the package index. Enabled by default.

• --save-index

  Save the 02packages.details.txt.gz file to the current directory.

• --upload

  Upload the index after creating it.

• --delete-all

  When a delete matches multiple objects, remove all of them instead of aborting.

• --dryrun

  Report what would be done without making any changes.

• --cli-pager [negatable]

  Page long output. Enabled by default; use --no-cli-pager to disable.

Commands

• upload (alias: add)

  Uploads the specified distribution tarball to S3 under the configured author path (D/DU/DUMMY by default).

    orepan2-s3 upload My-Package-1.0.0.tar.gz
  

  If you want to upload and index a distribution in a single step, use the inject command instead.

  The upload command will check the main module to see if there is a $GIT_DIRTY global variable defined that indicates whether the distribution has been committed. If the distribution is uncommitted the upload function will abort with an error message by default. Use --no-dirty-check or --force to upload a dirty distribution.

  Note:

  When using the CPAN::Maker::Bootstrapper framework the distribution status is automatically set in the Makefile so your module can include it as a global.

    GIT_DIRTY := $(shell $(GIT) describe --always --dirty --abbrev=40 2>/dev/null || echo 'unknown')
  

  ...then in your module:

    our $GIT_DIRTY = '51eb002566044d5af4c65ceff35848d3e462fbc8-dirty';
  

• inject

  Uploads the distribution tarball to S3 and updates the package details index (02packages.details.txt.gz).

    orepan2-s3 inject My-Package-1.0.0.tar.gz
  

• upload-index

  Uploads an HTML file as the DarkPAN's root index.html; defaults to the local index.html.

• upload-artifacts

  Uploads additional non-package artifacts defined in the index: files: section of your configuration file.

• delete

    orepan2-s3 delete My-Package-1.0.0.tar.gz
    orepan2-s3 delete My-Package
    orepan2-s3 -d My-Package-1.0.0.tar.gz delete
  

  Removes one or more distributions from the DarkPAN and updates the indexes accordingly. The distribution may be given as a positional argument or with --distribution. In a single run this command:

  • deletes the distribution tarball(s) from <prefix>/authors/id/<author_path>/;
  • deletes the associated documentation tree under docs/ (the create-docs output), if present;
  • regenerates 02packages.details.txt.gz, removing the packages that belonged to the deleted distribution(s), and uploads it;
  • unless --no-update-site-index is given, regenerates and uploads the HTML site index (index.html); and
  • unless --no-invalidate-index is given, invalidates the relevant CloudFront paths.

  If the argument ends in .tar.gz it is treated as an exact distribution filename. If the referenced object no longer exists in the bucket a warning is issued and only the documentation is removed.

  If the argument does not end in .tar.gz it is treated as a name prefix and may match several objects (for example every version of a distribution). When more than one object matches you must pass --delete-all, and you will be prompted to confirm before anything is removed:

    orepan2-s3 delete --delete-all My-Package
  

  Use --dryrun to see exactly which objects, docs, and index entries would be removed without modifying the bucket.

  Note: the package index, site index, and CloudFront invalidation are all updated automatically by default. You do not normally need to run create-site-index after a delete; pass --no-update-site-index and/or --no-invalidate-index if you want to suppress those steps.

• create-docs

  Extracts documentation (POD, README.md, and changelog content) from a distribution and creates a local docs.tar.gz archive. When --upload is specified, the POD and README are converted to HTML and uploaded to S3.

• create-site-index

  Generates the DarkPAN site's index.html. By default the generated HTML is written to STDOUT (or --output). When --upload is specified, the index is uploaded to the S3 bucket. If CloudFront is configured and invalidation is enabled, the configured paths are invalidated after upload.

• invalidate-index

  Invalidates CloudFront cache paths associated with package indices and documentation.

• download-index

  Downloads the 02packages.details.txt.gz file.

• list-packages

  Lists the distributions currently stored in the DarkPAN, grouped by distribution name and version.

• show

  Displays the contents of the current package index (02packages.details.txt.gz).

• dump-template

  Prints the default Template::Toolkit index template to STDOUT. Use this as a starting point for a custom template referenced by index: template: in your configuration file.

    orepan2-s3 dump-template > my-index.tt
  

Configuration File

The configuration file for orepan2-s3 is a JSON file that can contain multiple profiles (or none). Each profile represents a DarkPAN S3 repository. The format should look something like this:

{
    "default" : "bedrock",
    "tbc" : {
        "author_path": "D/DU/DUMMY",
        "AWS": {
            "profile" : "prod",
            "region" : "us-east-1",
            "bucket" : "tbc-cpan-mirror",
            "prefix" : "orepan2"
        },
        "CloudFront" : {
            "DistributionId" : "E2ABCDEFGHIJK"
        }
    },
    "bedrock" : {
        "author_path": "D/DU/DUMMY",
        "url" : "https://cpan.openbedrock.net/orepan2",
        "index" : {
            "template" : "/path/to/template",
            "files": {
               "src" : "dest"
            }
        },
        "AWS": {
            "profile" : "prod",
            "region" : "us-east-1",
            "bucket" : "cpan.openbedrock.net",
            "prefix" : "orepan2"
        },
        "CloudFront" : {
            "DistributionId" : "E2JKLMNOPQRXYZ",
            "InvalidationPaths" : []
       }
    }
}

Each profile can contain the keys described below. If you only have one profile you don't need to place it in a 'default' section.

The value for the 'default' key can be the name of a profile or a hash of the profile.

• author_path

  Overrides the default D/DU/DUMMY author path. For a personal DarkPAN you should all ldistributions in one path.

• index

  This section allows you to specify a custom template for the DarkPAN home page.

  • template

    The name of a template file that will be parsed and uploaded as /index.html. If you do not provide a template file a default template is used. The default template is a Template::Toolkit style template. To see the default template use the dump-template command:

      orepan2-s3 dump-template
    

    The templating process is provided with these variables:

    • utils

      A blessed reference to an object with one method (module_name) that returns a version of the module name suitable for use as unique CSS id.

    • repo

      A hash where each key is a DarkPAN distribution name and each value is an array of two-element arrays. Each inner array contains:

        [0] => Perl module name
        [1] => Module version
      

    • localtime

      The current time and date as a string.

    • pod_links

      A hash where the keys are distribution names and the values are links to the POD for a module.

    • readme_links

      A hash where the keys are distribution names and the values are links to a README for a module.

      NOTE: Sometimes the README and the POD will contain the same information.

  • files

    A hash of source/destination pairs that specify additional files you want uploaded to your S3 bucket.

    Example:

      "files": { 
         "/home/rlauer/git/some-project/foo.css" : "/css/foo.css",
         "/home/rlauer/git/some-project/foo.js" : "/javascript/foo.js"
      }
    

• AWS

  • profile

    The IAM profile that allows access to the S3 bucket and CloudFront.

  • region

    AWS region. Default: us-east-1

  • bucket

    S3 bucket name

  • prefix

    The prefix where the CPAN distribution files will be stored. Default: orepan2.

• CloudFront

  NOTE: Your profile must have the ability to invalidate the CloudFront cache!

  • DistributionId

    CloudFront distribution id

  • InvalidationPaths

    OrePAN2::S3 can optionally use CloudFront in front of the S3-backed DarkPAN. Because CloudFront caches objects, changes made in S3 may not be immediately visible to clients, depending on the caching behavior of your CloudFront distribution.

    When repository content changes, OrePAN2::S3 can automatically invalidate the configured CloudFront paths so clients receive the updated content.

    InvalidationPaths is an array of additional CloudFront paths to include whenever an invalidation is performed.

    Note: CloudFront invalidation pricing is controlled by AWS and may change. See the current AWS CloudFront pricing documentation for details.

• custom_sections

  This section contains key/value pairs where the key is the name of a variable that will be exposed to your template and the values are a two-element array that contains a regular expression and possible regexp flags. The script will use the regexp to filter your distributions and add them to a hash whose name is the key you provided.

  The purpose of this section is to allow you to possibly organize your distributions under possible HTML headings.

  Example:

    "custom_sections" : {
        "plugins" : ["^BLM\-(?!Startup)", "xsm"],
        "app_plugins" : ["^BLM\-Startup", "xsm"],
     }
  

  ...then in your template:

        <h1>Application Plugin Index</h1>
  
    [% FOREACH distribution = app_plugins.sort %]
          <h2>
           <span class="collapse-section-icon">&#9660;</span>
           [% distribution %]
           [% IF readme_links.$distribution %]
           <a title="README"  class='doc-link' href="[% readme_links.$distribution %]"><span class="material-symbols-outlined">docs</span></a>
           [% END %]
           [% IF pod_links.$distribution %]
           <a title="pod" class='doc-link' href="[% pod_links.$distribution %]"><span class="material-symbols-outlined">docs</span></a>
           [% END %]
          </h2>
  
          <ul class="collapsable" id="[% utils.module_name(distribution) %]">
    [% FOREACH module IN app_plugins.$distribution %]
            <li>[%  module.0 %]</li>
    [% END %]
          </ul>
    [% END %]
       <hr>
  

ROLES CONSUMED

• OrePAN2::S3::Role::Inject
• OrePAN2::S3::Role::Delete
• OrePAN2::S3::Role::Upload
• OrePAN2::S3::Role::UploadArtifacts

VERSION

This documentation refers to version 2.1.0.

AUTHOR

Rob Lauer - rlauer@treasurersbriefcase.com

SEE ALSO

OrePAN2, Amazon::S3::Lite, DarkPAN::Utils, CLI::Simple, Template

LICENSE

This program is free software; you can redistribute it and/or modify it under the same terms as Perl itself.