Artifact and Dependency Retention Strategies

Introduction: The Lifecycle of Digital Assets

In the modern software development lifecycle, we generate an immense volume of digital assets. Every time code is pushed, built, and tested, we create artifacts—compiled binaries, container images, documentation bundles, and test reports. Simultaneously, our projects rely on thousands of external dependencies pulled from registries like npm, PyPI, or Maven Central. If we do not manage these assets, our storage costs balloon, our build environments become cluttered, and we face significant security risks from stale, vulnerable versions of software.

Retention and migration strategies are the formal processes we use to decide how long to keep these assets, where to store them, and how to safely move them when our infrastructure changes. Without a clear strategy, organizations often default to "keep everything forever." While this might seem safe, it creates a massive attack surface. If an old container image contains a critical vulnerability, and it remains in your registry, it is a ticking time bomb waiting to be accidentally deployed by a developer or an automated script.

This lesson explores the technical and operational frameworks required to manage the lifecycle of your artifacts and dependencies. We will look at how to implement automated cleanup policies, how to bridge the gap between development and production registries, and how to ensure that your build process remains reproducible even as your underlying infrastructure shifts. By the end of this guide, you will be able to design a retention policy that balances storage efficiency, security posture, and developer productivity.

The Anatomy of an Artifact Lifecycle

To manage artifacts effectively, you must first understand that they are not static objects. They move through stages of maturity, and their retention requirements change at each stage. Typically, an artifact begins as a "snapshot" or "development build," moves through a "release candidate" phase, and finally matures into a "production-ready" version.

Stages of Maturity

1. Transient/Development Builds: These are the most numerous. They are created during local development or continuous integration (CI) runs for feature branches. They are rarely needed beyond the life of a pull request.
2. Release Candidates (RC): These represent stable versions intended for testing in staging environments. They require higher visibility but are not intended for long-term production use.
3. Production Releases: These are the "golden" assets. They are immutable, signed, and must be retained for compliance, audit, and potential rollback purposes.
4. Archival/Legacy: These are releases that are no longer actively supported but must be kept for regulatory or historical reasons.

Callout: The Immutability Principle A core rule of artifact management is that production artifacts should be immutable. Once a version (e.g., v1.2.4) is pushed to your production registry, it should never be overwritten. If you find a bug, you must increment the version (e.g., v1.2.5). Overwriting tags leads to "caching hell," where different parts of your infrastructure believe they are running different versions of the same code.

Designing a Retention Policy

A retention policy is not a one-size-fits-all document. It must be tailored to the specific needs of your engineering organization. When designing your policy, you should base your decisions on the balance between storage costs and the time-to-reproduce metric—the amount of time it takes to rebuild an artifact from source if the binary is lost.

Key Factors for Retention

• Compliance and Legal Requirements: Some industries, such as finance or healthcare, require you to keep build artifacts for years to satisfy audit requirements.
• Storage Cost: Storing terabytes of container images in cloud-based registries like Amazon ECR or Google Artifact Registry incurs monthly costs.
• Security Vulnerability Management: Keeping old artifacts increases the likelihood that a legacy version with a known exploit will be deployed.
• Operational Recovery: If your production systems fail, how quickly do you need to revert to a previous version? If the binary for that version is gone, you face a lengthy rebuild process.

Recommended Retention Tiers

We suggest implementing a tier-based system to automate the cleanup process. You can configure most modern artifact registries to apply these rules automatically.

Implementing Automated Cleanup

Manual cleanup is a recipe for failure. As your team grows, keeping track of what to delete becomes impossible. You must rely on automated policies integrated into your CI/CD platform or your registry provider.

Example: Configuring Lifecycle Policies in AWS ECR

If you are using AWS, ECR provides built-in lifecycle policies. These policies allow you to define rules based on the age of the image or the count of images.

{
  "rules": [
    {
      "rulePriority": 1,
      "description": "Keep only 5 most recent development images",
      "selection": {
        "tagStatus": "tagged",
        "tagPrefixList": ["dev-"],
        "countType": "imageCountMoreThan",
        "countNumber": 5
      },
      "action": {
        "type": "expire"
      }
    },
    {
      "rulePriority": 2,
      "description": "Delete all untagged images older than 7 days",
      "selection": {
        "tagStatus": "untagged",
        "countType": "sinceImagePushed",
        "countUnit": "days",
        "countNumber": 7
      },
      "action": {
        "type": "expire"
      }
    }
  ]
}

Explanation of the Code:

• Rule 1: This rule targets images prefixed with dev-. It ensures that we never keep more than 5 of these images. Once a 6th image is pushed, the oldest one is automatically deleted. This prevents "development noise" from filling up your registry.
• Rule 2: Untagged images are often the result of failed builds or intermediate layers. These are almost never useful. By deleting them after 7 days, you reclaim significant storage space without affecting your production deployment stability.

Dependency Management: The Hidden Danger

Artifacts aren't just the code you write; they are also the libraries you consume. Managing dependencies—the "third-party" code in your projects—requires a different approach than managing your own binaries. If you rely on public registries (like npmjs.com) directly, you are vulnerable to "left-pad" style incidents where packages are removed or modified, breaking your builds.

The Mirroring Strategy

The industry-standard approach is to use an internal artifact repository (like Artifactory, Sonatype Nexus, or a local registry proxy) to cache external dependencies. When a developer or build agent requests a dependency, the request goes to your internal proxy first. If the proxy doesn't have it, it fetches it from the public internet, caches it, and serves it to your internal network.

Note: Mirroring is not just about reliability. It is also about security. By proxying dependencies, you can scan them for vulnerabilities using tools like Snyk or OWASP Dependency-Check before they ever hit your developers' machines.

Dependency Retention Best Practices

1. Pin Versions: Never use latest or floating versions in production. Always specify the exact version (e.g., v1.2.3).
2. Lock Files: Commit your lock files (package-lock.json, Gemfile.lock, go.sum) to version control. These ensure that the exact same dependency tree is installed regardless of when or where the build happens.
3. Vulnerability Scanning: Configure your repository to automatically quarantine dependencies that have known security disclosures.
4. License Compliance: Use automated tools to ensure that the dependencies you are retaining don't violate your organization’s open-source license policies (e.g., avoiding GPL-licensed code in proprietary software).

Migration Strategies: Moving Between Registries

There will come a time when you need to migrate your artifacts. Perhaps you are moving from a self-hosted registry to a managed cloud service, or you are consolidating multiple registries into one global instance. Migration is a high-risk activity because it touches the foundation of your deployment process.

Step-by-Step Migration Process

1. Inventory and Audit: Before moving anything, run a script to export a list of all images and versions currently in your source registry.
2. Infrastructure Provisioning: Set up the destination registry with the same access control policies (IAM/RBAC) as the source.
3. The "Copy" Phase: Use tools like skopeo to copy images between registries without pulling them to a local machine. This is faster and uses less bandwidth.
4. Verification: Compare the checksums (SHAs) of the images in the source and destination to ensure data integrity.
5. Traffic Shifting: Update your CI/CD pipelines to point to the new registry. Do this in small batches, starting with non-critical services.
6. Decommissioning: Keep the old registry in a "read-only" state for a period (e.g., 30 days) before shutting it down completely.

Using Skopeo for Image Migration

skopeo is a command-line tool that performs operations on remote container images without needing a full Docker daemon. It is the gold standard for migrating between registries.

# Example command to copy an image from one registry to another
skopeo copy \
  docker://source-registry.com/my-app:v1.0.0 \
  docker://dest-registry.com/my-app:v1.0.0

Warning: When migrating, do not attempt to "re-build" the images in the new registry. Always move the existing binary blobs. Re-building introduces the risk of different build environments, different compiler versions, or updated base layers, which could change the hash of the image and break your production environment.

Common Pitfalls and How to Avoid Them

Even with the best intentions, teams often fall into traps that lead to broken builds or security incidents. Here are the most common mistakes we see in the field.

1. The "Latest" Tag Trap

Using the latest tag is convenient for local development but disastrous for production. If you push a new image and tag it as latest, your production orchestrator (like Kubernetes) might pull that new image during a rolling update, even if you didn't intend for it to be deployed.

• The Fix: Use semantic versioning (SemVer) and unique build IDs (e.g., v1.2.4-commit-sha).

2. Orphaned Dependencies

Developers often add dependencies that get used once and then forgotten. Over time, these bloat your application and increase your attack surface.

• The Fix: Periodically run dependency audits. Tools like npm-check or pip-audit can identify packages that are installed but never imported in your code.

3. Ignoring Build Metadata

Artifacts are useless if you don't know where they came from. If you have an image called app:v1.2.4 but no record of which commit created it, you cannot debug issues effectively.

• The Fix: Use labels in your container images or build artifacts to store metadata.
  • org.opencontainers.image.revision=a1b2c3d4
  • org.opencontainers.image.source=https://github.com/org/repo

4. Lack of Disaster Recovery for Registries

Many teams assume that because their registry is in the cloud, it is backed up. Cloud providers guarantee the availability of the service, but they do not guarantee that you won't accidentally delete your own images.

• The Fix: Enable cross-region replication for your registry. If your primary region goes down, your artifacts remain available in a secondary region.

Advanced Topic: Managing "Cold" Storage

As your artifact library grows, you will inevitably have thousands of images that you are required to keep for compliance but that you haven't touched in years. Storing these in a high-performance registry is a waste of money.

Moving to Cold Storage

Most enterprise-grade artifact repositories allow you to define "storage tiers." You can move artifacts that are older than 180 days to a cheaper storage tier, such as AWS S3 Glacier or equivalent.

• Cost Efficiency: Cold storage is significantly cheaper than standard registry storage.
• Accessibility: Note that retrieving from cold storage may take minutes or hours. This is acceptable for audit purposes, but not for emergency production rollbacks.
• Policy Automation: Ensure your retention policy automatically moves artifacts to cold storage based on age, rather than just deleting them.

Comparison: Registry vs. Repository vs. Cache

It is common to confuse these terms. Understanding the distinction is vital for designing your architecture.

• Registry: Think of this as a structured storage system for containerized applications. It understands layers, manifests, and tags.
• Repository: This is more flexible. It holds JAR files, NPM packages, NuGet packages, etc. It often provides features for dependency proxying and vulnerability scanning.
• Cache: This is ephemeral. If you delete your cache, your build might take longer, but your production environment should remain entirely unaffected.

Best Practices Checklist

To ensure your retention and migration strategy is effective, audit your current setup against these industry best practices:

1. Standardize Naming: Use a consistent naming convention for all artifacts (e.g., project-name/service-name:version).
2. Automate Everything: If you are doing manual cleanup, stop. Write a script or configure a policy today.
3. Sign Your Artifacts: Use tools like Cosign to sign your images. This ensures that you aren't just retaining any artifact, but one that you have verified as authentic.
4. Audit Regularly: Once a quarter, review your registry size and identify the top 10% of "space-wasters." Are those artifacts actually needed?
5. Document the "Why": For every retention rule you set, document the business reason. If an auditor asks why you deleted a build from three years ago, you should be able to point to the policy.
6. Test Migration Procedures: Don't wait for a forced migration (like a cloud provider shutdown) to test your migration scripts. Run a "fire drill" by moving a small, non-critical service to a new registry.

Common Questions (FAQ)

Q: Should I delete all old artifacts to save money?

A: No. You must balance cost with compliance. Deleting an artifact that is legally required for an audit can result in significant fines. Always consult with your legal or compliance team before setting "hard" deletion policies for long-term assets.

Q: What do I do if I need to roll back to an artifact that was deleted by a policy?

A: This is the primary risk of aggressive retention. You must ensure that your "Production" retention period is longer than your longest-expected recovery time objective. If you find yourself needing an old artifact, your only choice is to rebuild from the source code commit hash, which is why keeping your build environment code (the CI/CD pipeline definition) is just as important as the artifacts themselves.

Q: Is it safe to use a third-party registry as my primary storage?

A: It is common, but it creates a "vendor lock-in" risk. If you are worried about this, implement a secondary "backup" registry where you replicate your production images. Many companies use a primary cloud registry for speed and a secondary, independent registry for disaster recovery.

Q: How do I handle dependencies that are yanked from public registries?

A: This is why you should never pull directly from the internet. By using an internal proxy (like Artifactory or Nexus), you "pin" the dependency locally. Even if the original author deletes the package from the public internet, your internal copy remains safe, and your builds will continue to succeed.

Key Takeaways

1. Lifecycle Management is Mandatory: Artifacts are not static; they evolve. You must define clear retention tiers for development, staging, and production assets.
2. Automate to Avoid Human Error: Manual cleanup is prone to mistakes. Use built-in registry lifecycle policies to enforce your retention rules automatically.
3. Prioritize Immutability: Never overwrite tags. Once an artifact is released, treat it as a permanent, read-only record.
4. Mirror Your Dependencies: Never rely on public registries for production builds. Proxy and cache your dependencies locally to protect against external repository outages or malicious deletions.
5. Migration Requires Integrity: When moving artifacts, prioritize tools that preserve the original binary hash (like skopeo) rather than rebuilding from source.
6. Balance Cost and Compliance: Use cold storage tiers for long-term compliance requirements to minimize costs while maintaining accessibility for potential audits.
7. Treat Infrastructure as Code: Your retention policies, registry configurations, and CI/CD pipelines should all be stored in version control, allowing you to recreate your entire artifact management environment from scratch if necessary.

By following these strategies, you shift artifact management from an unpredictable "storage problem" to a structured, automated, and secure component of your engineering operations. You reduce security risk, lower costs, and ensure that your team can always reproduce the state of any production release, regardless of how long ago it was deployed.