style

border-bottom-glow

Managing Log Retention in AWS: S3 Lifecycle Rules and Terraform Best Practices

One of the easiest ways for cloud costs to grow unnoticed is through log retention. Many organizations understand the importance of collecting logs, fewer understand how to optimize and use the data you obtain from logs, and even fewer understand how long those logs should be kept. It is important to understand how often your company is accessing these logs and if your current storage strategy still aligns with business needs.

When working with a customer to review our logging strategy, we discovered a common pattern: logs were being retained longer than necessary, storage costs continued to grow, and there was no standardized lifecycle policy across environments.

To address this, we developed a formal log retention policy backed by Amazon S3 lifecycle management and Infrastructure as Code (IaC). The result was a more consistent approach to log management, reduced storage costs, and a retention strategy that better aligned with operational and compliance requirements.

Start with Purpose, Not Storage Classes

One of the biggest mistakes organizations make is starting with technology rather than business requirements.

Many retention policies are built around "what if" scenarios:

What if we need logs from two years ago?
What if an investigation requires historical data from 12 months ago?
What if someone asks for records we no longer have?

While these questions are reasonable, they often result in retaining everything indefinitely.

Instead, start with a few simple questions:

What is the actual purpose of these logs?
How often are they accessed?
When they are accessed, what business value do they provide?
Are there compliance requirements that dictate retention periods?
Who owns the decision for log retention?
How can we reduce storage costs without sacrificing anything?

While this varies by company, in many cases, teams discover that logs older than a few months are rarely accessed and provide limited operational value. They are there for assurance, and before you know it, you have terabytes upon terabytes of unwanted cost driving space and clutter

The end goal should be to retain logs long enough to support investigations, troubleshooting issues, compliance, and auditing, while avoiding unnecessary storage costs.

Different Environments Have Different Needs

Not every environment requires the same retention period.

Production logs often support audits, incident response, and security investigations. Development logs are typically used for short-term debugging and can usually be retained for much shorter periods.

Our retention strategy was built around environment-specific requirements. Below is a sample example of how this can look.

Production

Retention: 12 Months

Production logs require the longest retention because they support:

Security investigations
Incident response
Compliance and audit requirements
Historical operational analysis

Lifecycle Strategy:

Days 0–30: S3 Standard
Days 31–90: S3 Standard-IA
Days 91–365: S3 Glacier Flexible Retrieval
Delete after 12 months

QA

Retention: 90 Days

QA logs primarily support:

Release validation
Regression analysis
Historical troubleshooting

Lifecycle Strategy:

Days 0–60: S3 Standard-IA
Delete after 90 days

Development

Retention: 30 Days

Development logs are primarily used for:

Application debugging
Troubleshooting
Development support

Lifecycle Strategy:

Days 0–30: S3 Standard-IA
Delete after 30 days

By tailoring retention to each environment, we avoided applying expensive production-level retention requirements where they were not needed.

Choosing the Right S3 Storage Classes

Amazon S3 offers several storage classes, each designed for different access patterns.

The key lesson is that storage class selection should be driven by actual usage rather than assumptions.

Some questions to consider:

How often are logs frequently accessed?
Are there certain logs valued more than others?
How are our current s3 buckets setup?
Are they rarely accessed after the first month?
How quickly must they be retrieved?
Are retrieval costs acceptable?

For our use case:

S3 Standard

Used for newly created production logs where immediate access is important.

S3 Standard-IA

Used when logs are still occasionally accessed, but no longer require the urgency of standards.

S3 Glacier Flexible Retrieval

Used for long-term archival where access is rare, but retrieval remains possible if needed.

AWS provides a full breakdown of available S3 storage classes and their tradeoffs, which can be found here:
https://aws.amazon.com/s3/storage-classes/

Infrastructure as Code Makes Governance Easier

A retention policy is only valuable if it is consistently enforced.

Rather than manually configuring lifecycle rules across environments, we implemented our lifecycle policies using Terraform.

This provided several benefits:

Reusable lifecycle configurations
Version-controlled changes
Peer review through pull requests
Consistent deployment across environments
Reduced risk of configuration drift

Infrastructure as Code transformed retention management from a one-time configuration effort into a repeatable operational process.

Protect Yourself from Mistakes

Lifecycle policies automate deletion and storage transitions. That makes them powerful, but also potentially dangerous.

Before enabling automated expiration and transitions:

Enable S3 bucket versioning
Validate lifecycle rules in a non-production environment
Confirm transition timing calculations
Test object expiration behavior
Verify retrieval workflows
Ensure proper buckets are targeted

Versioning provides an additional layer of protection against accidental deletions or misconfigured lifecycle rules.

Test Before Production

We established a simple validation process before rollout:

Create a dedicated development bucket.
Archive development logs into the bucket.
Apply proposed lifecycle policies.
Validate storage class transitions.
Verify expiration behavior.
Document results.
Obtain approval before moving on to other dev buckets,
Promote to higher environments

This approach allowed us to identify configuration issues early while minimizing risk.

Governance Matters

Log retention should not be owned by a single engineer.

Successful retention policies require collaboration between:

DevOps teams
Security teams
Compliance teams
Infrastructure leadership

Clear ownership helps ensure retention periods remain aligned with business and regulatory requirements as systems evolve.

Final Thoughts

The most effective log retention policy is not necessarily the one that keeps logs the longest.

It is the one that balances:

Cost
Compliance
Security
Operational visibility
Simplicity

Organizations often retain data because of hypothetical future needs rather than actual business requirements. By understanding how logs are used, aligning retention periods with environment-specific needs, leveraging appropriate S3 storage classes and storage transitions, and implementing policies through IaC, teams can significantly reduce storage costs while maintaining the visibility they actually need.

Most importantly, remember that every organization is different. The lifecycle strategy described here worked well for our instance, but retention periods, storage classes, and archival timelines should always be driven by your own operational requirements and compliance obligations.