offgridtech.xyz

A Blog about IT, Security, Cloud and Off-Grid Technologies.

Optimizing AWS Costs with Cost Explorer, Lambda, and S3 Lifecycle Policies

Since the inception of this blog, I have been delving into AWS cloud services, exploring everything from compute and object storage to logging. Driven by curiosity, I ventured into various configurations to support my blog, aiming to expand my knowledge. As time went by, I not only amassed valuable insights but also a considerable bill for cloud services. It’s easy to imagine how organizations new to the cloud might face a similar initial shock at their spending.

Upon realizing this, I knew AWS offered numerous tools for cost optimization. I began by exploring AWS Billing and Cost Management and Cost Explorer to pinpoint which resources were incurring the most costs. After a careful evaluation, considering factors like risk management and the actual availability needed for my WordPress application, I identified several services contributing significantly to my expenses:

  1. EC2 Auto Scaling Group and RDS running 24/7: My blog doesn’t see a lot of traffic. I would be surprised if anyone is surfing my website at 2 am, so running this instance 24/7 doesn’t make sense.
  2. EC2 Auto Scaling minimum instance count of 2: With an application load balancer, having two instances for hot standby is a great means of redundancy but far to expensive for a blog that sees less than 100 viewers in 1 week.
  3. RDS running in Multi-AZ deployment: RDS is my largest expense. AWS charges a premium for their managed services, and for a small-scale blog, a Single-AZ deployment should be enough, especially when availability is not a great concern.
  4. S3 buckets for storing logs: While not yet a concern, failing to manage the accumulating log data could become costly if I decide that retaining CloudTrail logs is vital for security.

Solving These Problems

Addressing issues #2 and #3 was straightforward, requiring only a change in resource configuration. I switched the database from “multi-AZ” to “single-AZ” and set the desired number of instances in my Auto Scaling group to “1”. Additionally, I opted for a reserved instance for my database. Knowing that I intended to keep the blog page up for at least one more year, I was willing to commit to the contract to benefit from the significant savings offered. This decision not only reduced my monthly expenses but also aligned with my long-term plans for the blog, ensuring I could maintain it cost-effectively.

To tackle cost issue #1, I integrated AWS Lambda and Amazon EventBridge. This involved creating two Lambda functions: one to turn off instances in the evening and another to turn them on in the morning, both assigned the same IAM role and permissions. These functions were triggered daily through EventBridge, automating the process efficiently.

# Lambda Function to Start EC2 Instances and RDS Instance (Written in Python)

import boto3

def lambda_handler(event, context):
    # Initialize the clients
    autoscaling = boto3.client('autoscaling')
    rds = boto3.client('rds')

    # Start EC2 instances by setting the desired capacity of the Auto Scaling group
    # Replace 'your-desired-capacity' with the number of instances you want during active hours
    autoscaling.set_desired_capacity(
        AutoScalingGroupName='your-auto-scaling-group-name',
        DesiredCapacity=your-desired-capacity,  # e.g., 2
        HonorCooldown=False
    )
    
    # Start the RDS instance
    rds.start_db_instance(
        DBInstanceIdentifier='your-rds-instance-identifier'
    )

Although cost issue #4 isn’t immediate, managing data lifecycle is crucial. Now that I have an on-premises SIEM polling my CloudTrail bucket for logs, retaining logs in the bucket for more than 3 days seemed unnecessary. Configuring this was straightforward in the S3 dashboard by creating a lifecycle rule following an AWS guide.

Conclusion

While configuring AWS services to optimize costs, I learned the importance of regular audits and adjustments. Small changes, like adjusting instance availability and simplifying database deployment, can lead to significant savings without compromising functionality. Implementing lifecycle policies for S3 logs further ensures that costs are kept in check by automatically managing data retention. For fellow AWS users, my experience underscores the value of leveraging AWS’s tools for cost management and efficiency. As cloud technologies evolve, staying informed and adaptable is key to optimizing resources and maintaining security without breaking the bank.