AWS is a cloud service that offers a multitude of complex and high-value services and resources. You can build literally anything you want digitally using AWS.
But the most common issue beginners fail to consider when using the treasure trove of powerful services, is inevitably, cost.
Being aware of cost is critical at virtually every level of business and technology.
It must be considered at the business objective level, both short term and long term, at the technology solution architecture level, at the systems level, and all the way down to the features level.
In AWS the most effective manner to maintain, control, and monitor costs is by using AWS CloudWatch.
CloudWatch
CloudWatch is a powerful service that allows visual monitoring and management of the usage (and costs) of your AWS cloud services.
With CloudWatch you can set up billing alarms that will automatically notify you when a certain service’s costs surpass a budget threshold that you define.
This is very practical as you can get emails whenever a service may be costing you more than you expected and this allows you to have fine-grained control and visibility over your services costs as well as their usage.
While AWS provides a service better fit for monitoring costs, such as AWS Cost Explorer, CloudWatch lets you set billing alarms and is more easily integrateable with a wide array of AWS services.
Other practical applications for CloudWatch, which is as common as monitoring costs and usage, is monitoring logs, metrics and events.
It is often good practice to have your application logs go through CloudWatch and many services like AWS Lambda will by default stream logs to CloudWatch.
This allows you to view error logs or any type of log for any event coming from your Lambda functions for example, and offers a detailed explanation of why something is behaving in a certain way.
As great as this service is, there is one small problem.
All logs from every service are, by default, stored in CloudWatch for an indeterminate amount of time.
This default behaviour of the retention of your logs is designed that way in case you need to know what happened for certain events at any time in the past.
Cost Implication
The issue is that CloudWatch charges you for the retention of data — for the data stored. While it may not be much, CloudWatch charges you 3 cents per GB stored and 50 cents per data ingested.
This isn’t an issue until you learn that CloudWatch stores your data for all time. So while it may not sound expensive to store 10 GB of data per month at 3 cents per GB, that data aggregates and grows as your business and use cases scale, and at that point the costs can become substantial.
So what should you do?
The best option is to set the log data retention to an explicit expiry date.
Fixing The Issue
It is very easy to do this. Head over to the CloudWatch console in AWS. In the left sidebar you will see the Logs link, clicking on that will expand the sub items list and you can select Log Groups.
On this new page you will see a list of all the log groups that have automatically been created for you by AWS for various services.
Select the checkbox for any log group and at the top you will see the Actions dropdown select.
Clicking on that will show an option to edit retention setting(s). Click that and a modal will open, allowing you to modify the retention of the log data.
You can set whatever expiration period of time you need for this.
What is important is to not leave it at “never expire” which is the default as can be seen, unless you absolutely need this for compliance or other needs, and you are aware of the cost of storing that data every month.
Once you make a selection you can go ahead and click on save. A warning will appear. Confirm if you are ok with it and that’s all you need to do on your end — AWS will automatically delete the data after your specified retention period.
As an alternative, you can also delete one or many log groups if you no longer need them, so you can avoid the costs of keeping the data month after month.
Conclusion
To summarize, we went over the fact that AWS CloudWatch has a “hidden” feature by default that stores the log group data indefinitely. AWS also charges you for this storage of data and while it remains cheap (3 cents per GB), the data can grow in size quickly and you pay for it month after month. These costs can potentially grow and being aware of how to reduce the retention period is important — so we looked at a hands-on approach of configuring that option and let our minds be at ease in terms of costs from a very common service in AWS.
I hope you learned something of value and practical to make use of for your next project!
Click here to say hi 👋, collaborate with me, or explore my services 🤝.
Uriel