Search for a command to run...
Step-by-step guide to automating the cleanup of EC2 instances with AWS Systems Manager for better storage.
Cloud DevOps Engineer with hands-on experience in AWS and on-premise data centers. Continuously learning and growing my skills in cloud automation.
No comments yet. Be the first to comment.
Getting started with AWS can feel overwhelming. When I first explored cloud computing, I had the same questions many beginners ask today: • where do I start?• which AWS services should I learn first?•
In modern cloud architectures, controlling incoming traffic at the application layer is critical. Amazon Web Services Application Load Balancer (ALB) enables fine-grained routing using listener rules,
In this post, I’ll share how I optimized ECS task definitions and reduced CloudWatch log costs by 96% — from $478.15 to just $17.27 per month.
Step-by-Step Guide to Automating EC2, RDS, and Redis Start/Stop During Off-Hours with AWS Lambda
Deploying services on AWS EC2 instances using Docker over time can lead to high storage, memory, and CPU usage. Manually increasing volume size and running prune commands to clear overlay storage and reduce memory usage is not a sustainable solution. To address this, we can automate the cleanup of EC2 instances using AWS Systems Manager.
By setting up a maintenance window with a cron expression, you can automate the regular cleanup of instances without manual intervention. This setup can be customized to run daily, during off-hours, or at specific intervals such as every few days or weeks. While there are other scheduling options available, configuring a cron expression is typically the easiest. This automation helps maintain optimal storage utilization and performance. Below is a step-by-step guide with screenshots to assist you through each stage of the process.
Step1: In the AWS Management Console, navigate to EC2, locate the IAM role for your instance, and attach the AmazonSSMManagedInstanceCore policy. This grants Systems Manager the permissions needed to manage and clean up your EC2 instances.
Step2: After completing Step 1, connect to the EC2 instance using Session Manager. Verify the Docker and system agent paths by running the following commands: which docker and which systemctl.
Step3: Go to AWS Systems Manager, click on "Documents" in the left sidebar, and create a cleanup document as shown in the screenshots.
For AWS Linux-Based EC2 Instances(Confirm the Docker and system agent paths after Step 2):
schemaVersion: '2.2'
description: "Run Docker cleanup commands on EC2 instances"
mainSteps:
- action: aws:runShellScript
name: runDockerCleanup
inputs:
runCommand:
- /bin/docker stop $(/bin/docker ps -q)
- sleep 2
- /bin/docker system prune -a -f
- sleep 3
- /bin/docker volume ls -qf dangling=true | xargs -r /bin/docker volume rm
- sleep 3
- /bin/systemctl restart ecs
- sleep 3
- /bin/docker system prune -a -f
- sudo sync; sudo echo 3 > /proc/sys/vm/drop_caches
- /bin/systemctl status ecs
- /bin/docker ps -a
For Ubuntu-Based EC2 Instances(Confirm the Docker and system agent paths after Step 2):
schemaVersion: '2.2'
description: "Run Docker cleanup commands on EC2 instances"
mainSteps:
- action: aws:runShellScript
name: runDockerCleanup
inputs:
runCommand:
- /usr/bin/docker stop $(/bin/docker ps -q)
- sleep 2
- /usr/bin/docker system prune -a -f
- sleep 3
- /usr/bin/docker volume ls -qf dangling=true | xargs -r /bin/docker volume rm
- sleep 3
- /usr/bin/systemctl restart ecs
- sleep 3
- /usr/bin/docker system prune -a -f
- sudo sync; sudo echo 3 > /proc/sys/vm/drop_caches
- /usr/bin/systemctl status ecs
- /usr/bin/docker ps -a
NOTE: If your services include cron jobs*, avoid stopping running containers and restarting the ECS service as part of cleanup scripts. This can disrupt scheduled tasks. To modify the provided script for such scenarios, comment out the script lines* docker stop and systemctl restart ecs Use only the prune and dangling volume cleanup commands for such services.
Step4: Verify that your EC2 instance appears in Fleet Manager(found in the Systems Manager left sidebar), as shown in the screenshot below.
Step5: Set up a maintenance window to automate the cleanup using the created document, as shown in the screenshots below.
Step6: Navigate to the Systems Manager console, select "Maintenance Windows" from the left sidebar, choose your maintenance window, and click "Register Targets" to schedule the cleanup command on the selected EC2 instances, as shown in the screenshots below.
Step7: Register a "Tasks" to link the cleanup document with the maintenance window and the registered target instances, as shown in the screenshots below.
💡Note: Check Docker overlays and system storage usingdf -hand monitor memory usage withfree -hon your EC2 instances both before and after executing the cleanup commands. Automating cleanup commands helps prevent hitting storage and memory limits, which could cause loss of EC2 server access and poor performance due to full storage, memory, and CPU usage.