Prevent Storage Issues: Automated EC2 Cleanup Using AWS Systems Manager

Deploying services on AWS EC2 instances using Docker over time can lead to high storage, memory, and CPU usage. Manually increasing volume size and running prune commands to clear overlay storage and reduce memory usage is not a sustainable solution. To address this, we can automate the cleanup of EC2 instances using AWS Systems Manager.

By setting up a maintenance window with a cron expression, you can automate the regular cleanup of instances without manual intervention. This setup can be customized to run daily, during off-hours, or at specific intervals such as every few days or weeks. While there are other scheduling options available, configuring a cron expression is typically the easiest. This automation helps maintain optimal storage utilization and performance. Below is a step-by-step guide with screenshots to assist you through each stage of the process.

Step1: In the AWS Management Console, navigate to EC2, locate the IAM role for your instance, and attach the AmazonSSMManagedInstanceCore policy. This grants Systems Manager the permissions needed to manage and clean up your EC2 instances.

Step2: After completing Step 1, connect to the EC2 instance using Session Manager. Verify the Docker and system agent paths by running the following commands: which docker and which systemctl.

Step3: Go to AWS Systems Manager, click on "Documents" in the left sidebar, and create a cleanup document as shown in the screenshots.

• For AWS Linux-Based EC2 Instances(Confirm the Docker and system agent paths after Step 2):

    schemaVersion: '2.2'
    description: "Run Docker cleanup commands on EC2 instances"
    mainSteps:
      - action: aws:runShellScript
        name: runDockerCleanup
        inputs:
          runCommand:
            - /bin/docker stop $(/bin/docker ps -q)
            - sleep 2
            - /bin/docker system prune -a -f
            - sleep 3
            - /bin/docker volume ls -qf dangling=true | xargs -r /bin/docker volume rm
            - sleep 3
            - /bin/systemctl restart ecs
            - sleep 3
            - /bin/docker system prune -a -f
            - sudo sync; sudo echo 3 > /proc/sys/vm/drop_caches
            - /bin/systemctl status ecs
            - /bin/docker ps -a
  

• For Ubuntu-Based EC2 Instances(Confirm the Docker and system agent paths after Step 2):

schemaVersion: '2.2'
description: "Run Docker cleanup commands on EC2 instances"
mainSteps:
  - action: aws:runShellScript
    name: runDockerCleanup
    inputs:
      runCommand:
        - /usr/bin/docker stop $(/bin/docker ps -q)
        - sleep 2
        - /usr/bin/docker system prune -a -f
        - sleep 3
        - /usr/bin/docker volume ls -qf dangling=true | xargs -r /bin/docker volume rm
        - sleep 3
        - /usr/bin/systemctl restart ecs
        - sleep 3
        - /usr/bin/docker system prune -a -f
        - sudo sync; sudo echo 3 > /proc/sys/vm/drop_caches
        - /usr/bin/systemctl status ecs
        - /usr/bin/docker ps -a

NOTE: If your services include cron jobs*, avoid stopping running containers and restarting the ECS service as part of cleanup scripts. This can disrupt scheduled tasks. To modify the provided script for such scenarios, comment out the script lines* docker stop and systemctl restart ecs Use only the prune and dangling volume cleanup commands for such services.

• After the document is created, you can view and verify it as shown in the screenshots below.

Step4: Verify that your EC2 instance appears in Fleet Manager(found in the Systems Manager left sidebar), as shown in the screenshot below.

Step5: Set up a maintenance window to automate the cleanup using the created document, as shown in the screenshots below.

Step6: Navigate to the Systems Manager console, select "Maintenance Windows" from the left sidebar, choose your maintenance window, and click "Register Targets" to schedule the cleanup command on the selected EC2 instances, as shown in the screenshots below.

Step7: Register a "Tasks" to link the cleanup document with the maintenance window and the registered target instances, as shown in the screenshots below.

• Search for the created command document name in the command document search bar and select it.

• Select the target you registered earlier in the maintenance window. Set the concurrency value to 1 (tasks run on one instance at a time) and the error threshold value to 1 (task stops if one instance fails). This ensures sequential execution and halts on errors for accurate control.

• After registering the task, you can view and check the details, including description, tasks, and targets, as shown in the screenshots below.

• Check the history to verify the success or failure status of the cleanup commands executed on EC2 instances during the configured time period, as shown in the screenshot below.

💡

Note: Check Docker overlays and system storage using df -h and monitor memory usage with free -h on your EC2 instances both before and after executing the cleanup commands. Automating cleanup commands helps prevent hitting storage and memory limits, which could cause loss of EC2 server access and poor performance due to full storage, memory, and CPU usage.