Module 14 - Automation with Python

This repository contains a demo project created as part of my DevOps studies in the TechWorld with Nana – DevOps Bootcamp.

Demo Project: Data Backup & Restore

Technologies used: Python, Boto3, AWS

Project Description:

• Write a Python script that automates creating backups for EC2 Volumes
• Write a Python script that cleans up old EC2 Volume snapshots
• Write a Python script that restores EC2 Volumes

---

Prerequisites

• Python 3.12+ and uv for dependency management
• An AWS account with permissions to manage EC2 instances, EBS volumes and snapshots
• Boto3 — the AWS SDK for Python (EC2 client reference)
• schedule — for running the backups on a recurring basis

Install Python dependencies:

uv sync

Configure AWS credentials at ~/.aws/credentials:

[default]
aws_access_key_id = AKIA...
aws_secret_access_key = ...

And the default region at ~/.aws/config:

[default]
region = eu-central-1

All scripts in this project target eu-central-1. If you use a different region, update the region_name argument (and the AvailabilityZone in the restore script) accordingly.

---

Overview

Create EC2 instances

Launch 2 instances:

• Number of instances: 2
• Instance type: t2.micro
• Storage: 1 volume — 8 GiB

Tag the instances — name one dev and the other prod:

Confirm each instance has its volume attached:

Write a Python script that automates creating backups for EC2 Volumes

Boto3 docs:

• boto3.client
• describe_volumes
• create_snapshot

import boto3

ec2_client = boto3.client('ec2', region_name='eu-central-1')

volumes = ec2_client.describe_volumes()

for volume in volumes['Volumes']:
  new_snapshot = ec2_client.create_snapshot(
    VolumeId=volume['VolumeId']
  )
  print(new_snapshot)

Run

python3 volume-backups.py

Check snapshots in AWS Dashboard

Scheduler implementation

import boto3
import schedule
import time

ec2_client = boto3.client('ec2', region_name='eu-central-1')

def create_volume_snapshots():
  volumes = ec2_client.describe_volumes()
  for volume in volumes['Volumes']:
    new_snapshot = ec2_client.create_snapshot(
      VolumeId=volume['VolumeId']
    )
    print(new_snapshot)

schedule.every().day.do(create_volume_snapshots)

while True:
  schedule.run_pending()
  time.sleep(1)

For a quick demo — shorten the interval to a few seconds:

schedule.every(20).seconds.do(create_volume_snapshots)

Backup only Production Servers

First, tag the production volume. Navigate to the prod instance → Storage → select its attached volume:

Set the volume name to prod:

Then filter describe_volumes by that tag so only the prod volume is backed up (Filters reference):

  volumes = ec2_client.describe_volumes(
    Filters=[
      {
        'Name': 'tag:Name',
        'Values': ['prod']
      }
    ]
  )

python3 volume-backups.py

Check the source volume id — it should match the prod volume:

Write a Python script that cleans up old EC2 Volume snapshots

Doc: describe_snapshots

First, list the snapshots you own, sorted newest-first (reverse=True):

import boto3
from operator import itemgetter

ec2_client = boto3.client('ec2', region_name='eu-central-1')

snapshots = ec2_client.describe_snapshots(
  OwnerIds=['self']
)

sorted_by_date = sorted(snapshots['Snapshots'], key=itemgetter('StartTime'), reverse=True)

for snap in snapshots['Snapshots']:
  print(snap['StartTime'])

print('############')

for snap in sorted_by_date:
  print(snap['StartTime'])

With the list sorted newest-first, slicing [2:] skips the two most recent and deletes everything older:

Boto3 doc: delete_snapshot

import boto3
from operator import itemgetter

ec2_client = boto3.client('ec2', region_name='eu-central-1')

snapshots = ec2_client.describe_snapshots(
  OwnerIds=['self']
)

sorted_by_date = sorted(snapshots['Snapshots'], key=itemgetter('StartTime'), reverse=True)

for snap in sorted_by_date[2:]:
  response = ec2_client.delete_snapshot(
    SnapshotId=snap['SnapshotId']
  )
  print(response)

Run:

python3 cleanup-snapshots.py

Check the AWS dashboard — only the 2 most recent snapshots remain:

Cleanup for multiple volumes

import boto3
from operator import itemgetter

ec2_client = boto3.client('ec2', region_name='eu-central-1')

volumes = ec2_client.describe_volumes(
  Filters=[
    {
      'Name': 'tag:Name',
      'Values': ['prod']
    }
  ]
)

for volume in volumes['Volumes']:
  snapshots = ec2_client.describe_snapshots(
    OwnerIds=['self'],
    Filters=[
      {
        'Name': 'volume-id',
        'Values': [volume['VolumeId']]
      }
    ]
  )

  sorted_by_date = sorted(snapshots['Snapshots'], key=itemgetter('StartTime'), reverse=True)

  for snap in sorted_by_date[2:]:
    response = ec2_client.delete_snapshot(
      SnapshotId=snap['SnapshotId']
    )
    print(response)

Write a Python script that restores EC2 Volumes

Grab the prod instance id from the AWS dashboard:

Boto3 docs:

• boto3.resource
• describe_volumes — filtered by attachment.instance-id
• describe_snapshots
• create_volume
• Volume resource — used to poll vol.state
• Instance resource and its attach_volume action

Get the device id:

A new volume can't reuse a device name already in use, so change the device from /dev/xvda to /dev/xvdb.

Replace instance_id with your own prod instance id, and make sure AvailabilityZone matches the instance's AZ — a volume can only attach to an instance in the same Availability Zone. The script polls vol.state until the new volume is available, then attaches it.

import boto3
import time
from operator import itemgetter

ec2_client = boto3.client('ec2', region_name='eu-central-1')
ec2_resource = boto3.resource('ec2', region_name='eu-central-1')

instance_id = 'i-0b62c30dfaaf087c5'
device_id = '/dev/xvdb'

volumes = ec2_client.describe_volumes(
  Filters=[
    {
      'Name': 'attachment.instance-id',
      'Values': [instance_id]
    }
  ]
)

instance_volume = volumes['Volumes'][0]
print(instance_volume)

snapshots = ec2_client.describe_snapshots(
  OwnerIds=['self'],
  Filters=[
    {
      'Name': 'volume-id',
      'Values': [instance_volume['VolumeId']]
    }
  ]
)

latest_snapshot = sorted(snapshots['Snapshots'], key=itemgetter('StartTime'), reverse=True)[0]
print(latest_snapshot['StartTime'])

new_volume = ec2_client.create_volume(
  SnapshotId=latest_snapshot['SnapshotId'],
  AvailabilityZone='eu-central-1a',
  TagSpecifications=[
    {
      'ResourceType': 'volume',
      'Tags': [
        {
          'Key': 'Name',
          'Value': 'prod'
        }
      ]
    }
  ]
)

while True:
  vol = ec2_resource.Volume(new_volume['VolumeId'])
  print(vol.state)
  if vol.state == 'available':
    ec2_resource.Instance(instance_id).attach_volume(
      VolumeId=new_volume['VolumeId'],
      Device=device_id
    )
    break
  time.sleep(1)

Run:

python3 restore-volume.py

Check the restored volume on AWS — it's attached to the prod instance as a second device:

Clean up

Tear down everything created from the AWS console once you're done:

1. Instances
2. Snapshots
3. Volumes