Configuring a Python Environment to Automatically Run on EC2 Instance Startup

https://a0.awsstatic.com/libra-css/images/logos/aws_logo_smile_1200x630.png

Introduction

In this guide, we’ll walk you through the process of configuring a Python environment to automatically run on Amazon EC2 instance startup. This setup allows you to execute a Python script as soon as your EC2 is launched, making it a convenient solution for various automation tasks.

We’ll start by connecting to your EC2 instance and proceed to set up the necessary tools, clone a repository, create a Python virtual environment, and craft a custom startup script customized to your needs.

Additionally, we’ll explore a use case that highlights the potential of this automated setup in a broader architectural context. By integrating AWS Glue and EC2 instances, you can achieve a powerful combination for orchestrating complex data workflows, thereby leveraging the scalability and flexibility of Amazon Web Services.

Steps to Configure the EC2

Step 1: Connect to the EC2 Instance

Before you begin, make sure you’ve created an EC2 instance, for this demonstration, we’ll use a t2.medium Amazon Linux instance.

Connect to your EC2 instance using the AWS Management Console’s “Connect” feature. This method simplifies the connection process and allows you to access your EC2 instance with ease:

Once connected to the instance, execute all the code from the following steps.

Step 2: Install Required Packages

Install the necessary tools for setting up your environment:

sudo yum install git -y
sudo yum install python3-pip -y
pip3 install git-remote-codecommit

Note that in this example, we are using AWS CodeCommit, which is a managed source control service provided by Amazon Web Services. However, you can substitute this with other Git repository URLs or providers, depending on your project’s requirements.

Step 3: Clone the Repository

Clone the desired repository using the git command:

git clone codecommit::us-east-1://YourRepositoryName
cd YourRepositoryName

Be sure to replace YourRepositoryName with your repository's name.

It’s essential to note that the repository you are cloning should include the necessary Python files (.py) and a requirements.txt file.

• Python Executable Files (.py): The Python scripts that you intend to run during the EC2 instance startup should be present in the repository.
• requirements.txt: This file lists the Python packages and libraries required for your scripts.

Step 4: Create a Python Virtual Environment

Create a Python virtual environment to isolate your project’s dependencies and install all packages and libraries required:

python3 -m venv /home/ec2-user/venv
source /home/ec2-user/venv/bin/activate
pip3 install -r requirements.txt

Step 5: Create and Configure the Startup Script

Create and configure a script named startup.sh in the /home/ec2-user directory:

cd ..
vim startup.sh

Copy and paste the following content into the startup.sh file:

#!/bin/bash

source /home/ec2-user/venv/bin/activate
cd /home/ec2-user/YourRepositoryName
git pull

python3 your_script.py

# Check if the "Shutdown" tag is set to "True" to determine whether to shut down the instance
Shutdown="$(aws ec2 describe-tags --region "us-east-1" --filters "Name=resource-id,Values=your_instance_id" "Name=key,Values=Shutdown" --query 'Tags[*].Value' --output text)"
if [ $Shutdown == "True" ]
then
   sudo shutdown now -h
fi

To stop the instance, you need to create a ‘shutdown’ tag in the EC2 instance setup. Its value is used to determine whether the instance should shut down automatically.

Step 6: Make the Script Executable

Make the startup.sh script executable:

sudo chmod +x /home/ec2-user/startup.sh

Step 7: Configure the System Startup Script

Open the /etc/rc.d/rc.local file to configure the system's startup script:

sudo vim /etc/rc.d/rc.local

Copy and paste the following content into the /etc/rc.d/rc.local file:

#!/bin/bash

exec 1>/tmp/rc.local.log 2>&1
set -x
touch /var/lock/subsys/local
sh /home/ec2-user/startup.sh

exit 0

Step 8: Make the System Startup Script Executable

Make the /etc/rc.local file executable:

sudo chmod +x /etc/rc.d/rc.local

Step 9: Create the Log File

Create the log file /tmp/rc.local.log:

sudo touch /tmp/rc.local.log

Step 10: Reboot Manually

restart the instance to test:

sudo reboot

Step 11: Check the Log

Check the log generated by the startup script to ensure everything works as expected:

cat /tmp/rc.local.log

Use case

Now, let’s consider a broader architectural perspective. Imagine a scenario where a product deployment workflow is orchestrated using a combination of AWS Glue and EC2 instances.

AWS Glue:

Initially, AWS Glue performs the Extract, Transform, Load (ETL) process, working with data stored in Amazon S3. It processes, transforms, and aggregates this data as required, effectively preparing it for analysis.

Using this method in the end of the Glue job, you can trigger the EC2 instance:

def ec2_activate(instance_id, region):

    ec2 = boto3.client('ec2', region_name=region)
    cond = True
    while cond == True:
        response = ec2.describe_instances(InstanceIds=instance_id)
        instances = response['Reservations'][0]['Instances']
        instance_state = instances[0]['State']['Name']
        if (instance_state == 'running') or (instance_state == 'pending') or (instance_state == 'stopping'):
            continue
        else:
            ec2.start_instances(InstanceIds=instance_id)
            break

ec2_activate(instance_id, region)

EC2 Instance:

AWS Glue, after completing the ETL process, triggers the launch of an EC2 instance. This EC2 instance serves as the computing engine for more complex tasks.

This architecture offers a robust and streamlined approach to handling complex data workflows. By integrating AWS Glue and EC2 instances for specialized computations, you create an end-to-end solution that efficiently manages data, processes, and delivers valuable results while taking full advantage of AWS’s scalable and flexible infrastructure.

Conclusion

By following the steps outlined in this guide, you’ve successfully set up an automated Python environment to run on Amazon EC2 instance startup.

The advantage of running Python in this manner is the ability to leverage EC2 instance tags, allowing you to turn instances on and off as needed. By doing so, you can efficiently manage costs and resources across different areas and products.

This method represents a practical way to achieve this automation. Thank you for reading!