CI/CD Pipeline with Jenkins, AWS Fargate and Lambda

This guide walks through building a complete CI/CD pipeline from scratch. Every push to GitHub automatically builds a Docker image, runs tests, pushes to AWS ECR, and deploys to Fargate. A serverless Lambda function powers the backend via API Gateway.

Architecture Overview

GitHub push
    ↓ (webhook)
Jenkins (EC2)
    ↓ build & test
ECR (Docker image store)
    ↓ deploy
Fargate (Flask app)
    ↓ calls via API Gateway
Lambda (weather/time functions)

Prerequisites

An AWS account with admin access
A GitHub account
The AWS CLI installed locally
Docker installed locally

Step 1: Create an ECR Repository

ECR is AWS’s private Docker image registry — like Docker Hub but inside your account.

Go to AWS Console → ECR → Create repository
Set visibility to Private
Give it a name (e.g. makers-app)
Click Create repository
Note the full URI — you’ll need it later:

664047078509.dkr.ecr.eu-west-2.amazonaws.com/makers-app

Tip: Make sure you’re in the correct AWS region before creating anything. This guide uses eu-west-2 (London) throughout.

Step 2: Run the CloudFormation Template

CloudFormation creates all the infrastructure you need in one go — VPC, subnets, security groups, IAM roles, and a Jenkins EC2 instance with Jenkins pre-installed.

Go to CloudFormation → Create stack → Upload a template file
Upload resources/deploy_ec2_network_v2.json
Fill in the parameters:
- Stack name — e.g. makers-jenkins-stack
- KeyP — select your existing EC2 key pair
- InstanceType — t2.medium
- ECRRepoName — just the short name, e.g. makers-app (not the full URI)
On the final screen tick the IAM acknowledgement checkbox
Click Create stack and wait for CREATE_COMPLETE

Tip: The template is hardcoded to eu-west-2a. If you get a capacity error, edit the template and change eu-west-2a to eu-west-2b before re-uploading.

Tip: The CloudFormation template automatically creates an IAM role for the Jenkins EC2 with full ECR and ECS permissions. This means Jenkins can push images and deploy to Fargate without any hardcoded AWS credentials.

Once complete, go to the Outputs tab and note:

JenkinsURL — e.g. http://ec2-xx-xx-xx-xx.eu-west-2.compute.amazonaws.com:8080
InstanceDns — the EC2 public DNS

Step 3: Set Up Jenkins

Access Jenkins

Open the JenkinsURL from the CloudFormation Outputs in your browser.

Get the initial password

SSH into the EC2 instance:

chmod 400 your-key.pem
ssh -i your-key.pem ec2-user@<InstanceDns>

Then retrieve the password:

sudo cat /var/lib/jenkins/secrets/initialAdminPassword

Paste it into the browser, install suggested plugins, and create an admin user.

Step 4: Set Up ECS and Fargate

You need three things: a cluster, a task definition, and a service.

Create an ECS Cluster

Go to ECS → Create cluster
Name it (e.g. makers-cluster)
Infrastructure: Fargate
Click Create

Create a Task Definition

Go to Task Definitions → Create new task definition
Fill in:
- Family name — e.g. makers-task
- Launch type — Fargate
- CPU — 1 vCPU
- Memory — 3 GB
- Container name — makers-app
- Image URI — <your-ecr-uri>:latest
- Container port — 5000
Click Create

Tip: Start with at least 1 vCPU and 3 GB memory. Exit code 137 means the container was killed due to insufficient memory.

Create an ECS Service

Go to your cluster → Services → Create
Fill in:
- Launch type — Fargate
- Task definition — select your task definition
- Service name — e.g. makers-service
- Desired tasks — 1
Under Networking:
- Select the Lab VPC (created by CloudFormation)
- Select the LabSubnetPub1 subnet only
- Auto-assign public IP — Enabled
- Security group — ensure port 5000 is open inbound
Click Create

Tip: Always use the public subnet created by CloudFormation, not the default VPC subnets. Using the wrong subnet is a common reason the app isn’t reachable.

Tip: The service will show as failing until Jenkins pushes the first image to ECR. Come back to verify it’s running after the pipeline runs successfully.

Step 5: Create the App Repository

The Flask app needs its own private GitHub repository — separate from any course materials repo.

# Copy just the app folder
cp -r /path/to/source/app ~/Desktop/my-app
cd ~/Desktop/my-app

# Create .gitignore
echo "__pycache__/" > .gitignore
echo "*.pyc" >> .gitignore
echo ".env" >> .gitignore
echo "venv/" >> .gitignore

# Initialise git
git init
git add .
git commit -m "initial commit"

Create a private repository on GitHub (no README, no .gitignore, no licence), then push:

git remote add origin https://github.com/YOUR_USERNAME/your-repo.git
git push -u origin main

Tip: Keep the repo private. This forces you to learn credential management in Jenkins, which is standard practice in industry.

Step 6: Add GitHub Credentials to Jenkins

Jenkins needs a GitHub token to clone your private repository.

Generate a GitHub fine-grained token

Go to GitHub → Settings → Developer Settings → Personal Access Tokens → Fine-grained tokens → Generate new token
Set Repository access to only your app repo
Set permissions:
- Commit statuses: Read and write
- Contents: Read-only
- Metadata: Read-only
Generate and copy the token immediately — you won’t see it again

Add to Jenkins

Go to Manage Jenkins → Credentials → System → Global credentials → Add Credentials
Fill in:
- Kind — Username with password
- Username — your GitHub username
- Password — paste the token
- ID — github-credentials
Click Create

Step 7: Create the Jenkins Pipeline

Create a Jenkinsfile

Add a Jenkinsfile to your repo root:

pipeline {
    agent any
    environment {
        GIT_CREDENTIALS  = 'github-credentials'
        AWS_REGION       = 'eu-west-2'
        ECR_REPO         = '<your-ecr-uri>'
        ECS_CLUSTER      = '<your-cluster-name>'
        ECS_SERVICE      = '<your-service-name>'
    }

    stages {
        stage('Clone Repo') {
            steps {
                git branch: 'main', credentialsId: env.GIT_CREDENTIALS, url: 'https://github.com/YOUR_USERNAME/your-repo'
            }
        }

        stage('Install Dependencies') {
            steps {
                sh 'pip3 install -r requirements.txt'
            }
        }

        stage('Run Tests') {
            steps {
                sh 'python3 -m pytest tests/ -v'
            }
        }

        stage('Build Docker Image') {
            steps {
                sh 'docker build -t $ECR_REPO:latest .'
            }
        }

        stage('Push to ECR') {
            steps {
                sh '''
                    aws ecr get-login-password --region $AWS_REGION | \
                    docker login --username AWS --password-stdin $ECR_REPO
                    docker push $ECR_REPO:latest
                '''
            }
        }

        stage('Deploy to Fargate') {
            steps {
                sh '''
                    aws ecs update-service \
                        --cluster $ECS_CLUSTER \
                        --service $ECS_SERVICE \
                        --force-new-deployment \
                        --region $AWS_REGION
                '''
            }
        }
    }

    post {
        success {
            echo 'Pipeline succeeded! Your app is deploying to Fargate.'
        }
        failure {
            echo 'Pipeline failed. Check the console output above for errors.'
        }
    }
}

Tip: If your Jenkinsfile is at the repo root (i.e. the Dockerfile is also at the root), use requirements.txt and tests/ without any prefix. If the app is in a subdirectory, adjust paths accordingly.

Push the Jenkinsfile to GitHub:

git add Jenkinsfile
git commit -m "add Jenkinsfile"
git push

Create the pipeline in Jenkins

Jenkins → New Item → Pipeline
Give it a name and click OK
Under Pipeline, set Definition to Pipeline script from SCM
Fill in:
- SCM — Git
- Repository URL — your GitHub repo URL
- Credentials — github-credentials
- Branch — */main
- Script Path — Jenkinsfile
Save

Step 8: Fix the Deliberate Failing Test

The test suite includes a bug — the tests call /quote but the Flask route is /quotes. This is intentional: the pipeline will fail here so you experience CI catching a real bug.

Open tests/test_app.py and change all occurrences of:

client.get('/quote')

to:

client.get('/quotes')

Push the fix:

git add tests/test_app.py
git commit -m "fix: correct quote route in tests"
git push

Step 9: Set Up the GitHub Webhook

Webhooks make Jenkins trigger automatically on every push — no more clicking “Build Now” manually.

Add the webhook to GitHub

Go to your GitHub repo → Settings → Webhooks → Add webhook
Fill in:
- Payload URL — http://<InstanceDns>:8080/github-webhook/
- Content type — application/json
- Which events — Just the push event
Click Add webhook

Enable the trigger in Jenkins

Go to your pipeline → Configure
Under Build Triggers tick GitHub hook trigger for GITScm polling
Save

Tip: This only works because Jenkins is running on an EC2 with a public IP. It won’t work with Jenkins on localhost because GitHub can’t reach it.

Test it by pushing any change to your repo and watching Jenkins trigger automatically.

Step 10: Deploy the Lambda Function

Create the function

Go to AWS Lambda → Create function
Select Author from scratch
Fill in:
- Function name — e.g. makers-weather-lambda
- Runtime — Python 3.11
Click Create function

Add the code

Copy the contents of lambdas/your_first_lambda.py into the Lambda code editor, replacing the default placeholder. Click Deploy.

Test it

Click Test, create a test event using the default Hello World template, and run it. You should see a weather response in the output.

Step 11: Create the API Gateway

API Gateway provides the HTTP endpoint that the Flask app calls to invoke Lambda.

Go to API Gateway → Create API → HTTP API → Build
Click Add integration → Lambda → select your function
Configure routes:
- Method — GET
- Resource path — /weather
Leave stage name as $default
Click Create

Note the Invoke URL from the API overview page.

Test it from your terminal:

curl "https://YOUR_API_ID.execute-api.eu-west-2.amazonaws.com/weather?city=london"

Set up CORS

Go to Develop → CORS → Configure
Set:
- Access-Control-Allow-Origin — *
- Access-Control-Allow-Methods — GET
- Access-Control-Allow-Headers — content-type
Click Save

Step 12: Wire Lambda to the Flask App

The Flask app reads the Lambda URL from an environment variable. Add it to the ECS Task Definition.

Go to ECS → Task Definitions → your task → Create new revision
Click on the container, scroll to Environment variables
Add:
- Key — LAMBDA_WEATHER_URL
- Value — https://YOUR_API_ID.execute-api.eu-west-2.amazonaws.com/weather
Create the new revision
Go to your service → Update service → select the new revision → Force new deployment → Update

Once the new task is running, open http://<task-public-ip>:5000/lambda and click Get the weather.

Common Errors and Fixes

Error	Cause	Fix
`CREATE_FAILED` — no capacity in `eu-west-2a`	AZ doesn’t have `t2.medium` available	Change `eu-west-2a` to `eu-west-2b` in the template
`ServiceNotActiveException`	ECS service name in Jenkinsfile doesn’t match actual service name	Check the exact service name in ECS console and update `ECS_SERVICE` in Jenkinsfile
Exit code 137	Container killed due to out of memory	Increase task CPU to 1 vCPU and memory to 3 GB in task definition
Tests fail on `/quote` route	Deliberate bug — test calls `/quote` but route is `/quotes`	Fix test to use `/quotes`
App not reachable on port 5000	Wrong VPC or subnet, or port 5000 not open in security group	Use the Lab VPC public subnet and ensure inbound port 5000 is allowed
`remote origin already exists`	Already have a remote set	Run `git remote remove origin` then re-add

Bonus: Deploy the Time Lambda

Follow the same steps as the weather Lambda using lambdas/bonus_lambda.py:

Create a new Lambda function called tell-time
Deploy bonus_lambda.py code
Create a new API Gateway HTTP API with a GET route at /time
Add LAMBDA_TIME_URL as an environment variable in a new task definition revision
Update the ECS service

The /lambda page already has the “What time is it?” card built in — it just needs the LAMBDA_TIME_URL env var to activate.