CI/CD Pipeline Optimization: Build Caching, Parallel Jobs, and Deployment Strategies

Introduction

Slow CI/CD pipelines waste developer time and delay releases. This guide covers proven techniques to optimize pipeline performance including build caching, parallel job execution, and efficient deployment strategies across popular CI/CD platforms.

Build Caching

Why Caching Matters

Without caching:

Build 1: npm install (5 min) → tests (2 min) = 7 min
Build 2: npm install (5 min) → tests (2 min) = 7 min
Build 3: npm install (5 min) → tests (2 min) = 7 min
Total: 21 minutes

With caching:

Build 1: npm install (5 min) → tests (2 min) = 7 min
Build 2: npm install (10 sec cache) → tests (2 min) = 2.2 min
Build 3: npm install (10 sec cache) → tests (2 min) = 2.2 min
Total: 11.4 minutes (46% faster)

GitHub Actions Caching

What is GitHub Actions Caching?

GitHub Actions provides a built-in caching mechanism that stores files between workflow runs. This dramatically reduces build times by reusing previously downloaded dependencies, compiled artifacts, and other files that don’t change frequently.

How it works:

Cache Key - A unique identifier for your cache (usually based on file hashes)
Cache Path - The directory/files to cache
Restore Keys - Fallback keys if exact match not found
Automatic Cleanup - GitHub removes caches not accessed in 7 days

Key concepts:

Cache hit - Exact cache key match found, cache restored
Cache miss - No match, dependencies downloaded fresh
Partial restore - Restore key matches, provides base to build on

When to use caching:

Package dependencies (npm, pip, maven, etc.)
Build outputs that take time to generate
Docker layers
Test data or fixtures

Node.js dependencies example:

name: CI

on: [push]

jobs:
  build:
    runs-on: ubuntu-latest

    steps:
      - uses: actions/checkout@v3

      # Cache npm dependencies
      - name: Cache node modules
        uses: actions/cache@v3
        with:
          # What to cache
          path: ~/.npm

          # Cache key (unique per OS and package-lock.json hash)
          key: ${{ runner.os }}-node-${{ hashFiles('**/package-lock.json') }}

          # Fallback keys if exact match not found
          restore-keys: |
            ${{ runner.os }}-node-

      - name: Install dependencies
        run: npm ci  # If cache hit, this is very fast

      - name: Run tests
        run: npm test

How the cache key works:

key: ${{ runner.os }}-node-${{ hashFiles('**/package-lock.json') }}
# Example: "Linux-node-a3f2b1c8..."

# If package-lock.json changes:
# New hash → new key → cache miss → fresh install → new cache created

# If package-lock.json unchanged:
# Same hash → same key → cache hit → restore from cache (fast!)

Docker layer caching:

Docker builds in layers, and each layer can be cached. This is extremely powerful for reducing build times because unchanged layers don’t need to be rebuilt.

Why Docker caching matters:

Each Dockerfile instruction creates a layer
Layers are cached based on instruction and content
Changing one layer invalidates all subsequent layers
Proper caching can reduce 10-minute builds to 30 seconds

Example: If you only change application code, you don’t need to reinstall dependencies.

name: Docker Build

on: [push]

jobs:
  docker:
    runs-on: ubuntu-latest

    steps:
      - uses: actions/checkout@v3

      # Enable Docker BuildKit (required for advanced caching)
      - name: Set up Docker Buildx
        uses: docker/setup-buildx-action@v2

      # Cache Docker layers between runs
      - name: Cache Docker layers
        uses: actions/cache@v3
        with:
          path: /tmp/.buildx-cache
          key: ${{ runner.os }}-buildx-${{ github.sha }}
          restore-keys: |
            ${{ runner.os }}-buildx-

      # Build with cache
      - name: Build Docker image
        uses: docker/build-push-action@v4
        with:
          context: .
          push: false
          tags: myapp:latest

          # Use cached layers from previous builds
          cache-from: type=local,src=/tmp/.buildx-cache

          # Save new layers to cache
          cache-to: type=local,dest=/tmp/.buildx-cache-new,mode=max

      # Update cache (avoids cache bloat)
      - name: Move cache
        run: |
          rm -rf /tmp/.buildx-cache
          mv /tmp/.buildx-cache-new /tmp/.buildx-cache

Cache optimization tip:

# Bad: Changes to app code rebuild everything
COPY . .
RUN npm install
RUN npm run build

# Good: Dependencies cached separately
COPY package*.json ./
RUN npm install  # Cached until package.json changes
COPY . .
RUN npm run build  # Only this rebuilds when code changes

Multiple cache layers:

- name: Cache Go modules
  uses: actions/cache@v3
  with:
    path: ~/go/pkg/mod
    key: ${{ runner.os }}-go-${{ hashFiles('**/go.sum') }}

- name: Cache Go build
  uses: actions/cache@v3
  with:
    path: ~/.cache/go-build
    key: ${{ runner.os }}-go-build-${{ hashFiles('**/*.go') }}

GitLab CI Caching

Basic cache:

# .gitlab-ci.yml

stages:
  - build
  - test

cache:
  key: ${CI_COMMIT_REF_SLUG}
  paths:
    - node_modules/
    - .npm/

build:
  stage: build
  script:
    - npm ci --cache .npm --prefer-offline
    - npm run build
  artifacts:
    paths:
      - dist/

test:
  stage: test
  script:
    - npm test

Per-job caching:

build-frontend:
  stage: build
  cache:
    key: frontend-${CI_COMMIT_REF_SLUG}
    paths:
      - frontend/node_modules/
  script:
    - cd frontend
    - npm ci
    - npm run build

build-backend:
  stage: build
  cache:
    key: backend-${CI_COMMIT_REF_SLUG}
    paths:
      - backend/node_modules/
  script:
    - cd backend
    - npm ci
    - npm run build

Docker layer caching (GitLab):

build-image:
  stage: build
  image: docker:latest
  services:
    - docker:dind
  variables:
    DOCKER_DRIVER: overlay2
    DOCKER_BUILDKIT: 1
  before_script:
    - docker login -u $CI_REGISTRY_USER -p $CI_REGISTRY_PASSWORD $CI_REGISTRY
  script:
    # Pull previous image for cache
    - docker pull $CI_REGISTRY_IMAGE:latest || true

    # Build with cache
    - >
      docker build
      --cache-from $CI_REGISTRY_IMAGE:latest
      --tag $CI_REGISTRY_IMAGE:$CI_COMMIT_SHA
      --tag $CI_REGISTRY_IMAGE:latest
      .

    # Push new image
    - docker push $CI_REGISTRY_IMAGE:$CI_COMMIT_SHA
    - docker push $CI_REGISTRY_IMAGE:latest

Jenkins Caching

Declarative pipeline with caching:

pipeline {
    agent any

    environment {
        NPM_CACHE = "${WORKSPACE}/.npm-cache"
    }

    stages {
        stage('Install Dependencies') {
            steps {
                // Cache npm packages
                cache(maxCacheSize: 250, caches: [
                    arbitraryFileCache(
                        path: "${NPM_CACHE}",
                        cacheValidityDecidingFile: 'package-lock.json'
                    )
                ]) {
                    sh 'npm ci --cache ${NPM_CACHE}'
                }
            }
        }

        stage('Build') {
            steps {
                sh 'npm run build'
            }
        }

        stage('Test') {
            steps {
                sh 'npm test'
            }
        }
    }
}

Docker layer caching (Jenkins):

pipeline {
    agent any

    environment {
        DOCKER_BUILDKIT = '1'
    }

    stages {
        stage('Build Image') {
            steps {
                script {
                    // Pull previous image
                    sh 'docker pull myapp:latest || true'

                    // Build with cache
                    sh '''
                        docker build \
                          --cache-from myapp:latest \
                          --tag myapp:${GIT_COMMIT} \
                          --tag myapp:latest \
                          .
                    '''

                    // Push to registry
                    sh '''
                        docker push myapp:${GIT_COMMIT}
                        docker push myapp:latest
                    '''
                }
            }
        }
    }
}

Parallel Job Execution

What is parallel execution?

Instead of running jobs sequentially (one after another), parallel execution runs multiple jobs simultaneously. This is one of the most effective ways to reduce total pipeline time.

Sequential execution:

Lint (2 min) → Unit Tests (5 min) → Integration Tests (8 min) → Deploy (3 min)
Total: 18 minutes

Parallel execution:

┌─ Lint (2 min) ────────┐
├─ Unit Tests (5 min) ──┤→ Deploy (3 min)
└─ Integration Tests (8 min)─┘
Total: 11 minutes (39% faster)

When to use parallel jobs:

Independent tests (unit, integration, e2e)
Building multiple services (frontend, backend, mobile)
Testing across different environments (OS, versions)
Linting and security scans

Key principle: Jobs can run in parallel if they don’t depend on each other’s outputs.

GitHub Actions Parallelization

Matrix strategy:

Matrix builds automatically create multiple jobs from a single configuration. Perfect for testing across multiple versions, operating systems, or configurations.

How it works:

Define matrix variables (e.g., node-version, os)
GitHub creates one job per combination
All matrix jobs run in parallel

name: Test Matrix

on: [push]

jobs:
  test:
    runs-on: ubuntu-latest

    strategy:
      matrix:
        # Creates 9 jobs (3 versions × 3 OS = 9 combinations)
        node-version: [16, 18, 20]
        os: [ubuntu-latest, macos-latest, windows-latest]

    steps:
      - uses: actions/checkout@v3

      - name: Setup Node.js ${{ matrix.node-version }}
        uses: actions/setup-node@v3
        with:
          node-version: ${{ matrix.node-version }}

      - name: Install dependencies
        run: npm ci

      - name: Run tests
        run: npm test

# Result: All 9 jobs run simultaneously
# Time: ~5 minutes (instead of 45 minutes sequentially)

Parallel jobs:

jobs:
  # Run in parallel
  lint:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v3
      - run: npm ci
      - run: npm run lint

  unit-tests:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v3
      - run: npm ci
      - run: npm run test:unit

  integration-tests:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v3
      - run: npm ci
      - run: npm run test:integration

  # Deploy waits for all tests
  deploy:
    needs: [lint, unit-tests, integration-tests]
    runs-on: ubuntu-latest
    steps:
      - run: echo "Deploying..."

Conditional parallel execution:

jobs:
  changes:
    runs-on: ubuntu-latest
    outputs:
      frontend: ${{ steps.filter.outputs.frontend }}
      backend: ${{ steps.filter.outputs.backend }}
    steps:
      - uses: actions/checkout@v3
      - uses: dorny/paths-filter@v2
        id: filter
        with:
          filters: |
            frontend:
              - 'frontend/**'
            backend:
              - 'backend/**'

  build-frontend:
    needs: changes
    if: ${{ needs.changes.outputs.frontend == 'true' }}
    runs-on: ubuntu-latest
    steps:
      - run: echo "Building frontend..."

  build-backend:
    needs: changes
    if: ${{ needs.changes.outputs.backend == 'true' }}
    runs-on: ubuntu-latest
    steps:
      - run: echo "Building backend..."

GitLab CI Parallelization

Parallel keyword:

test:
  stage: test
  parallel: 5  # Run 5 instances in parallel
  script:
    - npm run test -- --shard=$CI_NODE_INDEX/$CI_NODE_TOTAL

# Or with matrix
test-matrix:
  stage: test
  parallel:
    matrix:
      - NODE_VERSION: [16, 18, 20]
        OS: [ubuntu, alpine]
  image: node:${NODE_VERSION}-${OS}
  script:
    - npm test

Independent jobs:

stages:
  - build
  - test
  - deploy

# Run in parallel (same stage)
build-frontend:
  stage: build
  script:
    - cd frontend && npm run build

build-backend:
  stage: build
  script:
    - cd backend && npm run build

# Run in parallel (same stage)
test-unit:
  stage: test
  script:
    - npm run test:unit

test-integration:
  stage: test
  script:
    - npm run test:integration

test-e2e:
  stage: test
  script:
    - npm run test:e2e

# Runs after all tests pass
deploy:
  stage: deploy
  script:
    - ./deploy.sh

Jenkins Parallelization

Parallel stages:

pipeline {
    agent any

    stages {
        stage('Build') {
            parallel {
                stage('Build Frontend') {
                    steps {
                        dir('frontend') {
                            sh 'npm ci'
                            sh 'npm run build'
                        }
                    }
                }

                stage('Build Backend') {
                    steps {
                        dir('backend') {
                            sh 'npm ci'
                            sh 'npm run build'
                        }
                    }
                }

                stage('Build Mobile') {
                    steps {
                        dir('mobile') {
                            sh 'npm ci'
                            sh 'npm run build'
                        }
                    }
                }
            }
        }

        stage('Test') {
            parallel {
                stage('Unit Tests') {
                    steps {
                        sh 'npm run test:unit'
                    }
                }

                stage('Integration Tests') {
                    steps {
                        sh 'npm run test:integration'
                    }
                }

                stage('E2E Tests') {
                    steps {
                        sh 'npm run test:e2e'
                    }
                }
            }
        }

        stage('Deploy') {
            steps {
                sh './deploy.sh'
            }
        }
    }
}

Dynamic parallel stages:

pipeline {
    agent any

    stages {
        stage('Test Microservices') {
            steps {
                script {
                    def services = ['auth', 'users', 'orders', 'payments']
                    def parallelStages = [:]

                    services.each { service ->
                        parallelStages[service] = {
                            stage("Test ${service}") {
                                sh "cd services/${service} && npm test"
                            }
                        }
                    }

                    parallel parallelStages
                }
            }
        }
    }
}

Optimization Techniques

1. Skip Unnecessary Steps

GitHub Actions:

jobs:
  build:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v3

      # Only run on specific paths
      - name: Check if build needed
        id: changes
        uses: dorny/paths-filter@v2
        with:
          filters: |
            src:
              - 'src/**'
              - 'package.json'

      - name: Build
        if: steps.changes.outputs.src == 'true'
        run: npm run build

      # Skip on draft PRs
      - name: Deploy
        if: github.event.pull_request.draft == false
        run: ./deploy.sh

GitLab CI:

build:
  stage: build
  script:
    - npm run build
  only:
    changes:
      - src/**
      - package.json

deploy:
  stage: deploy
  script:
    - ./deploy.sh
  except:
    variables:
      - $CI_MERGE_REQUEST_TITLE =~ /^WIP:/

2. Fail Fast

GitHub Actions:

jobs:
  test:
    strategy:
      fail-fast: true  # Stop all jobs on first failure
      matrix:
        node: [16, 18, 20]
    steps:
      - run: npm test

  # Or check before expensive operations
  lint:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v3
      - run: npm run lint

  test:
    needs: lint  # Only run if lint passes
    runs-on: ubuntu-latest
    steps:
      - run: npm test

3. Optimize Docker Builds

Multi-stage build optimization:

# Dependency stage (cached separately)
FROM node:18-alpine AS deps
WORKDIR /app
COPY package*.json ./
RUN npm ci --only=production

# Build stage
FROM node:18-alpine AS builder
WORKDIR /app
COPY package*.json ./
RUN npm ci
COPY . .
RUN npm run build

# Production stage (minimal)
FROM node:18-alpine
WORKDIR /app
COPY --from=deps /app/node_modules ./node_modules
COPY --from=builder /app/dist ./dist
CMD ["node", "dist/index.js"]

BuildKit cache mounts:

# syntax=docker/dockerfile:1

FROM node:18-alpine AS builder

WORKDIR /app

# Use cache mount for npm
RUN --mount=type=cache,target=/root/.npm \
    npm ci

COPY . .

# Use cache mount for build cache
RUN --mount=type=cache,target=/app/.next/cache \
    npm run build

4. Artifact Reuse

What are artifacts?

Artifacts are files produced by one job that can be reused by subsequent jobs. Instead of rebuilding the same files multiple times, you build once and share the output across jobs.

Why use artifacts:

Avoid redundant work - Build once, use multiple times
Consistency - All jobs use the exact same build output
Faster pipelines - No need to rebuild for each test/deploy stage
Debugging - Download artifacts to investigate failures

Common use cases:

Compiled applications
Built frontend assets
Test results and coverage reports
Docker images (via registries)

Without artifacts:

Build Job: checkout → install → build (5 min)
Test Job: checkout → install → build (5 min) → test (3 min)
Deploy Job: checkout → install → build (5 min) → deploy (2 min)
Total: 15 minutes of redundant building

With artifacts:

Build Job: checkout → install → build (5 min) → upload artifact
Test Job: download artifact (10 sec) → test (3 min)
Deploy Job: download artifact (10 sec) → deploy (2 min)
Total: Save 10 minutes

GitHub Actions example:

jobs:
  build:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v3
      - run: npm run build

      # Upload build artifacts
      - uses: actions/upload-artifact@v3
        with:
          name: build-output
          path: dist/
          retention-days: 1  # Auto-delete after 1 day

  test:
    needs: build  # Wait for build to complete
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v3

      # Download artifacts (fast - just downloads, no rebuild)
      - uses: actions/download-artifact@v3
        with:
          name: build-output
          path: dist/

      - run: npm test  # Test against built files

  deploy:
    needs: [build, test]  # Wait for both to complete
    runs-on: ubuntu-latest
    steps:
      # Only download what we need for deployment
      - uses: actions/download-artifact@v3
        with:
          name: build-output

      - run: ./deploy.sh  # Deploy the exact same build that was tested

GitLab CI:

build:
  stage: build
  script:
    - npm run build
  artifacts:
    paths:
      - dist/
    expire_in: 1 day

test:
  stage: test
  dependencies:
    - build
  script:
    - npm test  # Uses dist/ from build job

deploy:
  stage: deploy
  dependencies:
    - build
  script:
    - ./deploy.sh  # Uses dist/ from build job

5. Resource Optimization

Self-hosted runners (GitHub):

jobs:
  build:
    runs-on: self-hosted  # Faster, more resources
    steps:
      - run: npm run build

Larger runners (GitLab):

build:
  stage: build
  tags:
    - docker
    - large-cpu  # Custom runner tag
  script:
    - npm run build

Node labels (Jenkins):

pipeline {
    agent {
        label 'large-node'  // Use powerful node
    }

    stages {
        stage('Build') {
            steps {
                sh 'npm run build'
            }
        }
    }
}

Deployment Strategies

Choosing the right deployment strategy reduces risk and improves reliability. Different strategies offer different trade-offs between speed, safety, and resource usage.

1. Incremental Deployment

What is it?

Incremental deployment only deploys the components that actually changed. If you have a monorepo with multiple services, you don’t need to redeploy everything when only one service changes.

Benefits:

Faster deployments - Only deploy what changed
Lower risk - Fewer changes per deployment
Resource efficiency - Don’t waste CI/CD minutes on unchanged services
Parallel deploys - Changed services can deploy simultaneously

How it works:

Detect which files/directories changed
Only trigger deployments for affected services
Skip deployment for unchanged services

Real-world example:

Changed files: frontend/src/App.js
Result: Deploy only frontend (skip backend, mobile, api)
Time saved: 70% (only 1 of 4 services deployed)

GitHub Actions implementation:

# .github/workflows/deploy.yml
jobs:
  detect-changes:
    runs-on: ubuntu-latest
    outputs:
      # Output which services changed
      frontend: ${{ steps.filter.outputs.frontend }}
      backend: ${{ steps.filter.outputs.backend }}
    steps:
      - uses: actions/checkout@v3

      # Detect changes in specific paths
      - uses: dorny/paths-filter@v2
        id: filter
        with:
          filters: |
            frontend:
              - 'frontend/**'
            backend:
              - 'backend/**'

  deploy-frontend:
    needs: detect-changes
    # Only run if frontend files changed
    if: needs.detect-changes.outputs.frontend == 'true'
    runs-on: ubuntu-latest
    steps:
      - run: ./deploy-frontend.sh

  deploy-backend:
    needs: detect-changes
    # Only run if backend files changed
    if: needs.detect-changes.outputs.backend == 'true'
    runs-on: ubuntu-latest
    steps:
      - run: ./deploy-backend.sh

# If only frontend changed: backend deployment skipped automatically

2. Blue-Green Deployment

What is it?

Blue-green deployment runs two identical production environments (“blue” and “green”). At any time, one serves live traffic while the other is idle. To deploy, you update the idle environment and switch traffic instantly.

How it works:

Current state: Blue = live (serving users), Green = idle
Deploy: Update green environment with new version
Test: Run smoke tests on green
Switch: Route all traffic from blue to green instantly
Rollback: If issues, switch back to blue immediately

Benefits:

Zero downtime - Instant traffic switch
Easy rollback - Switch back if problems detected
Full testing - Test in production environment before going live
Safety - Old version stays running during deployment

Drawbacks:

Resource cost - Need 2× infrastructure
Database challenges - Both versions must work with same DB schema

Visual:

Before:        Blue (v1.0) ← 100% traffic    Green (idle)
Deploy:        Blue (v1.0) ← 100% traffic    Green (v1.1) deploying...
Test:          Blue (v1.0) ← 100% traffic    Green (v1.1) testing...
Switch:        Blue (v1.0)                    Green (v1.1) ← 100% traffic ✓
Cleanup:       Blue (deleted)                 Green (v1.1) ← 100% traffic

GitHub Actions implementation:

jobs:
  deploy:
    runs-on: ubuntu-latest
    steps:
      # Step 1: Deploy new version to green (idle) environment
      - name: Deploy to green environment
        run: |
          kubectl set image deployment/myapp \
            myapp=myapp:${{ github.sha }} \
            --namespace=green

      # Step 2: Wait for deployment to complete
      - name: Wait for rollout
        run: |
          kubectl rollout status deployment/myapp \
            --namespace=green \
            --timeout=5m

      # Step 3: Test green environment before switching traffic
      - name: Run smoke tests
        run: ./smoke-tests.sh https://green.example.com

      # Step 4: Switch all traffic from blue to green
      - name: Switch traffic to green
        run: |
          kubectl patch service myapp \
            -p '{"spec":{"selector":{"version":"green"}}}'

      # Step 5: Monitor for issues
      - name: Monitor for issues
        run: sleep 300  # Monitor for 5 minutes

      # Step 6: Clean up old blue environment
      - name: Cleanup blue environment
        if: success()
        run: kubectl delete deployment myapp --namespace=blue

# Rollback: Just switch service selector back to blue

3. Canary Deployment

What is it?

Canary deployment gradually rolls out changes to a small subset of users before deploying to everyone. If the canary (small group) shows problems, you can abort before affecting all users.

Name origin: Named after “canary in a coal mine” - miners used canaries to detect toxic gas. If the canary died, miners evacuated.

How it works:

Deploy to 10% of servers/users
Monitor metrics (errors, latency, user feedback)
If healthy: Increase to 25%, then 50%, then 100%
If problems: Rollback immediately, only 10% affected

Benefits:

Low risk - Limit blast radius to small percentage
Real user testing - Test with actual production traffic
Data-driven - Make decisions based on metrics
Gradual rollout - Increase confidence with each stage

Comparison to blue-green:

Blue-green: 0% → 100% instantly
Canary: 0% → 10% → 25% → 50% → 100% gradually

Visual:

Stage 1:  10% new version  |  90% old version   Monitor...
Stage 2:  25% new version  |  75% old version   Monitor...
Stage 3:  50% new version  |  50% old version   Monitor...
Stage 4:  100% new version |  0% old version    Complete!

If problems detected at any stage: Immediate rollback to 100% old version

When to use:

High-traffic applications (enough data at 10% for metrics)
Risk-averse deployments
When you can segment traffic (by user, region, etc.)

GitHub Actions implementation:

jobs:
  deploy-canary:
    runs-on: ubuntu-latest
    steps:
      # Stage 1: Deploy to 10% of traffic
      - name: Deploy canary (10%)
        run: |
          kubectl set image deployment/myapp-canary \
            myapp=myapp:${{ github.sha }}

          kubectl scale deployment/myapp-canary --replicas=1   # 10%
          kubectl scale deployment/myapp-stable --replicas=9   # 90%

      # Monitor metrics before proceeding
      - name: Monitor metrics
        run: ./check-metrics.sh  # Checks error rate, latency, etc.

      # Stage 2: Increase to 50% if metrics good
      - name: Increase to 50%
        if: success()
        run: |
          kubectl scale deployment/myapp-canary --replicas=5   # 50%
          kubectl scale deployment/myapp-stable --replicas=5   # 50%

      # Monitor again
      - name: Monitor metrics
        run: ./check-metrics.sh

      # Stage 3: Full rollout if all metrics healthy
      - name: Full rollout
        if: success()
        run: |
          # Update stable deployment to new version
          kubectl set image deployment/myapp-stable \
            myapp=myapp:${{ github.sha }}

          kubectl scale deployment/myapp-stable --replicas=10  # 100%
          kubectl scale deployment/myapp-canary --replicas=0   # 0%

# If any stage fails: Pipeline stops, canary scaled to 0, stable stays at 100%

Automated metrics checking:

#!/bin/bash
# check-metrics.sh

# Get error rate from Prometheus
ERROR_RATE=$(curl -s 'http://prometheus/api/v1/query?query=error_rate' | jq '.data.result[0].value[1]')

# Fail if error rate > 1%
if (( $(echo "$ERROR_RATE > 0.01" | bc -l) )); then
  echo "Error rate too high: $ERROR_RATE"
  exit 1
fi

echo "Metrics healthy, proceeding..."
exit 0

4. Feature Flags

Decouple deployment from release:

jobs:
  deploy:
    runs-on: ubuntu-latest
    steps:
      - name: Deploy with feature flag OFF
        run: |
          kubectl set image deployment/myapp myapp=myapp:${{ github.sha }}
          kubectl set env deployment/myapp \
            FEATURE_NEW_UI=false

      # Later, enable for subset of users
      - name: Enable for beta users
        run: |
          # Update feature flag service
          curl -X POST https://flags.example.com/api/flags/new-ui \
            -d '{"enabled": true, "rollout": 10}'

Complete Example Pipeline

Optimized GitHub Actions Workflow

name: Optimized CI/CD

on:
  push:
    branches: [main]
  pull_request:

env:
  NODE_VERSION: '18'

jobs:
  # Fast preliminary checks
  checks:
    runs-on: ubuntu-latest
    outputs:
      changed: ${{ steps.filter.outputs.src }}
    steps:
      - uses: actions/checkout@v3

      - uses: dorny/paths-filter@v2
        id: filter
        with:
          filters: |
            src:
              - 'src/**'
              - 'package*.json'

      - name: Lint
        if: steps.filter.outputs.src == 'true'
        uses: actions/setup-node@v3
        with:
          node-version: ${{ env.NODE_VERSION }}

      - uses: actions/cache@v3
        with:
          path: ~/.npm
          key: ${{ runner.os }}-node-${{ hashFiles('**/package-lock.json') }}

      - run: npm ci
      - run: npm run lint

  # Parallel builds
  build:
    needs: checks
    if: needs.checks.outputs.changed == 'true'
    runs-on: ubuntu-latest
    strategy:
      matrix:
        target: [frontend, backend, mobile]
    steps:
      - uses: actions/checkout@v3

      - uses: actions/setup-node@v3
        with:
          node-version: ${{ env.NODE_VERSION }}

      - uses: actions/cache@v3
        with:
          path: ~/.npm
          key: ${{ runner.os }}-node-${{ matrix.target }}-${{ hashFiles('**/package-lock.json') }}

      - run: |
          cd ${{ matrix.target }}
          npm ci
          npm run build

      - uses: actions/upload-artifact@v3
        with:
          name: ${{ matrix.target }}-build
          path: ${{ matrix.target }}/dist/

  # Parallel tests
  test:
    needs: build
    runs-on: ubuntu-latest
    strategy:
      fail-fast: true
      matrix:
        test-type: [unit, integration, e2e]
    steps:
      - uses: actions/checkout@v3

      - uses: actions/download-artifact@v3
        with:
          name: frontend-build
          path: frontend/dist/

      - uses: actions/setup-node@v3
        with:
          node-version: ${{ env.NODE_VERSION }}

      - uses: actions/cache@v3
        with:
          path: ~/.npm
          key: ${{ runner.os }}-node-${{ hashFiles('**/package-lock.json') }}

      - run: npm ci
      - run: npm run test:${{ matrix.test-type }}

  # Docker build with caching
  docker:
    needs: test
    if: github.ref == 'refs/heads/main'
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v3

      - uses: docker/setup-buildx-action@v2

      - uses: docker/login-action@v2
        with:
          registry: ghcr.io
          username: ${{ github.actor }}
          password: ${{ secrets.GITHUB_TOKEN }}

      - uses: docker/build-push-action@v4
        with:
          context: .
          push: true
          tags: |
            ghcr.io/${{ github.repository }}:latest
            ghcr.io/${{ github.repository }}:${{ github.sha }}
          cache-from: type=gha
          cache-to: type=gha,mode=max

  # Deploy
  deploy:
    needs: docker
    runs-on: ubuntu-latest
    environment: production
    steps:
      - uses: actions/checkout@v3

      - name: Deploy to Kubernetes
        run: |
          kubectl set image deployment/myapp \
            myapp=ghcr.io/${{ github.repository }}:${{ github.sha }}

      - name: Wait for rollout
        run: kubectl rollout status deployment/myapp

      - name: Smoke tests
        run: ./scripts/smoke-tests.sh

Monitoring Pipeline Performance

Metrics to Track

Key metrics:

Total pipeline duration
Time per stage
Cache hit rate
Parallel job efficiency
Deployment frequency
Lead time for changes
Change failure rate
Time to restore service

GitHub Actions insights:

# Available in GitHub UI
# Actions → Workflow → Insights

# Shows:
# - Average duration
# - Success rate
# - Job duration breakdown

GitLab CI analytics:

# Settings → CI/CD → Pipelines
# Shows:
# - Pipeline duration trends
# - Success rates
# - Job execution times

Optimization Checklist

Build optimization:

Caching dependencies
Caching build artifacts
Multi-stage Docker builds
BuildKit cache mounts
Parallel job execution
Skip unchanged paths
Fail fast on errors

Test optimization:

Parallel test execution
Test splitting/sharding
Run fast tests first
Skip tests for unchanged code
Cache test dependencies

Deployment optimization:

Incremental deployment
Blue-green deployment
Canary releases
Feature flags
Automated rollback

Conclusion

Optimizing CI/CD pipelines requires:

Caching - Dependencies, builds, Docker layers
Parallelization - Independent jobs run concurrently
Smart execution - Skip unnecessary steps
Efficient builds - Multi-stage Docker, BuildKit
Fast feedback - Fail fast, run quick tests first
Strategic deployment - Blue-green, canary, feature flags

Key takeaways:

Cache everything that can be cached
Run jobs in parallel when possible
Fail fast to save time
Use artifacts to avoid rebuilding
Monitor and optimize continuously
Deploy incrementally for safety

Well-optimized pipelines improve developer productivity, enable faster releases, and reduce infrastructure costs.

Before optimization:

Pipeline duration: 45 minutes
Deploy frequency: 2x per week
Developer waiting time: High

After optimization:

Pipeline duration: 8 minutes (82% faster)
Deploy frequency: 10x per day
Developer waiting time: Minimal

The investment in pipeline optimization pays dividends in team velocity and satisfaction.

Introduction#

Build Caching#

Why Caching Matters#

GitHub Actions Caching#

GitLab CI Caching#

Jenkins Caching#

Parallel Job Execution#

GitHub Actions Parallelization#

GitLab CI Parallelization#

Jenkins Parallelization#

Optimization Techniques#

1. Skip Unnecessary Steps#

2. Fail Fast#

3. Optimize Docker Builds#

4. Artifact Reuse#

5. Resource Optimization#

Deployment Strategies#

1. Incremental Deployment#

2. Blue-Green Deployment#

3. Canary Deployment#

4. Feature Flags#

Complete Example Pipeline#

Optimized GitHub Actions Workflow#

Monitoring Pipeline Performance#

Metrics to Track#

Optimization Checklist#

Conclusion#

Introduction

Build Caching

Why Caching Matters

GitHub Actions Caching

GitLab CI Caching

Jenkins Caching

Parallel Job Execution

GitHub Actions Parallelization

GitLab CI Parallelization

Jenkins Parallelization

Optimization Techniques

1. Skip Unnecessary Steps

2. Fail Fast

3. Optimize Docker Builds

4. Artifact Reuse

5. Resource Optimization

Deployment Strategies

1. Incremental Deployment

2. Blue-Green Deployment

3. Canary Deployment

4. Feature Flags

Complete Example Pipeline

Optimized GitHub Actions Workflow

Monitoring Pipeline Performance

Metrics to Track

Optimization Checklist

Conclusion