Home / Notebooks / DevOps
DevOps
intermediate

GitLab CI/CD Essentials

Essential GitLab CI/CD concepts for continuous integration and deployment

March 10, 2024
•
Updated regularly

GitLab CI/CD Essentials

Quick reference guide for GitLab CI/CD automation.

What is GitLab CI/CD?

GitLab CI/CD is a built-in continuous integration and deployment platform that:

  • Automates testing and deployment workflows
  • Defined in .gitlab-ci.yml at repository root
  • Runs on GitLab Runners (shared or self-hosted)
  • Integrated with GitLab (merge requests, issues)
  • Supports Docker, Kubernetes, and cloud platforms
  • Free tier with shared runners

    • Core Concepts

    Pipeline
├── Stages (sequential)
│   ├── Build
│   ├── Test
│   └── Deploy
└── Jobs (within stages)
    ├── Script (commands)
    ├── Artifacts (outputs)
    └── Cache (dependencies)

    Key Components

  • Pipeline: Collection of jobs organized in stages
  • Stage: Group of jobs that run in parallel
  • Job: Individual task with scripts to execute
  • Runner: Agent that executes jobs
  • Artifact: Files produced by jobs
  • Cache: Dependencies stored between pipeline runs

    • Basic Configuration

    Simple Pipeline

    # .gitlab-ci.yml
stages:
  - build
  - test

build-job:
  stage: build
  script:
    - echo "Building the application..."
    - npm install
    - npm run build

test-job:
  stage: test
  script:
    - echo "Running tests..."
    - npm test

    Complete CI/CD Pipeline

    # .gitlab-ci.yml
image: node:20

stages:
  - build
  - test
  - deploy

variables:
  NODE_ENV: production

cache:
  paths:
    - node_modules/

before_script:
  - npm ci

build:
  stage: build
  script:
    - npm run build
  artifacts:
    paths:
      - dist/
    expire_in: 1 week

test:lint:
  stage: test
  script:
    - npm run lint

test:unit:
  stage: test
  script:
    - npm test
  coverage: '/All files[^|]*\|[^|]*\s+([\d\.]+)/'
  artifacts:
    reports:
      coverage_report:
        coverage_format: cobertura
        path: coverage/cobertura-coverage.xml

deploy:production:
  stage: deploy
  script:
    - npm run deploy
  environment:
    name: production
    url: https://example.com
  only:
    - main

    Stages

    Default Stages

    # Default stages (if not specified)
stages:
  - .pre
  - build
  - test
  - deploy
  - .post

    Custom Stages

    stages:
  - install
  - lint
  - test
  - build
  - package
  - deploy
  - cleanup

install:
  stage: install
  script:
    - npm ci

lint:
  stage: lint
  script:
    - npm run lint

test:
  stage: test
  script:
    - npm test

build:
  stage: build
  script:
    - npm run build

deploy:
  stage: deploy
  script:
    - ./deploy.sh

    Jobs

    Basic Job Structure

    job-name:
  stage: test
  image: node:20
  variables:
    VAR_NAME: value
  before_script:
    - echo "Before script"
  script:
    - echo "Main script"
    - npm test
  after_script:
    - echo "After script"
  artifacts:
    paths:
      - coverage/
  only:
    - main

    Parallel Jobs

    test:
  stage: test
  parallel: 3
  script:
    - npm test -- --shard=$CI_NODE_INDEX/$CI_NODE_TOTAL

    Matrix Jobs

    test:
  stage: test
  parallel:
    matrix:
      - NODE_VERSION: ['16', '18', '20']
        OS: ['linux', 'macos']
  image: node:${NODE_VERSION}
  script:
    - npm test

    Scripts

    Multi-line Scripts

    build:
  script:
    - |
      echo "Building application..."
      npm ci
      npm run build
      echo "Build complete!"

    Script with Error Handling

    deploy:
  script:
    - set -e  # Exit on error
    - |
      if [ -z "$DEPLOY_TOKEN" ]; then
        echo "Error: DEPLOY_TOKEN not set"
        exit 1
      fi
    - ./deploy.sh

    Variables

    Global Variables

    variables:
  GLOBAL_VAR: "value"
  DATABASE_URL: "postgres://localhost:5432/db"

job:
  script:
    - echo $GLOBAL_VAR

    Job-specific Variables

    variables:
  GLOBAL_ENV: production

dev-deploy:
  variables:
    DEPLOY_ENV: development
  script:
    - echo "Deploying to $DEPLOY_ENV"

prod-deploy:
  variables:
    DEPLOY_ENV: production
  script:
    - echo "Deploying to $DEPLOY_ENV"

    Built-in Variables

    job:
  script:
    - echo "Pipeline ID: $CI_PIPELINE_ID"
    - echo "Commit SHA: $CI_COMMIT_SHA"
    - echo "Branch: $CI_COMMIT_BRANCH"
    - echo "Project: $CI_PROJECT_NAME"
    - echo "Runner: $CI_RUNNER_DESCRIPTION"

    Protected Variables

    # Set in GitLab UI: Settings > CI/CD > Variables
deploy:
  script:
    - echo "Using secret: $SECRET_KEY"
    - deploy --token $DEPLOY_TOKEN
  only:
    - main

    Rules and Conditions

    Using rules

    job:
  script:
    - echo "Running job"
  rules:
    - if: '$CI_COMMIT_BRANCH == "main"'
      when: always
    - if: '$CI_PIPELINE_SOURCE == "merge_request_event"'
      when: manual
    - when: never

# Run only on main branch
deploy:
  script:
    - ./deploy.sh
  rules:
    - if: '$CI_COMMIT_BRANCH == "main"'

# Run on tags only
release:
  script:
    - ./release.sh
  rules:
    - if: '$CI_COMMIT_TAG'

    Using only and except (Legacy)

    # Run only on main
deploy:
  script:
    - ./deploy.sh
  only:
    - main

# Run except on main
test:
  script:
    - npm test
  except:
    - main

# Run on specific branches
deploy:
  script:
    - ./deploy.sh
  only:
    - main
    - develop
    - /^release-.*$/

    Changes Detection

    # Run if specific files changed
test-api:
  script:
    - npm test -- api
  rules:
    - if: '$CI_PIPELINE_SOURCE == "merge_request_event"'
      changes:
        - api/**/*
        - package.json

# Run on any code changes
build:
  script:
    - npm run build
  rules:
    - changes:
        - src/**/*
        - package.json

    Artifacts

    Basic Artifacts

    build:
  script:
    - npm run build
  artifacts:
    paths:
      - dist/
      - build/
    expire_in: 1 week

    Artifacts with Reports

    test:
  script:
    - npm test
  artifacts:
    reports:
      junit: test-results.xml
      coverage_report:
        coverage_format: cobertura
        path: coverage/cobertura-coverage.xml
    paths:
      - coverage/
    when: always
    expire_in: 30 days

    Download Artifacts

    build:
  stage: build
  script:
    - npm run build
  artifacts:
    paths:
      - dist/

deploy:
  stage: deploy
  dependencies:
    - build
  script:
    - ls dist/
    - ./deploy.sh

    Cache

    NPM Cache

    cache:
  key: ${CI_COMMIT_REF_SLUG}
  paths:
    - node_modules/
    - .npm/

before_script:
  - npm ci --cache .npm --prefer-offline

    Multiple Caches

    cache:
  - key:
      files:
        - package-lock.json
    paths:
      - node_modules/
  - key:
      files:
        - Gemfile.lock
    paths:
      - vendor/ruby

    Cache Policy

    # Create cache
build:
  cache:
    key: build-cache
    paths:
      - node_modules/
    policy: push
  script:
    - npm ci

# Use cache
test:
  cache:
    key: build-cache
    paths:
      - node_modules/
    policy: pull
  script:
    - npm test

    Docker Integration

    Using Docker Image

    image: node:20-alpine

job:
  script:
    - node --version
    - npm --version

    Different Images per Job

    build:
  image: node:20
  script:
    - npm run build

test:
  image: node:20-alpine
  script:
    - npm test

deploy:
  image: alpine:latest
  script:
    - ./deploy.sh

    Docker-in-Docker (DinD)

    build-docker:
  image: docker:24
  services:
    - docker:24-dind
  variables:
    DOCKER_TLS_CERTDIR: "/certs"
  before_script:
    - docker login -u $CI_REGISTRY_USER -p $CI_REGISTRY_PASSWORD $CI_REGISTRY
  script:
    - docker build -t $CI_REGISTRY_IMAGE:latest .
    - docker push $CI_REGISTRY_IMAGE:latest

    Kaniko (Build without DinD)

    build:
  stage: build
  image:
    name: gcr.io/kaniko-project/executor:debug
    entrypoint: [""]
  script:
    - |
      echo "{\"auths\":{\"$CI_REGISTRY\":{\"username\":\"$CI_REGISTRY_USER\",\"password\":\"$CI_REGISTRY_PASSWORD\"}}}" > /kaniko/.docker/config.json
    - |
      /kaniko/executor \
        --context $CI_PROJECT_DIR \
        --dockerfile $CI_PROJECT_DIR/Dockerfile \
        --destination $CI_REGISTRY_IMAGE:$CI_COMMIT_TAG

    Services

    PostgreSQL Service

    test:
  image: node:20
  services:
    - postgres:15
  variables:
    POSTGRES_DB: test_db
    POSTGRES_USER: test_user
    POSTGRES_PASSWORD: test_password
    DATABASE_URL: "postgres://test_user:test_password@postgres:5432/test_db"
  script:
    - npm test

    Multiple Services

    integration-test:
  services:
    - postgres:15
    - redis:7-alpine
    - selenium/standalone-chrome:latest
  variables:
    POSTGRES_DB: test_db
    POSTGRES_PASSWORD: secret
    REDIS_URL: redis://redis:6379
  script:
    - npm run test:integration

    Environments

    Basic Environment

    deploy:production:
  stage: deploy
  script:
    - ./deploy.sh
  environment:
    name: production
    url: https://example.com
  only:
    - main

deploy:staging:
  stage: deploy
  script:
    - ./deploy.sh
  environment:
    name: staging
    url: https://staging.example.com
  only:
    - develop

    Dynamic Environments

    deploy:review:
  stage: deploy
  script:
    - ./deploy-review.sh
  environment:
    name: review/$CI_COMMIT_REF_NAME
    url: https://$CI_COMMIT_REF_SLUG.example.com
    on_stop: stop:review
  only:
    - merge_requests

stop:review:
  stage: deploy
  script:
    - ./stop-review.sh
  environment:
    name: review/$CI_COMMIT_REF_NAME
    action: stop
  when: manual
  only:
    - merge_requests

    Templates and Includes

    Local Templates

    # .gitlab-ci.yml
include:
  - local: '/templates/build.yml'
  - local: '/templates/deploy.yml'

variables:
  APP_NAME: myapp

    # templates/build.yml
.build-template:
  stage: build
  script:
    - npm run build
  artifacts:
    paths:
      - dist/

build:prod:
  extends: .build-template
  variables:
    NODE_ENV: production

    Remote Templates

    include:
  - remote: 'https://gitlab.com/awesome-project/raw/main/.gitlab-ci-template.yml'

    Extends (Inheritance)

    .deploy-template:
  stage: deploy
  before_script:
    - echo "Preparing deployment..."
  script:
    - ./deploy.sh
  only:
    - main

deploy:aws:
  extends: .deploy-template
  variables:
    PROVIDER: aws
  script:
    - ./deploy-aws.sh

deploy:gcp:
  extends: .deploy-template
  variables:
    PROVIDER: gcp
  script:
    - ./deploy-gcp.sh

    Runners

    Shared Runners

    # Uses GitLab.com shared runners
job:
  tags:
    - docker
  script:
    - npm test

    Specific Runner

    # Uses runner with specific tag
deploy:
  tags:
    - deploy
    - production
  script:
    - ./deploy.sh

    Shell Executor

    build:
  tags:
    - shell
  script:
    - npm ci
    - npm run build

    Pipeline Types

    Merge Request Pipelines

    test:
  script:
    - npm test
  rules:
    - if: '$CI_PIPELINE_SOURCE == "merge_request_event"'

    Scheduled Pipelines

    # Create schedule in GitLab UI: CI/CD > Schedules
cleanup:
  script:
    - ./cleanup-old-data.sh
  rules:
    - if: '$CI_PIPELINE_SOURCE == "schedule"'

    Manual Pipeline

    deploy:
  script:
    - ./deploy.sh
  when: manual
  only:
    - main

    Deployment Strategies

    Deploy to AWS

    deploy:aws:
  stage: deploy
  image: python:3.11
  before_script:
    - pip install awscli
  script:
    - aws s3 sync ./dist s3://$S3_BUCKET --delete
    - aws cloudfront create-invalidation --distribution-id $CLOUDFRONT_ID --paths "/*"
  environment:
    name: production
    url: https://example.com
  only:
    - main

    Deploy to Kubernetes

    deploy:k8s:
  stage: deploy
  image: bitnami/kubectl:latest
  before_script:
    - kubectl config set-cluster k8s --server="$KUBE_URL" --insecure-skip-tls-verify=true
    - kubectl config set-credentials admin --token="$KUBE_TOKEN"
    - kubectl config set-context default --cluster=k8s --user=admin
    - kubectl config use-context default
  script:
    - kubectl apply -f k8s/
    - kubectl rollout status deployment/myapp
  environment:
    name: production
  only:
    - main

    Deploy to Vercel

    deploy:vercel:
  stage: deploy
  image: node:20
  before_script:
    - npm install -g vercel
  script:
    - vercel --token $VERCEL_TOKEN --prod
  environment:
    name: production
  only:
    - main

    Testing

    Unit Tests

    test:unit:
  stage: test
  script:
    - npm run test:unit
  coverage: '/Lines\s+:\s+(\d+\.\d+)%/'
  artifacts:
    reports:
      junit: junit.xml
      coverage_report:
        coverage_format: cobertura
        path: coverage/cobertura-coverage.xml

    Integration Tests

    test:integration:
  stage: test
  services:
    - postgres:15
    - redis:7
  variables:
    DATABASE_URL: postgres://postgres:password@postgres/test_db
    REDIS_URL: redis://redis:6379
  script:
    - npm run test:integration

    E2E Tests

    test:e2e:
  stage: test
  services:
    - selenium/standalone-chrome:latest
  variables:
    SELENIUM_HOST: selenium__standalone-chrome
    SELENIUM_PORT: 4444
  script:
    - npm run test:e2e
  artifacts:
    when: on_failure
    paths:
      - cypress/screenshots/
      - cypress/videos/
    expire_in: 1 week

    Security Scanning

    SAST (Static Application Security Testing)

    include:
  - template: Security/SAST.gitlab-ci.yml

variables:
  SAST_EXCLUDED_PATHS: spec, test, tests, tmp

    Dependency Scanning

    include:
  - template: Security/Dependency-Scanning.gitlab-ci.yml

    Container Scanning

    include:
  - template: Security/Container-Scanning.gitlab-ci.yml

container_scanning:
  variables:
    CS_IMAGE: $CI_REGISTRY_IMAGE:$CI_COMMIT_SHA

    Performance Optimization

    Parallel Execution

    test:
  stage: test
  parallel: 5
  script:
    - npm test -- --shard=$CI_NODE_INDEX/$CI_NODE_TOTAL

    Interruptible Jobs

    test:
  script:
    - npm test
  interruptible: true

    DAG (Directed Acyclic Graph)

    stages:
  - build
  - test
  - deploy

build:
  stage: build
  script:
    - npm run build

test:unit:
  stage: test
  needs: [build]
  script:
    - npm run test:unit

test:integration:
  stage: test
  needs: [build]
  script:
    - npm run test:integration

deploy:
  stage: deploy
  needs: [test:unit, test:integration]
  script:
    - npm run deploy

    Debugging

    Debug Variables

    debug:
  script:
    - echo "Pipeline ID: $CI_PIPELINE_ID"
    - echo "Commit SHA: $CI_COMMIT_SHA"
    - echo "Branch: $CI_COMMIT_BRANCH"
    - echo "Tag: $CI_COMMIT_TAG"
    - env | sort

    CI Lint

    # Validate .gitlab-ci.yml locally
gitlab-ci-lint .gitlab-ci.yml

# Or use GitLab API
curl --header "Content-Type: application/json" \
  --data @.gitlab-ci.yml \
  https://gitlab.com/api/v4/ci/lint

    Best Practices

    Job Templates

    .test-template:
  image: node:20
  before_script:
    - npm ci
  cache:
    paths:
      - node_modules/

test:unit:
  extends: .test-template
  script:
    - npm run test:unit

test:integration:
  extends: .test-template
  script:
    - npm run test:integration

    Security

    # Use protected variables for secrets
deploy:
  script:
    - deploy --token $DEPLOY_TOKEN
  only:
    - main
  # Variables masked in logs

# Use environments for protection
deploy:production:
  environment:
    name: production
  only:
    - main

    Resource Management

    # Limit concurrent jobs
workflow:
  rules:
    - if: '$CI_PIPELINE_SOURCE == "merge_request_event"'
    - if: '$CI_COMMIT_BRANCH == "main"'

# Cancel redundant pipelines
workflow:
  auto_cancel:
    on_new_commit: interruptible

    Common Patterns

    Monorepo

    .build-microservice:
  script:
    - cd $SERVICE_PATH
    - npm ci
    - npm run build

build:service-a:
  extends: .build-microservice
  variables:
    SERVICE_PATH: services/service-a
  rules:
    - changes:
        - services/service-a/**/*

build:service-b:
  extends: .build-microservice
  variables:
    SERVICE_PATH: services/service-b
  rules:
    - changes:
        - services/service-b/**/*

    Release Pipeline

    release:
  stage: deploy
  image: node:20
  script:
    - npm ci
    - npx semantic-release
  only:
    - main
  variables:
    GIT_DEPTH: 0

    Tips

  • Use caching to speed up pipelines
  • Leverage artifacts to pass data between jobs
  • Use needs for faster pipelines (DAG)
  • Split long jobs into smaller parallel jobs
  • Use templates to avoid duplication
  • Set appropriate timeouts for jobs
  • Monitor pipeline minutes usage
  • Use protected branches and variables
  • Add retry logic for flaky tests
  • Document complex pipelines with comments

    • Quick Reference

    # Basic structure
image: node:20
stages: [build, test, deploy]

variables:
  VAR: value

cache:
  paths: [node_modules/]

before_script:
  - npm ci

job-name:
  stage: test
  script:
    - npm test
  artifacts:
    paths: [dist/]
  rules:
    - if: '$CI_COMMIT_BRANCH == "main"'

    Resources

  • GitLab CI/CD Documentation
  • .gitlab-ci.yml Reference
  • GitLab CI/CD Examples
  • GitLab CI/CD Variables
  • GitLab Runner Documentation

    • Topics

    GitLabCI/CDDevOpsAutomationTesting
    ← Back to Notebooks

    Found This Helpful?

    If you have questions or suggestions for improving these notes, I'd love to hear from you.

    Contact MeMore Notes