GitLab CI/CD Pipeline Mastery: A Comprehensive Guide

$1

Continuous Integration and Continuous Deployment (CI/CD) is at the heart of modern DevOps practices, and GitLab provides one of the most powerful pipeline systems available. In this comprehensive guide, we'll explore how to master GitLab CI/CD pipelines, from basic concepts to advanced implementations.

$1

Basic Pipeline Structure

Advanced Pipeline Features

Real-world Examples

Best Practices

Troubleshooting Guide

$1

$1

The .gitlab-ci.yml file is the backbone of GitLab CI/CD. Here's a basic structure:

``yaml

stages:

- build

- test

- deploy

variables:

APP_VERSION: "1.0.0"

DOCKER_REGISTRY: "registry.example.com"

build_job:

stage: build

script:

- echo "Building application..."

- npm install

- npm run build

artifacts:

paths:

- dist/

expire_in: 1 week

test_job:

stage: test

script:

- echo "Running tests..."

- npm run test

coverage: '/Coverage: \d+.\d+%/'

deploy_job:

stage: deploy

script:

- echo "Deploying application..."

- docker build -t $DOCKER_REGISTRY/myapp:$APP_VERSION .

- docker push $DOCKER_REGISTRY/myapp:$APP_VERSION

only:

- main

`

$1

1. Stages: Define the pipeline flow

2. Variables: Set global configuration

3. Jobs: Individual units of work

4. Artifacts: Pass data between jobs

5. Rules: Control job execution

$1

$1

`yaml

generate_jobs:

stage: .pre

script:

- python generate_pipeline.py

artifacts:

paths:

- generated-config.yml

include:

- local: 'generated-config.yml'

`

$1

`yaml

test:

parallel: 3

script:

- npm run test -- --split=$CI_NODE_INDEX/$CI_NODE_TOTAL

`

$1

`yaml

.deploy_template: &deploy_definition

script:

- kubectl apply -f k8s/

variables:

ENVIRONMENT: staging

deploy_staging:

<<: *deploy_definition

environment:

name: staging

url: https://staging.example.com

deploy_production:

<<: *deploy_definition

variables:

ENVIRONMENT: production

environment:

name: production

url: https://production.example.com

when: manual

only:

- tags

`

$1

$1

`yaml

image: node:16

stages:

- install

- lint

- test

- build

- deploy

cache:

paths:

- node_modules/

install_dependencies:

stage: install

script:

- npm ci

artifacts:

paths:

- node_modules/

lint_code:

stage: lint

script:

- npm run lint

- npm run prettier:check

unit_tests:

stage: test

script:

- npm run test:unit

coverage: '/All files[^|]\|[^|]\s+([\d\.]+)/'

integration_tests:

stage: test

services:

- mongo:4.4

variables:

MONGODB_URI: "mongodb://mongo:27017/test"

script:

- npm run test:integration

build_frontend:

stage: build

script:

- npm run build

artifacts:

paths:

- dist/

deploy_to_aws:

stage: deploy

image:

name: amazon/aws-cli

entrypoint: [""]

script:

- aws s3 sync dist/ s3://$S3_BUCKET/

- aws cloudfront create-invalidation --distribution-id $CF_DISTRIBUTION --paths "/*"

only:

- main

`

$1

`yaml

variables:

DOCKER_REGISTRY: "${CI_REGISTRY}/${CI_PROJECT_PATH}"

DOCKER_TAG: "${CI_COMMIT_SHA}"

stages:

- test

- build

- security

- deploy

.docker_login: &docker_login

before_script:

- docker login -u $CI_REGISTRY_USER -p $CI_REGISTRY_PASSWORD $CI_REGISTRY

test:

image: python:3.9

stage: test

script:

- pip install -r requirements.txt

- pytest --cov=app tests/

coverage: '/TOTAL.+ ([0-9]{1,3}%)/'

build:

stage: build

<<: *docker_login

script:

- docker build -t ${DOCKER_REGISTRY}:${DOCKER_TAG} .

- docker push ${DOCKER_REGISTRY}:${DOCKER_TAG}

security_scan:

stage: security

image: aquasec/trivy

script:

- trivy image ${DOCKER_REGISTRY}:${DOCKER_TAG}

deploy_k8s:

stage: deploy

image: bitnami/kubectl

script:

- kubectl set image deployment/myapp container=${DOCKER_REGISTRY}:${DOCKER_TAG}

environment:

name: production

url: https://api.example.com

`

$1

$1

Use cache effectively

Implement parallel jobs

Optimize Docker layers

Use shallow cloning

$1

`yaml

security_checks:

stage: security

parallel:

matrix:

- SCANNER: [sast, dast, dependency-scanning]

script:

- gitlab-security-scan $SCANNER

`

$1

Use environment variables

Implement review apps

Use manual deployments for production

Implement rollback strategies

$1

$1

1. Pipeline Timing Out

`yaml

long_running_job:

script:

- ./long-script.sh

timeout: 3 hours

`

2. Cache Issues

`yaml

cache:

key: ${CI_COMMIT_REF_SLUG}

paths:

- node_modules/

policy: pull-push

`

3. Job Dependencies

`yaml

deploy:

needs:

- job: build

artifacts: true

`

$1

$1

`yaml

metrics_job:

script:

- |

echo "pipeline_duration_seconds{pipeline=\"$CI_PIPELINE_ID\"} $CI_PIPELINE_DURATION" > metrics.txt

artifacts:

reports:

metrics: metrics.txt

`

$1

`yaml

review_app:

environment:

name: review/$CI_COMMIT_REF_NAME

url: https://$CI_ENVIRONMENT_SLUG.example.com

on_stop: stop_review_app

script:

- deploy_review_app.sh

``

$1

Mastering GitLab CI/CD pipelines requires understanding both basic concepts and advanced features. This guide provides a foundation for building efficient, secure, and scalable pipelines for your applications.

$1

1. GitLab CI/CD Documentation

2. Pipeline Examples Repository

3. GitLab Runner Documentation

4. CI/CD Best Practices Guide

---

Last Updated: February 2025