Skip to content
GitLab Drupal

GitLab CI for Drupal pipelines

This project contains a number of pre-configured tasks for GitLab CI/CD that allow to run very powerful pipelines which contain the following stage order:

We have a predefined execute job called LakeDrops, which looks like this:

These pre-configured tasks can easily be included into your Drupal project. A GitLab runner on any host can then be configured to run those pipelines. The following chapters describe the necessary steps and also all the options available for configuration.

Stages

The following chapters describes all stages and their jobs. The files:

.pre

To prepare and check, if the environment is good to go, we have a .pre stage, which contains the following jobs:

Sanity Checks

The sanity check gets important when you are using Crowdsec. It contains two little scripts, that check:

  • if the crowdsec.settings.yml is in the config split and
  • if it is not enabled in the core.extensions.yml.

Otherwise, it is conflicting with you local settings, since CrowdSec is a tool for production sites.

Except:

  • Commit message contains "Merge tag"
  • Commit message contains "Merge branch"
  • When the job is triggered by config auto export (CAE)

Validate Environment

This job extends the .prerequisites in test_deploy.yml.

This job executes a script, where several variables are checked and set accordingly.

Except:

  • Commit message contains "Merge tag"
  • Commit message contains "Merge branch"
  • When the job is triggered by config auto export (CAE)

Validate

This stage is currently not officially used.

Execute

This stage executes the build itself.

Build Prod

This stage is responsible for building the production website, which we want to finally deploy.

Build Prod Site

First we have to do some preparation. This is done in the before_script section.

We start a docker image for php with the specifies versions:

  • PHP_MAJOR_VERSION
  • PHP_MINOR_VERSION

In the script section we first update the environment by executing /usr/local/bin/update-env.

After that we install the production site with composer install without the tools for development. That is pretty it. We also add drupal/core-vendor-hardening.

The entire installation is saved as a build artefact called build-prod where all git files are excluded.

Build Prod Site NG

First we have to do some preparation. This is done in the before_script section.

We start a docker image for php with the specifies versions:

  • PHP_MAJOR_VERSION
  • PHP_MINOR_VERSION

In the script part we install two of our LakeDrops modules:

After that we execute:

  • composer lakedrops:config
  • composer lakedrops:docker4drupal

The entire installation is saved as a build artefact called build-prod-ng where all git files are excluded.

Rules, when the job should run:

  • $DISABLE_DEPLOYMENT_LOCAL != "1"

Build

Here, we define jobs, which build the Drupal website and does some additional checks. This stage is responsible for the database downloads as well.

Build Site

We start a docker image for php with the specifies versions:

  • PHP_MAJOR_VERSION
  • PHP_MINOR_VERSION

After that, we start to build the Drupal site with Composer:

Now we execute a composer install twice to ensure that all patches get applied correctly. After that we setup our own LakeDrops tools by composer lakedrops:scaffold and composer lakedrops:docker4drupal.

The next step is to pull all required Docker images with Docker Compose and start the services.

After everything has started, we create some directories, e.g. for styles, logs and JavaScript.

The entire installation is saved as a build artefact called build where all git files are excluded.

Rules, when the job should run:

  • the tests are enabled by $TESTSDISABLED == "no"

Build Theme

This job creates the required files for the styling of the website.

The following jobs must run before successfully:

Check 4 Updates

e start a docker image for php with the specifies versions:

  • PHP_MAJOR_VERSION
  • PHP_MINOR_VERSION

First we have to do some preparation. This is done in the before_script section.

The job stores the artefacts in check4updates.

Rules, when the job should run:

  • $CHECKUPDATES is set
  • we are on the develop branch

Check 4 Outdated Packages

We start a docker image for php with the specifies versions:

  • PHP_MAJOR_VERSION
  • PHP_MINOR_VERSION

The following jobs must run before successfully. Some are optional, which means, if they exist, this job waits for them to complete:

First we have to do some preparation. This is done in the before_script section.

First we perform a composer update which all dependencies without the development part.

Then we check for outdated packages:

1
2
composer outdated --minor-only --strict --no-interaction $OUTDATED_EXTRAS
composer outdated --patch-only --strict --no-interaction $OUTDATED_EXTRAS

The variable $OUTDATED_EXTRAS contains the ignored package like: --ignore=drupal/somemodule --ignore=drupal/another

Rules, when the job must not run:

  • the commit message contains SKIP_check4outdated and $CHECKUPDATES != "yes"
  • the commit message contains "Merge branch 'develop' into 'main'"
  • the commit message contains "Merge branch 'release' into 'main'"

Rules, when the job should run:

  • $IGNORE_COMPOSER_EXTENDED_AUDIT == "0"

Check 4 Security Vulnerability Advisories

We start a docker image for php with the specifies versions:

  • PHP_MAJOR_VERSION
  • PHP_MINOR_VERSION

The following jobs must run before successfully. Some are optional, which means, if they exist, this job waits for them to complete:

First we have to do some preparation. This is done in the before_script section.

First we perform a composer update which all dependencies without the development part.

Then we check for outdated packages:

1
2
composer diagnose --no-interaction || true
composer audit --no-dev --locked --format=table

Rules, when the job must not run:

  • the commit message contains SKIP_check4security and $CHECKUPDATES != "yes"
  • the commit message contains "Merge branch 'develop' into 'main'"
  • the commit message contains "Merge branch 'release' into 'main'"

Rules, when the job should run:

  • $IGNORE_COMPOSER_AUDIT == "0"

Download DB

This job executes a script, that removes the SQL file $]PROJECT_NAME}.sql, if it exists already. After that an Ansible script is executed, which dumbs the current Drupal database of the live stage.

The file is saved as a build artefact, which contains $]PROJECT_NAME}.sql.

Rules, when the job must not run:

  • $DISABLE_DEPLOYMENT != "0"
  • $DISABLE_DEPLOYMENT_LOCAL != "1"
  • $DBREQUIRED == "no"
  • $INITIALINSTALL == "yes"
  • $TESTSDISABLED == "yes"
  • $LOCALDBFILE != "none"

Rules, when the job should run:

  • the tests are enabled by $DISABLE_CI_TESTS != "1"

Download DB NG

This is very similar to Download DB, but does not use Ansible. It uses Docker Compose instead to execute the drush sql:dump command to dumb the current live database to the file $]PROJECT_NAME}.sql.

The file is saved as a build artefact, which contains $]PROJECT_NAME}.sql.

Rules, when the job must not run:

  • $DISABLE_DEPLOYMENT != "0"
  • $DISABLE_DEPLOYMENT_LOCAL != "0"
  • $DBREQUIRED == "no"
  • $INITIALINSTALL == "yes"
  • $TESTSDISABLED == "yes"
  • $LOCALDBFILE != "none"

Rules, when the job should run:

  • the tests are enabled by $DISABLE_CI_TESTS != "1"

Prepare

This stage is used to prepare the system for testing. This stage is also used for internal purpose and for proprietary tools.

Import DB

This job extends .importdb, which itself defines the anchor import_db_default.

The following jobs must run before successfully:

Rules, when the job must not run:

  • $DISABLE_DEPLOYMENT != "0"
  • $DISABLE_DEPLOYMENT_LOCAL != "1"
  • $DBREQUIRED == "no"
  • $INITIALINSTALL == "yes"
  • $TESTSDISABLED == "yes"
  • $LOCALDBFILE != "none"

Rules, when the job should run:

  • the tests are enabled by $DISABLE_CI_TESTS != "1"
Import DB Default Anchor

We start a docker image for php with the specifies versions:

  • PHP_MAJOR_VERSION
  • PHP_MINOR_VERSION

A script copies the available SQL file into the PHP container, drops the current DB and imports the file. If the variable SKIPCONFIGIMPORT is no, a drush config-import gets executed as well. After that the UPDATE_DB_COMMAND gets called as well as a cache reset.

Import DB NG

This job extends .importdbNG, which itself defines the anchor import_db_default.

The following jobs must run before successfully:

Rules, when the job must not run:

  • $DISABLE_DEPLOYMENT != "0"
  • $DISABLE_DEPLOYMENT_LOCAL != "0"
  • $DBREQUIRED == "no"
  • $INITIALINSTALL == "yes"
  • $TESTSDISABLED == "yes"
  • $LOCALDBFILE != "none"

Rules, when the job should run:

  • the tests are enabled by $DISABLE_CI_TESTS != "1"

Import Local DB

This job extends .importdbLocal, which itself defines the anchor import_db_default.

The following jobs must run before successfully:

A before script is executed, which copies the ${LOCALDBFILE} to ${PROJECT_NAME}.sql.gz and unzips that file.

Rules, when the job must not run:

  • $DBREQUIRED == "no"
  • $TESTSDISABLED == "yes"
  • $LOCALDBFILE != "none"

Rules, when the job should run:

  • the tests are enabled by $DISABLE_CI_TESTS != "1"

Update DB

We start a docker image for php with the specifies versions:

  • PHP_MAJOR_VERSION
  • PHP_MINOR_VERSION

If the variable SKIPCONFIGIMPORT is no, a drush config-import gets executed as well. After that the UPDATE_DB_COMMAND gets called as well as a cache reset.

The following jobs must run before successfully:

Rules, when the job must not run:

  • $DBREQUIRED == "yes"
  • $DBUPDREQUIRED == "no"
  • $INITIALINSTALL == "yes"
  • $TESTSDISABLED == "yes"

Rules, when the job should run:

  • the tests are enabled by $DISABLE_CI_TESTS != "1"

Postprocess

Check 4 Updates Commit

We start a docker image for php with the specifies versions:

  • PHP_MAJOR_VERSION
  • PHP_MINOR_VERSION

The following jobs must run before successfully. Some are optional, which means, if they exist, this job waits for them to complete:

First we have to do some preparation. This is done in the before_script section.

Rules, when the job should run:

  • $CHECKUPDATES is set
  • we are on the develop branch

Test

After the preparation and the setup of a Drupal database, we can execute tests. This happens in this stage.

All tests use the two Internal Test Stages.

Test Code Style

This job uses PHP Code Sniffer and stores the artefacts in phpcs.

Rules, when the job should run:

  • the code style tests are enabled by $DISABLE_CI_TEST_CODESTYLE != "1"

Test PHPUnit

This job uses PHP Unit and stores the artefacts in phpunit.

Rules, when the job should run:

  • the code style tests are enabled by $DISABLE_CI_TEST_PHPUNIT != "1"

Test Backstop

This job uses Backstop and stores the artefacts in backstop.

The following jobs must run before successfully. Some are optional, which means, if they exist, this job waits for them to complete:

Rules, when the job should run:

  • the backstop tests are enabled by $DISABLE_CI_TEST_BACKSTOP != "1"

Test Cypress E2E

This job uses Cypress and stores the artefacts in cypresse2e. The tests are executed in headless mode.

The following jobs must run before successfully. Some are optional, which means, if they exist, this job waits for them to complete:

Rules, when the job should run:

  • the cypress tests are enabled by $DISABLE_CI_TEST_CYPRESSE2E != "1"

Internal Test Stages

Every test stage needs internal reoccurring definitions.

.test_lakedrops_default

We start a docker image for php with the specifies versions:

  • PHP_MAJOR_VERSION
  • PHP_MINOR_VERSION

The following jobs must run before successfully:

But this job can start running when the Build Site has finished, so other test jobs can run in parallel.

After running, we copy the logs from the php container to the current directory.

.test_lakedrops_theme

The following jobs must run before successfully:

In the before_script part, we copy the $THEME_CSS_PATH into the php container, if the $THEME_BUILD variable ist set to "yes".

In after_script we remove $THEME_CSS_PATH again.

Deploy

This stage handles the deployment of the produced build artefact.

Deploy

The following jobs must run before successfully:

This job executes an Ansible script, which does the deployment.

Rules, when the job must not run:

  • $DISABLE_DEPLOYMENT != "0"
  • $DISABLE_DEPLOYMENT_LOCAL != "1"

Deploy NG

The following jobs must run before successfully:

First we have to do some preparation. This is done in the before_script section.

This job does not use Ansible. A lot of commands are executed in the script section. Here is a summary:

  • create the directories, where the site gets deployed to:
    • app
    • db
    • files
    • redis
    • backup
  • sets the ownership and permissions of these files
  • start all the docker service with Docker Compose
  • execute several drush commands
  • create config auto export directory
  • restart cron jobs

To see all the commands, visit test_deploy.yml.

Rules, when the job must not run:

  • $DISABLE_DEPLOYMENT != "0"
  • $CI_COMMIT_BRANCH != "main" && $ENFORCE_DEPLOYMENT != "1"

Finalize

Here we can do cleanups and remove some garbage the build process produces.

Shut Down Docker Project

We start a docker image for php with the specifies versions:

  • PHP_MAJOR_VERSION
  • PHP_MINOR_VERSION

The following jobs must run before successfully:

This job simply stops and removes the docker container, which are started for the build by:

docker compose down

Release Stage

This stage is responsible for creating a new release.

Release Job

This job extends the internal job .merge.

Rules, when the job should run:

  • $CI_COMMIT_MESSAGE contains CREATE_RELEASE
  • we are on the develop branch

Rules, when the job must not run:

  • if the reference is a tag
  • $CI_COMMIT_MESSAGE contains Merge tag
  • if it is triggered by config auto export (CAE)
  • $CHECKUPDATES is set
  • $TRIGGERTASK is set

Internal Stages

Here we have some internal stages, which cannot be executed alone. The purpose is to collect repeating execution.

.prepareaccess

This internal job, you find in mixins.yml.

Mainly, we care about getting access to:

  • the host $CI_SERVER_HOST with ssh via $SSH_PRIVATE_KEY
  • GitLab Login via $GITLAB_ACCESS_TOKEN and $GITLAB_PRIVATE_TOKEN
  • Docker configuration via $DOCKER_AUTH_CONFIG

.preparecomposerplugins

This internal job, you find in mixins.yml.

The following points will be prepared for Composer:

  • the version >2, if $DOWNGRADE_COMPOSER is not set
  • the configuration allow-plugins
  • the configuration audit.abandoned, depend on $IGNORE_COMPOSER_ABANDONED_AUDIT
  • unsupported module versions, which are allowed to use

.merge

First we have to do some preparation. This is done in the before_script section.

We start a docker image for php with the specifies versions:

  • PHP_MAJOR_VERSION
  • PHP_MINOR_VERSION

We create a clean directory and clone the git project $CI_PROJECT_PATH for branch $SOURCE_BRANCH from $CI_SERVER_HOST right into it.

After that we execute the merge script, which does a few checks and use the gitlab tool to merge.

Finally, we remove the directory we created above.

.retry-for-system-issues

This little job ste the amount of retries to "2", when:

  • runner_system_failure
  • stuck_or_timeout_failure

.cache_paths

All internal jobs concerning cache will use this job here. It defines the paths:

  • .docker-init/
  • assets/
  • drush/
  • files/
  • keys/
  • settings/
  • vendor/
  • web/core/
  • web/libraries/
  • web/modules/contrib/
  • web/profiles/contrib/
  • web/sites/
  • web/themes/contrib/
  • .ahoy.yml
  • .env
  • docker-compose.yml

Configuration

GitLab Runner

To setup a GitLab runner, they need to be installed first - see also our Ansible role. Then go to the Drupal project on GitLab into Settings / CI/CD / Runners and follow the instructions there. Make sure that the runner gets tagged with default in GitLab.

As a result, a file /etc/gitlab-runner/config.toml get created and this should be edited to look like this:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
[[runners]]
  name = "Name of runner"
  url = "https://gitlab.lakedrops.com/"
  token = "TOKEN"
  executor = "docker"
  [runners.cache]
    [runners.cache.s3]
    [runners.cache.gcs]
  [runners.docker]
    tls_verify = false
    image = "registry.lakedrops.com/docker/gitlab-drupal-ci:php-7.4"
    privileged = true
    disable_entrypoint_overwrite = false
    oom_kill_disable = false
    disable_cache = false
    volumes = ["/var/run/docker.sock:/var/run/docker.sock", "/cache"]
    cache_dir = "/cache"
    shm_size = 0

The image is tagged for the PHP version the project uses. For projects with different PHP versions a different GitLab runner should be configured or the image being overwritten for each task in the .gitlab-ci.yml of the respective project.

GitLab Variables

Some variables are required for proper access control, and you have to provide them in the project configuration by going to Settings / CI/CD / Variables and adding these variables:

  • SSH_PRIVATE_KEY: A private key being generated elsewhere just for this purpose. This is necessary for cloning private Git repositories.
  • GITLAB_ACCESS_TOKEN: For authenticating with the LakeDrops Gitlab package repository, you have to create an access tokeb for the user who runs the pipeline and provide it in this variable.

GitLab CI instructions in Drupal project

The Drupal project should be setup with the Drupal Development Environment composer plugin and then get a .gitlab-ci.yml file for the pipeline configuration:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
variables:
  COMPOSE_PROJECT_NAME: myproject_$CI_COMMIT_REF_SLUG
  ENVIRONMENT_NAME: myproject/$CI_COMMIT_REF_NAME
  THEME_CSS_PATH: web/themes/custom/mytheme/css

include:
  - project: 'gitlab-ci-cd/drupal'
    ref: master
    file: '/test-and-deploy.yml'

Validate Environment:
  extends: '.prerequisites'

Build Prod Site:
  extends: '.build-prod'

Build Site:
  extends: '.build'

Download DB:
  stage: build
  tags:
    - default
  variables:
    GIT_STRATEGY: none
  script: |
    if [[ "$DBREQUIRED" == "yes" ]]; then
      echo "Scripts have not been developed yet"
    fi
  cache: {}
  dependencies:
    - 'Validate Environment'
  artifacts:
    name: dbdump
    when: always
    paths:
      - ${CI_PROJECT_NAME}.sql
  except:
    refs:
      - tags
    variables:
      - $CI_COMMIT_MESSAGE =~ /^Merge tag /i
      - $CAE
      - $DISABLE_CI_TESTS

Import DB:
  extends: '.importdb'

Update DB:
  extends: '.updatedb'

Build Theme:
  extends: '.theme'
  before_script:
    - cd web/themes/custom/mytheme

Test Code Style:
  extends: '.codestyle'

Test PHPUnit:
  extends: '.phpunit'

Test Behat:
  extends: '.behat'

Test Backstop:
  extends: '.backstop'

Deploy:
  stage: deploy
  tags:
    - default
  variables:
    GIT_STRATEGY: none
  environment:
    name: ${ENVIRONMENT_NAME}
  script:
    - echo "Scripts have not been developed yet"
  cache: {}
  dependencies:
    - 'Build Prod Site'
    - 'Build Theme'
  except:
    refs:
      - tags
    variables:
      - $CI_COMMIT_MESSAGE =~ /^Merge tag /i
      - $CAE

The tasks for downloading the database dump and to deploy the site finally, depend on your hosting environment and have not been generalized, unless you're using our Ansible environment too, then please refer to Using Ansible to Dump DB and Using Ansible for Deployment below. Otherwise, you have to write the scripts for those two tasks yourselves.

Variables

  • COMPOSE_PROJECT_NAME: a string only unique project name which will be used to identify caches and Docker containers.
  • ENVIRONMENT_NAME: the environment name for the GitLab UI.
  • THEME_CSS_PATH: the relative path to the theme's css path where the generated artefact can be found.
  • PHP_MAJOR_VERSION: the major PHP version, defaults to 10.
  • PHP_MINOR_VERSION: the minor PHP version, default to 4.
  • COMPOSER_DOWNGRADE: by default, composer 2 is being used. To use composer 1 instead, set this variable to 1.
  • INITIAL_INSTALL: if set to yes, the initial Drupal site installation gets triggered and all DB and test tasks will be skipped. Can also be used as [INITIAL_INSTALL] in the Git commit message.
  • PULL_DB: if set to yes, the download of a fresh database dump will be forced, regardless of any other conditions. Can also be used as [PULL_DB] in the Git commit message.
  • DISABLE_CI_TESTS: if this variable is set to any value, the tasks in the test stage will be skipped. This is useful e.g. in a development environment where you push and run pipelines often but don't want to run the tests every single time.
  • $DISABLE_GITLAB_CI_TESTS: if this variable is set to "1", no GitLab Tests will be executed, see: GitLab Templates.
  • DISABLE_CI_TEST_BACKSTOP: if this variable is set to any value, the visual regression tests with Backstop will be skipped.
  • DISABLE_CI_TEST_BEHAT: if this variable is set to any value, the Behat tests will be skipped.
  • DISABLE_CI_TEST_CODESTYLE: if this variable is set to any value, the PHP CS tests will be skipped.
  • DISABLE_CI_TEST_CYPRESSE2E:: if this variable is set to any value, the Cypress tests will be skipped.
  • DISABLE_CI_TEST_PHPUNIT: if this variable is set to any value, the PHP unit tests will be skipped.
  • CAE: This variable is being used by the Drupal module Config auto export
  • RESET_LOCALE: if set to yes, the interface translations will be wiped completely and built from scratch. Can also be used as [RESET_LOCALE] in the Git commit message.

Usage

This chapter is incomplete so far and needs more attention.

Initial installation of a Drupal site

This pipeline covers both, the initial installation and later updates of a Drupal site. The latter is the default, because it happens regularely whereas the initial installation only happens once.

Therefore, if you run the pipeline the first time to initially install the Drupal site, either add [INITIAL_INSTALL] to your commit message or define the variable INITIAL_INSTALL with the value yes when triggering the pipeline from the GitLab UI.

When either of these conditions apply, no database will be downloaded, imported or updated and all tests will be skipped.

Handling of the database

To build and test your Drupal site prior to deployment, a database with content for this project is required and this is handled by the pipeline pretty smart.

First, it needs to be decided, if a fresh dump of the database needs to be collected or if the already existing database from the previous pipeline run can be re-used. Here is how the prepared pipelines make that decision:

A fresh database is being pulled if one of the following conditions apply, tested in the given order:

  • if the pipeline run for the master branch
  • if the commit message contains the string [PULL_DB]
  • if the variable PULL_DB is set to yes
  • if no database container from a previous pipeline exists
  • if the database from the previous pipeline doesn't contain any user data yet

If a new database is required, the task Download DB in the build stage will create a dump and make it available as an artifact for subsequent tasks. The task Import DB will then import that dump in the prepare stage.

Otherwise, those two tasks will be skipped and the task Update DB will be executed in the prepare stage instead.

Example

We provide a Demo Drupal 10 project which uses this GitLab CI framework. If you want to give it a try, login to our GitLab, go to the project and click on "Request access". Once we've accepted your request, you can clone the project, make some changes and push them back to the project. This will trigger the pipeline thet you can watch running.

The deployed demo site is available online.

The setup for this project is very simple. Because we wanted to protect the pipeline configuration, we removed the default .gitlab-ci.yml file from the repository and configured the usage of the example pipeline configuration that you can review in the Drupal GitLab CI Project.

Screenshot

Also, all the variables have been configured in the GitLab project settings, because developers should not have access to them:

Screenshot

GitLab CI for a single Drupal module

For building and testing a single Drupal.org, custom or contrib module, we make use of the official Drupal project GitLab Templates. To include the templates, we provide a GitLab CI file private-modules, which can be included in your own gitlab-ci.yml file. This includes the following stages:

  • build
  • validate
  • test

To get a complete documentation for this powerful project, see here.

This file private-modules.yml includes another CI template called composer-packages.yml. Here you can store your module as a composer package in your own GitLab installation. The versioning is also included. Now you can use it in a composer.json file. You have to define the repository like this:

1
2
3
4
"<your-name>": {
      "type": "composer",
      "url": "https://<your-domain>/api/v4/group/<your-group>/-/packages/composer/"
    },