GitLab CI for Drupal pipelines

This project contains a number of pre-configured tasks for GitLab CI/CD that allow to run very powerful pipelines which contain the following stages:

• validation
• buildprod
• build
• prepare
• test
• deploy

These pre-configured tasks can easily be included into your Drupal project. A GitLab runner on any host can then be configured to run those pipelines. The following chapters describe the necessary steps and also all the options available for configuration.

Configuration

GitLab Runner

To setup a GitLab runner, they need to be installed first - see also our Ansible role. Then go to the Drupal project on GitLab into Settings / CI/CD / Runners and follow the instructions there. Make sure that the runner gets tagged with default in GitLab.

As a result, a file /etc/gitlab-runner/config.toml get created and this should be edited to look like this:

[[runners]]
  name = "Name of runner"
  url = "https://gitlab.lakedrops.com/"
  token = "TOKEN"
  executor = "docker"
  [runners.cache]
    [runners.cache.s3]
    [runners.cache.gcs]
  [runners.docker]
    tls_verify = false
    image = "registry.lakedrops.com/docker/gitlab-drupal-ci:php-7.4"
    privileged = true
    disable_entrypoint_overwrite = false
    oom_kill_disable = false
    disable_cache = false
    volumes = ["/var/run/docker.sock:/var/run/docker.sock", "/cache"]
    cache_dir = "/cache"
    shm_size = 0

The image is tagged for the PHP version the project uses. For projects with different PHP versions a different GitLab runner should be configured or the image being overwritten for each task in the .gitlab-ci.yml of the respective project.

GitLab Variables

Some variables are required for proper access control, and you have to provide them in the project configuration by going to Settings / CI/CD / Variables and adding these variables:

• SSH_PRIVATE_KEY: A private key being generated elsewhere just for this purpose. This is necessary for cloning private Git repositories.
• GITLAB_ACCESS_TOKEN: For authenticating with the LakeDrops Gitlab package repository, you have to create an access tokeb for the user who runs the pipeline and provide it in this variable.

GitLab CI instructions in Drupal project

The Drupal project should be setup with the Drupal Development Environment composer plugin and then get a .gitlab-ci.yml file for the pipeline configuration:

variables:
  COMPOSE_PROJECT_NAME: myproject_$CI_COMMIT_REF_SLUG
  ENVIRONMENT_NAME: myproject/$CI_COMMIT_REF_NAME
  THEME_CSS_PATH: web/themes/custom/mytheme/css

include:
  - project: 'gitlab-ci-cd/drupal'
    ref: master
    file: '/test-and-deploy.yml'

Validate Environment:
  extends: '.prerequisites'

Build Prod Site:
  extends: '.build-prod'

Build Site:
  extends: '.build'

Download DB:
  stage: build
  tags:
    - default
  variables:
    GIT_STRATEGY: none
  script: |
    if [[ "$DBREQUIRED" == "yes" ]]; then
      echo "Scripts have not been developed yet"
    fi
  cache: {}
  dependencies:
    - 'Validate Environment'
  artifacts:
    name: dbdump
    when: always
    paths:
      - ${CI_PROJECT_NAME}.sql
  except:
    refs:
      - tags
    variables:
      - $CI_COMMIT_MESSAGE =~ /^Merge tag /i
      - $CAE
      - $DISABLE_CI_TESTS

Import DB:
  extends: '.importdb'

Update DB:
  extends: '.updatedb'

Build Theme:
  extends: '.theme'
  before_script:
    - cd web/themes/custom/mytheme

Test Code Style:
  extends: '.codestyle'

Test PHPUnit:
  extends: '.phpunit'

Test Behat:
  extends: '.behat'

Test Backstop:
  extends: '.backstop'

Deploy:
  stage: deploy
  tags:
    - default
  variables:
    GIT_STRATEGY: none
  environment:
    name: ${ENVIRONMENT_NAME}
  script:
    - echo "Scripts have not been developed yet"
  cache: {}
  dependencies:
    - 'Build Prod Site'
    - 'Build Theme'
  except:
    refs:
      - tags
    variables:
      - $CI_COMMIT_MESSAGE =~ /^Merge tag /i
      - $CAE

The tasks for downloading the database dump and to deploy the site finally, depend on your hosting environment and have not been generalized, unless you're using our Ansible environment too, then please refer to Using Ansible to Dump DB and Using Ansible for Deployment below. Otherwise, you have to write the scripts for those two tasks yourselves.

Variables

• COMPOSE_PROJECT_NAME: a string only unique project name which will be used to identify caches and Docker containers.
• ENVIRONMENT_NAME: the environment name for the GitLab UI.
• THEME_CSS_PATH: the relative path to the theme's css path where the generated artefact can be found.
• PHP_MAJOR_VERSION: the major PHP version, defaults to 7.
• PHP_MINOR_VERSION: the minor PHP version, default to 4.
• COMPOSER_DOWNGRADE: by default, composer 2 is being used. To use composer 1 instead, set this variable to 1.
• INITIAL_INSTALL: if set to yes, the initial Drupal site installation gets triggered and all DB and test tasks will be skipped. Can also be used as [INITIAL_INSTALL] in the Git commit message.
• PULL_DB: if set to yes, the download of a fresh database dump will be forced, regardless of any other conditions. Can also be used as [PULL_DB] in the Git commit message.
• DISABLE_CI_TESTS: if this variable is set to any value, the tasks in the test stage will be skipped. This is useful e.g. in a development environment where you push and run pipelines often but don't want to run the tests every single time.
• DISABLE_CI_TEST_BACKSTOP: if this variable is set to any value, the visual regression tests with Backstop will be skipped.
• DISABLE_CI_TEST_BEHAT: if this variable is set to any value, the Behat tests will be skipped.
• DISABLE_CI_TEST_CODESTYLE: if this variable is set to any value, the PHP CS tests will be skipped.
• DISABLE_CI_TEST_PHPUNIT: if this variable is set to any value, the PHP unit tests will be skipped.
• CAE: This variable is being used by the Drupal module Config auto export
• RESET_LOCALE: if set to yes, the interface translations will be wiped completely and built from scratch. Can also be used as [RESET_LOCALE] in the Git commit message.

Stages and Tasks

It's important to use exactly these task names as most of them are also being used to define dependencies:

• validation
  • Validate Environment
• buildprod
  • Build Prod Site
• build
  • Build Site
  • Download DB
• prepare
  • Import DB
  • Update DB
  • Build Theme
• test
  • Test Code Style
  • Test PHPUnit
  • Test Behat
  • Test Backstop
• deploy
  • Deploy

Using Ansible

If you also use an Ansible inventory for your hosts, you can of course integrate them into the GitLab pipelines too. To make this work, you need a separate GitLab runner tagged with ansible and a configuration like this:

[[runners]]
  name = "Name of ansible runner"
  url = "https://gitlab.lakedrops.com/"
  token = "TOKEN"
  executor = "docker"
  environment = ["DOCKER_AUTH_CONFIG={\"auths\":{\"registry.lakedrops.com\":{\"auth\":\"YOURAUTHTOKEN\"}}}"]
  [runners.custom_build_dir]
  [runners.cache]
    [runners.cache.s3]
    [runners.cache.gcs]
  [runners.docker]
    tls_verify = false
    hostname = "Ansible-INVENTORYNAME"
    image = "registry.lakedrops.com/ansible-inventories/INVENTORYNAME:latest"
    privileged = true
    disable_entrypoint_overwrite = false
    oom_kill_disable = false
    disable_cache = false
    volumes = ["/home/gitlab-runner/.ssh/id_rsa:/root/.ssh/id_rsa", "/home/gitlab-runner/.a/variables.yml:/root/.ansible/secrets", "/home/gitlab-runner/.a/inventories/INVENTORYNAME:/root
/.ansible", "/etc/ansible.yml:/etc/ansible.yml", "/var/log/ansible:/var/log/ansible"]
    pull_policy = "always"
    shm_size = 0

The used placeholders in this example are:

• TOKEN: the token of the GitLab runner being generated during setup.
• YOURAUTHTOKEN: your auth token to the LakeDrops GitLab Docker registry.
• INVENTORYNAME: the name of the Ansible inventory to be used.

To learn more about the Ansible imeages for an inventory and how to set up some configuration files, please have a look into Docker for Ansible.

Once you have the GitLab runner for Ansible setup for the Drupal project, you can adjust your GitLab CI configuration:

Using Ansible to Dump DB

You can replace the task above with this simple instruction:

Download DB:
  extends: '.dumpdb'

Using Ansible for Deployment

You need a couple of additional variables and then you can replace the deploy task above with this simple instruction:

variables:
  HOST_NAME: myhostname
  SITE_NAME: mysiteid

Deploy:
  extends: '.deploy'

The host name needs to be set to how the destination host is known in the Ansible inventory, and the site name needs to be set to the ID of the Drupal site in your inventory.

Usage

This chapter is incomplete so far and needs more attention.

Initial installation of a Drupal site

This pipeline covers both, the initial installation and later updates of a Drupal site. The latter is the default, because it happens regularely whereas the initial installation only happens once.

Therefore, if you run the pipeline the first time to initially install the Drupal site, either add [INITIAL_INSTALL] to your commit message or define the variable INITIAL_INSTALL with the value yes when triggering the pipeline from the GitLab UI.

When either of these conditions apply, no database will be downloaded, imported or updated and all tests will be skipped.

Handling of the database

To build and test your Drupal site prior to deployment, a database with content for this project is required and this is handled by the pipeline pretty smart.

First, it needs to be decided, if a fresh dump of the database needs to be collected or if the already existing database from the previous pipeline run can be re-used. Here is how the prepared pipelines make that decision:

A fresh database is being pulled if one of the following conditions apply, tested in the given order:

• if the pipeline run for the master branch
• if the commit message contains the string [PULL_DB]
• if the variable PULL_DB is set to yes
• if no database container from a previous pipeline exists
• if the database from the previous pipeline doesn't contain any user data yet

If a new database is required, the task Download DB in the build stage will create a dump and make it available as an artifact for subsequent tasks. The task Import DB will then import that dump in the prepare stage.

Otherwise, those two tasks will be skipped and the task Update DB will be executed in the prepare stage instead.

Example

We provide a Demo Drupal 9 project which uses this GitLab CI framework. If you want to give it a try, login to our GitLab, go to the project and click on "Request access". Once we've accepted your request, you can clone the project, make some changes and push them back to the project. This will trigger the pipeline thet you can watch running.

The deployed demo site is available online.

The setup for this project is very simple. Because we wanted to protect the pipeline configuration, we removed the default .gitlab-ci.yml file from the repository and configured the usage of the example pipeline configuration that you can review in the Drupal GitLab CI Project.

Also, all the variables have been configured in the GitLab project settings, because developers should not have access to them: