Skip to content
GitLab Drupal

GitLab CI for Drupal pipelines

This project contains a number of pre-configured tasks for GitLab CI/CD that allow to run very powerful pipelines which contain the following stage order:

We have a predefined execute job called LakeDrops, which looks like this:

These pre-configured tasks can easily be included into your Drupal project. A GitLab runner on any host can then be configured to run those pipelines. The following chapters describe the necessary steps and also all the options available for configuration.

Stages

The following chapters describes all stages and their jobs. The files:

.pre

To prepare and check, if the environment is good to go, we have a .pre stage, which contains the following jobs:

Sanity Checks

The sanity check gets important when you are using Crowdsec. It contains two little scripts, that check:

  • if the crowdsec.settings.yml is in the config split and
  • if it is not enabled in the core.extensions.yml.

Otherwise, it is conflicting with you local settings, since CrowdSec is a tool for production sites.

Except:

  • Commit message contains "Merge tag"
  • Commit message contains "Merge branch"
  • When the job is triggered by config auto export (CAE)

Validate Environment

This job extends the .prerequisites in test_deploy.yml.

This job executes a script, where several variables are checked and set accordingly.

Except:

  • Commit message contains "Merge tag"
  • Commit message contains "Merge branch"
  • When the job is triggered by config auto export (CAE)

Validate

This stage is currently not officially used.

Execute

This stage executes the build itself.

Build Prod

This stage is responsible for building the production website, which we want to finally deploy.

Build Prod Site

First we have to do some preparation. This is done in the before_script section.

We start a docker image for php with the specifies versions:

  • PHP_MAJOR_VERSION
  • PHP_MINOR_VERSION

In the script section we first update the environment by executing /usr/local/bin/update-env.

After that we install the production site with composer install without the tools for development. That is pretty it. We also add drupal/core-vendor-hardening.

The entire installation is saved as a build artefact called build-prod where all git files are excluded.

Build Prod Site NG

First we have to do some preparation. This is done in the before_script section.

We start a docker image for php with the specifies versions:

  • PHP_MAJOR_VERSION
  • PHP_MINOR_VERSION

In the script part we install two of our LakeDrops modules:

After that we execute:

  • composer lakedrops:config
  • composer lakedrops:docker4drupal

The entire installation is saved as a build artefact called build-prod-ng where all git files are excluded.

Rules, when the job should run:

  • $DISABLE_DEPLOYMENT_LOCAL != "1"

Build

Here, we define jobs, which build the Drupal website and does some additional checks. This stage is responsible for the database downloads as well.

Build Site

We start a docker image for php with the specifies versions:

  • PHP_MAJOR_VERSION
  • PHP_MINOR_VERSION

After that, we start to build the Drupal site with Composer:

Now we execute a composer install twice to ensure that all patches get applied correctly. After that we setup our own LakeDrops tools by composer lakedrops:scaffold and composer lakedrops:docker4drupal.

The next step is to pull all required Docker images with Docker Compose and start the services.

After everything has started, we create some directories, e.g. for styles, logs and JavaScript.

The entire installation is saved as a build artefact called build where all git files are excluded.

Rules, when the job should run:

  • the tests are enabled by $TESTSDISABLED == "no"

Build Theme

This job creates the required files for the styling of the website.

The following jobs must run before successfully:

Check 4 Updates

e start a docker image for php with the specifies versions:

  • PHP_MAJOR_VERSION
  • PHP_MINOR_VERSION

First we have to do some preparation. This is done in the before_script section.

The job stores the artefacts in check4updates.

Rules, when the job should run:

  • $CHECKUPDATES is set
  • we are on the develop branch

Check 4 Outdated Packages

We start a docker image for php with the specifies versions:

  • PHP_MAJOR_VERSION
  • PHP_MINOR_VERSION

The following jobs must run before successfully. Some are optional, which means, if they exist, this job waits for them to complete:

First we have to do some preparation. This is done in the before_script section.

First we perform a composer update which all dependencies without the development part.

Then we check for outdated packages:

1
2
composer outdated --minor-only --strict --no-interaction $OUTDATED_EXTRAS
composer outdated --patch-only --strict --no-interaction $OUTDATED_EXTRAS

The variable $OUTDATED_EXTRAS contains the ignored package like: --ignore=drupal/somemodule --ignore=drupal/another

Rules, when the job must not run:

  • the commit message contains SKIP_check4outdated and $CHECKUPDATES != "yes"
  • the commit message contains "Merge branch 'develop' into 'main'"
  • the commit message contains "Merge branch 'release' into 'main'"

Rules, when the job should run:

  • $IGNORE_COMPOSER_EXTENDED_AUDIT == "0"

Check 4 Security Vulnerability Advisories

We start a docker image for php with the specifies versions:

  • PHP_MAJOR_VERSION
  • PHP_MINOR_VERSION

The following jobs must run before successfully. Some are optional, which means, if they exist, this job waits for them to complete:

First we have to do some preparation. This is done in the before_script section.

First we perform a composer update which all dependencies without the development part.

Then we check for outdated packages:

1
2
composer diagnose --no-interaction || true
composer audit --no-dev --locked --format=table

Rules, when the job must not run:

  • the commit message contains SKIP_check4security and $CHECKUPDATES != "yes"
  • the commit message contains "Merge branch 'develop' into 'main'"
  • the commit message contains "Merge branch 'release' into 'main'"

Rules, when the job should run:

  • $IGNORE_COMPOSER_AUDIT == "0"

Download DB

This job executes a script, that removes the SQL file $]PROJECT_NAME}.sql, if it exists already. After that an Ansible script is executed, which dumbs the current Drupal database of the live stage.

The file is saved as a build artefact, which contains $]PROJECT_NAME}.sql.

Rules, when the job must not run:

  • $DISABLE_DEPLOYMENT != "0"
  • $DISABLE_DEPLOYMENT_LOCAL != "1"
  • $DBREQUIRED == "no"
  • $INITIALINSTALL == "yes"
  • $TESTSDISABLED == "yes"
  • $LOCALDBFILE != "none"

Rules, when the job should run:

  • the tests are enabled by $DISABLE_CI_TESTS != "1"

Download DB NG

This is very similar to Download DB, but does not use Ansible. It uses Docker Compose instead to execute the drush sql:dump command to dumb the current live database to the file $]PROJECT_NAME}.sql.

The file is saved as a build artefact, which contains $]PROJECT_NAME}.sql.

Rules, when the job must not run:

  • $DISABLE_DEPLOYMENT != "0"
  • $DISABLE_DEPLOYMENT_LOCAL != "0"
  • $DBREQUIRED == "no"
  • $INITIALINSTALL == "yes"
  • $TESTSDISABLED == "yes"
  • $LOCALDBFILE != "none"

Rules, when the job should run:

  • the tests are enabled by $DISABLE_CI_TESTS != "1"

Prepare

This stage is used to prepare the system for testing. This stage is also used for internal purpose and for proprietary tools.

Import DB

This job extends .importdb, which itself defines the anchor import_db_default.

The following jobs must run before successfully:

Rules, when the job must not run:

  • $DISABLE_DEPLOYMENT != "0"
  • $DISABLE_DEPLOYMENT_LOCAL != "1"
  • $DBREQUIRED == "no"
  • $INITIALINSTALL == "yes"
  • $TESTSDISABLED == "yes"
  • $LOCALDBFILE != "none"

Rules, when the job should run:

  • the tests are enabled by $DISABLE_CI_TESTS != "1"
Import DB Default Anchor

We start a docker image for php with the specifies versions:

  • PHP_MAJOR_VERSION
  • PHP_MINOR_VERSION

A script copies the available SQL file into the PHP container, drops the current DB and imports the file. If the variable SKIPCONFIGIMPORT is no, a drush config-import gets executed as well. After that the UPDATE_DB_COMMAND gets called as well as a cache reset.

Import DB NG

This job extends .importdbNG, which itself defines the anchor import_db_default.

The following jobs must run before successfully:

Rules, when the job must not run:

  • $DISABLE_DEPLOYMENT != "0"
  • $DISABLE_DEPLOYMENT_LOCAL != "0"
  • $DBREQUIRED == "no"
  • $INITIALINSTALL == "yes"
  • $TESTSDISABLED == "yes"
  • $LOCALDBFILE != "none"

Rules, when the job should run:

  • the tests are enabled by $DISABLE_CI_TESTS != "1"

Import Local DB

This job extends .importdbLocal, which itself defines the anchor import_db_default.

The following jobs must run before successfully:

A before script is executed, which copies the ${LOCALDBFILE} to ${PROJECT_NAME}.sql.gz and unzips that file.

Rules, when the job must not run:

  • $DBREQUIRED == "no"
  • $TESTSDISABLED == "yes"
  • $LOCALDBFILE != "none"

Rules, when the job should run:

  • the tests are enabled by $DISABLE_CI_TESTS != "1"

Update DB

We start a docker image for php with the specifies versions:

  • PHP_MAJOR_VERSION
  • PHP_MINOR_VERSION

If the variable SKIPCONFIGIMPORT is no, a drush config-import gets executed as well. After that the UPDATE_DB_COMMAND gets called as well as a cache reset.

The following jobs must run before successfully:

Rules, when the job must not run:

  • $DBREQUIRED == "yes"
  • $DBUPDREQUIRED == "no"
  • $INITIALINSTALL == "yes"
  • $TESTSDISABLED == "yes"

Rules, when the job should run:

  • the tests are enabled by $DISABLE_CI_TESTS != "1"

Postprocess

Check 4 Updates Commit

We start a docker image for php with the specifies versions:

  • PHP_MAJOR_VERSION
  • PHP_MINOR_VERSION

The following jobs must run before successfully. Some are optional, which means, if they exist, this job waits for them to complete:

First we have to do some preparation. This is done in the before_script section.

Rules, when the job should run:

  • $CHECKUPDATES is set
  • we are on the develop branch

Test

After the preparation and the setup of a Drupal database, we can execute tests. This happens in this stage.

All tests use the two Internal Test Stages.

Test Code Style

This job uses PHP Code Sniffer and stores the artefacts in phpcs.

Rules, when the job should run:

  • the code style tests are enabled by $DISABLE_CI_TEST_CODESTYLE != "1"

Test PHPUnit

This job uses PHP Unit and stores the artefacts in phpunit.

Rules, when the job should run:

  • the code style tests are enabled by $DISABLE_CI_TEST_PHPUNIT != "1"

Test Backstop

This job uses Backstop and stores the artefacts in backstop.

The following jobs must run before successfully. Some are optional, which means, if they exist, this job waits for them to complete:

Rules, when the job should run:

  • the backstop tests are enabled by $DISABLE_CI_TEST_BACKSTOP != "1"

Test Cypress E2E

This job uses Cypress and stores the artefacts in cypresse2e. The tests are executed in headless mode.

The following jobs must run before successfully. Some are optional, which means, if they exist, this job waits for them to complete:

Rules, when the job should run:

  • the cypress tests are enabled by $DISABLE_CI_TEST_CYPRESSE2E != "1"

Internal Test Stages

Every test stage needs internal reoccurring definitions.

.test_lakedrops_default

We start a docker image for php with the specifies versions:

  • PHP_MAJOR_VERSION
  • PHP_MINOR_VERSION

The following jobs must run before successfully:

But this job can start running when the Build Site has finished, so other test jobs can run in parallel.

After running, we copy the logs from the php container to the current directory.

.test_lakedrops_theme

The following jobs must run before successfully:

In the before_script part, we copy the $THEME_CSS_PATH into the php container, if the $THEME_BUILD variable ist set to "yes".

In after_script we remove $THEME_CSS_PATH again.

Deploy

This stage handles the deployment of the produced build artefact.

Deploy

The following jobs must run before successfully:

This job executes an Ansible script, which does the deployment.

Rules, when the job must not run:

  • $DISABLE_DEPLOYMENT != "0"
  • $DISABLE_DEPLOYMENT_LOCAL != "1"

Deploy NG

The following jobs must run before successfully:

First we have to do some preparation. This is done in the before_script section.

This job does not use Ansible. A lot of commands are executed in the script section. Here is a summary:

  • create the directories, where the site gets deployed to:
    • app
    • db
    • files
    • redis
    • backup
  • sets the ownership and permissions of these files
  • start all the docker service with Docker Compose
  • execute several drush commands
  • create config auto export directory
  • restart cron jobs

To see all the commands, visit test_deploy.yml.

Rules, when the job must not run:

  • $DISABLE_DEPLOYMENT != "0"
  • $CI_COMMIT_BRANCH != "main" && $ENFORCE_DEPLOYMENT != "1"

Finalize

Here we can do cleanups and remove some garbage the build process produces.

Shut Down Docker Project

We start a docker image for php with the specifies versions:

  • PHP_MAJOR_VERSION
  • PHP_MINOR_VERSION

The following jobs must run before successfully:

This job simply stops and removes the docker container, which are started for the build by:

docker compose down

Release Stage

This stage is responsible for creating a new release.

Release Job

This job extends the internal job .merge.

Rules, when the job should run:

  • $CI_COMMIT_MESSAGE contains CREATE_RELEASE
  • we are on the develop branch

Rules, when the job must not run:

  • if the reference is a tag
  • $CI_COMMIT_MESSAGE contains Merge tag
  • if it is triggered by config auto export (CAE)
  • $CHECKUPDATES is set
  • $TRIGGERTASK is set

Internal Stages

Here we have some internal stages, which cannot be executed alone. The purpose is to collect repeating execution.

.prepareaccess

This internal job, you find in mixins.yml.

Mainly, we care about getting access to:

  • the host $CI_SERVER_HOST with ssh via $SSH_PRIVATE_KEY
  • GitLab Login via $GITLAB_ACCESS_TOKEN and $GITLAB_PRIVATE_TOKEN
  • Docker configuration via $DOCKER_AUTH_CONFIG

.preparecomposerplugins

This internal job, you find in mixins.yml.

The following points will be prepared for Composer:

  • the version >2, if $DOWNGRADE_COMPOSER is not set
  • the configuration allow-plugins
  • the configuration audit.abandoned, depend on $IGNORE_COMPOSER_ABANDONED_AUDIT
  • unsupported module versions, which are allowed to use

.merge

First we have to do some preparation. This is done in the before_script section.

We start a docker image for php with the specifies versions:

  • PHP_MAJOR_VERSION
  • PHP_MINOR_VERSION

We create a clean directory and clone the git project $CI_PROJECT_PATH for branch $SOURCE_BRANCH from $CI_SERVER_HOST right into it.

After that we execute the merge script, which does a few checks and use the gitlab tool to merge.

Finally, we remove the directory we created above.

.retry-for-system-issues

This little job ste the amount of retries to "2", when:

  • runner_system_failure
  • stuck_or_timeout_failure

.cache_paths

All internal jobs concerning cache will use this job here. It defines the paths:

  • .docker-init/
  • assets/
  • drush/
  • files/
  • keys/
  • settings/
  • vendor/
  • web/core/
  • web/libraries/
  • web/modules/contrib/
  • web/profiles/contrib/
  • web/sites/
  • web/themes/contrib/
  • .ahoy.yml
  • .env
  • docker-compose.yml