Continuous Integration in Cypress

 

 What you’ll learn

• How to run Cypress tests in Continuous Integration
• How to configure Cypress in various CI Providers
• How to record tests to the Cypress Dashboard
• How to run tests in parallel on CI

• Basics

  Running Cypress in Continuous Integration is almost the same as running it locally in your terminal. You generally only need to do two things:

  1. Install Cypress

     npm install cypress --save-dev
     

  2. Run Cypress

     cypress run
     

  Depending on which CI provider you use, you may need a config file. You’ll want to refer to your CI provider’s documentation to know where to add the commands to install and run Cypress. For more configuration examples check out our examples.

  Boot your server

  Challenges

  Typically you will need to boot a local server prior to running Cypress. When you boot your web server, it runs as a long running process that will never exit. Because of this, you’ll need it to run in the background - else your CI provider will never move onto the next command.
  Backgrounding your server process means that your CI provider will continue to execute the next command after executing the signal to start your server.

  Many people approach this situation by running a command like the following:

  npm start & cypress run // Do not do this
  

  The problem is - what happens if your server takes time to boot? There is no guarantee that when the next command runs (cypress run) that your web server is up and available. So your Cypress test may start and try to visit your local server before it is ready to be visited.

  Solutions

  Luckily, there are some solutions for this. Instead of introducing arbitrary waits (like sleep 20) you can use a better option.
  wait-on module
  Using the wait-on module, you can block the cypress run command from executing until your server has booted.

  npm start & wait-on http://localhost:8080
  

  cypress run
  

  Most CI providers will automatically kill background processes so you don’t have to worry about cleaning up your server process once Cypress finishes.

  However, if you’re running this script locally you’ll have to do a bit more work to collect the backgrounded PID and then kill it after cypress run.

  start-server-and-test module

  If the server takes a very long time to start, we recommend trying the start-server-and-test module.

  npm install --save-dev start-server-and-test
  

  In your package.json scripts, pass the command to boot your server, the url your server is hosted on and your Cypress test command.

  {
    ...
    "scripts": {
      "start": "my-server -p 3030",
      "cy:run": "cypress run",
      "test": "start-server-and-test start http://localhost:3030 cy:run"
    }
  }
  

  In the example above, the cy:run command will only be executed when the URL http://localhost:3030 responds with an HTTP status code of 200. The server will also shut down when the tests complete.

  Gotchas

  When working with webpack-dev-server that does not respond to HEAD requests, use an explicit GET method to ping the server like this:

  {
    "scripts": {
      "test": "start-server-and-test start http-get://localhost:3030 cy:run"
    }
  }
  

  When working with local https in webpack, set an environment variable to allow local certificate:

  {
    "scripts": {
      "start": "my-server -p 3030 --https",
      "cy:run": "cypress run",
      "cy:ci": "START_SERVER_AND_TEST_INSECURE=1 start-server-and-test start https-get://localhost:3030 cy:run"
    }
  }
  

  Record tests

  Cypress can record your tests and make the results available in the Cypress Dashboard, which is a service that gives you access to recorded tests - typically when running Cypress tests from your CI provider. The Dashboard provides you insight into what happened when your tests ran.

  Recording tests allow you to:

  • See the number of failed, pending and passing tests.
  • Get the entire stack trace of failed tests.
  • View screenshots taken when tests fail and when using cy.screenshot().
  • Watch a video of your entire test run or a clip at the point of test failure.
  • See which machines ran each test when parallelized.

  To record tests:

  1. Set up your project to record
  2. Pass the --record flag to cypress run within CI.

  cypress run --record --key=abc123
  

  Read the full guide on the Dashboard Service.

  Run tests in parallel

  Cypress can run tests in parallel across multiple machines.

  You’ll want to refer to your CI provider’s documentation on how to set up multiple machines to run in your CI environment.

  Once multiple machines are available within your CI environment, you can pass the --parallel flag to have your tests run in parallel.

  cypress run --record --key=abc123 --parallel
  

  Read the full guide on parallelization.

  Examples

  Cypress should run on all CI providers. We have provided some example projects and configuration for some CI providers to help you get started.

  CI Provider	Example Project	Example Config
  AWS Amplify Console	cypress-example-kitchensink	amplify.yml
  AppVeyor	cypress-example-kitchensink	appveyor.yml
  Azure / VSTS CI / TeamFoundation	cypress-example-kitchensink	azure-ci.yml
  BitBucket	cypress-example-kitchensink	bitbucket-pipelines.yml
  Buildkite	cypress-example-kitchensink	.buildkite/pipeline.yml
  CircleCI	cypress-example-kitchensink	.circleci/config.yml
  CodeShip Basic (has cy.exec() issue)
  CodeShip Pro	cypress-example-docker-codeship
  Concourse
  Docker	cypress-docker-images
  GitLab	cypress-example-kitchensink	.gitlab-ci.yml
  Jenkins	cypress-example-kitchensink	Jenkinsfile
  Semaphore
  Shippable	cypress-example-kitchensink	shippable.yml
  TravisCI	cypress-example-kitchensink	.travis.yml

  Travis

  Example .travis.yml config file

  language: node_js
  node_js:
    - 10
  addons:
    apt:
      packages:
        # Ubuntu 16+ does not install this dependency by default, so we need to install it ourselves
        - libgconf-2-4
  cache:
    # Caches $HOME/.npm when npm ci is default script command
    # Caches node_modules in all other cases
    npm: true
    directories:
      # we also need to cache folder with Cypress binary
      - ~/.cache
  install:
    - npm ci
  script:
    - $(npm bin)/cypress run --record
  

  Caching folders with npm modules saves a lot of time after the first build.