Skip to main content

CI workflow

This article walks through the process of running end-to-end tests against a matrix of Grafana versions.

Why run end-to-end tests against a matrix of Grafana versions​

Due to Grafana’s dependency sharing mechanism, many plugin-related issues only emerge at runtime. For example, if a plugin invokes a function, component or class that is unavailable in the Grafana runtime environment, any page loading that part of the plugin will crash. These runtime-specific issues are beyond the scope of unit tests but can be effectively identified through end-to-end testing.

To maintain reliability and compatibility, plugin developers must regularly perform end-to-end tests across all supported Grafana versions. The e2e-versions GitHub Action simplifies this process by automatically resolving supported Grafana versions based on your plugin's grafanaDependency, while also including Grafana's main development branch. Integrating this Action into your CI workflows ensures your plugin remains stable and compatible with both older and newer versions of Grafana, giving you confidence in its functionality across versions."

The e2e-versions Action​

The e2e-versions GitHub Action generates a matrix of Grafana image names and versions for use in end-to-end testing a Grafana plugin within a GitHub workflow. The Action supports two modes:

  • plugin-grafana-dependency: This mode resolves the most recent grafana-dev image and returns all the latest patch releases of Grafana Enterprise since the version specified as grafanaDependency in the plugin.json file. To prevent the initiation of too many jobs, the output is capped at 6 versions. This is the default mode.
  • version-support-policy: In this mode, the action resolves versions based on Grafana's plugin compatibility support policy. It retrieves the latest patch release for each minor version within the current major Grafana version. Additionally, it includes the most recent release for the latest minor version of the previous major Grafana version.

For detailed information on configuring the inputs, visit the e2e-versions GitHub page.

Example workflows​

All Grafana plugins created with grafana/create-plugin version 4.7.0 or later automatically include end-to-end tests against a Grafana version matrix as part of their ci.yml workflow. If your plugin was created with an earlier version, you can use the following example workflows to set up and run end-to-end tests with a Grafana version matrix in a separate GitHub workflow:

note

The following examples are generic and based on frontend and backend plugins. Depending on the specifics of your plugin, you may need to modify or remove certain steps in the playwright-tests job before integrating them into your plugin’s workflow.

Backend plugin workflow

.github/workflows/e2e.yml
name: E2E tests
on:
  pull_request:
  schedule:
    - cron: '0 11 * * *' #Run e2e tests once a day at 11 UTC

permissions:
  contents: read
  id-token: write

jobs:
  resolve-versions:
    name: Resolve Grafana images
    runs-on: ubuntu-latest
    timeout-minutes: 3
    outputs:
      matrix: ${{ steps.resolve-versions.outputs.matrix }}
    steps:
      - name: Checkout
        uses: actions/checkout@v4
      - name: Resolve Grafana E2E versions
        id: resolve-versions
        uses: grafana/plugin-actions/e2e-version@main

  playwright-tests:
    needs: resolve-versions
    timeout-minutes: 60
    strategy:
      fail-fast: false
      matrix:
        GRAFANA_IMAGE: ${{fromJson(needs.resolve-versions.outputs.matrix)}}
    name: e2e ${{ matrix.GRAFANA_IMAGE.name }}@${{ matrix.GRAFANA_IMAGE.VERSION }}
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4

      - name: Setup Node.js environment
        uses: actions/setup-node@v4
        with:
          node-version-file: .nvmrc

      - name: Install npm dependencies
        run: npm ci

      - name: Install Mage
        uses: magefile/mage-action@v3
        with:
          install-only: true

      - name: Build binaries
        run: mage -v build:linux

      - name: Build frontend
        run: npm run build

      - name: Install Playwright Browsers
        run: npx playwright install --with-deps

      - name: Start Grafana
        run: |
          docker compose pull
          GRAFANA_VERSION=${{ matrix.GRAFANA_IMAGE.VERSION }} GRAFANA_IMAGE=${{ matrix.GRAFANA_IMAGE.NAME }} docker compose up -d

      - name: Wait for grafana server
        uses: grafana/plugin-actions/wait-for-grafana@main

      - name: Run Playwright tests
        id: run-tests
        run: npx playwright test

      # Uncomment this step to upload the Playwright report to Github artifacts.
      # If your repository is public, the report will be public on the Internet so beware not to expose sensitive information.
      # - name: Upload artifacts
      #   uses: actions/upload-artifact@v4
      #   if: ${{ (always() && steps.run-tests.outcome == 'success') || (failure() && steps.run-tests.outcome == 'failure') }}
      #   with:
      #     name: playwright-report-${{ matrix.GRAFANA_IMAGE.NAME }}-v${{ matrix.GRAFANA_IMAGE.VERSION }}-${{github.run_id}}
      #     path: playwright-report/
      #     retention-days: 30

Frontend plugin workflow

.github/workflows/e2e.yml
name: E2E tests
on:
  pull_request:
  schedule:
    - cron: '0 11 * * *' #Run e2e tests once a day at 11 UTC

permissions:
  contents: read
  id-token: write

jobs:
  resolve-versions:
    name: Resolve Grafana images
    runs-on: ubuntu-latest
    timeout-minutes: 3
    outputs:
      matrix: ${{ steps.resolve-versions.outputs.matrix }}
    steps:
      - name: Checkout
        uses: actions/checkout@v4
      - name: Resolve Grafana E2E versions
        id: resolve-versions
        uses: grafana/plugin-actions/e2e-version@main

  playwright-tests:
    needs: resolve-versions
    timeout-minutes: 60
    strategy:
      fail-fast: false
      matrix:
        GRAFANA_IMAGE: ${{fromJson(needs.resolve-versions.outputs.matrix)}}
    name: e2e ${{ matrix.GRAFANA_IMAGE.name }}@${{ matrix.GRAFANA_IMAGE.VERSION }}
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4

      - name: Setup Node.js environment
        uses: actions/setup-node@v4
        with:
          node-version-file: .nvmrc

      - name: Install npm dependencies
        run: npm ci

      - name: Build frontend
        run: npm run build

      - name: Install Playwright Browsers
        run: npx playwright install --with-deps

      - name: Start Grafana
        run: |
          docker compose pull
          GRAFANA_VERSION=${{ matrix.GRAFANA_IMAGE.VERSION }} GRAFANA_IMAGE=${{ matrix.GRAFANA_IMAGE.NAME }} docker compose up -d

      - name: Wait for grafana server
        uses: grafana/plugin-actions/wait-for-grafana@main

      - name: Run Playwright tests
        id: run-tests
        run: npx playwright test

      # Uncomment this step to upload the Playwright report to Github artifacts.
      # If your repository is public, the report will be public on the Internet so beware not to expose sensitive information.
      # - name: Upload artifacts
      #   uses: actions/upload-artifact@v4
      #   if: ${{ (always() && steps.run-tests.outcome == 'success') || (failure() && steps.run-tests.outcome == 'failure') }}
      #   with:
      #     name: playwright-report-${{ matrix.GRAFANA_IMAGE.NAME }}-v${{ matrix.GRAFANA_IMAGE.VERSION }}-${{github.run_id}}
      #     path: playwright-report/
      #     retention-days: 30

Playwright report​

The end-to-end tooling generates a Playwright HTML test report for every Grafana version that is being tested. In case any of the tests fail, a Playwright trace viewer is also generated along with the report. The Upload artifacts step in the example workflows uploads the report to GitHub as an artifact.

To find information on how to download and view the report, refer to the Playwright documentation.