How to validate Sigma rules with GitHub Actions for improved security monitoring

Monitoring your identity provider’s logs is critical to identify potential security threats. These logs are vital for a security team, who may store them in a specialized tool like Grafana Loki for enhanced accessibility and analysis.

The ability to pinpoint specific patterns within these logs is key — and by crafting these patterns into Loki queries, you can conduct focused searches across logs. You can set alerts to streamline this process, but conducting checks manually can become cumbersome over time.

Enter Sigma, an open standard for defining pattern detection rules. By employing YAML-formatted rules, Sigma allows you to specify the exact log patterns you’re searching for, thereby optimizing your security operations. What makes Sigma rules particularly powerful is their versatility; they’re generic and can be converted into queries for various backends. An example of this is the pySigma-backend-loki project, which translates Sigma rules into LogQL queries, further enhancing the efficiency and effectiveness of your security monitoring strategy.

This blog post introduces a new GitHub Action, developed by the Grafana Labs Security Operations team, that automates the validation of Sigma rules against our previously developed JSON Schema. We’ll delve into the tool’s history and how it came to be, and then walk through how to integrate it into your GitHub Actions’ workflows.

A (little more than a) bit of history

In the summer of 2023, the Grafana SecOps team was working on an internal project that integrated pySigma and pySigma-backend-loki, building on the foundational work of the Sigma contributors. 

During the project’s progression, we identified a need for better validation within the Sigma repository, where all Sigma rules are stored and maintained. The existing schema, crafted from the Sigma specification repository, required updates. Moreover, it was formatted in Rx Schema, a standard that was quite unfamiliar to me at the time. Upon investigation, we found the Rx Schema format to be outdated and seemingly no longer maintained.

Based on our experience with JSON Schema, we decided to contribute to the Sigma project by creating a JSON Schema for Sigma rules, building on top of the Sigma specification. Sigma rules are written in YAML, so the JSON Schema can help validate their contents. This work involved significant collaboration and iteration with the Sigma maintainers, leading up to the successful merging of our contributions.

The schema is also available on the Schema Store, which is a collection of independent JSON Schemas that can be used in your IDE or JSON tooling. For more information, visit the Schema Store website.

Included in our contribution was a small Bash script, alongside the check-jsonschema project, which allowed us to validate all the rules against the newly introduced JSON Schema. This process surfaced two Sigma rules with invalid related UUIDs, which we promptly addressed. Now, every new commit that alters the rules undergoes validation against this JSON Schema.

A few months ago, in a team discussion, the idea emerged to develop a GitHub Action for workflows to validate Sigma rules, including those in private repositories. None of us had prior experience in this area, so we embarked on a learning journey, eventually creating a GitHub Action. Initially, we expanded the Bash script for use in GitHub Actions, but faced a few challenges in its functionality and maintainability. Ultimately, we rewrote the script in Python to utilize the pathlib module and other features, enhancing the script’s effectiveness.

Our efforts received positive feedback from the Sigma maintainers, particularly Nasreddine Bencherchali, who supported the idea of keeping the script within the Sigma rules repository and fetching the schema directly from the Sigma specification repository. This collaborative effort took time to refine, but ultimately succeeded.

We donated the sigma-rules-validator project to SigmaHQ, and after some revisions and bug fixes, it was officially published and is now available to the community on GitHub Marketplace.

How to validate Sigma rules

Now, we’ll walk through an example of how to validate Sigma rules using the sigma-rules-validator project. The project’s README provides comprehensive instructions, and the Sigma repository itself serves as an excellent example of the action’s application. 

yaml Copy

steps:
   - uses: SigmaHQ/sigma-rules-validator@v1

The process involves enumerating .yml files in the repository’s root directory, validating them against the JSON Schema file, and then outputting the validation results. For a more focused approach, placing Sigma files in a specific directory, like rules, and specifying this directory for validation can streamline the process, and helps prevent non-Sigma rules from being enumerated.

yaml Copy

steps:
   - uses: SigmaHQ/sigma-rules-validator@v1
     with:
       paths: './rules'

You can also pass multiple directories, if you happen to organize different rules in different directories.

yaml Copy

steps:
   - uses: SigmaHQ/sigma-rules-validator@v1
     with:
       paths: |-
         ./rules
         ./custom-rules

For those with a JSON Schema within their Sigma rules repository or intending to validate against a custom JSON Schema, the action offers the flexibility to accommodate these scenarios. The following examples show how this can be done.

yaml Copy

steps:
   - uses: SigmaHQ/sigma-rules-validator@v1
     with:
       paths: './'
       schemaFile: './sigma-schema.json'

yaml Copy

steps:
   - uses: SigmaHQ/sigma-rules-validator@v1
     with:
       paths: './'
       schemaURL: 'https://raw.githubusercontent.com/SigmaHQ/sigma-specification/main/sigma-schema.json'

We hope this initiative will aid in crafting valid Sigma rules and we welcome any feedback, feature requests, or contributions through issues or pull requests. You can also reach out to us in the #security channel of our Grafana Community Slack.