Tests

Rules can get complex, and might have to handle many complex cases. Luckily ORL has a test type for integration level testing and a test command that executes all the tests located in a directory.

Test file

The test file is basically a single invocation of orl remediate plus 2 possible comparisons: remediated_workspace, and expected_report. At least one of those comparisons is required per test.

type: Test
version: v1
metadata:
  name: My Test

  # Tests are executed in priority order in the same way that rules are
  priority: 1

spec:
  # Each test loads a single rule-space from the defined path
  # relative to the test file
  rulespace: path/to/the/rules/to/test

  # Any number of named test cases can be defined.  Each loads a
  # workspace, remediates, and checks the results
  cases:
    - name: First Case

      # Workspace and Rule-space are both filtered by this language
      language: terraform

      # Workspaces can be loaded from paths relative to this test file.
      # or can be defined as a single inline string.  Only one is allowed.
      workspace:
        path: path/to/workspace/to/test

        # or
        # inline: |
        #   my file

      # (Optional) A workspace to diff with the patched workspace against.
      # Any diffs = a failure.  It can be a relative path, or a single
      # inline string.
      remediated_workspace:
        path: path/to/workspace/after/fix

        # or
        # inline: |
        #   my patched file

      # (Optional) Expectations to test the report against.  All fields
      # from the report can be checked.  Omit items you don't care to check.
      # Everything defined is exact match
      expected_report:
        rules_applied: 5
        changes: 1
        findings: 1
        fixes: 1
        errors: []
        rules:
          # This is the first rule we care about, not necessarily the first
          # rule that is in the report.
          - name: "test-rule"
            findings: 1
            fixes: 1
            changes: 1
            errors: []

          # Ordering is relative.  This means that "test_rule" must be before
          # "third-rule", but there can be any number of rules executed
          # between these two rules.
          - name: third-rule
            findings: 0

Recommended Directory Layout

The recommended layout is as follows:

This directory structure contains 3 rules. Each rule has a workspace to remediate, and the expected results of the remediation for comparison. It also contains the actual ORL rule.

Up a level are all the test files so that a single run or orl test /my-rules can execute all the tests.