The Automated Trials

Automated tests, matrix, cache and artifacts

Why automate tests?

Imagine a world without automated tests: every code change must be verified manually by a human. It's slow, error-prone, and doesn't scale.

Automated tests bring several crucial advantages:

• Early detection: errors are caught before a human reviews the code
• Confidence: you can modify existing code without fearing you'll break everything
• Speed: hundreds of checks in a few seconds
• Consistency: the same tests run every time, without human oversight
• Living documentation: tests show how the code is supposed to work

Types of automated trials

There are several levels of tests, from fastest to most thorough. Each has a specific role.

Linting - The trial of form

Linting checks that the code follows style and formatting rules. It doesn't test whether the code works - it checks that it's well written.

Example checks:

• Does the script start with a shebang (#!/usr/bin/env bash)?
• Is the indentation consistent?
• Are there missing or extra spaces?
• Are variables properly named?

Linting is ultra-fast and catches the simplest errors. It always runs first.

Unit tests - The trial of correctness

Unit tests verify that each individual function produces the correct result. You isolate a small part of the code and test its inputs/outputs.

Example: if you have a function that adds two numbers, a unit test verifies that add 2 3 returns 5.

# Simple unit test example
result=$(./calculate.sh addition 2 3)
if [ "$result" = "5" ]; then
    echo "PASS: addition 2 + 3 = 5"
else
    echo "FAIL: expected 5, got $result"
fi

Integration tests - The trial of coherence

Integration tests verify that multiple components work together. A unit test checks a function in isolation. An integration test verifies that functions collaborate correctly.

Example: checking that the calculation function and the display function produce the correct result when combined.

End-to-end tests - The complete trial

End-to-end tests simulate the complete journey of a user. They test the entire application, from input to output.

Example: running the complete script with arguments, checking that the output file is correct, that error messages are appropriate, etc.

The test matrix - Test everywhere at once

Your code must work on different operating systems and different tool versions. The test matrix lets you run the same tests on all combinations in parallel.

strategy:
  matrix:
    os: [ubuntu-latest, windows-latest, macos-latest]
    node-version: [18, 20, 22]

With this configuration, GitHub Actions creates 9 jobs (3 OS x 3 versions). Each combination is tested independently.

You use the matrix values in the workflow:

jobs:
  test:
    runs-on: ${{ matrix.os }}
    strategy:
      matrix:
        os: [ubuntu-latest, windows-latest, macos-latest]
        node-version: [18, 20, 22]
    steps:
      - uses: actions/checkout@v4
      - uses: actions/setup-node@v4
        with:
          node-version: ${{ matrix.node-version }}
      - run: npm test

The cache - Speed up the trials

Downloading dependencies on every workflow run is slow and wastes resources. The actions/cache action lets you save these files between runs.

steps:
  - uses: actions/checkout@v4

  - name: Cache dependencies
    uses: actions/cache@v4
    with:
      path: ~/.npm
      key: ${{ runner.os }}-npm-${{ hashFiles('**/package-lock.json') }}
      restore-keys: |
        ${{ runner.os }}-npm-

  - run: npm install
  - run: npm test

How it works:

• path: the folder to cache
• key: a unique identifier based on the system and the dependency file
• restore-keys: fallback keys if the exact key is not found

If the package-lock.json file hasn't changed, dependencies are restored from cache instead of being re-downloaded. That's much faster.

Artifacts - Keep the results

Artifacts are files produced during the workflow that you want to save after execution. Most common use cases: test reports, coverage files, logs.

steps:
  - run: npm test -- --coverage

  - name: Save coverage report
    uses: actions/upload-artifact@v4
    with:
      name: coverage-report
      path: coverage/
      retention-days: 30

After the workflow runs, you'll find the artifacts in the "Actions" tab of GitHub, downloadable directly.

To retrieve an artifact in a later job:

  - uses: actions/download-artifact@v4
    with:
      name: coverage-report

Required status checks - Protect the main branch

The true power of automated tests is making them mandatory. With required status checks, you can prevent any merge into main if tests don't pass.

Configuration on GitHub:

1. Go to Settings > Branches
2. Add a Branch protection rule for main
3. Check Require status checks to pass before merging
4. Select the required checks (your workflow job names)

# Example: the "tests" job must pass before any merge
name: CI
on:
  pull_request:
    branches: [main]

jobs:
  tests:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - run: npm test

With this configuration, if someone opens a Pull Request to main and the tests fail, the "Merge" button will be blocked. Nobody can merge broken code.

Test coverage

Test coverage measures the percentage of your code that is actually tested. It's a useful but imperfect indicator.

• 100% coverage doesn't mean your code is perfect - it means every line is executed at least once
• Low coverage is a warning sign - entire parts of your code are never tested
• A good target: aim for 70-80% for a standard project

Coverage tools generate reports showing which lines are tested (in green) and which are not (in red).

Common patterns in test workflows

Run the linter before the tests

Linting is fast. If the code is poorly formatted, there's no point running expensive tests. Put the linter first:

jobs:
  lint:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - run: npm run lint

  tests:
    needs: lint
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - run: npm test

The needs keyword creates a dependency: the tests job only starts if lint succeeded.

Fail fast vs run all

By default, in a matrix, if one job fails, GitHub cancels the others. This is fail-fast mode:

strategy:
  fail-fast: true  # default: stops everything as soon as a job fails
  matrix:
    os: [ubuntu-latest, windows-latest]

If you want to see all results even when some fail:

strategy:
  fail-fast: false  # continues even if a job fails
  matrix:
    os: [ubuntu-latest, windows-latest]

Test only on Pull Requests

To save resources, you can limit tests to Pull Requests:

on:
  pull_request:
    branches: [main]

This avoids re-running tests on every push to every branch. Tests only trigger when someone proposes to merge into main.

Hands-on exercise - Create an automated test suite

Step 1 - Create the repository

mkdir automated-trials
cd automated-trials
git init -b main

Step 2 - Create the calculation script

Create a calculate.sh file that performs simple math operations:

cat > calculate.sh << 'EOF'
#!/usr/bin/env bash
# calculate.sh - Simple command-line calculator
# Usage: ./calculate.sh   

operation="$1"
a="$2"
b="$3"

if [ -z "$operation" ] || [ -z "$a" ] || [ -z "$b" ]; then
    echo "Usage: ./calculate.sh   "
    echo "Operations: addition, subtraction, multiplication"
    exit 1
fi

case "$operation" in
    addition)
        echo $(( a + b ))
        ;;
    subtraction)
        echo $(( a - b ))
        ;;
    multiplication)
        echo $(( a * b ))
        ;;
    *)
        echo "Error: unknown operation '$operation'"
        exit 1
        ;;
esac
EOF

Make it executable:

chmod +x calculate.sh

Test it manually:

./calculate.sh addition 2 3
# Should display: 5

./calculate.sh multiplication 4 7
# Should display: 28

Step 3 - Create the test script

Create a test-calculate.sh file that automatically verifies each operation:

cat > test-calculate.sh << 'EOF'
#!/usr/bin/env bash
# test-calculate.sh - Automated tests for calculate.sh

SCRIPT="./calculate.sh"
ERRORS=0
TOTAL=0

check() {
    local description="$1"
    local expected="$2"
    local actual="$3"
    TOTAL=$((TOTAL + 1))

    if [ "$expected" = "$actual" ]; then
        echo "  PASS: $description"
    else
        echo "  FAIL: $description (expected: '$expected', got: '$actual')"
        ERRORS=$((ERRORS + 1))
    fi
}

echo "=== Tests for calculate.sh ==="
echo ""

echo "-- Addition tests --"
check "2 + 3 = 5"       "5"  "$($SCRIPT addition 2 3)"
check "0 + 0 = 0"       "0"  "$($SCRIPT addition 0 0)"
check "100 + 200 = 300"  "300" "$($SCRIPT addition 100 200)"

echo ""
echo "-- Subtraction tests --"
check "10 - 3 = 7"      "7"  "$($SCRIPT subtraction 10 3)"
check "5 - 5 = 0"       "0"  "$($SCRIPT subtraction 5 5)"

echo ""
echo "-- Multiplication tests --"
check "4 x 7 = 28"      "28" "$($SCRIPT multiplication 4 7)"
check "0 x 100 = 0"     "0"  "$($SCRIPT multiplication 0 100)"
check "3 x 3 = 9"       "9"  "$($SCRIPT multiplication 3 3)"

echo ""
echo "=== Result: $((TOTAL - ERRORS))/$TOTAL tests passed ==="

if [ "$ERRORS" -gt 0 ]; then
    exit 1
fi

exit 0
EOF

Make it executable and test it:

chmod +x test-calculate.sh
./test-calculate.sh

You should see all tests pass.

Step 4 - Create the GitHub Actions workflow

Create the folder and workflow file:

mkdir -p .github/workflows

# .github/workflows/trials.yml
name: Automated Trials

on:
  push:
    branches: [main]
  pull_request:
    branches: [main]

jobs:
  lint:
    name: Form check (lint)
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4

      - name: Check the shebang of calculate.sh
        run: |
          first_line=$(head -n 1 calculate.sh)
          if [ "$first_line" != "#!/usr/bin/env bash" ]; then
            echo "ERROR: calculate.sh must start with #!/usr/bin/env bash"
            exit 1
          fi
          echo "OK: shebang correct"

      - name: Check the shebang of test-calculate.sh
        run: |
          first_line=$(head -n 1 test-calculate.sh)
          if [ "$first_line" != "#!/usr/bin/env bash" ]; then
            echo "ERROR: test-calculate.sh must start with #!/usr/bin/env bash"
            exit 1
          fi
          echo "OK: shebang correct"

  tests:
    name: Tests (${{ matrix.os }})
    needs: lint
    runs-on: ${{ matrix.os }}
    strategy:
      fail-fast: false
      matrix:
        os: [ubuntu-latest, macos-latest]
    steps:
      - uses: actions/checkout@v4

      - name: Make scripts executable
        run: |
          chmod +x calculate.sh
          chmod +x test-calculate.sh

      - name: Run the tests
        run: ./test-calculate.sh

Step 5 - Commit everything

git add calculate.sh test-calculate.sh .github/workflows/trials.yml
git commit -m "Add the calculator, tests and CI workflow"

Step 6 - Verify the structure

# The calculation script exists and works
./calculate.sh addition 10 20

# The tests pass
./test-calculate.sh

# The workflow exists
cat .github/workflows/trials.yml

Step 7 - Run the verification

bash verifier.sh

.\verifier.ps1

Concept summary

YAML summary