Usage Guide

This guide explains how to use the plugin effectively both during local development and in Continuous Integration (CI) pipelines.

Quick Start

Install the plugin using pip:

pip install pytest-html-plus

Like your usual approach, run your pytest

pytest (OR)
pytest -n auto (OR)
pytest -n auto --reruns 1

This will: - Generate a combined JSON test report called final_report.json - Create a visual HTML report inside the report_output/ folder

Output Capture Policy

By default, captured stdout and stderr are included for every reported test. To reduce JSON and HTML report size for verbose suites, select an output policy with --plus-output.

The supported values are:

  • all: Include stdout and stderr for every test. This is the default and preserves the existing behavior.

  • failed-only: Include stdout and stderr only for failed tests, setup/teardown errors, and expected failures (XFAIL). Ordinary skips and passing tests do not include captured output.

  • none: Do not include captured stdout or stderr in the generated reports.

The policy controls the top-level stdout and stderr report fields. It does not change retry attempt errors or traces.

For example:

pytest --plus-output=failed-only

Reusable Profiles

If you want to reuse the same reporting options across local runs and CI jobs, define a named profile in pyproject.toml and activate it with --plus-profile.

[tool.pytest-html-plus.profiles.ci]
html-output = "ci-report"
json-report = "ci.json"
capture-screenshots = "failed"
output = "failed-only"
generate-xml = true
xml-report = "ci.xml"

pytest --plus-profile=ci

Profile keys must match existing pytest-html-plus CLI option names without the leading --. Regular CLI flags still win, so you can override a profile for a specific run:

pytest --plus-profile=ci --json-report=override.json

An explicit CLI option overrides the selected profile. If neither is supplied, the default value is all.

The JSON report (final_report.json)

The JSON file contains rich, structured test metadata that you can use beyond HTML reporting:

Use Cases:

  • 📊 Internal Dashboards: - Feed data into tools like Grafana, Tableau, or custom React dashboards. - Track test pass/fail trends over time, flaky test rates, test durations, and more. - Combine with Git metadata (commit hash, branch, author) to analyze test health by developer or feature area.

  • 🔁 Automated Analytics: - Run periodic jobs to parse the JSON and detect:

    • Slow tests

    • Most flaky tests

    • Recently added tests with high failure rates

    • Tag-based trends (e.g., @smoke, @login, etc.)

  • ⚠️ Slack/Email Notifications: - Send summaries directly from JSON (e.g., “3 failures in checkout flow”). - Include direct links to Playwright traces or screenshots in messages.

  • 📂 Storing Historical Test Data: - Archive reports from each CI run into S3, GCS, or internal storage. - Useful for audits, traceability, or debugging intermittent failures over time.

  • 🔌 Custom Integrations: - Push data into test case management systems (e.g., TestRail, Xray). - Trigger Jira ticket creation when critical tests fail.

Structure Preview:

The report has a structure like:

{
  "filters": {
    "skipped": 1,
    "untracked": 75,
    "failed": 5,
    "total": 77,
    "passed": 71,
    "marker_counts": {
      "skip": 1,
      "parametrize": 5,
      "jira": 1,
      "link": 1
    }
  },
  "results": [
    {
      "test": "test_skipped_example",
      "nodeid": "tests/unit/test_convert_json_to_junit_xml.py::test_skipped_example",
      "status": "skipped",
      "duration": 0.0001316650000262598,
      "trace": null,
      "error": null,
      "markers": [
        "skip"
      ],
      "file": "tests/unit/test_convert_json_to_junit_xml.py",
      "line": 13,
      "stdout": "",
      "stderr": "",
      "timestamp": "2026-07-07T06:16:43.404084Z",
      "screenshot": "screenshots",
      "logs": [],
      "worker": "gw0",
      "links": [],
      "attempts": [
        {
          "status": "skipped",
          "trace": null,
          "error": null,
          "duration": 0.0001316650000262598,
          "timestamp": "2026-07-07T06:16:43.404107Z"
        }
      ],
      "attempt_count": 1,
      "flaky": false,
      "attempt_statuses": [
        "skipped"
      ],
      "first_failure_index": null,
      "first_failure": null
    }
  ]
}

You can easily parse this using Python, JavaScript, or any JSON-compatible tool.

Retry / flaky metadata

Each test result includes retry-aware metadata:

  • attempts: List of recorded attempts for the test.

  • attempt_count: Total number of attempts made for the test.

  • attempt_statuses: Status of each attempt in order.

  • flaky: true when the test failed in an earlier attempt but eventually passed.

  • first_failure_index: Index of the first failed attempt, or null if the test never failed.

  • first_failure: Failure details from the first failed attempt, or null if unavailable.

Metadata Schema

Alongside the main JSON report, the plugin writes a lightweight metadata file named plus_metadata.json (in the root directory). This captures high-value, actionable context for the run and is also rendered at the top of the HTML report.

File: <root>/plus_metadata.json

Example

{
  "report_title": "report_output",
  "environment": "staging",
  "branch": "feature/login-flow",
  "commit": "e1b6737f858a7ceb1da88de2ed5d368ee6206408",
  "python_version": "3.11.7",
  "pytest_version": "8.3.3",
  "generated_at": "2025-08-20T12:34:56.123456"
}

Fields

  • report_title (string) The title shown in the HTML header. By default this is derived from the --html-output folder name (e.g., report_output). Can be overridden with --html-output.

  • environment (string) Target environment (e.g., staging, prod-sim). Auto-detected from common CLI flags if present (--env or --environment). Defaults to "NA" unless additionally –rp-env` if not provided.

  • branch (string) Git branch at test time. Falls back to "NA" if git info isn’t available (e.g., not a repo).

  • commit (string) Full commit SHA for traceability. Falls back to "NA" if unavailable.

  • python_version (string) Python interpreter version used for the run (e.g., 3.11.7).

  • generated_at (ISO 8601 string) Timestamp when metadata was created, e.g., 2025-08-20T12:34:56.123456.

Behavior & Notes

  • Zero-config: All fields are collected automatically where possible.

  • Overrides: - Title: --plus-report-title="My Nightly Report" - Environment: pass your usual flag (--env or --environment); if your project has it already, the plugin will pick it up, else pass –rp-env

  • Non-git folders / CI without checkout: Branch/commit gracefully become "NA" (no failures).

  • xdist: Metadata is written once (on the controller), not per worker.

  • Portability: The HTML report reads this file at render time and shows a compact, copy-ready “Run Metadata” section at the top.

Tip

Keep metadata lean and high-value. We intentionally avoid low-actionability fields (e.g., full pip freeze or OS package lists) to keep reports fast, portable, and CI-artifact friendly.