Run and write tests

In short

• piighost has more than a thousand automated tests. They run in a few seconds without downloading any AI model.
• The tests that load real models are kept apart. They run every night on GitHub.
• Before any merge, a blocking check verifies formatting, types, code security and documentation.
• Some of the tests are run by no automated job, because the libraries they need are missing. See What runs nowhere in CI.

The terms are defined in the glossary.

Run the tests

addopts excludes the integration marker by default and disables the anyio and langsmith_plugin plugins (pyproject.toml:166). With asyncio_mode = "auto", an async def test needs no decorator.

Check

uv run pytest -q must end with no failed and no error. The deselected tests (deselected) are the integration tests.

What make lint checks

make lint modifies nothing and fails at the first problem (Makefile:11-16):

1. ruff format --check .: formatting.
2. ruff check .: style rules, including mandatory type annotations.
3. pyrefly check src tests examples docs/tools: type checking, on explicit paths.
4. bandit -c pyproject.toml -r src examples: code security.
5. python skills/piighost-docs/scripts/audit.py: audit of the documentation pages.

How the suite is organized

Write a test

1. Put the file in the folder of the stage under test, on the model of the neighboring test.
2. Use ExactMatchDetector({"Claire Dubois": "PERSON"}) as the detector, so that no model loads.
3. For a storage backend, use InMemoryConversationMemory(), or fakeredis for Redis as in tests/conversation_memory/test_redis.py:21-27.
4. If the test requires an optional dependency, call pytest.importorskip("module_name").
5. If the test loads a real model (torch, gliner2, spacy, transformers), mark it @pytest.mark.integration.
6. For a new detector, also add its constructor to DETECTORS in tests/components/detector/test_contract.py.

Check

uv run pytest path/to/test.py -v
make lint

What GitHub CI runs

A change that touches only .md files, docs/ or LICENSE does not trigger ci.yml (paths-ignore).

What runs nowhere in CI

The dev group installs the extras config, argon2, crypto, redis, mistral, langchain, pydantic-ai, observation, fuzzy, sqlalchemy (pyproject.toml:150-151). It installs neither llama-index, nor gliner2, nor spacy, nor transformers, nor presidio.

• In the tests job, the tests that request these libraries through importorskip are skipped.
• In integration.yml, these libraries are installed, but -m integration selects only the tests marked integration.

The skipped and unmarked tests therefore run in no job. They include:

• tests/integrations/llama_index/ (both files)
• the gliner2, transformers, presidio and spacy entries of tests/components/detector/test_contract.py
• tests/components/guard/test_gliner2_guard.py
• tests/components/detector/ner/test_presidio.py, and the unmarked tests of test_spacy.py and test_transformers.py
• tests/config/test_presidio_detector.py

To run them locally:

uv sync --all-extras --all-groups
uv run pytest -rs
uv sync

Then go back to the default environment with uv sync, and run make lint again. An environment loaded with every extra can hide an import or type error that CI would see.

Pitfalls

• A skipped test looks like a passed one. Run pytest -rs to see the list and the reason for each skip.
• The observation conftest installs a global tracer for the whole session. The clear-text tracing warning is therefore filtered in pyproject.toml:172-174, and only its own tests check it.
• pyrefly receives explicit paths. Without them, it checks nothing in a git worktree and fails anyway (Makefile:8-10).
• The Redis tests use fakeredis. The behavior of a real server or of a cluster is not covered.

See also Add or replace a component for the contract and import tests.