Add or replace a pipeline component

In short

• piighost is a sequence of interchangeable stages. You replace a detector or a rule without touching the other stages.
• Each stage follows a contract written once. Any part that respects this contract can take its place.
• The configuration file builds these parts. The parts, for their part, know nothing about the configuration file.
• The heavy building blocks (AI models, databases) are installed only if you ask for them.
• The type of the chosen placeholder is checked before execution, so an incompatible combination is rejected early.

This page is mainly for developers. For the flow of a message, read Protect a message before it is sent to the model. The terms are defined in the glossary.

How the code is split

Each stage lives in a package of src/piighost/components/. Its base.py declares the port, that is a Protocol marked runtime_checkable, named Any*. The pipeline depends on the port, never on a concrete class. An object satisfies the port as soon as it has the right method, without inheritance.

When several adapters share a skeleton, this skeleton lives in a Base* template (Template Method pattern). The adapter then supplies only the step that varies, for example _key for a linker or _reduce for an overlap resolver.

Which ports have a template

The ports without a template explain this absence in their docstring. Their implementations differ in their whole mechanism, not in a single step.

The shared NER detector

BaseNERDetector (components/detector/ner/base.py) carries the pass common to the models, that is label mapping, a confidence threshold applied whatever the model, and splitting a text that is too long into overlapping chunks. A text longer than max_chars is split if auto_chunk is active (by default). Otherwise, the detector raises TextTooLongError.

One-way coupling between configuration and core

config/ imports the core and builds it. No core module imports piighost.config at runtime. Each configuration model inherits from _ComponentConfig, which forbids any undeclared key. Each model also exposes a build() that imports its adapter at the last moment. There is neither a builder registry nor a from_config method.

You choose the type of a component with the type key. For example, DetectorConfig is a union discriminated on type, meaning the value of type picks the configuration model to use.

Optional dependencies loaded on demand

The core depends only on typing-extensions. Everything else is an extra of pyproject.toml. A module that needs an extra checks its presence with importlib.util.find_spec and raises an ImportError that names the extra to install. The packages expose these names lazily through a __getattr__ that reads a dictionary of name to module. from piighost import AnonymizationPipeline therefore loads neither torch nor langchain.

Typed placeholders

The placeholder factories carry a preservation tag (components/placeholder/tags.py). These tags are subclasses of str that exist only for the type checker. They say whether the placeholder keeps the value type, the identity, the shape, and whether it can be found in a text. The middleware requires PreservesRecognizableIdentity, that is a placeholder that identifies a single value and that can be found. Passing a mask factory to the middleware therefore becomes a typing error.

Add a detector

1. Copy the closest adapter. For a NER model, start from components/detector/ner/spacy.py and inherit from BaseNERDetector. Otherwise, start from components/detector/regex.py and implement async def detect(self, text: str) -> list[Detection].
2. If the module imports a heavy dependency, keep the import in the module, behind a find_spec test, and expose the class through the package __getattr__.
3. Add the extra in [project.optional-dependencies] of pyproject.toml, then in the all extra.
4. Write the configuration model next to its neighbors in config/models/detector_model.py. It carries a type: Literal["..."], validated fields and a build() that imports the adapter locally.
5. Add this model to the DetectorConfig union of config/models/detector.py.
6. Add a constructor to the DETECTORS list of tests/components/detector/test_contract.py.
7. If the module checks that its dependency is present (step 2), add the (module, dependency, extra) line to OPTIONAL_DEPENDENCY_GUARDS in tests/regression/test_imports.py.

Check

uv run pytest tests/components/detector/test_contract.py tests/regression/test_imports.py tests/config
make lint

For your detector, the contract test must report the same thing as for the other detectors, that is the same position (span) counted in code points, the same text read back from the source and the same external label. A code point is a Unicode character counted once, as Python does. The emoji 😀 therefore counts as one, not as two as in JavaScript.

Pitfalls

• A detector can return overlapping detections. The port allows it. The overlap resolver arbitrates them, and it is always active.
• A heavy import at package level breaks the minimal install. test_missing_optional_dependency_names_its_extra catches it for the modules listed in OPTIONAL_DEPENDENCY_GUARDS. test_every_module_imports_cleanly does not see it if the test environment already has all the extras.
• A misspelled configuration key is rejected, not ignored, thanks to extra="forbid". This is intended.
• The threshold of a NER detector applies even if the model ignores it. Do not rely on the model to filter.

Tests

No test enforces the one-way coupling. The rule "the core never imports piighost.config" holds through code review. To check it, grep -rn "piighost.config" src/piighost --include=*.py | grep -v "^src/piighost/config\|^src/piighost/cli" must be empty.

To run the tests, see Run and write tests. For the configuration, see Configure a pipeline.