Configure a pipeline by file, catalog and command line

In short

• A text file (TOML or JSON) describes the whole protection chain, that is what to look for, how to replace it, where to keep the conversation memory.
• The file never contains a secret. Keys and passwords come from the server's environment variables.
• Pattern lists (e-mails, card numbers, etc.) can come from an online registry, the catalog. A pinned version is downloaded once, then read locally.
• The piighost validate command checks a file without starting anything. It fits an automatic check before going to production.
• A typo in the file is rejected, never ignored.

There is no graphical interface. All configuration goes through this file and the command line. The terms are defined in the glossary. Each section and each key of the file is listed in the configuration reference of the technical guide. The configuration tutorial builds a file step by step.

Choose how to load the configuration

source is a file path or a catalog reference (catalog:piighost/generic). The .json suffix selects the JSON reader, any other suffix the TOML reader (config/settings.py:54-69).

Write a minimal file

The smallest valid file declares only a detector (examples/config/detector_only.toml):

[detector]
type = "regex"
patterns = { EMAIL = '[a-z0-9._%+-]+@[a-z0-9.-]+\.[a-z]{2,}' }

The entity linker, the anonymizer and the overlap resolver take their default value. An address becomes <<EMAIL:1>>. Other examples are in examples/config/, namely pipeline.toml, thread_redis.toml, thread_sqlalchemy.toml, minimal.json.

Rules to know

BR-CFG-01 BR-CFG-01 Configure a pipeline by file, catalog and command line When the file contains an undeclared key, then loading fails with ConfigValidationError.. When the file contains an undeclared key, then loading fails with ConfigValidationError. For example, [detectr] instead of [detector] is rejected. 2 locations

BR-CFG-02 BR-CFG-02 Configure a pipeline by file, catalog and command line When the file declares a [memory] section, then it describes a conversation pipeline.. When the file declares a [memory] section, then it describes a conversation pipeline. load_pipeline rejects it with this configuration declares a memory; use load_thread_pipeline. Conversely, load_thread_pipeline rejects a file without [memory]. 1 location · 17 direct tests

BR-CFG-03 BR-CFG-03 Configure a pipeline by file, catalog and command line When token_memo_ttl is set without a [memory] section, then validation fails.. When token_memo_ttl is set without a [memory] section, then validation fails. Only a conversation pipeline keeps this cache. 1 location

BR-CFG-04 BR-CFG-04 Configure a pipeline by file, catalog and command line When a value is given in several places, then explicit arguments take precedence, then the PIIGHOST_* variables, then the file or catalog.. When a value is given in several places, then explicit arguments take precedence, then the PIIGHOST_* variables, then the file or catalog. 1 location

BR-CFG-05 BR-CFG-05 Configure a pipeline by file, catalog and command line When a variable targets a subkey, such as PIIGHOST_DETECTOR__TYPE, then it has no effect, because no nested delimiter (the __ that separates a section from its key) is configured.. When a variable targets a subkey, such as PIIGHOST_DETECTOR__TYPE, then it has no effect, because no nested delimiter (the __ that separates a section from its key) is configured. You override a whole section with a JSON object, for example PIIGHOST_DETECTOR='{"type": "exact", "values": {"Patrick": "PERSON"}}'. 1 location

BR-CFG-06 BR-CFG-06 Configure a pipeline by file, catalog and command line When a secret is missing, then the ConfigError error occurs at build time, not at validation.. When a secret is missing, then the ConfigError error occurs at build time, not at validation. piighost validate therefore accepts a file whose secrets are not supplied yet. 2 locations · 4 direct tests

BR-CFG-07 BR-CFG-07 Configure a pipeline by file, catalog and command line When a catalog reference ends with eight hexadecimal characters (a commit), then the response is cached on disk and never downloaded again.. When a catalog reference ends with eight hexadecimal characters (a commit), then the response is cached on disk and never downloaded again. A reference that ends with a tag, or has no selector (latest), is downloaded again at each build. 1 location · 3 direct tests

BR-CFG-08 BR-CFG-08 Configure a pipeline by file, catalog and command line When a regex detector combines catalogs and inline patterns, then the catalogs merge in order, then the inline patterns.. When a regex detector combines catalogs and inline patterns, then the catalogs merge in order, then the inline patterns. On the same label, the last one wins, so an inline pattern overrides any catalog. 1 location · 11 direct tests

BR-CFG-09 BR-CFG-09 Configure a pipeline by file, catalog and command line When a catalog is named generic, us, eu or fr without a catalog prefix, then it is rejected with the catalog reference that replaces it (catalog:piighost/generic).. When a catalog is named generic, us, eu or fr without a catalog prefix, then it is rejected with the catalog reference that replaces it (catalog:piighost/generic). 1 location · 11 direct tests

Supply the secrets

Secrets are read from the environment only. Never write them in the file.

The non-secret variables read elsewhere are PIIGHOST_CATALOG_URL (private registry), XDG_CACHE_HOME (root of the catalog cache), PIIGHOST_API_URL and PIIGHOST_HOOK_LOG (Claude Code hooks, see Plug the protection into an agent).

Pull patterns from the catalog

The catalog is the only source of patterns. The library ships none. A reference is written namespace/name, with an optional :tag or :commit selector, and the optional catalog: prefix. A reference written with the 1.x hub: prefix still works, and so does PIIGHOST_HUB_URL when PIIGHOST_CATALOG_URL is unset.

• Origin: https://catalog.piighost.dev, or PIIGHOST_CATALOG_URL. Only http and https are accepted.
• Timeout: 10 seconds (catalog.py:57).
• Cache: $XDG_CACHE_HOME/piighost/catalog/, otherwise ~/.cache/piighost/catalog/. The file name is a SHA-256 digest of the URL. The 1.x cache, under piighost/hub/, is not read, so each pinned reference is downloaded once more.
• A regex detector takes only the ?part=detector part. If the reference describes a model detector, loading raises CatalogPayloadError. In that case, load the whole configuration with load_config("catalog:…").

Check from the command line

The piighost command needs the config extra (typer). Without it, it prints The piighost CLI requires typer. Install it with: pip install piighost[config] and exits with code 1.

The options of anonymize are --config <file>, --api <url> (mutually exclusive, otherwise Pass at most one of --config and --api.), --thread-id (default default), --json. Without --config or --api, the command runs a regex detector on catalog:piighost/generic:fab51b33 (e-mail, URL, IPv4, card number).

Check

uv run piighost validate examples/config/pipeline.toml
echo "Write to claire.dubois@example.com" | uv run piighost anonymize --config examples/config/detector_only.toml

The first command prints OK: examples/config/pipeline.toml. The second prints Write to <<EMAIL:1>>.

Pitfalls

• validate does not prove that the pipeline starts. Secrets, catalog groups and models are read only at build time (BR-CFG-06 BR-CFG-06 Configure a pipeline by file, catalog and command line When a secret is missing, then the ConfigError error occurs at build time, not at validation.).
• Building a file that names a catalog group calls the network on the first run. A server without outbound access fails with CatalogUnreachableError, unless the cache is already filled.
• A non-writable cache is silently ignored (catalog.py:298-308). The pipeline then downloads again at each start.
• anonymize catches only ConfigError and CatalogError. A missing extra or a guard rail that blocks (PIIRemainingError) surfaces as a full Python traceback.
• --thread-id is default by default. Two calls without an identifier share the same conversation on a conversation pipeline.

Where the rules live

Tests

See also Store conversations and protect traces for the [memory] section.