Configuration reference

Module: piighost.config

A configuration file describes a whole pipeline declaratively. piighost reads it as TOML or JSON, depending on the file suffix. It then validates it with Pydantic and builds the pipeline the file describes. This page documents every section and every component type.

from piighost.config import load_config, load_pipeline, load_thread_pipeline

The config extra is required (pip install "piighost[config]"). It pulls in pydantic-settings. Unknown keys are rejected, so a typo fails validation rather than being ignored. A component type can need an extra of its own, named in the Extra column of the table that documents it.

Entry points

load_config parses and validates a file into a PipelineConfig without building any component, so no model loads. load_pipeline builds a stateless AnonymizationPipeline and raises ConfigError if the file declares a [memory] section, since a memory describes a thread pipeline. load_thread_pipeline builds a ThreadAnonymizationPipeline and raises ConfigError if the file declares no [memory] section.

from piighost.config import load_pipeline, load_thread_pipeline

stateless = load_pipeline("pipeline.toml")  # no [memory]
thread = load_thread_pipeline("thread.toml")  # has [memory]

File format

The suffix picks the parser. A .json suffix is read as JSON, whatever its case. Any other suffix is read as TOML. The two formats carry the same schema. A section is a TOML table or a JSON object.

[detector]
type = "regex"
patterns = { EMAIL = '[a-z0-9._%+-]+@[a-z0-9.-]+\.[a-z]{2,}' }

[linker]
type = "exact"

[anonymizer.placeholder]
type = "redact"

{
  "detector": { "type": "regex", "patterns": { "EMAIL": "[a-z0-9._%+-]+@[a-z0-9.-]+\\.[a-z]{2,}" } },
  "linker": { "type": "exact" },
  "anonymizer": { "placeholder": { "type": "redact" } }
}

Environment overrides

Every top-level key accepts an override from an environment variable prefixed PIIGHOST_, whether it holds a scalar or a whole section. PIIGHOST_NAME overrides the name scalar. PIIGHOST_DETECTOR overrides the [detector] section with a JSON object. If it is not valid JSON, the variable is rejected as a validation error. Overrides layer above the file key by key. An environment value wins over the file value, and the keys it leaves out keep the file value.

export PIIGHOST_NAME="local-en"
export PIIGHOST_DETECTOR='{"type": "exact", "values": {"Patrick": "PERSON"}}'

No nested delimiter is configured. A variable such as PIIGHOST_DETECTOR__TYPE therefore names no field. It is ignored without an error rather than reaching the type key. A section is overridden by its JSON object only.

Secrets are never read from the file. Each is read from its own environment variable at build time, and a missing one raises ConfigError from build().

Sections

The top-level keys of a PipelineConfig.

[detector]

Discriminated on type. Required.

type = "regex"

Matches confidential data by one regex per label, pulled from inline patterns, catalog groups in catalogs, or both. The groups merge first, then inline patterns. An inline pattern therefore overrides a catalog pattern on the same label. At least one inline pattern or one catalog is required. At load time, each pattern is validated as a compilable regex. It is then compiled under re.ASCII, so \d matches 0-9 and \w stops at the first non-ASCII character. A pattern written with \w therefore matches prénom@corp.com from nom onwards. To take every letter in, scope the Unicode flag to the class, (?u:\w), or name a range. The EMAIL pattern of catalog:piighost/generic names the Latin range À-ɏ this way. See Limitations for what each choice misses.

[detector]
type = "regex"
catalogs = ["catalog:piighost/generic", "catalog:piighost/fr"]
patterns = { EMPLOYEE_ID = 'EMP-[0-9]{4}' }

A group is fetched from the catalog when the config is built, not when it is parsed. A reference pinned to a commit is fetched once, then read from the on-disk cache. PIIGHOST_CATALOG_URL points at a private registry, and PIIGHOST_HUB_URL from 1.x is still read when it is unset. The names generic, us, eu and fr are refused. See Catalog groups for the groups that replace them.

type = "composite"

Runs child detectors together and merges their detections.

[detector]
type = "composite"

[[detector.detectors]]
type = "regex"
catalogs = ["catalog:piighost/generic"]

[[detector.detectors]]
type = "exact"
values = { Patrick = "PERSON" }

type = "exact"

Finds occurrences of literal values, each mapped to a label.

[detector]
type = "exact"
values = { Patrick = "PERSON", Lyon = "LOCATION" }

type = "chunked"

Wraps a detector with a splitter that cuts long text into overlapping chunks.

[detector]
type = "chunked"
chunk_size = 2000
chunk_overlap = 200

[detector.detector]
type = "spacy"
model = "en_core_web_sm"

Model-backed detectors

Each needs its own extra, and every one but presidio needs a model. labels accepts a list or an {emitted: internal} map. max_concurrency caps concurrent inferences. None leaves them unbounded.

[detector]
type = "gliner2"
model = "fastino/gliner2-multi-v1"
labels = ["PERSON", "LOCATION"]
threshold = 0.5
max_chars = 2000

The gliner2 and transformers detectors take max_chars, the longest text one inference sees. A longer text is split into overlapping chunks. Each chunk is scanned separately, then the spans are mapped back. Without this key, the whole text goes to the model in one pass. A model with a short window then truncates the text, and a long document can exhaust memory.

The transformers detector passes aggregation_strategy to its token-classification pipeline, which groups sub-word tokens into whole entities.

The presidio detector takes no model key, since the config path builds Presidio's default English AnalyzerEngine with its default recognizers. Another language, a custom recognizer, or a custom NLP engine is the programmatic path, constructing the engine and passing it to PresidioDetector.

The llm detector reads its provider credential from the provider's own environment variable, never from the file.

[linker]

Optional. Defaults to ExactEntityLinker. One linker exists, so type names it rather than discriminating a union.

[linker]
type = "exact"

[anonymizer]

Optional. Defaults to an Anonymizer with a label-counter factory. When present it carries one [anonymizer.placeholder] table selecting the placeholder factory, discriminated on type.

[anonymizer.placeholder]
type = "label_counter"

The middleware needs a delimited factory, meaning redact, label, label_counter, or label_hash. The mask factory produces P***, which keeps no delimiters and has no recognizer.

[overlap_resolver]

Optional in the file, but the stage runs either way. Omitting the section builds a ConfidenceOverlapResolver. There is no supported way to disable the stage, since the render stage assumes disjoint spans.

[overlap_resolver]
type = "merge"

merge hides every character a detector flagged. With confidence, a regex at confidence 1.0 that found Wirth beats a model that found Loni M. Wirth. Loni M. is then sent in clear. Choose merge when a leak costs more than a masked neighbour word, as in a document with rules and a model together.

[expander]

Optional, and disabled when omitted. One expander exists, so type names it rather than discriminating a union.

[expander]
type = "word_boundary"
case_sensitive = false

[entity_resolver]

Optional. Discriminated on type.

[entity_resolver]
type = "fuzzy"
threshold = 0.85

[guard]

Optional. Discriminated on type. Re-checks the de-identified output for residual confidential data and refuses it when any remains.

type = "detector"

Re-runs a detector on the output. Carries a nested [guard.detector] config.

[guard]
type = "detector"

[guard.detector]
type = "regex"
patterns = { EMAIL = '[a-z0-9._%+-]+@[a-z0-9.-]+\.[a-z]{2,}' }

type = "llm"

Prompts a chat model to find residual PII.

type = "moderation"

Scores the output with a Mistral moderation model. The credential is read from MISTRAL_API_KEY at build time, and build() raises ConfigError when it is unset.

type = "gliner2"

Classifies the output with a GLiNER2 guardrail model that runs in the process, so it needs no credential. The checkpoint is downloaded on the first build, then read from the Hugging Face cache.

[guard]
type = "gliner2"
threshold = 0.5

[override]

Optional. Forces detections through a deny list, whose values are always masked, and vetoes them through an allow list, whose values are always left in clear. Each list is a detector config, [override.deny_list] and [override.allow_list], and both are optional. The 1.x keys, whitelist, blacklist and their strategies, are refused at load time with the key that replaces them, see Upgrading to 2.0.

[override]
allow_list_strategy = "value"

[override.deny_list]
type = "regex"
patterns = { CODENAME = 'ACME-[A-Z]+' }

[override.allow_list]
type = "exact"
values = { "public@corp.com" = "EMAIL" }

[observation_redactor]

Optional. A placeholder factory config, with the same type values as [anonymizer.placeholder]. It redacts the payloads sent to a tracing backend, so a trace holds tokens, not raw values.

[observation_redactor]
type = "label"

Without this section, the clear text and the detection values are traced. A live tracer then emits a PIIGhostSecurityWarning. The pipeline's trace_clear_text flag silences that warning, but it has no key in a configuration file. A file-built pipeline therefore cannot acknowledge clear-text tracing. Passing trace_clear_text=True to the pipeline is the programmatic path.

[memory]

Optional. Its presence makes the pipeline a ThreadAnonymizationPipeline keeping per-thread state. Discriminated on type.

The token_memo_ttl scalar goes with this section, but it sits at the top level, because it bounds the pipeline's own memoized token map and not the store. Setting it without a [memory] raises, because a stateless pipeline memoizes nothing. Multi-instance deployment explains why it matters on a multi-worker deployment.

type = "in_memory"

A process-local store, lost on restart and not shared across workers.

[memory]
type = "in_memory"

type = "redis"

A persistent, multi-worker store. Optionally, it keys each stored message with a hasher and encrypts each stored value with a cipher.

Configure both [memory.hasher] and [memory.cipher], or neither. With neither, the backend stores the mapping in clear and warns. With exactly one, build() raises ConfigError.

The hasher, [memory.hasher], is discriminated on type.

The cipher, [memory.cipher], has one type.

The hasher reads its pepper from PIIGHOST_HASH_PEPPER and the cipher reads its base64 key from PIIGHOST_CIPHER_KEY, both at build time. A missing or malformed value raises ConfigError.

[memory]
type = "redis"
url = "redis://localhost:6379/0"
namespace = "piighost"
ttl = 3600

[memory.hasher]
type = "argon2"

[memory.cipher]
type = "aesgcm"

type = "sqlalchemy"

A durable, multi-worker store backed by any SQLAlchemy-supported database (SQLite, PostgreSQL, ...). It reads the database URL from an environment variable rather than the config file, so the URL and its password stay out of version control. An optional hasher and cipher protect the stored values exactly as they do for Redis.

Configure both [memory.hasher] and [memory.cipher], or neither, exactly as for Redis. With neither, the backend stores the mapping in clear and warns. With exactly one, build() raises ConfigError.

The URL must use an async driver, for example postgresql+asyncpg://... or sqlite+aiosqlite://.... A missing environment variable raises ConfigError at build time. Call await memory.create_schema() once at startup to create the table.

[memory]
type = "sqlalchemy"
url_env = "PIIGHOST_DATABASE_URL"
table_name = "piighost_conversation_messages"

[memory.hasher]
type = "argon2"

[memory.cipher]
type = "aesgcm"

Full example

The keys of examples/config/pipeline.toml, a stateless pipeline pulling a catalog group, adding one inline pattern, and enabling several optional stages. The file itself carries the same keys with a comment on each stage.

[detector]
type = "regex"
catalogs = ["catalog:piighost/generic"]
patterns = { EMPLOYEE_ID = 'EMP-[0-9]{4}' }

[overlap_resolver]
type = "confidence"

[expander]
type = "word_boundary"

[entity_resolver]
type = "fuzzy"
threshold = 0.85

[linker]
type = "exact"

[anonymizer.placeholder]
type = "label_counter"

[override.deny_list]
type = "regex"
patterns = { CODENAME = 'ACME-[A-Z]+' }

[guard]
type = "detector"

[guard.detector]
type = "regex"
patterns = { EMAIL = '[a-z0-9._%+-]+@[a-z0-9.-]+\.[a-z]{2,}' }

[observation_redactor]
type = "label"

The same content in JSON, chosen by a .json suffix, is equivalent. A table becomes an object, an inline table becomes a nested object, and an array of tables becomes an array of objects.

Errors

ConfigFileError and ConfigValidationError are subclasses of ConfigError, so catching ConfigError covers all three. The classes live in piighost.exceptions, so a caller can catch them without the config extra.

See also

• examples/config/ in the repository for six runnable files, detector_only.toml, minimal.toml, minimal.json, pipeline.toml, thread_redis.toml and thread_sqlalchemy.toml, all six loaded by examples/config/run.py.
• Command-line interface for validating a file from the shell.
• Detectors reference for the detector each type builds.
• LangChain middleware reference for driving a thread pipeline in an agent.
• Configure a pipeline by file, catalog and command line, for the configuration rules BR-CFG-01 to BR-CFG-09 and where each lives in the code.