Data models reference

Module: piighost.models

The value objects the pipeline stages exchange. A detector returns Detection, and the linker groups detections into Entity. A Detection and an Entity carry their position as a Span. A splitter cuts a long text into Chunk. Pure Python, no external dependency.

from piighost.models import Chunk, Detection, Entity, Span

Detection, Entity and Span are also re-exported from the package root, so from piighost import Detection resolves the same class.

The dataclass contract

The four models are declared @dataclass(frozen=True, slots=True). Span and Detection add order=True. For a caller that means:

• Immutable. Assigning to a field raises FrozenInstanceError. Build a modified copy with dataclasses.replace, which is how ChunkedDetector remaps a chunk detection onto the original text.
• Slotted. An instance carries no __dict__, so no attribute outside the declared fields can be set on it.
• Compared by value and hashable. Two instances with equal fields are equal and hash alike. A Detection can therefore sit in a set, and an Entity can key the tokens mapping of an Anonymization.
• Sortable for Span and Detection only. Both compare in field order. Entity and Chunk declare no ordering, so comparing two of them raises TypeError.
• Validated at construction. Every invariant is checked in __post_init__, so an invalid instance never exists. Each exception derives from PIIGhostError.

from dataclasses import replace

from piighost.models import Detection, Span

detection = Detection(span=Span(0, 7), text="Patrick", label="PERSON", confidence=1.0)
moved = replace(detection, span=detection.span.shift(10))
# moved == Detection(span=Span(10, 17), text="Patrick", label="PERSON", confidence=1.0)

Span

Module: piighost.models.span

A half-open character range over a text, [start, end), mirroring the slice text[start:end]. It is the geometric primitive the detect and render stages share.

Fields

Ordering is (start, end), so a list of spans sorts left to right. The render stage relies on that order to apply its edits without shifting an offset it has not processed yet.

Properties

length (property)

The number of characters covered, end - start. len(span) returns the same value.

Methods

overlaps(other) -> bool

Whether the two ranges share at least one character. Half-open semantics mean two adjacent ranges such as Span(0, 5) and Span(5, 10) do not overlap.

contains(other) -> bool

Whether other is fully enclosed by this span, bounds included.

shift(offset) -> Span

A copy translated by offset characters. It remaps a span found on a chunk or on normalized text back onto the original text. A shift that would push start below zero raises an error in the constructor rather than clamp the span. The bug therefore surfaces.

extract(text) -> str

The substring of text this span covers.

from piighost.models import Span

span = Span(9, 26)
span.length  # 17
span.extract("write to alice@example.com")  # "alice@example.com"
span.overlaps(Span(26, 30))  # False, the two ranges are adjacent
span.shift(-9)  # Span(0, 17)

Validation

Both derive from SpanError. An empty range is refused because a detection always covers at least one character.

Detection

Module: piighost.models.detection

One occurrence of confidential data a detector found. It carries a span, the matched text, a label and a confidence.

Fields

Ordering is (span, text, label, confidence), so detections sort by position first. The overlap-resolver stage relies on that order.

Methods

overlaps(other) -> bool

Whether this detection's span overlaps the other's. It delegates to Span.overlaps.

to_dict() -> dict[str, str | int | float]

The detection as a flat, JSON-ready dict, the span flattened into start and end. The shape is one level, so a store or a wire format serializes it without knowing the model. The CLI prints detections in this shape under piighost anonymize --json.

from_dict(data) -> Detection (classmethod)

A detection rebuilt from the flat dict to_dict produces.

from piighost.models import Detection, Span

detection = Detection(span=Span(0, 7), text="Patrick", label="PERSON", confidence=1.0)
detection.to_dict()
# {"start": 0, "end": 7, "text": "Patrick", "label": "PERSON", "confidence": 1.0}

Validation

It derives from DetectionError.

Entity

Module: piighost.models.entity

The detections identified as the same value, grouped by the link stage. The group shares one token and restores to one value.

Fields

Properties

label (property)

The shared label of the grouped detections.

text (property)

The canonical value, taken from the first occurrence.

spans (property)

The span of every occurrence, in detection order.

The label, the canonical text and the spans are derived from the detections rather than stored. Nothing can therefore drift out of sync, and the value is held in one place.

from piighost.models import Detection, Entity, Span

first = Detection(span=Span(0, 7), text="Patrick", label="PERSON", confidence=1.0)
second = Detection(span=Span(20, 27), text="Patrick", label="PERSON", confidence=0.8)
entity = Entity(detections=(first, second))

entity.label  # "PERSON"
entity.text  # "Patrick"
entity.spans  # (Span(0, 7), Span(20, 27))

Validation

Both derive from EntityError.

Chunk

Module: piighost.models.chunk

A contiguous slice of a larger text, with its offset in that text.

Fields

Properties

end (property)

The exclusive end offset in the original text, start + len(text).

A splitter produces chunks. Every AnySplitter in piighost.text returns the chunks in order. RecursiveCharacterTextSplitter overlaps consecutive chunks, so a value sitting on a boundary is still seen whole in one chunk. ChunkedDetector runs its wrapped detector on chunk.text, then shifts each detection by chunk.start to remap it onto the original text.

from piighost.text import RecursiveCharacterTextSplitter

splitter = RecursiveCharacterTextSplitter(chunk_size=20, chunk_overlap=5)
chunks = splitter.split("Patrick lives in Lyon and works in Paris.")
# chunks[0] == Chunk(text="Patrick lives in", start=0)
# chunks[1] == Chunk(text="in Lyon and works in", start=14)

Models defined elsewhere

Two more frozen dataclasses travel with the pipeline, each documented on the page of the component that produces it.

See also

• Detectors reference for the detectors that produce a Detection.
• Pipeline reference for the stages these models pass through.
• Extending piighost for building them in your own component.
• CLI reference for the JSON output built from to_dict.