Skip to content
piighostdocs
Technical docs
On this page

Conversational pipeline

You will build a ThreadAnonymizationPipeline that keeps a stable token for the same value from one message to the next. A value seen in message 1 keeps its <<PERSON:1>> in message 2, instead of restarting from scratch at each call. You assemble the pipeline with an in-RAM memory, send two messages of the same thread, then erase the thread.

1. Assemble the pipeline

ThreadAnonymizationPipeline takes the same components as AnonymizationPipeline (detector, linker, anonymizer), plus a conversation memory. Only the detector is required. The code below passes only the detector, and the other components keep their defaults. The memory accumulates each message's detections, thread by thread. The pipeline can thus assign tokens over the whole thread rather than over one isolated message.

The default memory, InMemoryConversationMemory, keeps that state in a process dictionary. Nothing survives a restart and nothing is shared across processes. This memory therefore suits development and tests. The detector stays simple here with ExactMatchDetector, which spots known values. The result is thus verifiable, with no model.

import asyncio

from piighost.components.detector import ExactMatchDetector
from piighost.pipeline import ThreadAnonymizationPipeline

detector = ExactMatchDetector({"Patrick": "PERSON", "Paris": "LOCATION"})
pipeline = ThreadAnonymizationPipeline(detector)

2. De-identify two messages of the same thread

anonymize takes the text and a thread_id. The thread_id is required. There is no shared default thread, so two callers cannot fall into the same thread and leak each other's confidential data. The code sends two messages on the thread "thread-42".

async def main() -> None:
    first = await pipeline.anonymize("Patrick lives in Paris.", thread_id="thread-42")
    print(first.text)

    second = await pipeline.anonymize("Does Patrick love Paris?", thread_id="thread-42")
    print(second.text)


asyncio.run(main())

The output should be:

<<PERSON:1>> lives in <<LOCATION:1>>.
Does <<PERSON:1>> love <<LOCATION:1>>?

Patrick keeps <<PERSON:1>> from the first message to the second, and Paris keeps <<LOCATION:1>>. With a plain AnonymizationPipeline, each call would restart at <<PERSON:1>> with no link to the previous message. The thread memory is what makes the number stable.

3. Restore a value

deanonymize rebuilds the thread's tokens from its memory. It therefore restores any text carrying these tokens, including a model reply the pipeline never de-identified.

async def main() -> None:
    restored = await pipeline.deanonymize("Hello <<PERSON:1>>!", thread_id="thread-42")
    print(restored)


asyncio.run(main())

The output should be:

Hello Patrick!

4. Forget a thread

forget_thread erases a thread's memory and returns the count of what was dropped. Useful to honor an erasure request or to free RAM at the end of a conversation.

async def main() -> None:
    forgotten = await pipeline.forget_thread(thread_id="thread-42")
    print(forgotten)


asyncio.run(main())

The output should be:

Forgotten(messages=2, detections=4)

How it works

ThreadAnonymizationPipeline wraps the base pipeline with a per-thread memory. On each message it caches the detections, then assigns tokens over the union of the whole thread's detections, not the current message alone. A value therefore gets one token for the whole thread. Rendering stays per message. Only the current message's positions are replaced, because each message counts its positions from its own start.

See also

Edit this page Report a problem