Problem
When replicating or snapshotting data between PostgreSQL databases, null bytes (0x00) in text/varchar columns can cause invalid byte sequence for encoding "UTF8": 0x00 errors on the target database. This happens because PostgreSQL's binary COPY protocol validates for null bytes, even though they may exist in the source data (e.g. from historical bugs in older PostgreSQL versions).
Proposal
Add an optional sanitizer processor that strips null bytes from string column values before they are passed to the target processor. This should:
- Be configurable via env var (
PGSTREAM_PROCESSOR_STRIP_NULL_CHAR_BYTES=true) or YAML (modifiers.strip_null_char_bytes: true)
- Follow the existing processor decorator pattern
- Log a warning with schema, table, column, and primary key info when null bytes are stripped
- Have no performance impact when disabled
Problem
When replicating or snapshotting data between PostgreSQL databases, null bytes (0x00) in text/varchar columns can cause
invalid byte sequence for encoding "UTF8": 0x00errors on the target database. This happens because PostgreSQL's binary COPY protocol validates for null bytes, even though they may exist in the source data (e.g. from historical bugs in older PostgreSQL versions).Proposal
Add an optional sanitizer processor that strips null bytes from string column values before they are passed to the target processor. This should:
PGSTREAM_PROCESSOR_STRIP_NULL_CHAR_BYTES=true) or YAML (modifiers.strip_null_char_bytes: true)