How It Works

~30 min read
↻ Updated: May 17, 2026

🦉 The Outbox Without an Outbox

The idea #

When a service writes a row and must also tell other services about it, the usual answer is an outbox table. This post shows a way to get the same guarantee without one. The event goes into Postgres’ own replication stream, inside the same transaction as the row. By the end you will know how the event goes from the transaction to Kafka, which LSN the consumer may confirm without losing events, and how long Kafka can be down before something breaks.

We ship this at FamPay as factlib1, a Go library, and its consumer OwlPost. The post explains both line by line.

The outbox pattern needs three things: durability, order, and resumable delivery after a consumer disconnects. Every database already provides them, for replication. To keep a standby in sync, the database writes every change to a write-ahead log, ships those bytes to replicas in order, and tracks how far each replica has caught up.

So the outbox table is a second copy of replication, written in application code on the same database. It has its own polling loop, its own high-water mark (processed = true), and its own cleanup job.

Since 9.6 (September 2016)2, Postgres lets the application emit its own events into the replication stream. pg_logical_emit_message writes a blob into the WAL atomically with the surrounding transaction; the same logical decoding that logical replication uses reads it on the other end. The producer call is one line:

// pkg/outbox/producer/producer.go
sqlQuery := "SELECT pg_logical_emit_message(true, $1, $2::bytea)"
err = a.conn.Exec(ctx, sqlQuery, a.prefix, protoBytes)

If the transaction rolls back, the message is never delivered. If it commits, the message goes through the WAL like any replicated change. The replication slot’s confirmed_flush_lsn stores the consumer’s position, so there is no separate table, index, or vacuum job.

1. The dual-write fallacy # 

func CreateUser(ctx context.Context, u User) error {
    if err := db.Save(ctx, u); err != nil {
        return err
    }
    return kafka.Produce(ctx, "user.created", u)
}

This code has one failure mode: the DB commit succeeds but the event never reaches Kafka. It happens two ways:

  1. Kafka returns an error. The user exists in your DB, but downstream services never receive the event. The request returns 500, the user retries, and now you have two users (or a UNIQUE violation, depending on your schema).
  2. The process dies before the Kafka call. OOM, kill -9, panic, or a draining K8s node. The outcome is the same, and there is no error to log.

No atomic operation spans your database and your message broker. Kafka transactions (KIP-983) do not help. They cover a producer’s writes across Kafka topics only, and Postgres is outside them. The fix is to make event emission part of the same atomic write as the business data. That is the outbox pattern.

2. The outbox table and its costs # 

BEGIN;
  INSERT INTO users (id, ...) VALUES (...);
  INSERT INTO outbox (id, aggregate_type, payload, created_at, processed)
  VALUES (gen_random_uuid(), 'user', '{...}'::jsonb, now(), false);
COMMIT;

A separate process polls:

SELECT id, aggregate_type, payload
FROM outbox
WHERE processed = false
ORDER BY created_at
LIMIT 100;

For each row it produces to Kafka, then:

UPDATE outbox SET processed = true WHERE id = ANY($1);
-- or
DELETE FROM outbox WHERE id = ANY($1);

This works. The costs are polling, vacuum, and cleanup, as described below.

The polling-latency / scan-cost tradeoff #

Poll every 100 ms and lag stays ~100 ms, at the cost of 864,000 SELECT scans per day per worker even when no events exist. Poll every 5 s and you add 5 s of p99 latency to every downstream. 1 second is the usual compromise.

Vacuum, indexes, and cleanup #

UPDATE outbox SET processed = true writes a new row version; the old tuple becomes dead and stays until autovacuum removes it. You need a partial index:

CREATE INDEX outbox_unprocessed_idx ON outbox (created_at) WHERE processed = false;

It makes that update not-HOT (Heap-Only Tuple; after the update the row no longer matches the index predicate), so index bloat grows with table bloat. At 10K events/sec:

10,000 inserts/sec      → 10K live rows added per second
+ 10,000 updates/sec    → 10K dead rows per second
                        → vacuum has to reclaim ~864M dead tuples/day

DELETE instead of UPDATE leaves a dead tuple too, so the bloat and the vacuum work are the same. The table only shrinks when autovacuum keeps up with your inserts, which usually means tuning autovacuum_vacuum_scale_factor down to ~0.01 and autovacuum_vacuum_cost_limit up to 5000 for this table. And it stays unbounded without a cleanup job:

DELETE FROM outbox WHERE processed = true AND created_at < now() - interval '7 days';

Run that on a 10 GB table at peak and p99 latency goes up. The safe options are a chunked delete (LIMIT + pg_sleep) or partition-by-day + DROP PARTITION. Both need extra code, alerts, and runbooks.

Polling vs change-data-capture #

The other option is Debezium tailing the WAL for INSERTs on outbox: no polling, ~10 ms latency, row-level semantics. The cost is running Debezium, a JVM service on Kafka Connect, often with a schema registry. That fits if you already run that stack.

The event is in the WAL when you commit. The next section reads it from there, without the table or the Connect tier.

3. pg_logical_emit_message #

The function signature4:

pg_logical_emit_message(
    transactional boolean,
    prefix        text,
    content       text   [, flush boolean DEFAULT false]
) → pg_lsn

pg_logical_emit_message(
    transactional boolean,
    prefix        text,
    content       bytea  [, flush boolean DEFAULT false]
) → pg_lsn

It emits a text or binary logical-decoding message that plugins receive through WAL. transactional = true makes it visible to decoders only on COMMIT; false is decoded right away, even if the transaction later rolls back. factlib uses transactional = true, so the message is flushed with the surrounding COMMIT.

factlib depends on three properties:

  1. Atomic with the surrounding transaction. If you ROLLBACK, the message is never delivered.5 This is the outbox guarantee, without the table.
  2. Decoded by pgoutput / wal2json like any row change. Same START_REPLICATION connection, same confirmed_flush_lsn.
  3. Zero on-disk table footprint after WAL recycling. The bytes stay in WAL until every replication slot has decoded past them (each slot’s restart_lsn marks the oldest WAL it still needs, so a checkpoint recycles anything below the minimum), then Postgres recycles them like any other WAL record. There is nothing to vacuum or clean up.

Debezium-style row-level CDC decodes row writes on an outbox table. factlib inserts an application-defined message into the WAL and decodes only that message. (The underlying logical-decoding framework went GA in 9.4; pgoutput was added in 10 with built-in logical replication.)

4. How factlib emits #

factlib’s producer (pkg/outbox/producer/producer.go) is 99 lines end-to-end. The hot path is Emit(), reproduced below with the per-error metrics.EmitFailures.WithLabelValues(...) calls collapsed:

// pkg/outbox/producer/producer.go
func (a *PostgresAdapter) Emit(ctx context.Context, fact *common.Fact) (string, error) {
    if err := fact.Validate(); err != nil {
        return "", errors.Wrap(err, "failed to validate fact")
    }
    eventId, err := uuid.NewV7()
    if err != nil {
        return "", errors.Wrap(err, "failed to generate event ID")
    }
    outboxEvent := &pb.OutboxEvent{
        Id:            eventId.String(),
        AggregateType: fact.AggregateType,
        AggregateId:   fact.AggregateID,
        EventType:     fact.EventType,
        Payload:       fact.Payload,
        Metadata:      fact.Metadata,
        TraceInfo: &pb.TraceInfo{
            TraceId: fact.TraceInfo.TraceId,
            SpanId:  fact.TraceInfo.SpanId,
            Metadata: map[string]string{
                "parent_op":  fact.TraceInfo.ParentOp,
                "is_sampled": fact.TraceInfo.IsSampled,
            },
        },
        CreatedAt: time.Now().UTC().UnixNano(),
    }
    protoBytes, err := proto.Marshal(outboxEvent)
    if err != nil {
        return "", errors.Wrap(err, "failed to marshal proto event")
    }
    sqlQuery := "SELECT pg_logical_emit_message(true, $1, $2::bytea)"
    err = a.conn.Exec(ctx, sqlQuery, a.prefix, protoBytes)
    return outboxEvent.Id, err
}

WithTxn(txn) binds the producer to a transaction. Right above the function:

func (a *PostgresAdapter) WithTxn(txn postgres.SQLExecutor) (postgres.OutboxProducer, error) {
    return &PostgresAdapter{ conn: txn, logger: a.logger, prefix: a.prefix }, nil
}

Application code:

func CreateUser(ctx context.Context, db *pgxpool.Pool, u User) error {
    return pgx.BeginFunc(ctx, db, func(tx pgx.Tx) error {
        if _, err := tx.Exec(ctx, "INSERT INTO users ...", u.ID, ...); err != nil {
            return err
        }
        producer, _ := factlibProducer.WithTxn(postgres.GetPgxTxn(tx))
        fact, _ := common.NewFact("user", u.ID, "user.created", payloadBytes, nil)
        fact.TraceInfo = &common.TraceInfo{}  // required; Emit dereferences it
        _, err := producer.Emit(ctx, fact)
        return err
    })
}

The producer shares the pgx.Tx with the business INSERT, and Emit never commits on its own. The binding is only a convention. WithTxn accepts any SQLExecutor, and factlib also provides GetPgxPool and GetPgxConn. Pass one of those and the SELECT runs in autocommit, so the event is committed even if the business write later fails. Always pass the transaction.

UUIDv7 for event IDs. A 48-bit millisecond timestamp prefix, then random bits. Two advantages over UUIDv4:

  1. B-tree locality on the consumer side. Whatever ledger the consumer keeps for “events I have seen” (Postgres, Mongo, or Cassandra) gets inserts in time order, which keeps the right side of the btree hot and the rest cold. v4 inserts are uniformly random and bloat the btree over time.
  2. Time ordering. Sort events by ID and you get an approximate timeline without a created_at index. This helps when you debug a replay.

ULID6 gives the same (128-bit, leading millisecond timestamp). We pick v7 because it is part of the UUID standard and round-trips through any Postgres / pgx uuid column without custom serde.

One marshal, one SQL call. Emit does not retry. If the SELECT pg_logical_emit_message(...) fails, the surrounding transaction goes into the aborted state and rolls back, and the caller decides whether to retry the whole business operation.

Latency observability. Each emit records the factlib_event_processing_seconds histogram; the SELECT itself runs in microseconds.

Same emit in Python. factlib has a Python version in python/factlib/ with the same SQL and protobuf bytes:

# python/factlib/index.py
def emit(self):
    cursor = connection.cursor()
    try:
        sql_query = "SELECT pg_logical_emit_message(true, %s, %s::bytea)"
        cursor.execute(sql_query, (self._prefix, self._event.SerializeToString()))
    finally:
        cursor.close()

The Django version uses django.db.connection, so the emit is part of the current ORM transaction when emit() runs inside an atomic() block. The wire format is “bytes in WAL with a prefix”, so any language can emit and any can consume. A Django service emits a payments-user event, OwlPost (Go) sends it to Kafka, and consumers in any language read the same proto.

To check that an emit is written to WAL, create a slot before running the producer (cmd/e2e/main.go, which needs wal_level = logical and connects to postgres://postgres:postgres@localhost:5432/westeros); a slot only receives changes made after it is created. Then read the slot with pg_logical_slot_peek_binary_changes, which does not consume the changes:

-- before: table, publication, slot
CREATE TABLE users (id bigserial PRIMARY KEY, data bytea);
CREATE PUBLICATION demo_pub;
SELECT pg_create_logical_replication_slot('demo_peek', 'pgoutput');

-- run the producer (INSERT + emit), then peek at the bytes.
-- You get BEGIN, MESSAGE and COMMIT; the INSERT is not sent,
-- because the publication has no tables.
SELECT lsn, encode(data, 'hex')
FROM pg_logical_slot_peek_binary_changes(
    'demo_peek', NULL, NULL,
    'proto_version', '1', 'publication_names', 'demo_pub', 'messages', 'true'
);

-- cleanup:
SELECT pg_drop_replication_slot('demo_peek');
DROP PUBLICATION demo_pub;
DROP TABLE users;

5. How OwlPost consumes #

OwlPost (cmd/owlpost/) opens a logical-replication connection, filters the WAL for our prefix, deserialises the protobuf, and produces to Kafka. The connection setup needs two flags:

// pkg/postgres/wal.go — NewWALSubscriber
replUrl := fmt.Sprintf("%s?replication=database", cfg.DatabaseURL)
replConn, err := pgconn.Connect(ctx, replUrl)

?replication=database enters replication mode (it can still run regular SQL, unlike replication=true). factlib opens two connections because after START_REPLICATION, that socket only streams CopyData. replConn does streaming; queryConn runs the SELECT EXISTS(SELECT 1 FROM pg_replication_slots ...) checks.

Setting up the slot #

Two one-time, idempotent operations that pkg/postgres/wal.go runs on every boot (%s is fmt.Sprintf interpolation):

-- ensurePublication
CREATE PUBLICATION %s;
-- ensureReplicationSlot
SELECT pg_create_logical_replication_slot('%s', 'pgoutput');

A publication is a set of tables whose row changes get streamed. We only want logical-decoding messages, but pgoutput requires a publication before it starts, so we make an empty one.

A replication slot is the durability primitive. It keeps WAL until the consumer acks past that LSN (log sequence number, a byte offset into the WAL). This gives at-least-once delivery: if OwlPost crashes for an hour, WAL accumulates for an hour and we resume where we left off. The risk is that a slot whose consumer never returns keeps WAL until the disk fills.

Starting replication # 

// pkg/postgres/wal.go — startReplication
err = pglogrepl.StartReplication(ctx, w.replConn, w.cfg.ReplicationSlotName, w.xLogPos,
    pglogrepl.StartReplicationOptions{
        PluginArgs: []string{
            "proto_version '1'",
            fmt.Sprintf("publication_names '%s'", w.cfg.PublicationName),
            "messages 'true'",   // ← the important one
        },
    })

messages 'true' is a pgoutput plugin arg (added in PG 147) that tells it to decode logical-decoding messages alongside row changes. Without it, the subscriber silently drops pg_logical_emit_message calls, with no error.

w.xLogPos is the start position. On every boot it’s the slot’s confirmed_flush_lsn:

SELECT confirmed_flush_lsn FROM pg_replication_slots WHERE slot_name = $1;

That single column is the entire durability state of the consumer.

The receive loop # 

for {
    rawMsg, err := w.replConn.ReceiveMessage(receiveCtx)
    switch msg := rawMsg.(type) {
    case *pgproto3.CopyData:
        switch msg.Data[0] {
        case pglogrepl.PrimaryKeepaliveMessageByteID:
            // keepalive; reply if requested
        case pglogrepl.XLogDataByteID:
            xld, _ := pglogrepl.ParseXLogData(msg.Data[1:])
            newXLogPos := xld.WALStart + pglogrepl.LSN(len(xld.WALData))
            logicalMsg, _ := pglogrepl.Parse(xld.WALData)
            w.processLogicalMessage(ctx, logicalMsg, newXLogPos)
        }
    }
}

The loop handles two message types:

processLogicalMessage filters by prefix with a type switch:

func (w *WALSubscriber) processLogicalMessage(
    ctx context.Context,
    msg pglogrepl.Message,
    xLogPos pglogrepl.LSN,
) {
    if ldm, ok := msg.(*pglogrepl.LogicalDecodingMessage); ok {
        if ldm.Prefix == w.cfg.OutboxPrefix {
            w.handleMessage(ctx, ldm.Content, xLogPos)
        }
    }
}

We decode the WAL stream of an entire database but act only on our prefix. Other services on the same database each subscribe with their own prefix and ignore the rest. The prefix works as the routing key.

Kafka adapter #

OutboxConsumer.RegisterHandler(prefix, handler) registers one handler per prefix (and an ack callback). For OwlPost the handler is the Kafka adapter:

// pkg/outbox/consumer/kafka.go — KafkaEventHandler
topic := fmt.Sprintf("%s.%s", event.OutboxPrefix, event.Outbox.AggregateType)
key := []byte(event.Outbox.AggregateId)
value, _ := proto.Marshal(&event.Outbox)
headers := map[string]string{
    "event_id":   event.Outbox.Id,
    "event_type": event.Outbox.EventType,
    "LSN":        event.XLogPos.String(),
}
producer.Produce(ctx, topic, key, value, headers)

Three details:

6. Distributed tracing through the WAL #

Trace context must be passed through the outbox explicitly. Otherwise the producer’s Sentry / OTel span ends at the database write, and a disconnected span starts at the Kafka consume. factlib stores it in the protobuf:

// pkg/proto/outbox.proto
message TraceInfo {
  string trace_id = 1;
  string span_id  = 2;
  map<string, string> metadata = 3;   // parent_op, is_sampled
}

The producer fills TraceInfo from its tracer (the Python client reads the active Sentry span from the hub; the Go client does the same). On the consumer, KafkaEventHandler copies those fields into Kafka headers, so Service B opens a child span with the same trace ID, and Sentry / Jaeger / Tempo show the two spans in one trace. Every language uses the same protobuf, so this works across languages.

7. Crash safety: the LSN ack pipeline #

Whether events can be lost depends on when OwlPost tells Postgres that it may discard WAL. This section covers which LSN to confirm, out-of-order Kafka acks, and what a crash or a Kafka outage costs.

OwlPost produces to Kafka asynchronously (franz-go, acks=all, idempotent producer), so each partition stays ordered and a success callback means the record is durably replicated.

Which LSN to confirm #

OwlPost tags each event with WALStart + len(WALData) (section 5), sends the tag in the LSN header, and confirms it once Kafka acks. In the run below, event A’s frame starts at 0/15A0A88 and is 253 bytes long, so the tag is 0/15A0B85.

WALStart + len(WALData) is the end position in physical replication, where WALData holds WAL bytes. Here WALData holds one pgoutput message, so the sum is a WAL position plus a frame length. For a transactional message, WALStart is the end of the message record; in this run, A’s COMMIT record starts right there. A 225-byte event is sent in a 253-byte frame, so the sum is 205 bytes past the end of that COMMIT.

After a restart, Postgres sends a transaction again only if its COMMIT record starts at or after confirmed_flush_lsn. So if another transaction commits inside those 205 bytes, confirming this LSN skips it. This can happen with concurrent transactions: B writes its message first, A commits, and B commits right after.

Fig. 1Which LSN is safe to confirm?
WAL records of two concurrent transactions A and B drawn to scale, with three possible ACK positions and what Postgres sends again after a restart
Records drawn to scale, 8-byte aligned. LSNs, sizes and frames come from one run on Postgres 17.7 (pgoutput, proto_version 1, messages 'true'). The restart outcome of each choice was checked on the same server.

I checked each choice the same way: confirm the LSN, disconnect, and stream again from the slot. With WALStart + len(WALData), Postgres never sent B again. Two choices are safe:

Message LSNs follow write order, and the stream follows commit order. In this run, B’s frames arrive second, but its message LSN (0/15A08C0) is lower than A’s (0/15A0A88). Commit end LSNs rise in stream order.

Acks arrive out of order #

A later event can be acked while an earlier one is still in flight on another partition, so the slot must not follow the latest ack. factlib’s public main does follow it today: listenEventAck sets w.xLogPos = *ackPos on every ack. The safe rule is to advance to the contiguous acked prefix.

The single WAL reader passes events to the producer in stream order, and pending keeps that order. Tag each event with its message LSN (xld.WALStart, Figure 1). Each Kafka success marks one event done; then safe advances to the end of the longest run of acked events at the front of pending:

// the ack rule OwlPost needs (sketch; not in factlib main yet)
func (w *WALSubscriber) onKafkaAck(lsn pglogrepl.LSN) {
    // NOTE: pending is an unbounded slice today; a fixed ~1000-slot ring
    // buffer would cap the in-flight set and back-pressure the WAL reader.
    w.mu.Lock()
    w.acked[lsn] = true
    for len(w.pending) > 0 && w.acked[w.pending[0]] {
        w.safe = w.pending[0]          // this event and all before it are in Kafka
        delete(w.acked, w.pending[0])
        w.pending = w.pending[1:]
    }
    w.mu.Unlock()
}

// 1s tick:
w.SendStandbyStatusUpdate(w.safe)      // everything <= safe is durable
Fig. 2Kafka acks out of order
You ack six events on three Kafka partitions in any order; one slot follows the latest ack, the other the contiguous acked prefix; a crash shows which events are lost or sent twice
Six events from six transactions; keys hash to a 3-partition topic. Both rules use the message LSN from Figure 1, and here a status update follows every ack. At a crash, records Kafka has not acked are gone.

Even in stream order, safe can move backwards, because message LSNs follow write order. This does no harm, because Postgres ignores a confirm below the slot’s current position. OwlPost does not have to stop and wait for in-flight events before it confirms. The rule has two effects:

The cost is a bounded replay window (events since the last status update), which the consumer dedupes on event.Id.

Producer crash before COMMIT #

With transactional=true, logical decoding emits a transaction’s records only on COMMIT. A panic before COMMIT means ROLLBACK, so no commit record is written to the WAL and no events are sent, the same as with the outbox table.

Consumer crash after the Kafka ack #

OwlPost can crash after Kafka stored an event but before the next status update sends its LSN. On restart, Postgres sends everything from confirmed_flush_lsn again, so up to one tick of events reaches Kafka twice: about 10,000 at 10K events/sec with a 1-second tick. Nothing before the slot is lost.

This is at-least-once by design, so the consumer must be idempotent. Dedupe on event.Id (a UUIDv7), in the same transaction as the side effect:

INSERT INTO processed_events (id, processed_at)
VALUES ('019e0b6e-6684-7b4d-9a61-2f8e4c0d5b17', now())
ON CONFLICT (id) DO NOTHING;
-- INSERT 0 1: first copy, handle it
-- INSERT 0 0: second copy, skip it

processed_events(id uuid PRIMARY KEY, processed_at timestamptz) is a table you TTL-prune. An in-memory LRU of recent IDs can be checked before the table, but only the table keeps its data after a consumer restart. A Bloom filter uses the least memory for a membership check, but it has false positives. A negative result proves an event is new, and a positive result does not prove it was handled. Keep the table as the source of truth.

Kafka down for hours #

Fig. 3How long Kafka can be down before something breaks
WAL kept behind a stalled slot grows with outage length and WAL rate, against the 1 GiB alert, the 5 GiB page, max_slot_wal_keep_size, and free disk
WAL per event uses the record layout in section 8: one 500-byte event per commit. The 500 GiB of free disk is a sample. Other WAL is the rest of the cluster’s writes; a stalled slot keeps all of it. Invalidation under max_slot_wal_keep_size and the error after it were checked on Postgres 17.7.

While Kafka is down, OwlPost reads up to 1000 events into the w.events channel (a buffered Go channel; sends block past that), plus whatever franz-go’s producer can buffer. Once both buffers are full, the WAL receive loop stops reading and Postgres keeps WAL behind the slot. The LSN does not advance.

If the outage lasts hours, two things happen:

When Kafka is back, OwlPost reads the backlog, Postgres recycles the WAL, and no data is lost.

8. Ordering & throughput #

Ordering #

Logical decoding gives a total commit order across all transactions on the publisher database. A single-partition topic would keep that strict total order; the per-partition guarantee is usually enough.

One consumer is fine if it keeps up with your event rate; add more partitions only when you need parallel consumers. Repartitioning a live topic changes which partition each key maps to, so choose the partition count for about a year of growth at the start.

Throughput — napkin math #

Each Emit is one extra SELECT pg_logical_emit_message(...) on top of the business transaction. Three costs:

At 10K events/sec:

WAL bytes   ≈ 616 B × 10,000  ≈ 6.2 MB/sec
            ≈ 22 GB/hour
            ≈ 530 GB/day

A modern NVMe sustains 1–3 GB/sec sequential writes; the EBS gp3 volume here did 574 MB/sec. For small events, WAL bandwidth is not the limit. The limit depends on payload size and how you commit. Measured on a 16-core EC2 box (EBS gp3, durable synchronous_commit = on)12:

small events, one emit per commit    fsync-bound      ~25–70K emits/sec
small events, batched (~512/commit)  CPU / WAL locks  ~390K emits/sec
large events (≥100 KB)               WAL bandwidth    ~SSD write rate

A single connection does ~1,000 emits/sec, about the volume’s single-stream fsync rate. Concurrency, and especially batching events into one transaction, share one fsync between many emits. At ~512 per commit the disk is 12% busy. So one node can handle 100K events/sec with batching. On the consumer side, OwlPost batches WAL reads and fans the work out across Kafka partitions or workers. This keeps per-key order but gives up strict global order.

Cost breakdown vs the outbox table #

Outbox table (poll) factlib (logical msg)
Producer SQL 1 INSERT (heap tuple ~24 B header + payload + 2 index entries) 1 SELECT (~616 B WAL)
Producer round-trips 1 1
Consumer query rate 10/sec poll (100 ms) 0 (push via WAL)
Index writes per event 2 (PK + partial-on-processed) 0
Vacuum cost proportional to event rate none
End-to-end latency poll interval (100 ms–5 s) WAL flush + Kafka produce (single-digit ms on a same-VPC pgx connection, derived)
Consumer parallelism N workers (SKIP LOCKED / partitioned table) 1 reader per slot; fan out via Kafka

The producer cost is about the same. The consumer side differs. The outbox table costs more to poll but parallelizes at the database read: N workers with SELECT ... FOR UPDATE SKIP LOCKED, or a partitioned table. A logical slot has exactly one reader, so factlib parallelizes after Kafka.

Limitations #

Cross-database transactions #

pg_logical_emit_message is per-database. If your business transaction spans multiple Postgres clusters, you have no atomic write to begin with, and factlib does not help. You need 2PC or a saga.

Schema evolution #

Standard protobuf evolution rules apply (additive optional fields, never re-use field numbers, never change types), and you need a shared schema registry across consumer languages. factlib does not solve this, like any protobuf-based event bus.

Long Postgres transactions #

With the proto_version '1' plugin arg factlib uses today, logical decoding does not send a transaction’s records until COMMIT, so a 30-second transaction delays its own events by 30 seconds. Transactions that commit meanwhile are not held up. The outbox table behaves the same. Keep transactions short and move long-running work outside them.

Newer pgoutput protocols help: v2 (PG 14) allows streaming of large in-progress transactions with the streaming option on. Once the decoded changes of all in-progress transactions exceed logical_decoding_work_mem, the largest transaction is streamed before its COMMIT13. v3 (PG 15) adds two-phase commit; v4 (PG 16) adds parallel apply14. Switching factlib past v1 is deferred because it needs care around rolled-back streamed messages on the consumer.

Production gotchas #

What this post showed #

  1. pg_logical_emit_message(true, …) makes the event part of the business transaction. COMMIT delivers it, ROLLBACK drops it, and there is no outbox table to write, poll, or vacuum.
  2. The replication slot’s confirmed_flush_lsn replaces the outbox’s processed flag. OwlPost advances it, and Postgres recycles the WAL before it.
  3. The LSN OwlPost confirms must be the message’s WALStart or the COMMIT’s end LSN. WALStart + len(WALData) adds a frame length to a WAL position and can skip a transaction that committed right after.
  4. Kafka acks arrive out of order across partitions, so the slot must advance only to the end of the contiguous acked prefix.
  5. A crash causes at most one status-update interval of duplicates, which consumers drop by event.Id. During a Kafka outage WAL accumulates, so size max_slot_wal_keep_size from your WAL rate.
  6. The outbox table has two advantages: it works on any database, and N workers can poll it. factlib is Postgres-only, with one WAL reader per slot.

Further reading #

  1. factlib — github.com/fampay-inc/factlib ↩︎

  2. Postgres 9.6 release notes — September 2016 ↩︎

  3. KIP-98 — Exactly Once Delivery and Transactional Messaging ↩︎

  4. pg_logical_emit_message — Postgres 17 docs ↩︎

  5. Rollback still writes to WAL. The XLOG_LOGICAL_MESSAGE record is written to WAL when the function runs, and ROLLBACK only adds an abort record after it. Logical decoding buffers the transaction and drops it on abort, so ROLLBACK prevents delivery but the write still happens. The bytes stay in WAL until normal recycling. The outbox table behaves the same way: a rolled-back INSERT still wrote a tuple and WAL that vacuum later reclaims. ↩︎

  6. ULID specification ↩︎

  7. Postgres 14 release notes ↩︎

  8. Disk Full Failure — Postgres 17 docs: “If the disk holding the WAL files grows full, database server panic and consequent shutdown might occur.” ↩︎

  9. xlogrecord.h — postgres/postgres REL_17_0 ↩︎

  10. replication/message.h — postgres/postgres REL_17_0 ↩︎

  11. Measured on a local PostgreSQL 17 (wal_level = logical, synchronous_commit = on), 10,000 iterations of BEGIN → INSERT → pg_logical_emit_message → COMMIT timing the emit SELECT, over a unix socket and TCP loopback on an Apple M3 Max. ↩︎

  12. Throughput measured on a 16-core EC2 instance (EBS gp3, ~574 MB/sec sequential write, single-stream fdatasync ~1,100/sec) running PostgreSQL 15 with wal_level = logical, synchronous_commit = on, over a unix socket. Load from a concurrent Go emitter (pgx, one pg_logical_emit_message per commit unless batched), cross-checked against pgbench within ~2%. ↩︎

  13. Streaming of Large Transactions for Logical Decoding — Postgres 17 docs; proto_version and streaming options ↩︎

  14. logicalproto.h — postgres/postgres REL_17_0 ↩︎

Tags:
Share: twitter/x linkedin