An idempotent Kafka consumer with one extra table

2026-07-14 · #kafka

With auto-commit turned off and offsets committed after processing, Kafka gives at-least-once delivery: after a crash or a rebalance the same record can arrive again. The handler has to tolerate that.

The simplest approach I know is a table of processed message ids, written in the same database transaction as the business change.

CREATE TABLE processed_message (
    message_id  uuid PRIMARY KEY,
    processed_at timestamptz NOT NULL DEFAULT now()
);

INSERT INTO processed_message (message_id)
VALUES (?)
ON CONFLICT DO NOTHING;

If the insert reports zero affected rows, the message was already handled: skip the work and commit the offset. If it reports one, do the work in the same transaction. Either both the marker and the change are stored, or neither.

Details

The id must come from the producer (a header or a field in the payload), not from topic, partition and offset, because a re-published message gets a new offset. Old rows can be removed by a scheduled job once they are older than the topic retention.