A Kafka-like queue on MySQL¶
Everybody knows you should not build a message queue on a database. I just did, and I think it was the right call.
What I needed was small. Events have to be kept for a few days. A client that reconnects must be able to say "I have seen up to message 45,610" and get the rest. The volume is a few tens of messages per second.
Kafka does that, and it is what I would pick for real volume. But a Kafka cluster is something you operate. It needs ZooKeeper, monitoring and someone who knows what to do when a broker is unhappy. For tens of messages per second that is absurd, and we already run MySQL.
So kfka keeps the part of Kafka I needed. Messages are rows in one table, and the client holds its own offset.
kfkaManager.addListener((msg) ->
{
// handle message
},
new KfkaPredicate()
.topic("chat")
.lastSeenMessageId(45_610) // next message will be 45,611
.addPropertyMatch("userId", 123));
A new listener can also rewind, and get up to the last 100 messages before the live ones. That makes it a good fit behind server-sent events and websockets, where the browser drops the connection all the time and wants to continue where it was.
Since it is a table, you can filter on your own columns, here userId. Kafka cannot do that for you. Old messages are deleted on a timer after the retention time.
What it is not¶
It is not Kafka. There are no partitions and no consumer groups, and the throughput is whatever one MySQL table gives you. If the volume grows by a factor of a hundred, this is the wrong tool and I will have to say so.
I still feel a bit like I am getting away with something.
Update, November 2026: it never had to be replaced. It has carried every use we have had for it, also for the largest customers.