RabbitMQ: Message persistence and Metrics

May 17, 2021

1. Persistence

We've noticed our RabbitMQ lost messages when the container had to be restarted.

Three things to consider to not lose stuff on a restart:

Your exchanges and queues should be durable
Your messages should be persistent
If you run RabbitMQ via Docker, make sure to set a fixed hostname

Links to RabbitMQ's documentation:

Queue durability
Message durability (scroll down to Message durability)
Persistence Configuration

I will show you how I enabled queue and message durability in a Java project based on Micronaut.

1.1. Queue and Exchange durability

1.1.1. Java Client

The RabbitMQ Java client's method com.rabbitmq.client.Channel#queueDeclare takes a boolean parameter durable. Set it to true when you declare your queue, and you're done. The queue will now survive a restart.

Same for exchanges in the method com.rabbitmq.client.Channel#exchangeDeclare.

1.1.2. Micronaut

You will most likely have an implementation of ChannelInitializer, as shown in their documentation.

There, you will find the declaration calls.

1.1.3. CLI

rabbitmqadmin declare exchange name=my-new-exchange type=fanout durable=true
rabbitmqadmin declare queue name=my-new-queue durable=true

1.2. Message durability

1.2.1. Java Client

Now for the message durability, the method com.rabbitmq.client.Channel#basicPublish takes a parameter props of the type BasicProperties. In there, you need to set deliveryMode to the magic hard-coded value 2. Well.

1.2.2. Micronaut

For a Micronaut producer, find the class annotated with @RabbitClient. There, you can set another annotation @RabbitProperty as follows:

@RabbitClient("exchange-name")
@RabbitProperty(name = "deliveryMode", value = "2")

1.2.3. CLI

If you like to use the RabbitMQ cli, you can do it as follows:

rabbitmqadmin -u rabbitmq -p password publish exchange=my-exchange routing_key=my-routing-key properties="{\"delivery_mode\":2}" payload='test'

1.3. Set a fixed hostname for RabbitMQ container

Rabbitmq uses the hostname as part of the folder name in the mnesia directory. Maybe add a --hostname some-rabbit to your docker run?

github.com

This is how the mnesia directory looks like after a few restarts if you don't set a hostname:

Docker generates a new hostname every container run, and RabbitMQ takes it to create the directory structure.

Solution: Set the hostname via --hostname rabbitmq or in your docker-compose.yml via hostname: rabbitmq.

2. Metrics

RabbitMQ has a big tutorial on how to use their built-in prometheus-compatible metrics.

For my needs, adding this to my Prometheus scrape_configs was enough:

  - job_name: 'rabbitmq-server'
    static_configs:
    - targets:
        - 'rabbitmq:15692'

I use the rabbitmq:3.8-management docker image, which seems to already have the metrics endpoint available.

$ docker ps
63ae8f15f137 rabbitmq:3.8-management "docker-entrypoint.s…" 10 minutes ago Up 9 minutes rabbitmq

$ docker exec -it rabbitmq rabbitmq-plugins list
Listing plugins with pattern ".*" ...
 Configured: E = explicitly enabled; e = implicitly enabled
 | Status: * = running on rabbit@rabbitmq
 |/
[  ] rabbitmq_amqp1_0                  3.8.16
[  ] rabbitmq_auth_backend_cache       3.8.16
[  ] rabbitmq_auth_backend_http        3.8.16
[  ] rabbitmq_auth_backend_ldap        3.8.16
[  ] rabbitmq_auth_backend_oauth2      3.8.16
[  ] rabbitmq_auth_mechanism_ssl       3.8.16
[  ] rabbitmq_consistent_hash_exchange 3.8.16
[  ] rabbitmq_event_exchange           3.8.16
[  ] rabbitmq_federation               3.8.16
[  ] rabbitmq_federation_management    3.8.16
[  ] rabbitmq_jms_topic_exchange       3.8.16
[E*] rabbitmq_management               3.8.16
[e*] rabbitmq_management_agent         3.8.16
[  ] rabbitmq_mqtt                     3.8.16
[  ] rabbitmq_peer_discovery_aws       3.8.16
[  ] rabbitmq_peer_discovery_common    3.8.16
[  ] rabbitmq_peer_discovery_consul    3.8.16
[  ] rabbitmq_peer_discovery_etcd      3.8.16
[  ] rabbitmq_peer_discovery_k8s       3.8.16
[E*] rabbitmq_prometheus               3.8.16
[  ] rabbitmq_random_exchange          3.8.16
[  ] rabbitmq_recent_history_exchange  3.8.16
[  ] rabbitmq_sharding                 3.8.16
[  ] rabbitmq_shovel                   3.8.16
[  ] rabbitmq_shovel_management        3.8.16
[  ] rabbitmq_stomp                    3.8.16
[  ] rabbitmq_top                      3.8.16
[  ] rabbitmq_tracing                  3.8.16
[  ] rabbitmq_trust_store              3.8.16
[e*] rabbitmq_web_dispatch             3.8.16
[  ] rabbitmq_web_mqtt                 3.8.16
[  ] rabbitmq_web_mqtt_examples        3.8.16
[  ] rabbitmq_web_stomp                3.8.16
[  ] rabbitmq_web_stomp_examples       3.8.16

Note that the official examples use a different docker image: pivotalrabbitmq/rabbitmq-prometheus