Pub/Sub

Publisher#

// ...
type Publisher interface {
	// Publish publishes provided messages to the given topic.
	//
	// Publish can be synchronous or asynchronous - it depends on the implementation.
	//
	// Most publisher implementations don't support atomic publishing of messages.
	// This means that if publishing one of the messages fails, the next messages will not be published.
	//
	// Publish does not work with a single Context.
	// Use the Context() method of each message instead.
	//
	// Publish must be thread safe.
	Publish(topic string, messages ...*Message) error
	// Close should flush unsent messages if publisher is async.
	Close() error
}
// ...

Full source: github.com/ThreeDotsLabs/watermill/message/pubsub.go

Publishing multiple messages#

Most publishers implementations don’t support atomic publishing of messages. This means that if publishing one of the messages fails, the next messages won’t be published.

Async publish#

Publish can be synchronous or asynchronous - it depends on the implementation.

Close()#

Close should flush unsent messages if the publisher is asynchronous. It is important to not forget to close the subscriber. Otherwise you may lose some of the messages.

Subscriber#

// ...
type Subscriber interface {
	// Subscribe returns an output channel with messages from the provided topic.
	// The channel is closed after Close() is called on the subscriber.
	//
	// To receive the next message, `Ack()` must be called on the received message.
	// If message processing fails and the message should be redelivered `Nack()` should be called instead.
	//
	// When the provided ctx is canceled, the subscriber closes the subscription and the output channel.
	// The provided ctx is passed to all produced messages.
	// When Nack or Ack is called on the message, the context of the message is canceled.
	Subscribe(ctx context.Context, topic string) (<-chan *Message, error)
	// Close closes all subscriptions with their output channels and flushes offsets etc. when needed.
	Close() error
}
// ...

Full source: github.com/ThreeDotsLabs/watermill/message/pubsub.go

Ack/Nack mechanism#

It is the Subscriber’s responsibility to handle an Ack and a Nack from a message. A proper implementation should wait for an Ack or a Nack before consuming the next message.

Important Subscriber’s implementation notice: Ack/offset to message’s storage/broker must be sent after Ack from Watermill’s message. Otherwise there is a chance to lose messages if the process dies before the messages have been processed.

Close()#

Close closes all subscriptions with their output channels and flushes offsets, etc. when needed.

At-least-once delivery#

Watermill is built with at-least-once delivery semantics. That means when some error occurs when processing a message and an Ack cannot be sent, the message will be redelivered.

You need to keep it in mind and build your application to be idempotent or implement a deduplication mechanism.

Unfortunately, it’s not possible to create a universal middleware for deduplication, so we encourage you to build your own.

Universal tests#

Every Pub/Sub is similar in most aspects. To avoid implementing separate tests for every Pub/Sub, we’ve created a test suite which should be passed by any Pub/Sub implementation.

These tests can be found in pubsub/tests/test_pubsub.go.

Built-in implementations#

To check available Pub/Sub implementations, see Supported Pub/Subs .

Implementing custom Pub/Sub#

See Implementing custom Pub/Sub for instructions on how to introduce support for a new Pub/Sub.

We will also be thankful for submitting pull requests with the new Pub/Sub implementations.

You can also request a new Pub/Sub implementation by submitting a new issue .