mogoz

Message Queue

tags
Task Queue , Queues, Scheduling and orchestrator , Message Passing
  • One of more popular Messaging Patterns
  • Async communication on an infrastructural level
  • The Queue itself is just a data structure/container and usually needs a Broker (Queue Manager)
  • This is different from distributed Task Queue , Message Queue compliment Task queues.

Message Broker (Queue Manager)

Sort of middleware that can handle things like routing, validate, store, handle backpressure, deliver stuff etc.

FAQ

NAK/ NACK?

Polling

  • User submits a request via the web application
  • Backend handles that message by adding a message to the message queue.
  • Consumer gets message, makes HTTP call to another web service to actually submit the request.
  • From there, the polling logic takes over
  • Subsequent messages on the queue each represent a polling attempt to retrieve the results.
  • If job has no results, consumer places message back on the queue
    • We can delay the next polling attempt (configurable).
    • Delay logic can use TTL and dead letter definitions etc.

Error Handling

Fail Fast

Bail out

Dead Letter Queue

  • Failed messages will be written to a DLQ
  • Now you can inspect items in the DLQ and after making any required changes you can now re-process the message from the DLQ
  • Almost always needs manual intervention or code to handle stuff written to DLQ