High Volume Webhooks

Here we describe what happens when when your webhook generates a high volume of events.

  2 minute read  

In this guide we cover:

  • What happens when your webhook generates a high volume of events

Why do we treat high volume webhooks differently?

For a non-high-volume webhook we store every event along with its payload and all the responses that we have received from the webhook consumer for three days. That doesn’t sound like a lot, but when you have lots of products and they’re frequently updated, and you have a webhook that generates events for every product update, you can get to millions of events very quickly.

This leads to us having so many webhook events being stored that we can’t efficiently query them.

When is a webhook marked as high volume?

If at the time we generate a new payload for an event, there are more than a certain number of events still awaiting processing for that webhook, then we will mark that webhook as high volume.

Currently that threshold is 50,000 yet to be delivered, but that may change in the future.

What changes for high volume webhooks?

When a payload is successfully delivered, we will immediately remove all record of the event. On a non-high-volume webhook, we would keep the record of that event and all the responses we’d received from the webhook consumer for three days.

If the webhook has extra deduplication from the “Allow Skip” option, then when a payload is generated, we will not save the skipped event.

How does this interact with “Allow Skip” deduplication?

Because successful deliveries result in the event being immediately removed, deduplication with successful deliveries will not be possible. Deduplication with as yet undelivered events will still occur.

Doesn’t this make debugging things hard?

It does when you want to see what was successfully sent or skipped. We expect that most of your debugging of that kind of issue would be completed before your webhook event volume gets high enough to flag your webhook as high volume. Note that failed webhook events will be kept for three days, and retriable webhook events will be kept until they are either successful or failed.