Notifications

Hyperwallet servers communicate entity and status changes to the connector using webhooks. A webhook is a notification that is received by the connector in the '/webhook/notifications' endpoint.

Storing and processing

Hyperwallet does not guarantee that notifications are delivered in the same order as the events that generated them. Out-of-order delivery could cause issues such as reverting a KYC status that was already verified.

To avoid such issues, and following Hyperwallet recommendations, the connector stores every incoming notification in a processing queue and filters out duplicates, obsolete entries, and superseded ones before processing them.

Processing pipeline

When the connector receives a notification it immediately enqueues it in the NotificationEntity table with a PENDING status, unless it falls into one of the discard cases below. A periodic Quartz job (Notification Processing Job) then picks up PENDING and eligible RETRYING entries in batches and processes them asynchronously.

This design decouples webhook reception latency from downstream Mirakl and Hyperwallet call latency: the HTTP response to Hyperwallet is always fast, and any slow or failing downstream call is handled transparently by the retry mechanism.

Discard rules on enqueue

The connector discards an incoming notification at enqueue time in three cases:

  • Duplicate: a notification with the same webhook token is already stored in the database.

  • Obsolete: a notification for the same object (same objectToken and notificationType) already exists in the database with a later creation date — the incoming notification is older and therefore irrelevant.

  • Superseded queue entry: when a newer notification arrives for the same object, any existing PENDING or RETRYING entry for that object is marked OUTDATED before the new notification is saved. Entries in terminal states (SUCCESS, FAILED, OUTDATED) are never modified.

Diagram Description automatically generated

Notification lifecycle — status values

Every NotificationEntity row carries a status field that reflects the current stage of processing:

Status Meaning

PENDING

Received and enqueued; waiting to be picked up by the processing job.

RETRYING

A processing attempt failed. The connector has computed a nextRetryDate using the exponential back-off formula and will retry when that date is reached.

SUCCESS

Processed successfully; no further action needed.

FAILED

All retry attempts have been exhausted. An email alert is sent to the operator. The entry is never retried again.

OUTDATED

Superseded by a newer notification for the same object while still PENDING or RETRYING. The entry is permanently skipped.

NotificationEntity table

The connector stores the following information in the NotificationEntity table:

Database field Data type Notes

id

Long

Autogenerated ID.

webhookToken

String

Token of the notification.

objectToken

String

Token of the related item (seller, payment, etc.).

creationDate

Date

Creation date of the notification as reported by Hyperwallet.

receptionDate

Date

Date on which the connector received the notification.

notificationType

String

Type derived from the prefix of objectToken (the segment before the first -, e.g. usr in usr-abc123). Possible values: USR, PMT, STK, TRM, UNK.

status

String

Current lifecycle status (see table above). Defaults to PENDING.

retryCounter

Integer

Number of processing attempts made so far. Defaults to 0.

lastRetryDate

Date

Timestamp of the most recent failed attempt. null for PENDING entries.

nextRetryDate

Date

Earliest timestamp at which the connector will attempt reprocessing. null for PENDING and terminal-status entries.

Notification types

The connector derives the notification type from the prefix of the object token — the segment before the first - character (e.g. usr in usr-abc123):

  • USR — Sellers

  • STK — Stakeholders

  • PMT — Payments (invoices)

  • TRM — Bank accounts

  • UNK — Unknown (object token prefix could not be parsed; stored to avoid processing failure)

Querying and housekeeping

Two endpoints are available to query or remove notifications stored in the database.

Endpoint Method Description

/webhooks/notifications

GET

Query notifications stored in the database within the specified date range.

/webhooks/notifications

DELETE

Remove notifications stored in the database within the specified date range.

Both endpoints accept from and to query parameters (ISO-8601 date-time format).

Example requests:

curl --location --request GET \
  'http://localhost:8080/webhooks/notifications?from=2021-04-27T10:30:00.000-00:00&to=2023-04-27T10:30:00.000-00:00'

curl --location --request DELETE \
  'http://localhost:8080/webhooks/notifications?from=2021-04-27T10:30:00.000-00:00&to=2023-04-27T10:30:00.000-00:00'

Notification Cleanup Job

The connector includes a dedicated Quartz job (NotificationCleanupJob) that automatically deletes terminal-state notifications older than the configured retention period. The job:

  • Runs on the schedule defined by PAYPAL_HYPERWALLET_NOTIFICATIONS_CLEANUP_CRON_EXPRESSION (default: 0 0 1 * * ? — daily at 01:00 UTC).

  • Deletes all NotificationEntity rows whose receptionDate is older than PAYPAL_HYPERWALLET_NOTIFICATIONS_CLEANUP_RETENTION_DAYS (default: 90 days) and whose status is FAILED, SUCCESS, or OUTDATED.

  • Never touches rows in PENDING or RETRYING state — only rows that can no longer transition are eligible for deletion.

  • Uses @DisallowConcurrentExecution — only one instance of the job runs at a time.

The job can also be triggered on demand via the REST API (see REST API).

Notification Processing Job

The connector includes a dedicated Quartz job (NotificationProcessJob) that processes enqueued notifications in batches. The job:

  • Runs on the schedule defined by PAYPAL_HYPERWALLET_NOTIFICATIONS_PROCESSING_CRON_EXPRESSION (default: every 30 seconds).

  • Fetches up to PAYPAL_HYPERWALLET_NOTIFICATIONS_BATCH_SIZE (default: 100) entries whose status is PENDING or RETRYING and whose nextRetryDate is in the past (or null).

  • Processes entries in ascending creationDate order to preserve relative ordering.

  • Uses @DisallowConcurrentExecution — only one instance of the job runs at a time.

The job can also be triggered on demand via the REST API (see REST API).

Retry mechanism

The connector retries the processing of notifications that could not be delivered to Mirakl or Hyperwallet due to transient errors (for example, connection issues or temporary service unavailability).

How retries work

All notifications are stored in the NotificationEntity table from the moment they arrive (see Storing and processing). When processing a notification fails, the connector:

  1. Increments the retryCounter on the entity.

  2. Sets status to RETRYING.

  3. Computes a nextRetryDate using an exponential back-off formula:

    nextRetryDate = now + initialRetryDelay × backoffMultiplier^(retryCounter - 1)

With the default values (initialRetryDelay = PT1M, backoffMultiplier = 2.0) the retry schedule for a notification that fails on every attempt is:

Attempt Delay before next retry

1st failure

1 minute

2nd failure

2 minutes

3rd failure

4 minutes

4th failure

8 minutes

5th failure (final)

— (marked FAILED)

The Notification Processing Job only picks up RETRYING entries whose nextRetryDate is in the past, ensuring the back-off window is respected.

When all retry attempts for a notification are exhausted, the entry is set to FAILED and the connector sends an email alert to the operator. Failed entries are never retried again automatically, but they can be managed through the Failed Notifications Management API.

Incoming notifications while retrying

When a new notification is received and a PENDING or RETRYING entry for the same object already exists, the connector checks the incoming notification’s creation date:

  • Identical: same webhook token — the incoming notification is a duplicate and is discarded.

  • Newer: later creation date — the incoming notification is enqueued and the existing PENDING/RETRYING entry is marked OUTDATED (it will be skipped by the job).

  • Older: earlier creation date — the incoming notification is obsolete and is discarded without touching the existing entry.

Failed Notifications Management API

The connector exposes a management REST API that lets operators inspect and manage notifications that are in FAILED or RETRYING state. This API is gated behind the hmc.toggle-features.management-api feature toggle.

HTTP Method Path Description

GET

/management/failed-notifications/

Returns a paged list of FAILED and RETRYING notifications. Supports optional type (notification type, e.g. USR) and target (object token) query parameters for filtering.

GET

/management/failed-notifications/{notificationToken}

Returns a single notification by its webhook token. Returns 404 if not found.

POST

/management/failed-notifications/

Adds a new entry directly with FAILED status (useful for manual re-injection).

PUT

/management/failed-notifications/{notificationToken}

Updates mutable fields (e.g. retryCounter, program) of an existing entry. Returns 404 if not found.

PUT

/management/failed-notifications/

Replaces the entire list of FAILED and RETRYING entries — deletes all existing entries and persists the provided list.

DELETE

/management/failed-notifications/{notificationToken}

Deletes a single entry by webhook token. Returns 404 if not found.
Both FAILED and RETRYING notifications are returned by GET /management/failed-notifications/. Compare the retryCounter field against the configured PAYPAL_HYPERWALLET_MAX_AMOUNT_OF_NOTIFICATION_RETRIES value to determine whether a notification has exhausted all retries or is still being reattempted.

Email alerts

The connector sends an email to the operator when all retry attempts for a notification are exhausted. The email contains the following information:

Subject: [HMC] Technical error occurred when processing the notification <NOTIFICATION_TOKEN>

Body: There was an error processing the notification <NOTIFICATION_TOKEN> and the operation
could not be completed. The maximum number of attempts has been reached, therefore it will
not try to re-process the notification anymore. Please check the logs for further information.

Configuration

The retry behaviour is controlled by the following environment variables (see also Notification configuration variables):

  • PAYPAL_HYPERWALLET_MAX_AMOUNT_OF_NOTIFICATION_RETRIES — maximum number of processing attempts per notification. Default: 5.

  • PAYPAL_HYPERWALLET_NOTIFICATION_INITIAL_RETRY_DELAY — ISO-8601 duration for the delay before the first retry. Default: PT1M (1 minute).

  • PAYPAL_HYPERWALLET_NOTIFICATION_RETRY_BACKOFF_MULTIPLIER — multiplier applied to the delay on each subsequent failure. Default: 2.0.

  • PAYPAL_HYPERWALLET_NOTIFICATIONS_PROCESSING_CRON_EXPRESSION — cron expression that controls how often the processing job runs. Default: 0/30 * * * * ? (every 30 seconds).

  • PAYPAL_HYPERWALLET_NOTIFICATIONS_BATCH_SIZE — maximum number of notifications processed per job execution. Default: 100.

Cleanup job configuration

The following variables control the Notification Cleanup Job:

  • PAYPAL_HYPERWALLET_NOTIFICATIONS_CLEANUP_CRON_EXPRESSION — cron expression that controls when the cleanup job runs. Default: 0 0 1 * * ? (daily at 01:00 UTC).

  • PAYPAL_HYPERWALLET_NOTIFICATIONS_CLEANUP_RETENTION_DAYS — number of days after which terminal-state (FAILED, SUCCESS, OUTDATED) notifications are eligible for deletion. Default: 90.