TopicsDelivery
Delivery
Schedule and retries
Events wait in a table of the tenant database. The job “PXM Log Exporter: ship events” delivers them every five minutes, in order, and sends only events that are at least a minute old; as the platform’s scheduler starts the job up to a minute late, an event reaches the receiver one to seven minutes after it happened. A failing sink is retried after 5, 10, 20, 40 and then every 60 minutes, and the other sink keeps going. The job also runs during the automation downtime the platform sets for maintenance, so the heartbeat keeps coming; while the tenant itself is down, nothing runs, and the next run catches up.
Dropped events
Events are dropped, and counted in exporter.dropped, when they are older than the retention (retention),
when the queue is full (cap, the oldest go first), or when one page request or REST call produced more
events than Maximum events per PXM request (request-cap). The per-request limit keeps a request’s first
events and drops the later ones, except events about the exporter itself, the request’s login and the events
at its end: a refused call, rejected input or an application error. Active Script jobs write their events while
they run and have no such limit. The limits are on the Limits page, each with its default and its range.
When a batch counts as delivered
A batch counts as delivered once the receiver has taken it. Until then the exporter sends the batch again, so
an event can arrive twice. Deduplicate on its id. A receiver has taken a batch when:
- HTTPS: it answered the body that holds the events with a 2xx status.
- TLS stream: it read the batch to the end and closed the connection. The protocol has no acknowledgement, so every batch has a connection of its own: the exporter ends its side after the batch and waits up to 30 seconds for the receiver to close. A receiver that goes away before it has read the whole batch resets the connection, and the exporter sends the whole batch again. If a receiver keeps ending connections before a batch is through, lower Events per delivery on the Limits page (default 500). The batch also fails if the receiver sends data back, as a web server does. The outgoing proxy ends the tunnel as soon as the exporter ends its side, so through the proxy the exporter keeps the connection open and counts the batch once the receiver ends its TLS session with close_notify. Without a close_notify the batch is sent again, as it is when rsyslog is stopped or killed. If a receiver sends close_notify while it exits with part of the batch unread, the batch still counts as delivered.
The exporter cannot see what a stream receiver does with a batch after reading it. A collector that holds events in memory loses them when it restarts. Use the HTTPS sink where every event must arrive.
Refused events
A receiver that refuses an event (HTTP 400, 413 or 422, or 403 or 406 as a filter in front of it may answer) in
three runs in a row gets that event skipped when it took an earlier event of the same run or takes the next event
on its own: exporter.dropped with the reason rejected names the sink, the event’s seq and the receiver’s
answer. Until it is skipped, each refused event holds its sink back for three runs, so a refused event that recurs,
such as exporter.heartbeat, puts the sink further behind each time. Refused events in a row at the head of the
queue are not skipped: the sink keeps failing on them until the receiver takes them, or until the retention or the
queue’s cap drops them. A filter in front of the receiver that refuses events by their content therefore lets
anyone who can put text into an event, such as a user agent or a user name typed at the login, have single events
skipped, and a run of them stall the sink. Fix such refusals at the receiver. If a receiver refuses every event,
its configuration is wrong, and its sink keeps failing until that is fixed.
Sequence numbers
Every queued event carries seq, its place in the tenant’s queue. It grows with gaps, and an event sent again
keeps it. Per sink, each heartbeat reports sent and dropped since installation and last_seq, the seq
up to which every event was delivered or skipped. Between two heartbeats, the events that arrived with a seq
above the earlier last_seq and up to the later one are as many as sent grew; fewer means events were lost
on the way (see Detection rules). When Update Data Model reinstalls the queue’s table
while it is empty, seq starts again at 1. A heartbeat whose last_seq is lower than the previous one marks
the restart, and the lost-events query for Sentinel under Detection rules skips the interval between those
two heartbeats. Until the events from before the restart are older than the query’s look-back, old and new
events can share a seq, and the query can miss a loss.
Saves the platform rolls back
Events come from the platform’s own hooks as a change is saved. A save the platform rolls back after a database error can therefore still produce its event.
Centric PXM™ is a trademark of Centric Software, Inc.