Question

Choose the clock before assigning an event to a period

Arrived today and happened today are different statements.

About 1 min read

Before you use it

Use when
  • Late messages or delayed processing move events into the wrong reporting day.

Keep this limit in mind

  • Source clocks and timestamps can be wrong. A streaming watermark estimates progress; it does not make late arrival impossible.

An idea to explore.

Try this idea.

Inspect the example, try the idea in your situation, then check your next move.

Keep reading

A question to ask

Does this report concern when work happened or when the system received it? · Which time zone and cutoff define the period? · How will late events correct or annotate an already issued result?

Why it matters

Keep occurrence time and processing time separate. Decide which clock answers the business question, and define what happens when an event arrives after a reporting cutoff. Ordering records by arrival cannot reconstruct the event sequence by itself.

An example

A service event occurs before midnight but reaches reporting after midnight. Its business date follows the agreed occurrence-time rule, not an accidental queue delay.

Check your result

A deliberately late test event lands in the expected period or follows the documented correction route.

Save and practise this idea

My learning

Read. Recall. Repeat.

Save this idea and return to it on any device.

Use email
Keep exploring

Connected ideas

You are readingChoose the clock before assigning an event to a period

Evidence and sources
Supports

Event time records when an event occurred, while processing time records when a system processes it; arrival order need not follow event order.

Choosing the reporting clock and late-data correction policy remains a domain decision.

Apache Beam Programming Guide · Section 8.4

All sources (1)
Edition and review

Version 0.1.0

No review details are supplied for this edition.

Read this idea as JSON · Release details