PEPPOL Platform — Interview Revision Sheet¶
The one-line summary¶
A multi-service e-invoicing compliance platform handling UBL generation, transport, and webhook relay across 32 clients, achieving 99.86% same-day delivery at 7K invoices/day via event-driven architecture, automated reconciliation, and a vendor-contractually-justified 24-hour ambiguity window.
The Three Services — and why three, not one¶
Why three services specifically¶
Each service owns a distinct trust and data boundary. That's the real reason — not "separation of concerns" as a generic label.
- Aggregator needs per-merchant DB access and business-level validation rules. No other service should touch merchant data.
- Relay needs a single stable vendor-facing identity ("Sunrise"). The vendor has no concept of merchants — everything comes from one entity. Relay shouldn't know merchant internals.
- Watchdog needs to be decoupled from both so that a stuck invoice in either Aggregator or Relay doesn't block reconciliation of other invoices. It's the external observer that can act when neither of the other two can.
Service 1 — Aggregator¶
- Connected to per-merchant DB instances (each merchant has their own DB)
- Runs business-level validation: jurisdiction checks, company number validity, etc.
- Builds canonical payload in the structure the rest of the pipeline expects
- Writes atomically to the outbox in a single transaction covering: Status (NOTINIT → Processing), Outbox row, ReconKey, PeppolId snapshot on the invoice record
- A separate worker picks up the Outbox row and dispatches to the RabbitMQ exchange
Service 2 — Relay (Relay and Dispatcher)¶
- The only service that faces the vendor
- Receives from its declared queue on the exchange
- Persists its own record including the correlation ID
- Has its own status state machine: persisted → dispatched → awaiting-webhook → terminal
- Dispatches to the vendor network
- Receives webhooks, uses the correlation ID to attribute delivery/failure back to the correct merchant invoice, writes status
Service 3 — Watchdog¶
- Runs every hour
- Only job: resolve invoices stuck in ambiguous state (Processing > 10 minutes)
- Not a yes/no check against Relay — reads Relay's actual status for the invoice
- Resolution logic: see below
Two IDs — do not conflate them¶
| ID | Purpose | Scope |
|---|---|---|
| Correlation ID | Originates at Aggregator, flows through to vendor, round-tripped on webhook, stored by Relay | Cross-system attribution — how a webhook gets matched to a merchant invoice |
| MQ Idempotency Key | Used to detect redelivered RabbitMQ messages | Prevents duplicate processing of the same MQ delivery |
These are two distinct keys solving two distinct problems. If asked about "the idempotency token" — clarify which one.
The Locking Mechanism¶
The atomic write¶
Single DB transaction covering: 1. Status update: NOTINIT → Processing 2. Outbox row insert 3. ReconKey set 4. PeppolId snapshot on invoice table (audit record — merchant's PeppolId can change later, this captures which ID the invoice was actually sent to)
Only one writer wins. The guard condition is WHERE status = 'NOTINIT' — rowcount 1 means winner, rowcount 0 means loser backs off.
Lock-after-read — the deliberate choice¶
Read and payload build happen before the lock is acquired. Two concurrent requests can both read and build the payload in memory, but only one wins the atomic write.
Why not lock-before-read (the alternative tried first): Lock-before-read was implemented originally. It prevented wasted read+compute on the race loser. But it meant every failure mode — including a process crash mid-validation — needed an explicit compensating step to reverse the status, because the lock was held before any real work happened. That added permanent code complexity to prevent a cheap, rare problem.
The actual tradeoff accepted: sub-100ms wasted read-and-compute on the rare genuine concurrent first-touch, in exchange for a simpler process with fewer failure-recovery paths. The NOTINIT guard means this race only happens on genuine first-touch — retries of the same invoice won't re-enter the race, the row is no longer NOTINIT.
The escape hatch if scale demands it: introduce an intermediate status between NOTINIT and Processing and lock earlier. Documented, available, not needed yet.
Engineering principle behind the choice: absorb a small, rare, bounded inefficiency instead of adding permanent code complexity to prevent it.
Failure Modes and Resolution¶
Exception during processing (process alive)¶
Handler explicitly releases the lock — reverts status, invoice becomes eligible for retry. Process is alive, it self-heals via its own exception handler. No external intervention needed.
Process crash (process dead)¶
Nothing releases the lock. Invoice stays in Processing with no heartbeat. Process is dead, nothing local survived to do the cleanup. Only an external observer can resolve it. This is what Watchdog exists for.
The clean line: alive process self-heals, dead process needs an external observer.
Watchdog resolution logic¶
Runs every hour. For any invoice in Processing for more than 10 minutes:
- Reads Relay's actual status for the invoice (not a boolean exists/doesn't exist)
- If Relay never received it: by design, no record = never dispatched (Relay only persists on receipt). Lock released immediately, invoice free to retry.
- If Relay has a record: queries vendor API using the correlation ID for current delivery status. Depending on response: release lock for retry, or leave as-is (still in flight vendor-side).
Why hourly cadence specifically¶
- Webhook worst-case arrival: 6-7 minutes → 10-minute stuck-threshold is the minimum floor
- At 7K invoices/day in an 8-hour workday: ~145 invoices per 10 minutes
- At ~1% infra failure rate: ~1.5 genuinely-stuck invoices per 10-minute window
- Running more frequently would mostly query empty results
- Hourly gives ~8 stuck invoices per run — a sane reconciliation batch
- Tunable to ~11 minutes with no impact — nobody's needed to yet
The 24-Hour Ambiguity Ceiling¶
An invoice can remain in ambiguous reconciliation for up to 24 hours — roughly 24 Watchdog cycles.
Why 24 hours specifically: Vendor contractual guarantee — within 24 hours, any invoice they received will reach a terminal status on their side (delivered to recipient's AP, or failed in their systems). Past 24 hours, waiting longer serves no purpose — the invoice is reopened for resend.
The residual double-send risk: In a genuine edge case, a vendor could declare terminal-failed at hour 23, you resend, and the original somehow still lands late. This hasn't happened. The safeguards (correlation ID checks, vendor status queries before resend) mean it would require a combined failure on your side and the vendor AP side simultaneously. If it does happen, the only remedy is a credit note for the duplicate invoice total. This is an accepted residual risk, not a gap being hidden.
Don't soften this under interview pressure. "Here's the residual risk we accepted and here's the compensating control" is a stronger answer than claiming perfect exactly-once delivery.
Numbers to Know Cold¶
| Metric | Value |
|---|---|
| Daily volume | 7K invoices/day |
| Clients | 32 |
| Same-day delivery rate | 99.86% |
| Stuck-invoice threshold | 10 minutes |
| Watchdog cadence | Hourly |
| Worst-case detection time | ~70 minutes |
| Ambiguity ceiling | 24 hours |
| Max reconciliation cycles before resend | ~24 |
Why RabbitMQ, not Azure Service Bus¶
- Portable — same configuration works on VPS and Azure VMs
- Self-hosted — no managed service dependency, no vendor lock-in on the messaging layer
- 7K/day is steady throughput, not bursty — no need for the elasticity Service Bus provides
- Translatable later if needed — abstraction is in place
Infrastructure — What to Know¶
VNet isolation — what it actually means¶
VNet secures the VM perimeter (NSG rules restrict inbound ports per subnet). Docker Swarm overlay network handles container-to-container routing within that perimeter. These are two different layers — not competing, not the same thing. VNet = VM-level isolation. Overlay = container-level routing.
Service Principal over Managed Identity¶
Containers don't reliably access IMDS (Azure Instance Metadata Service) to inherit VM identity. Service Principal provides explicit credentials that work anywhere — Azure VMs, VPS, local dev. Cloud-agnostic by design.
Known issue: DefaultAzureCredential tries Managed Identity first on startup, IMDS call times out — causes ~19s delay on first request. Fix: exclude unused credential providers via DefaultAzureCredentialOptions.
Cloud-agnostic storage and secrets¶
IDocumentStoreinterface abstracts Blob Storage — swap the implementation without touching domain codeAddAzureKeyVault()layered on ASP.NET Core config pipeline — replaces.envfiles, no credentials on disk
CI/CD flow¶
Push to master → GitHub Actions → login to GHCR with PAT → build image, push :latest and :<sha> → SSH into Swarm manager via appleboy/ssh-action → docker service update --with-registry-auth --image ...:latest → Swarm rolling update. Stack file changes still require manual docker stack deploy.
Follow-up Questions to Be Ready For¶
"What happens if an invoice is sent twice?" Accepted residual risk. Correlation ID checks and vendor status queries before resend make this require a combined failure on both sides simultaneously. If it happens, issue a credit note. This hasn't occurred.
"What does the reconciliation job actually do?" Runs hourly. Finds invoices in Processing > 10 minutes. Reads Relay's status. If Relay has no record, releases lock. If Relay has a record, queries vendor API for current status, acts accordingly.
"Why hourly for Watchdog?" Volume math: 1.5 genuinely stuck invoices per 10-minute window at 1% failure rate. Hourly gives a sane 8-invoice batch to reconcile. Tunable, nobody's needed to.
"How do you prevent duplicate invoice delivery?"
Two mechanisms: MQ idempotency key on Relay's consumer (detect redelivered messages), correlation ID round-tripped through vendor (attribute webhooks to the correct invoice). The NOTINIT guard prevents double-processing at the Aggregator stage. At-most-once delivery past the 24-hour ceiling is handled by resend-as-new with vendor status pre-check.
"What does the outbox give you?" Atomic write: outbox row written in the same DB transaction as the domain state change. You never have a committed invoice status update with no corresponding outbox entry, or vice versa. Eliminates the dual-write problem between DB and message broker.
"Why lock-after-read instead of lock-before-read?"
Lock-before-read was tried first. It prevented wasted compute but required a compensating status-reversal step on every failure mode including crashes, adding permanent complexity to guard against a rare cheap problem. Lock-after-read accepts sub-100ms wasted compute on genuine concurrent first-touch in exchange for a simpler process. The NOTINIT guard bounds the race to first-touch only.