Razorpay Payment Production Hardening¶

Purpose¶

Extends Razorpay Production-Readiness Audit - read that first. That audit identified P1 ("No server-to-server webhook endpoint exists" - order creation depended entirely on the customer's browser successfully calling /verify) as the single highest-value gap for real production payment safety, and left refund execution, reconciliation, and the payment/refund financial state machine unreviewed. This tranche implements the webhook, event idempotency, browser/webhook convergence, out-of-order-event handling, real Razorpay refund execution, refund reconciliation, and hardens the financial state machine against concurrent settlement/refund races - as an isolated payment PR. No real Razorpay credentials were used, no live payment/refund was executed, nothing was deployed.

Payment state machine¶

(none) --CreateRazorpayOrderCommandHandler--> InventoryReservation.Pending
Pending --TryClaimForCompletionAsync (browser /verify, signature-verified)--> Completed
                                                                    + PaymentTransaction.Verified
Pending --payment.failed webhook / expiry sweep--> Released

PaymentTransaction itself (ECommerce.Domain.Entities.PaymentTransaction) has no Pending/ Failed construction path in this codebase (pre-existing, unchanged design - see its own doc comment): a PaymentTransaction row only ever comes into existence already Verified, created exactly once, inside the same database transaction as Order/SellerOrder, after the Razorpay signature has already been checked. An invalid signature or a failed payment never creates any PaymentTransaction row at all - the "payment failure" state is represented entirely by the InventoryReservation going Released, not by a PaymentTransaction row in a Failed status. PaymentTransaction.Status transitions from there via ApplyRefund only: Verified -> PartiallyRefunded -> Refunded (or Verified -> Refunded directly for a full refund) - see "Refund state machine" below. There is no method that moves RefundedAmount backwards or Status from Refunded/PartiallyRefunded back to Verified - "Refunded must never become Paid again" is structurally enforced by the absence of any such method, not by a runtime check.

Browser `/verify` vs webhook race¶

The core primitive: IInventoryReservationRepository.TryClaimForCompletionAsync(razorpayOrderId) (Infrastructure/Repositories/Repositories.cs) - a single atomic conditional UPDATE:

UPDATE InventoryReservations
SET Status='Completed', UpdatedAt=@now
WHERE RazorpayOrderId=@orderId AND Status='Pending'
-- affected==1 -> this caller won the completion. affected==0 -> already completed/released.

VerifyAndCreateOrderCommandHandler.CompleteVerifiedPaymentAsync calls this before constructing any Order/SellerOrder/PaymentTransaction row, inside the ambient transaction (deliberately no transaction of its own - it must roll back together with everything downstream). If the claim is lost, the handler looks up the PaymentTransaction by ProviderOrderId (GetByProviderOrderIdAsync) and returns the winning caller's OrderNumber rather than throwing - so a double-submitted /verify (client retry after a lost response, or a genuine race with a concurrent call) converges on the same order number instead of erroring or double-creating.

This is the same primitive both a duplicate browser /verify call and the webhook's own payment.captured handling would need to share if a future change lets the webhook itself construct an order (it currently does not - see "Known limitations"). Today the webhook's payment.captured handler only ever reads (GetByProviderPaymentIdAsync), never claims, so the three required orderings resolve as:

Ordering	Outcome
webhook before browser verify	No `PaymentTransaction` yet -> webhook logs `AwaitingBrowserCompletion_NoShippingAddressAvailable` (reconciliation candidate, not an error) and does nothing else. Browser `/verify` still runs normally afterward and completes the reservation exactly once.
browser verify before webhook	`PaymentTransaction` already exists -> webhook returns `AlreadyConfirmedByBrowser`, a no-op.
both "simultaneously"	Reduces to one of the two orderings above, whichever the two concurrent database operations actually interleave to - `TryClaimForCompletionAsync`'s atomicity is what makes exactly one of them win when it matters (a genuine double `/verify`); the webhook path never contends for the claim itself in the current design.

Test coverage: RazorpayWebhookAndRefundHardeningTests.WebhookBeforeBrowserVerify_*, BrowserVerifyBeforeWebhook_*, VerifyAndWebhookEitherOrder_*, ProviderCaptureWithLocalTimeout_* (browser /verify succeeded server-side but the client never saw the response - identical to browser-before-webhook from the server's perspective).

Webhook architecture¶

Endpoint: POST /api/webhooks/razorpay (RazorpayWebhookController), deliberately outside /api/payment (the customer-facing, JWT-authenticated controller) and [AllowAnonymous] - this endpoint is authenticated by Razorpay's own HMAC signature, never a Vayzo session/JWT.

Raw body: the action reads Request.Body directly via StreamReader with no [FromBody] parameter, so signature verification runs against the exact bytes Razorpay signed, before any JSON parsing/model binding could reformat them.

Signature verification (RazorpayWebhookSignature.Verify, PaymentFeatures-adjacent file RazorpayWebhookFeatures.cs): HMAC-SHA256(rawBody, Razorpay:WebhookSecret), hex-encoded, lower-cased, compared against the X-Razorpay-Signature header via CryptographicOperations.FixedTimeEquals - the same constant-time-comparison fix the prior audit applied to the checkout /verify signature check (P2 in the prior audit), reused here.

Fail-closed on the enable flag itself: Razorpay:WebhookEnabled is checked FIRST, before the request body is read, before signature verification, before any MediatR dispatch. It must be explicitly true - a real, correctly-configured Razorpay:WebhookSecret is deliberately NOT sufficient on its own to accept webhooks. If the flag is missing or not true, the controller returns 503 immediately and never reaches signature verification or dispatch.

Fail-closed on missing/placeholder config: if Razorpay:WebhookSecret is absent or looks like a placeholder (YOUR_...), the controller returns 503 and never attempts verification. Program.ValidateRuntimeConfiguration additionally refuses to start outside Development if Razorpay:WebhookEnabled=true and the secret is missing/placeholder - the webhook can never be "on" with an unusable secret in a real environment.

No secrets/signatures/raw payload logged: the controller logs only "signature verification failed" on a 401, never the signature or body; the handler logs event type/entity id, never the raw payload.

Always 200 once signature verification passes: the handler durably records the event (see "Event idempotency" below) before any business-logic branch runs, so a processing failure loses no event; returning 200 regardless of the internal processing outcome avoids Razorpay's own retry storm stacking on top of Vayzo's own idempotency ledger. A processing failure is visible via the event row's Failed status for operator/reconciliation follow-up, not via the HTTP response code.

Recognized event types: payment.captured, payment.failed, refund.processed, refund.failed. Anything else is recorded (for the idempotency ledger and audit trail) but marked Ignored and not further processed.

Event idempotency¶

RazorpayWebhookEvent (new entity/table, migration AddRazorpayWebhookAndRefundHardening): PayloadHash (SHA-256 hex of the exact raw body - unique index, the real idempotency key), EventType, EntityId, Status (Received/Processed/Failed/Ignored), ReceivedAt, ProcessedAt, ProcessingNote.

PayloadHash rather than any Razorpay-supplied delivery-id header was chosen deliberately - a delivery-id header is not guaranteed to exist across every Razorpay webhook integration path, while "the exact same bytes were already recorded" is always true for a genuine replay and always false for a distinct, later event (even one for the same payment/refund id - e.g. payment.captured followed later by a distinct refund.processed for the same underlying payment has two different raw bodies and therefore two different hashes, which is correct: they are different events that must both be recorded).

RazorpayWebhookEventRepository.TryRecordAsync is the idempotent-insert pattern already established by MarketplaceNotificationRepository.TryAddAsync/EmailDeliveryRepository.TryEnqueueAsync: insert, catch DbUpdateException for a MySQL "Duplicate entry" / SQLite "UNIQUE constraint failed" message, detach only the failed entity, return false. A concurrent duplicate delivery therefore always results in exactly one Processed/Ignored/Failed row and zero exceptions surfaced to the caller.

First valid webhook processes normally.
Exact replay (identical raw body) -> TryRecordAsync returns false -> handler logs and returns, a harmless no-op.
Concurrent duplicates (two simultaneous deliveries of the identical body) -> exactly one INSERT wins the unique-index race; the loser's TryRecordAsync catches the duplicate-key exception and no-ops. Proven against real MySQL/InnoDB in MySqlRazorpayHardeningConcurrencyTests.MySql_TwoConcurrentIdenticalWebhookDeliveries_ExactlyOneEventRowIsRecorded.
Failed processing is safely reconcilable without losing the event: the event row is inserted and committed before the business-logic switch runs; a processing exception is caught, marks the row Failed with the exception type name as ProcessingNote, and still returns 200 - the event itself was never dropped, only its processing outcome, which is visible for operator follow-up.

Out-of-order events¶

payment.failed after payment.captured/verify already succeeded: HandlePaymentFailedAsync checks for an existing PaymentTransaction first; if found, returns OutOfOrder_PaymentAlreadyVerified and does nothing - the reservation is never released out from under an already-completed order.
payment.failed with no prior verify: releases the InventoryReservation via the existing TryReleaseByRazorpayOrderIdAsync (customerId null - the webhook has no authenticated customer context), faster than waiting for the reservation's own expiry sweep.
Refund terminal events (refund.processed/refund.failed) arriving out of order or as a stale duplicate for an already-Completed/Failed RazorpayRefundAttempt: TryMarkTerminalFromWebhookAsync is a conditional UPDATE ... WHERE Status NOT IN ('Completed','Failed') - an older or duplicate resolution can never overwrite an already-terminal one. The handler surfaces this as AlreadyTerminal_IdempotentNoOp.
Accepted transitions: Pending/Retrying/Processing/UnknownOutcome -> Completed, Pending/Retrying/Processing/UnknownOutcome -> Failed (both only via the webhook or a winning dispatch-worker finalize). Never accepted: Completed -> anything, Failed -> anything (a failed refund attempt is not retried automatically by a stale webhook; a brand-new RazorpayRefundAttempt would be required, which this tranche's dispatcher does not create automatically - see "Known limitations").

Refund execution¶

RazorpayRefundAttempt (new entity/table) is the out-of-transaction provider-dispatch queue row - the refund counterpart to EmailDeliveryAttempt, reusing its claim/lease design exactly (see Email Delivery Hardening):

Pending/Retrying --TryClaimAsync--> Processing (ClaimToken/ClaimedBy/ClaimedAt/LeaseExpiresAt)
Processing --TryFinalizeCompletedAsync--> Completed (ProviderRefundId recorded)
Processing --TryFinalizeFailureAsync (transient, attempts<max)--> Retrying
Processing --TryFinalizeFailureAsync (permanent, or attempts>=max)--> Failed
Processing --lease expired, swept--> UnknownOutcome (terminal, never auto-retried)

Created alongside, not instead of, the existing internal accounting: InspectReturnCommandHandler still creates RefundTransaction and calls PaymentTransaction.ApplyRefund exactly as before (unchanged) - RazorpayRefundAttempt is purely additive, enqueued in the same atomic inspection/refund transaction right after. RefundTransaction/PaymentTransaction.RefundedAmount remain the authoritative internal accounting; RazorpayRefundAttempt tracks whether the money has actually been asked to move via Razorpay, separately.

Amount/currency derived server-side: AmountPaise is computed from RefundTransaction.TotalRefundAmount (itself computed server-side from order-time snapshots, never client input) at enqueue time, converted via checked((long)decimal.Round(amount * 100m, 0, MidpointRounding.AwayFromZero)).

Full vs partial refund: both go through the identical path - RefundTransaction.TotalRefundAmount is whatever the inspection computed (full order value or a partial line/quantity amount); RazorpayRefundAttempt.AmountPaise mirrors it exactly. See RazorpayWebhookAndRefundHardeningTests.FullRefund_* / PartialRefund_*.

Idempotency key: "vzr-" + refundTransactionId:N (36 chars) - deterministic, derived from RefundTransactionId, never regenerated per dispatch attempt, sent to Razorpay's Create Refund API as the idempotency key so a retried dispatch of the same attempt is deduplicated by Razorpay itself, not only by this database. Verify Razorpay's current idempotency-key length/character limit before enabling real refund execution - see "External prerequisites" below.

Transport header (P1-2 remediation): HttpRazorpayTransport.PostAsync sends this key as the X-Refund-Idempotency HTTP header - the documented Razorpay refund-idempotency mechanism. An earlier revision of this tranche sent X-Payment-Idempotency instead (Razorpay's order creation idempotency header, unrelated to refunds) - Razorpay's refund API never recognized that header as an idempotency key at all, so provider-side dedup for a retried/duplicated refund call was silently absent; only this database's own RefundTransactionId/IdempotencyKey unique indexes were actually protecting against a duplicate provider refund. Both layers are required together (see "Duplicate/ repeated refund request cannot create duplicate provider refunds" above) - the DB-level uniqueness alone cannot prevent two DIFFERENT RazorpayRefundAttempt rows (e.g. a genuine retry that re-enqueues) from each independently reaching the provider; only a stable, correctly-named provider idempotency header does that.

Duplicate/repeated refund request cannot create duplicate provider refunds at two layers:

RazorpayRefundAttempt.RefundTransactionId and IdempotencyKey are both unique-indexed - TryEnqueueAsync rejects a second attempt for the same RefundTransaction outright (proven sequentially in SQLite and under a genuine concurrent race against real MySQL/InnoDB in MySqlRazorpayHardeningConcurrencyTests.MySql_TwoSimultaneousRefundRequests_*).
Even if a retry of the same already-enqueued attempt reaches the provider twice (e.g. a dispatch worker's own retry after a transient failure), the deterministic IdempotencyKey means Razorpay's own API is expected to return the original refund rather than creating a second one.

Provider timeout/ambiguous outcome never blindly retried: if a dispatch worker's lease expires before it finalizes (crash, hang, dropped connection mid-call), the row is swept to UnknownOutcome by the same background-sweep mechanism EmailDeliveryBackgroundService uses, and is never automatically reclaimed or retried - retrying an attempt whose outcome is unknown risks a genuine duplicate provider-side refund that this database cannot prevent once it has lost track of the in-flight call. UnknownOutcome requires manual operator investigation (check Razorpay's dashboard/ API for the actual outcome, then requeue or resolve by hand).

Prevents total refunded exceeding captured payment: enforced upstream, unchanged, by PaymentTransaction.ApplyRefund (RefundExceedsPaymentException if cumulative refunds would exceed the original captured Amount) - a RazorpayRefundAttempt is only ever enqueued after ApplyRefund already succeeded in the same transaction, so an over-refund attempt is rejected before any RazorpayRefundAttempt row is even created (proven in RefundExceedingRemainingRefundableAmount_ThrowsAndEnqueuesNoAttempt_TransactionRolledBack).

Dispatch: RazorpayRefundDispatchBackgroundService (structurally identical to EmailDeliveryBackgroundService) sweeps stale leases, claims candidates, calls IRazorpayClient.CreateRefundAsync, finalizes. RazorpayClient/IRazorpayTransport mirror SesEmailSender/ISesTransport exactly - "checked enabled+configured at call time, returns a Disabled()-style result rather than throwing" - gated by Razorpay:RefundExecutionEnabled (default false).

Refund webhook / reconciliation¶

refund.processed/refund.failed webhooks resolve a RazorpayRefundAttempt via TryMarkTerminalFromWebhookAsync - this is the authoritative resolution path for a Pending/Processing/Submitted/UnknownOutcome attempt regardless of what the dispatch-time HTTP response itself looked like: "local refund completion is based on authoritative provider state." TryMarkTerminalFromWebhookAsync's own conditional UPDATE (WHERE Status NOT IN (Completed, Failed)) is what actually enforces this - it can resolve the row from ANY non-terminal status, not only Processing, which is exactly what makes the correlation fix below able to close the early-webhook race regardless of how far the local dispatch had progressed.

Correlation: refund-webhook-correlation P1 (two-step, deterministic)¶

refund.processed/refund.failed can genuinely arrive before the dispatching worker (or the inline ineligible-checkout compensation path in PaymentFeatures.cs) has persisted ProviderRefundId via TryMarkSubmittedAsync/TryFinalizeCompletedAsync - Razorpay's own webhook delivery has no ordering guarantee relative to the synchronous HTTP response the dispatcher is still waiting on. Before this fix, ProcessRazorpayWebhookEventCommandHandler. HandleRefundTerminalAsync correlated only via GetByProviderRefundIdAsync, which finds nothing in that race - the webhook was durably acknowledged as NoMatchingRefundAttempt, and the local attempt could be left Submitted/Processing indefinitely with no further reconciliation path. Fixed with a two-step correlation, in order:

ProviderRefundId (unchanged, first choice) - the normal case for a webhook arriving after the worker has already persisted it.
Fallback, only when step 1 finds nothing: the stable, Vayzo-owned merchant reference - RazorpayRefundAttempt.IdempotencyKey (already deterministic per RefundTransactionId, already sent to Razorpay as the X-Refund-Idempotency header). RazorpayClient.CreateRefundAsync now also sends it in the refund creation payload's notes object (key: RazorpayRefundCorrelation.NotesKey = "vayzo_refund_attempt_ref") - a documented Razorpay Refunds API field that round-trips onto the created refund entity and is echoed back verbatim in every webhook for it, unlike the idempotency header (which Razorpay never returns anywhere). Looked up via the new IRazorpayRefundAttemptRepository.GetByIdempotencyKeyAsync, backed by the pre-existing unique DB index on IdempotencyKey - can never ambiguously match more than one row, even when several attempts share the same payment_id and amount (e.g. two equal-amount partial refunds against the same payment).

Validation before mutation, regardless of which step found the row: the webhook's own payment_id, amount, and currency (where present in the payload) are cross-checked against the correlated attempt before any state change. Any mismatch fails closed - no mutation, logged as a warning, the event still marked processed (a mismatch is not itself a processing error to surface as Failed). payment_id + amount are never used as the correlation key on their own - only as a validation cross-check against a row already uniquely identified via ProviderRefundId or the stable merchant reference. A refund created entirely outside Vayzo (e.g. manually via the Razorpay dashboard, or an unrecognized/malformed merchant reference) correlates to no row either way - NoMatchingRefundAttempt, not an error. An already-Completed/Failed attempt cannot be re-resolved (see "Out-of-order events" above) - AlreadyTerminal_IdempotentNoOp.

Worker vs webhook race: no ClaimToken is checked by the webhook's finalize (it is provider-driven confirmation, not a dispatch worker finalizing its own claim), so a stale worker finalize arriving after the webhook already resolved the row fails its own conditional UPDATE (TryMarkSubmittedAsync requires Status == Processing; the webhook has already moved it past that) - the webhook is always the authoritative final word, regardless of which one physically commits first at the database level. See MySqlRazorpayHardeningConcurrencyTests. MySql_WorkerFinalizeRacesEarlyArrivalWebhookFinalize_WebhookIsAlwaysAuthoritative for the real InnoDB proof.

PaymentReconciliationBackgroundService: a lighter-weight sweep that finds InventoryReservation rows stuck Pending well past normal checkout duration (Razorpay:ReconciliationStalePendingMinutes, default 30 - deliberately well past the ordinary reservation expiry, so this only fires on a genuinely ambiguous case, not routine cart abandonment) and logs them at Warning for operator triage. Deliberately does not auto-call Razorpay to resolve these in this pass - see "Known limitations": an automated resolution risks abandoning a paying customer's order if it guesses wrong, and the correct resolution (constructing a complete Order, including shipping address) cannot be done from payment state alone. IRazorpayClient.GetPaymentStatusAsync exists as the provider-status-lookup primitive a future, more complete reconciliation flow would use; it is intentionally not wired to any automatic state mutation yet, gated by Razorpay:ReconciliationEnabled (default false).

External calls outside DB transactions: both CreateRefundAsync and GetPaymentStatusAsync are called only from the two background services above, never from inside a request-scoped BeginTransactionAsync block - consistent with RazorpayClient's own doc comment and the same constraint already honored for SES sends.

Refund vs settlement race¶

Sequential correctness is proven and unchanged: refund-after-settlement-generated and refund-after-settlement-settled both correctly produce a SettlementAdjustment without mutating the (possibly already-payout-issued) settlement snapshot itself - this was already true before this tranche (ReturnsFlowTests.RefundAfterSettlementSettled_*) and is now additionally proven to commit atomically alongside the new RazorpayRefundAttempt enqueue in the same transaction (RazorpayWebhookAndRefundHardeningTests.RefundAfterSettlementGenerated_*) - the provider-dispatch enqueue and the settlement adjustment can never commit independently of each other.

FIXED (P1 settlement-vs-refund concurrency tranche) - the true concurrent settlement-generation-vs-refund-inspection race identified above (GenerateSettlementCommandHandler and InspectReturnCommandHandler both reading "does a settlement exist" without any lock, able to both observe "no settlement yet" and each proceed down the "not yet generated" branch) is now closed by IOrderRepository.LockSellerOrderForFinancialUpdateAsync - a real MySQL/InnoDB row lock (SELECT ... FOR UPDATE) both handlers acquire as the very first statement of their own transaction (GenerateSettlementCommandHandler did not previously run inside an explicit transaction at all - it now does), before evaluating any settlement/refund state. See Seller Settlements "Settlement vs refund concurrency" for the full design (why the lock must be strictly the first statement, given MySQL's default REPEATABLE READ snapshot semantics) and backend/tests/ECommerce.Tests/Application/MySqlSettlementRefundConcurrencyTests.cs for the real-InnoDB proof (refund-first, settlement-first, and true-simultaneous-start races all converge on exactly one consistent financial outcome; idempotent retry; unchanged behavior when only one side is involved).

Financial state machine (summary)¶

Transition	Guard
`InventoryReservation` Pending -> Completed	`TryClaimForCompletionAsync` atomic conditional UPDATE - exactly one caller wins
`InventoryReservation` Pending -> Released	`TryReleaseByRazorpayOrderIdAsync` / `TryClaimAndReleaseReservationAsync` - conditional on still-Pending
`PaymentTransaction` (created) Verified	Only construction path; no Pending/Failed factory exists
`PaymentTransaction` Verified -> PartiallyRefunded/Refunded	`ApplyRefund` - `RefundExceedsPaymentException` if cumulative would exceed `Amount`; one-directional, no un-refund method exists
`RazorpayRefundAttempt` Pending -> Processing	`TryClaimAsync` - conditional on Pending/due-Retrying, one claimant wins
`RazorpayRefundAttempt` Processing -> Completed/Retrying/Failed	`TryFinalize*Async` - conditional on matching `ClaimToken`, stale claimant always loses
`RazorpayRefundAttempt` Processing -> UnknownOutcome	Stale-lease sweep - conditional on `LeaseExpiresAt<=now`; terminal, never auto-reclaimed
`RazorpayRefundAttempt` (any non-terminal) -> Completed/Failed via webhook	`TryMarkTerminalFromWebhookAsync` - conditional on not already Completed/Failed
`RazorpayWebhookEvent` (insert)	`TryRecordAsync` - conditional on unique `PayloadHash`, exact replay rejected

Tests¶

All new tests are in backend/tests/ECommerce.Tests/:

Application/RazorpayWebhookAndRefundHardeningTests.cs (32 tests, SQLite) - signature verification (valid/invalid/tampered/missing-header), payload-hash determinism, malformed/missing- field webhooks never persisted, unrecognized event types recorded-but-ignored, exact-replay no-op, a genuinely concurrent (Task.WhenAll, dedicated shared-cache multi-connection SQLite database) duplicate-webhook race, webhook-before-verify / verify-before-webhook / provider- capture-with-local-timeout convergence, out-of-order payment.failed, refund-terminal-webhook idempotency and non-overwrite, refund enqueue duplicate rejection, full/partial refund dispatch, over-refund rejection, provider-timeout/ambiguous-outcome sweep-to-UnknownOutcome, transient-vs- permanent dispatch failure classification, stale-webhook-cannot-overwrite-terminal-state, cumulative-over-refund structural impossibility, and refund-after-settlement-generated atomicity.
Api/RazorpayWebhookControllerTests.cs (6 tests) - missing/placeholder secret -> 503 and mediator never called; invalid/missing signature -> 401 and mediator never called; valid signature -> 200 with the exact raw body dispatched; malformed-but-validly-signed body still returns 200 (the controller never parses JSON itself).
Application/MySqlRazorpayHardeningConcurrencyTests.cs (4 tests, [Trait("Category","mysql-smoke")], MYSQL_SMOKE_CONNSTR-gated, never silently skipped) - real InnoDB-row-locking proof of: two concurrent workers claiming the same RazorpayRefundAttempt (exactly one wins, exactly one simulated provider call), stale-lease sweep then reclaim (never two live owners), two simultaneous refund-enqueue requests for the same RefundTransaction (exactly one insert wins), two concurrent identical webhook deliveries (exactly one event row).
Application/VerifyAndCreateOrderCommandHandlerTests.cs (+9 tests, RC payment-integrity blocker P1-C, SQLite) - captured+exact order/amount/currency succeeds; authorized-but-not-captured fails and never creates an order; provider-reported failure fails; mismatched provider order id, amount, and currency each fail; provider lookup failure fails closed; invalid signature still fails before any provider lookup is attempted; retry after an already-completed payment remains idempotent (no second order/capture) with the P1-C check active.
Application/CheckoutIdempotencyTests.cs (8 tests, RC payment-integrity blocker P2, SQLite) - same key sequential retry returns the same Razorpay order/reservation; a genuinely concurrent (Task.WhenAll, dedicated shared-cache multi-connection SQLite database) same-key race produces exactly one payment attempt/provider order; different keys produce separate legitimate attempts; the same key string can never be reused across two different customers to access each other's attempt; a Failed (pre-provider-order) attempt can be reclaimed under the same key; a Completed attempt replay is rejected once its reservation is Completed/Released/expired.
Api/OrdersControllerLegacyEndpointTests.cs (8 tests, RC payment-integrity blockers P1-A/P1-B/ P2) - no controller action is mapped to a bare POST api/[controller] route; CreateOrderCommand/ CreateOrderCommandHandler no longer exist anywhere in the Application assembly; GetMyOrders/ GetOrder/GetInvoice/Cancel retain their original HTTP verbs; a real order created and verified through the hardened checkout path is still readable via both GET order-history and GET order-detail handlers, with cross-customer detail access still rejected; PaymentController. CreateOrder throws CheckoutIdempotencyKeyRequiredException before ever calling IMediator when the Idempotency-Key header is missing or whitespace.

Not covered by a new real-MySQL test in this pass: the browser-/verify-vs-webhook TryClaimForCompletionAsync race on InventoryReservations itself is proven at the SQLite tier (pre-existing VerifyAndCreateOrderCommandHandlerTests.Handle_CalledTwiceForSameRazorpayOrder_*, new RazorpayWebhookAndRefundHardeningTests convergence tests above) but not re-proven against real MySQL in this tranche - MySqlInventoryReservationConcurrencyTests.cs already exists for that table and would be the natural place to extend, but its existing dev-DB-schema-drift workaround (a full DDL rebuild of InventoryReservations/InventoryReservationLines, unrelated to this tranche's own migration) made extending it safely, in the time available, lower priority than the two genuinely new tables this tranche introduces. Recommended P3 follow-up.

Validation performed¶

dotnet build ECommerce.sln --configuration Release - 0 warnings, 0 errors.
Full default backend suite (--filter "Category!=mysql-smoke") - 799/799 passed (761 pre-existing
38 new).
Payment-focused filter (FullyQualifiedName~Payment|Razorpay|Refund, excluding mysql-smoke) - 107/107 passed.
Real-MySQL concurrency suite (--filter "Category=mysql-smoke", against the local ecommerce_mysql container) - 14/14 passed (10 pre-existing + 4 new).
Migration AddRazorpayWebhookAndRefundHardening applied directly to the live dev MySQL container (via an extracted SQL script - dotnet ef database update cannot be used directly against this dev database due to pre-existing, unrelated migration-history drift from other in-progress branches) and verified via SHOW TABLES LIKE 'Razorpay%' and SHOW INDEX ... WHERE Non_unique=0.

External Razorpay prerequisites (before enabling any of this for real)¶

A staging Razorpay account/test-mode credentials (rzp_test_*) with a webhook configured in the Razorpay dashboard pointing at the staging deployment's POST /api/webhooks/razorpay, subscribed to at minimum payment.captured, payment.failed, refund.processed, refund.failed.
The webhook secret Razorpay generates for that webhook endpoint, provided via Razorpay:WebhookSecret (real secret manager / environment variable in staging/production, never committed - matches the existing Razorpay:KeySecret convention).
Verify Razorpay's current documented idempotency-key length/character constraints for the Create Refund API before enabling Razorpay:RefundExecutionEnabled - this tranche's IdempotencyKey (36 chars, vzr- + 32 hex) was sized conservatively but was not verified against a live API call.
Razorpay:WebhookEnabled=true and Razorpay:RefundExecutionEnabled=true (and, once a reconciliation flow beyond logging is implemented, Razorpay:ReconciliationEnabled=true) only after the above are in place - Program.ValidateRuntimeConfiguration will refuse to start otherwise outside Development.
Operational alert recommended: alert on any RazorpayRefundAttempt row reaching UnknownOutcome, and on any RazorpayWebhookEvent reaching Failed - both indicate a state that will not self-heal and needs a human to check Razorpay's own dashboard/API for the authoritative outcome.
Operational alert recommended: alert on any InventoryReservation surfaced by PaymentReconciliationBackgroundService's stale-Pending sweep (Warning-level log, see "Refund webhook / reconciliation" above) - this is the "browser timeout after provider capture" scenario and currently requires manual resolution.

Provider-authoritative verification (RC payment-integrity blocker P1-C)¶

An independent financial/security audit of the release candidate found that VerifyAndCreateOrderCommandHandler validated the Razorpay HMAC callback signature but never independently confirmed Razorpay's own record of the payment before completing checkout - a valid signature only proves the caller knows razorpayOrderId/razorpayPaymentId and the shared secret, not that the payment was actually captured, for the right amount, in the right currency, against the right order.

Fixed: after signature validation, before the atomic reservation claim (TryClaimForCompletionAsync) or any Order/PaymentTransaction is created, the handler now calls IRazorpayClient. GetPaymentDetailsAsync (a new method, deliberately separate from the existing reconciliation-only GetPaymentStatusAsync - see IRazorpayClient's own doc comment for why) and fails closed (PaymentVerificationFailedException, one type with a ReasonCode) unless ALL of:

the lookup itself succeeds (ProviderLookupFailed otherwise - network/timeout/5xx/malformed response, or credentials not configured, are never treated as "probably fine");
the provider's order_id equals this reservation's Razorpay order id (OrderIdMismatch);
the provider's currency equals the expected currency (CurrencyMismatch);
the provider's amount (minor units) equals the server-authoritative amount recomputed from the reservation's own frozen line items - never the client-supplied request (AmountMismatch);
the provider's status is captured, not merely authorized (NotCaptured) and not failed (ProviderReportedFailure).

Placed before the claim deliberately: claiming first and failing this check after would strand the reservation Completed with no Order/PaymentTransaction ever created, and TryReleaseByRazorpayOrderIdAsync only operates on Pending rows. A rejection here instead leaves the reservation untouched, so the existing Handle() outer catch releases it exactly like any other failure mode (insufficient stock, coupon errors, total-mismatch, etc.) - no new failure philosophy, just one more check with the same fail-closed shape. See VerifyAndCreateOrderCommandHandlerTests.cs's "provider verification" test group for the full matrix (captured+exact match succeeds; authorized/failed/mismatched order-id/amount/currency/ provider-lookup-failure all fail closed; a bad signature still fails before the provider is ever queried; retrying an already-completed payment remains exactly as idempotent as before this change).

Checkout idempotency (RC payment-integrity blocker P2)¶

The same audit found POST /api/payment/create-order had no durable client idempotency key - a client retry (network timeout, double-tap, etc.) could create a second, separate Razorpay order and InventoryReservation for the same logical checkout attempt.

Fixed with a new CheckoutIdempotencyRecords table (one row per (CustomerId, IdempotencyKey), a real DB unique index - see CheckoutIdempotencyRecordConfiguration) and a claim/replay state machine in CreateRazorpayOrderCommandHandler.ResolveOrCreateAsync - see CheckoutIdempotencyRecord's own doc comment for the full Claimed/Completed/Failed walk. Summary:

The client supplies an Idempotency-Key header (required - PaymentController.CreateOrder rejects a missing one with 400 before ever calling the command); one key per logical checkout attempt, reused only for a genuine retry of that same attempt.
The idempotent-insert-and-catch-duplicate pattern is the same one IRazorpayWebhookEventRepository.TryRecordAsync already established for webhook dedup - the real guard is the database's own unique index, so this is correct across multiple API replicas, not just within one process.
The claim happens BEFORE any call to Razorpay, so a losing concurrent request never causes a second provider order - it waits briefly (bounded, ~5s) for the winner to finish and replays its result, converging both callers on the same Razorpay order/reservation.
A Failed claim (the attempt failed before any provider order existed - e.g. an empty cart or a policy-acceptance gate) can be reclaimed by a later retry under the same key. A Completed claim whose reservation has since actually completed payment, expired, or been released cannot be replayed - the client gets a clear CheckoutIdempotencyConflictException (AlreadyCompleted / AttemptExpired) telling it to mint a new key for a genuinely new attempt, never a stale/re-used order.
Scoped to (CustomerId, IdempotencyKey) together, always - the same literal key string from two different customers can never collide or cross-resolve to the other's attempt.

See CheckoutIdempotencyTests.cs for the full regression matrix (sequential retry replays the same order; genuine Task.WhenAll concurrent retry - two separate SQLite connections against a shared-cache in-memory database, not just two calls against one shared connection - produces exactly one provider order; different keys are separate attempts; the same key string never crosses customers; a failed attempt can be retried under the same key; a completed/expired/released attempt fails closed under the same key).

Known limitations¶

The webhook cannot construct an Order on its own. payment.captured arriving before the browser's own /verify call is recorded and flagged (AwaitingBrowserCompletion_ NoShippingAddressAvailable) but does not create an order, because the shipping address is only collected at /verify time in this codebase's checkout flow, not at Razorpay order-creation time. If the browser never completes /verify for a payment Razorpay did capture, the order is never created automatically - PaymentReconciliationBackgroundService's stale-Pending sweep surfaces this for manual operator resolution, but does not resolve it automatically. A structural fix (collecting/persisting the shipping address at order-creation time, before payment) is out of scope for this tranche.
~~The settlement-generation-vs-refund-inspection concurrent race~~ FIXED - see "Refund vs settlement race" above and Seller Settlements "Settlement vs refund concurrency".
RazorpayRefundAttempt.UnknownOutcome rows are never automatically retried or resolved - by design (retrying risks a duplicate provider-side refund), but this means production needs the operational alert above; there is no automated path from UnknownOutcome back to a terminal state today.
PaymentReconciliationBackgroundService only logs - it does not call IRazorpayClient.GetPaymentStatusAsync or take any corrective action automatically. Building an automated (or semi-automated, operator-confirmed) reconciliation action on top of the existing GetPaymentStatusAsync primitive is recommended future work, not implemented here.
No real-MySQL concurrency test for the InventoryReservation.TryClaimForCompletionAsync race itself in this tranche - see "Tests" above.
Idempotency-key length/format was not verified against a live Razorpay API call - see "External prerequisites" above.
A checkout-idempotency claim that fails AFTER a real Razorpay order was already created (the HTTP call to Razorpay succeeded, but the subsequent reservation-save transaction then failed for an unrelated reason - e.g. a stock check discovered only inside that transaction) is still marked Failed and can be reclaimed. A retry under the same key then creates a genuinely SECOND Razorpay order; the first one is simply left orphaned (unused, no reservation ever points at it) rather than cancelled. This is a pre-existing characteristic of the checkout flow (the original code never attempted to void/cancel a Razorpay order on a downstream failure either) and is not a regression introduced by the idempotency work - just not solved by it. Orphaned Razorpay orders are inert (never charged unless a payment is separately made against them) and are not one of the four confirmed release blockers; automatically cancelling them is recommended future work.

7a. FIXED (post-1cd70c4 payment/security hardening review): the compensating writes that mark a claim Failed (CreateNewCheckoutAsync's catch) or release a reservation (VerifyAndCreateOrderCommandHandler's two ReleaseReservationAsync call sites) used to reuse the caller's own CancellationToken - the one ASP.NET Core binds to HttpContext.RequestAborted. That token is very often already canceled by the time either catch block runs, because the single most common way to reach one is the client itself aborting the in-flight request (a timeout, a flaky mobile network, a closed tab) - an entirely ordinary event, not a rare crash. Reusing an already-canceled token made the compensating write throw immediately without ever executing, which is strictly worse than the scenario in point 7 above: instead of ending up Failed (reclaimable, at worst wasting one orphaned provider-side order), the claim was left stuck in Claimed forever - and unlike a Pending InventoryReservation, a Claimed CheckoutIdempotencyRecord has no ExpiresAt/reaper to self-heal it. Every subsequent retry under the SAME idempotency key (the frontend deliberately reuses one key across retries of the same logical attempt - see frontend/src/utils/checkoutIdempotency.js) would then wait out MaxClaimWaitAttempts (~5s) and fail closed with AttemptInProgress, indefinitely, until an operator manually repaired the row. Both compensating writes now always use CancellationToken.None instead - the request's DbContext/connection is still open regardless of the token's cancellation state (ASP.NET Core only tears down the request's DI scope after the action method returns), so the write itself succeeds normally either way. See PaymentCompensatingCleanupCancellationTests.cs for the regression coverage.

Residual risk (documented, not fixed - genuinely P3): a true process crash (pod OOM-kill, an unhandled fatal exception that bypasses the catch entirely, a hard node failure) between the claim insert and either compensating write still leaves a Claimed row with no automatic reaper, and a genuinely orphaned Razorpay order if the crash happened after the provider call succeeded. Fixing this fully would mean adding a time-based sweep for stale Claimed rows (mirroring PaymentReconciliationBackgroundService's stale-Pending sweep) - deliberately not done in this same change, to keep the fix isolated and reviewable; recommended future work.