vishal maurya

LLD Series: Payment Gateway with Routing, Retries and Webhooks (Low-Level Design)

Overview

Part of the Interview Prep series. Dotted-underline words are glossary terms: hover for the meaning or open the Glossary. Track your progress on the Interview Prep tracker. Series index: Start here.

1. Problem statement and how it is asked

Typical phrasing: "Design a payment gateway (Razorpay / PayU / Stripe style). Merchants submit payments by card, UPI, net-banking or wallet. The gateway routes to a bank/acquirer, falls back if one is down, retries safely, never double-charges, tracks a payment state machine and notifies merchants through webhooks."

  • Asked at: PayU, Visa, Mastercard, NPCI, Amex - this is the "home turf" question for payments companies; expect it as the senior/hard round or as an extended follow-up to wallet.
  • Interviewer is checking: Strategy/Factory for payment methods, routing + fallback that knows which failures are safe to retry, idempotency, an explicit state machine, async results (UPI callbacks), webhooks (signed, retried), and money-safety reasoning.

2. Requirements clarification

Functional

  1. pay(request): merchant, order id, amount, method (CARD / UPI / NET_BANKING / WALLET), method details, idempotency key.
  2. Validate method-specific details (Luhn for cards, VPA format, bank code).
  3. Route to a capable connector (acquirer/bank adapter). Support static priority and success-rate based routing.
  4. Retry transient errors on the same connector with exponential backoff, then fail over to the next connector. Never retry or fail over on a hard decline.
  5. Circuit breaker per connector to stop hammering a failing bank.
  6. Payment lifecycle: CREATED -> PROCESSING -> (SUCCESS | FAILED | PENDING) ; PENDING -> (SUCCESS | FAILED) ; SUCCESS -> REFUNDED.
  7. Async confirmation endpoint (bank callback) that is idempotent.
  8. Webhooks to merchants on SUCCESS/FAILED/REFUNDED: HMAC-signed, retried with backoff, dead-lettered when exhausted.
  9. Refund of a successful payment (once).

Non-functional: no double charge (idempotency + state machine); low latency (routing in-memory); extensibility (new method/bank/routing rule without touching the orchestrator); observability (attempt trail per payment); PCI hygiene (raw PAN never stored/logged, masked in Instrument).

Assumptions: connectors are synchronous adapters (a real one wraps HTTP/ISO 8583 with timeouts); a TECHNICAL_ERROR means the bank definitely did not process the request (e.g. connection refused/5xx before processing). Ambiguous timeouts are treated in follow-ups (status inquiry before retry). Amounts in minor units.

Out of scope: settlement/reconciliation files, 3-D Secure/OTP step, tokenisation vault, fraud engine, multi-currency/FX, chargebacks, persistence (discussed).

3. Entities, relationships, class diagram

 PaymentRequest ---> PaymentService (orchestrator) ---creates---> Payment (state machine)
                        |  |   |   |   |                           | status, attempts[], bankRef
                        |  |   |   |   +--publishes--> PaymentEvent
                        |  |   |   |                        |
     +------------------+  |   |   +----------+             v
     v                     v   v              v      <<iface>> PaymentListener
 PaymentMethodFactory   RoutingStrategy   RetryPolicy     |-- WebhookDispatcher (HMAC, retries)
   |  creates/gets          ^   ^          + Sleeper      |-- AuditListener (not shown)
   v                        |   |
 <<iface>> PaymentMethod  StaticPriority  SuccessRate   HealthRegistry --> ConnectorHealth (circuit breaker + success rate)
   ^   ^    ^    ^
 Card Upi NetBanking Wallet                 <<iface>> BankConnector  (adapter per acquirer/bank)
  |  prepare(details) -> Instrument           ^           ^            ^
  |  (validate, mask)                     HDFC adapter  ICICI adapter  NPCI-UPI adapter   (here: ScriptedConnector fakes)

Payment state machine:

 CREATED --> PROCESSING --+--> SUCCESS --> REFUNDED
                          |        ^
                          +--> PENDING --+--> FAILED
                          |        (bank callback)
                          +--> FAILED

FAILED and REFUNDED are terminal. Encoded in PaymentStatus.next(); Payment.transition rejects anything else.

4. Design patterns and why

PatternWhereWhy / alternative rejected
StrategyPaymentMethod (validation + sanitising per method), RoutingStrategy, RetryPolicyBehaviour varies independently. Rejected: switch(methodType) in the orchestrator; every new method edits the hot path.
Factory (registry)PaymentMethodFactoryOne place maps type -> strategy; register() supports plug-ins.
AdapterBankConnectorEach bank has its own protocol; the orchestrator sees one interface.
State machinePaymentStatus + PaymentMoney-critical lifecycle; illegal transitions throw; callbacks use compare-and-transition so duplicates are no-ops. Rejected: free-form string statuses.
ObserverPaymentListener, WebhookDispatcherMerchant notifications, audit, analytics are side effects that must not be in the payment path or break it.
Circuit breakerConnectorHealthFail fast to the next bank instead of burning latency on a dead one; half-open after cooldown.
Idempotent receiverbyIdempotencyKey mapSafe client retries (network timeouts are the norm).
(Mention) Chain of Responsibilityfallback chain over connectorsThe ordered fallback loop is a CoR; kept as a loop for readability.
(Mention) Template Method / DecoratorRetryingConnector wrapperAlternative: decorate each connector with retry/timeout/metrics; we keep retry in the orchestrator because failover needs cross-connector knowledge.

5. Complete Java code

Default package; javac *.java && java Main. Fake bank connectors (ScriptedConnector) play scripted outcomes so every path is deterministic.

5.1 Enums, state machine, exceptions

// Enums.java
enum PaymentMethodType { CARD, UPI, NET_BANKING, WALLET }

/** What a bank/acquirer answered. Only TECHNICAL_ERROR is safe to retry / fail over. */
enum AuthOutcome { APPROVED, DECLINED, TECHNICAL_ERROR, PENDING }
// PaymentStatus.java
import java.util.EnumSet;
import java.util.Set;

/** The payment state machine. Illegal transitions are impossible by construction. */
public enum PaymentStatus {
    CREATED, PROCESSING, PENDING, SUCCESS, FAILED, REFUNDED;

    public Set<PaymentStatus> next() {
        switch (this) {
            case CREATED:    return EnumSet.of(PROCESSING);
            case PROCESSING: return EnumSet.of(SUCCESS, FAILED, PENDING);
            case PENDING:    return EnumSet.of(SUCCESS, FAILED);
            case SUCCESS:    return EnumSet.of(REFUNDED);
            default:         return EnumSet.noneOf(PaymentStatus.class);   // FAILED, REFUNDED are terminal
        }
    }
    public boolean canMoveTo(PaymentStatus to) { return next().contains(to); }
}
// GatewayException.java
public class GatewayException extends RuntimeException {
    public GatewayException(String m) { super(m); }
}
// PaymentValidationException.java
public class PaymentValidationException extends GatewayException {
    public PaymentValidationException(String m) { super(m); }
}

5.2 Request, instrument and payment methods (Strategy + Factory)

// PaymentRequest.java
import java.util.Map;

public final class PaymentRequest {
    public final String merchantId, orderId, idempotencyKey;
    public final long amountMinor;
    public final PaymentMethodType methodType;
    public final Map<String, String> details;     // card number, vpa, bank code ...

    public PaymentRequest(String merchantId, String orderId, String idempotencyKey, long amountMinor,
                          PaymentMethodType methodType, Map<String, String> details) {
        if (amountMinor <= 0) throw new PaymentValidationException("Amount must be positive");
        if (idempotencyKey == null || idempotencyKey.isBlank()) throw new PaymentValidationException("Idempotency key required");
        this.merchantId = merchantId; this.orderId = orderId; this.idempotencyKey = idempotencyKey;
        this.amountMinor = amountMinor; this.methodType = methodType; this.details = Map.copyOf(details);
    }
    /** Same key + different fingerprint => client bug (409 Conflict). Sensitive details excluded from the hash input shown in logs. */
    String fingerprint() { return merchantId + "|" + orderId + "|" + amountMinor + "|" + methodType + "|" + details.hashCode(); }
}
// Instrument.java
import java.util.Map;

/** Validated, sanitised view of how the customer wants to pay. Never carries the raw PAN/CVV past the method class. */
public final class Instrument {
    public final PaymentMethodType type; public final String maskedRef; public final Map<String, String> attributes;
    public Instrument(PaymentMethodType type, String maskedRef, Map<String, String> attributes) {
        this.type = type; this.maskedRef = maskedRef; this.attributes = Map.copyOf(attributes);
    }
    public String attr(String k) { return attributes.getOrDefault(k, ""); }
}
// PaymentMethod.java
import java.util.Map;

/** Strategy: validates method-specific details and produces a sanitised Instrument. */
public interface PaymentMethod {
    PaymentMethodType type();
    Instrument prepare(Map<String, String> details);     // throws PaymentValidationException
}
// CardPaymentMethod.java
import java.util.Map;

public final class CardPaymentMethod implements PaymentMethod {
    @Override public PaymentMethodType type() { return PaymentMethodType.CARD; }

    @Override public Instrument prepare(Map<String, String> d) {
        String pan = d.getOrDefault("pan", "").replaceAll("\\s", "");
        if (!pan.matches("\\d{13,19}") || !luhn(pan)) throw new PaymentValidationException("Invalid card number");
        if (!d.getOrDefault("cvv", "").matches("\\d{3,4}")) throw new PaymentValidationException("Invalid CVV");
        if (!d.getOrDefault("expiry", "").matches("(0[1-9]|1[0-2])/\\d{2}")) throw new PaymentValidationException("Invalid expiry");
        String network = pan.startsWith("4") ? "VISA" : pan.startsWith("5") ? "MASTERCARD" : "OTHER";
        return new Instrument(type(), "****" + pan.substring(pan.length() - 4), Map.of("network", network));
    }

    static boolean luhn(String s) {
        int sum = 0; boolean dbl = false;
        for (int i = s.length() - 1; i >= 0; i--) {
            int n = s.charAt(i) - '0';
            if (dbl && (n *= 2) > 9) n -= 9;
            sum += n; dbl = !dbl;
        }
        return sum % 10 == 0;
    }
}
// UpiPaymentMethod.java
import java.util.Map;

public final class UpiPaymentMethod implements PaymentMethod {
    @Override public PaymentMethodType type() { return PaymentMethodType.UPI; }
    @Override public Instrument prepare(Map<String, String> d) {
        String vpa = d.getOrDefault("vpa", "");
        if (!vpa.matches("[\\w.\\-]{2,}@[a-zA-Z]{2,}")) throw new PaymentValidationException("Invalid VPA");
        return new Instrument(type(), vpa, Map.of("handle", vpa.substring(vpa.indexOf('@') + 1)));
    }
}
// NetBankingPaymentMethod.java
import java.util.Map;
import java.util.Set;

public final class NetBankingPaymentMethod implements PaymentMethod {
    private static final Set<String> BANKS = Set.of("HDFC", "ICICI", "SBI", "AXIS");
    @Override public PaymentMethodType type() { return PaymentMethodType.NET_BANKING; }
    @Override public Instrument prepare(Map<String, String> d) {
        String bank = d.getOrDefault("bank", "");
        if (!BANKS.contains(bank)) throw new PaymentValidationException("Unsupported bank " + bank);
        return new Instrument(type(), bank, Map.of("bank", bank));
    }
}
// WalletPaymentMethod.java
import java.util.Map;

public final class WalletPaymentMethod implements PaymentMethod {
    @Override public PaymentMethodType type() { return PaymentMethodType.WALLET; }
    @Override public Instrument prepare(Map<String, String> d) {
        String id = d.getOrDefault("walletId", "");
        if (id.isBlank()) throw new PaymentValidationException("walletId required");
        return new Instrument(type(), id, Map.of());
    }
}
// PaymentMethodFactory.java
import java.util.EnumMap;
import java.util.Map;

/** Registry-style factory: new method = new class + register(); service untouched. */
public final class PaymentMethodFactory {
    private final Map<PaymentMethodType, PaymentMethod> methods = new EnumMap<>(PaymentMethodType.class);

    public static PaymentMethodFactory withDefaults() {
        PaymentMethodFactory f = new PaymentMethodFactory();
        f.register(new CardPaymentMethod()); f.register(new UpiPaymentMethod());
        f.register(new NetBankingPaymentMethod()); f.register(new WalletPaymentMethod());
        return f;
    }
    public void register(PaymentMethod m) { methods.put(m.type(), m); }
    public PaymentMethod get(PaymentMethodType t) {
        PaymentMethod m = methods.get(t);
        if (m == null) throw new PaymentValidationException("Unsupported payment method " + t);
        return m;
    }
}

5.3 Bank connectors (Adapter) and health (circuit breaker)

// AuthRequest.java
public final class AuthRequest {
    public final String paymentId; public final long amountMinor; public final Instrument instrument;
    public AuthRequest(String paymentId, long amountMinor, Instrument instrument) {
        this.paymentId = paymentId; this.amountMinor = amountMinor; this.instrument = instrument;
    }
}
// AuthResponse.java
public final class AuthResponse {
    public final AuthOutcome outcome; public final String code, bankRef;
    private AuthResponse(AuthOutcome o, String code, String ref) { this.outcome = o; this.code = code; this.bankRef = ref; }
    public static AuthResponse approved(String ref)   { return new AuthResponse(AuthOutcome.APPROVED, "00", ref); }
    public static AuthResponse declined(String code)  { return new AuthResponse(AuthOutcome.DECLINED, code, null); }
    public static AuthResponse technical(String code) { return new AuthResponse(AuthOutcome.TECHNICAL_ERROR, code, null); }
    public static AuthResponse pending(String ref)    { return new AuthResponse(AuthOutcome.PENDING, "PENDING", ref); }
}
// BankConnector.java
/** Adapter to one acquirer / bank / network. One implementation per integration (HDFC, ICICI, NPCI-UPI ...). */
public interface BankConnector {
    String name();
    boolean supports(PaymentMethodType type);
    AuthResponse authorize(AuthRequest request);
    boolean refund(String bankRef, long amountMinor);
}
// ScriptedConnector.java
import java.util.Set;
import java.util.concurrent.ConcurrentLinkedQueue;
import java.util.concurrent.atomic.AtomicInteger;
import java.util.function.Supplier;

/** Test double: plays back a script of responses, then falls back to a default behaviour. */
public final class ScriptedConnector implements BankConnector {
    private final String name; private final Set<PaymentMethodType> supported;
    private final ConcurrentLinkedQueue<AuthResponse> script = new ConcurrentLinkedQueue<>();
    private final Supplier<AuthResponse> fallback;
    private final AtomicInteger calls = new AtomicInteger();

    public ScriptedConnector(String name, Set<PaymentMethodType> supported, Supplier<AuthResponse> fallback) {
        this.name = name; this.supported = supported; this.fallback = fallback;
    }
    public ScriptedConnector thenReturn(AuthResponse... rs) { for (AuthResponse r : rs) script.add(r); return this; }
    public int calls() { return calls.get(); }

    @Override public String name() { return name; }
    @Override public boolean supports(PaymentMethodType t) { return supported.contains(t); }
    @Override public AuthResponse authorize(AuthRequest r) {
        int n = calls.incrementAndGet();
        AuthResponse scripted = script.poll();
        return scripted != null ? scripted : fallback.get();
    }
    @Override public boolean refund(String ref, long amt) { return true; }
}
// ConnectorHealth.java
import java.time.Clock;
import java.time.Duration;
import java.time.Instant;
import java.util.concurrent.atomic.AtomicInteger;
import java.util.concurrent.atomic.AtomicLong;

/** Circuit breaker + success-rate counters for one connector. Lock-free (atomics). */
public final class ConnectorHealth {
    private final int failureThreshold; private final Duration cooldown;
    private final AtomicInteger consecutiveFailures = new AtomicInteger();
    private final AtomicLong ok = new AtomicLong(), total = new AtomicLong();
    private volatile Instant openUntil = Instant.MIN;

    public ConnectorHealth(int failureThreshold, Duration cooldown) { this.failureThreshold = failureThreshold; this.cooldown = cooldown; }

    /** CLOSED or half-open (cooldown elapsed) => allow a trial request. */
    public boolean allowRequest(Clock clock) { return !Instant.now(clock).isBefore(openUntil); }

    public void recordSuccess() { consecutiveFailures.set(0); ok.incrementAndGet(); total.incrementAndGet(); }
    public void recordFailure(Clock clock) {
        total.incrementAndGet();
        if (consecutiveFailures.incrementAndGet() >= failureThreshold) openUntil = Instant.now(clock).plus(cooldown);
    }
    public double successRate() { long t = total.get(); return t == 0 ? 1.0 : (double) ok.get() / t; }
    public boolean isOpen(Clock clock) { return !allowRequest(clock); }
}
// HealthRegistry.java
import java.time.Duration;
import java.util.Map;
import java.util.concurrent.ConcurrentHashMap;

public final class HealthRegistry {
    private final Map<String, ConnectorHealth> byConnector = new ConcurrentHashMap<>();
    private final int threshold; private final Duration cooldown;
    public HealthRegistry(int threshold, Duration cooldown) { this.threshold = threshold; this.cooldown = cooldown; }
    public ConnectorHealth of(String connectorName) {
        return byConnector.computeIfAbsent(connectorName, n -> new ConnectorHealth(threshold, cooldown));
    }
}

5.4 Routing and retry

// RoutingStrategy.java
import java.util.List;

/** Strategy: order the capable connectors for this payment. First = preferred, rest = fallbacks. */
public interface RoutingStrategy {
    List<BankConnector> route(PaymentRequest request, Instrument instrument, List<BankConnector> capable, HealthRegistry health);
}
// StaticPriorityRouting.java
import java.util.List;

/** Keep configured order (e.g. cheapest MDR first). */
public final class StaticPriorityRouting implements RoutingStrategy {
    @Override public List<BankConnector> route(PaymentRequest r, Instrument i, List<BankConnector> capable, HealthRegistry h) {
        return capable;
    }
}
// SuccessRateRouting.java
import java.util.ArrayList;
import java.util.Comparator;
import java.util.List;

/** Smart routing: best live success-rate first (stable sort keeps configured order on ties). */
public final class SuccessRateRouting implements RoutingStrategy {
    @Override public List<BankConnector> route(PaymentRequest r, Instrument i, List<BankConnector> capable, HealthRegistry h) {
        List<BankConnector> sorted = new ArrayList<>(capable);
        sorted.sort(Comparator.comparingDouble((BankConnector c) -> h.of(c.name()).successRate()).reversed());
        return sorted;
    }
}
// RetryPolicy.java
public final class RetryPolicy {
    private final int maxAttemptsPerConnector; private final long baseDelayMs;
    public RetryPolicy(int maxAttemptsPerConnector, long baseDelayMs) { this.maxAttemptsPerConnector = maxAttemptsPerConnector; this.baseDelayMs = baseDelayMs; }
    public int maxAttempts() { return maxAttemptsPerConnector; }
    /** Exponential backoff before attempt n (n >= 2): base * 2^(n-2). Production adds jitter. */
    public long delayBefore(int attempt) { return baseDelayMs << (attempt - 2); }
}
// Sleeper.java
/** Injectable so tests and the demo do not really sleep. */
public interface Sleeper { void sleep(long millis); }

5.5 Payment aggregate, events, webhooks (Observer)

// Payment.java
import java.util.ArrayList;
import java.util.List;

/** Aggregate with a guarded state machine. All mutation goes through synchronized methods of this object (per-payment lock). */
public final class Payment {
    private final String id, fingerprint; private final PaymentRequest request; private final Instrument instrument;
    private PaymentStatus status = PaymentStatus.CREATED;
    private final List<String> attempts = new ArrayList<>();
    private String connectorName, bankRef, failureReason;

    Payment(String id, PaymentRequest request, Instrument instrument) {
        this.id = id; this.request = request; this.instrument = instrument; this.fingerprint = request.fingerprint();
    }
    public String id() { return id; }
    public PaymentRequest request() { return request; }
    public Instrument instrument() { return instrument; }
    String fingerprint() { return fingerprint; }
    public synchronized PaymentStatus status() { return status; }
    public synchronized String bankRef() { return bankRef; }
    public synchronized String connectorName() { return connectorName; }
    public synchronized String failureReason() { return failureReason; }
    public synchronized List<String> attempts() { return List.copyOf(attempts); }

    synchronized void transition(PaymentStatus to) {
        if (!status.canMoveTo(to)) throw new GatewayException("Illegal transition " + status + " -> " + to + " for " + id);
        status = to;
    }
    /** Atomic compare-and-transition: used by webhooks/callbacks that may be duplicated or race. */
    synchronized boolean transitionIf(PaymentStatus expected, PaymentStatus to) {
        if (status != expected) return false;
        transition(to);
        return true;
    }
    synchronized void addAttempt(String connector, String result) { attempts.add(connector + ":" + result); }
    synchronized void recordRoute(String connector, String ref) { this.connectorName = connector; this.bankRef = ref; }
    synchronized void recordFailure(String reason) { this.failureReason = reason; }
    @Override public synchronized String toString() {
        return id + " " + request.orderId + " " + status + (failureReason != null ? " (" + failureReason + ")" : "")
                + " via " + connectorName + " trail=" + attempts;
    }
}
// PaymentEvent.java
public final class PaymentEvent {
    public final String paymentId, merchantId, orderId, bankRef, reason; public final PaymentStatus status; public final long amountMinor;
    public PaymentEvent(Payment p) {
        this.paymentId = p.id(); this.merchantId = p.request().merchantId; this.orderId = p.request().orderId;
        this.status = p.status(); this.amountMinor = p.request().amountMinor; this.bankRef = p.bankRef(); this.reason = p.failureReason();
    }
}
// PaymentListener.java
/** Observer. Called synchronously after every state change, outside any payment lock. Must not throw. */
public interface PaymentListener { void onEvent(PaymentEvent event); }
// WebhookClient.java
/** HTTP port to a merchant endpoint. Returns true on 2xx. */
public interface WebhookClient { boolean post(String url, String jsonBody, String signature); }
// WebhookDispatcher.java
import javax.crypto.Mac;
import javax.crypto.spec.SecretKeySpec;
import java.nio.charset.StandardCharsets;
import java.util.Map;
import java.util.Set;
import java.util.concurrent.ConcurrentHashMap;

/** Observer that notifies merchants of terminal outcomes: signed payload, retried with backoff, dead-lettered when exhausted. */
public final class WebhookDispatcher implements PaymentListener {
    public static final class Endpoint {
        final String url, secret; final WebhookClient client;
        public Endpoint(String url, String secret, WebhookClient client) { this.url = url; this.secret = secret; this.client = client; }
    }
    private static final Set<PaymentStatus> NOTIFY_ON = Set.of(PaymentStatus.SUCCESS, PaymentStatus.FAILED, PaymentStatus.REFUNDED);
    private final Map<String, Endpoint> endpoints = new ConcurrentHashMap<>();
    private final Sleeper sleeper; private final int maxAttempts;

    public WebhookDispatcher(Sleeper sleeper, int maxAttempts) { this.sleeper = sleeper; this.maxAttempts = maxAttempts; }
    public void register(String merchantId, Endpoint e) { endpoints.put(merchantId, e); }

    @Override public void onEvent(PaymentEvent e) {
        Endpoint ep = endpoints.get(e.merchantId);
        if (ep == null || !NOTIFY_ON.contains(e.status)) return;
        String body = "{\"paymentId\":\"" + e.paymentId + "\",\"orderId\":\"" + e.orderId + "\",\"status\":\"" + e.status
                + "\",\"amount\":" + e.amountMinor + "}";
        String sig = hmacSha256Hex(ep.secret, body);
        for (int attempt = 1; attempt <= maxAttempts; attempt++) {
            if (attempt > 1) sleeper.sleep(100L << (attempt - 2));
            boolean ok = false;
            try { ok = ep.client.post(ep.url, body, sig); } catch (RuntimeException ignored) { }
            System.out.println("    [webhook] " + e.paymentId + " " + e.status + " attempt " + attempt + (ok ? " delivered" : " failed"));
            if (ok) return;
        }
        System.out.println("    [webhook] " + e.paymentId + " moved to dead-letter queue for manual replay");
    }

    static String hmacSha256Hex(String secret, String data) {
        try {
            Mac mac = Mac.getInstance("HmacSHA256");
            mac.init(new SecretKeySpec(secret.getBytes(StandardCharsets.UTF_8), "HmacSHA256"));
            StringBuilder sb = new StringBuilder();
            for (byte b : mac.doFinal(data.getBytes(StandardCharsets.UTF_8))) sb.append(String.format("%02x", b));
            return sb.toString();
        } catch (Exception ex) { throw new IllegalStateException(ex); }
    }
}

5.6 Orchestrator

// PaymentService.java
import java.time.Clock;
import java.util.*;
import java.util.concurrent.ConcurrentHashMap;
import java.util.concurrent.CopyOnWriteArrayList;
import java.util.concurrent.atomic.AtomicLong;
import java.util.stream.Collectors;

/**
 * Orchestrator. Thread-safety:
 *  - idempotency: ConcurrentHashMap.putIfAbsent => exactly one thread processes a (merchant,key); others get the same Payment.
 *  - state changes: per-Payment monitor (Payment.transition*), so unrelated payments never contend.
 *  - listeners/webhooks are invoked after the transition, never while holding a payment lock.
 */
public final class PaymentService {
    private final PaymentMethodFactory methods;
    private final List<BankConnector> connectors;
    private final RoutingStrategy routing;
    private final RetryPolicy retry;
    private final HealthRegistry health;
    private final Sleeper sleeper;
    private final Clock clock;
    private final Map<String, Payment> byIdempotencyKey = new ConcurrentHashMap<>();
    private final Map<String, Payment> byId = new ConcurrentHashMap<>();
    private final List<PaymentListener> listeners = new CopyOnWriteArrayList<>();
    private final AtomicLong seq = new AtomicLong();

    public PaymentService(PaymentMethodFactory methods, List<BankConnector> connectors, RoutingStrategy routing,
                          RetryPolicy retry, HealthRegistry health, Sleeper sleeper, Clock clock) {
        this.methods = methods; this.connectors = List.copyOf(connectors); this.routing = routing;
        this.retry = retry; this.health = health; this.sleeper = sleeper; this.clock = clock;
    }

    public void addListener(PaymentListener l) { listeners.add(l); }

    public Payment pay(PaymentRequest req) {
        Instrument instrument = methods.get(req.methodType).prepare(req.details);       // validation first, no side effects
        Payment fresh = new Payment("PAY-" + seq.incrementAndGet(), req, instrument);
        String key = req.merchantId + ":" + req.idempotencyKey;
        Payment existing = byIdempotencyKey.putIfAbsent(key, fresh);
        if (existing != null) {
            if (!existing.fingerprint().equals(fresh.fingerprint()))
                throw new GatewayException("Idempotency key reused with a different request: " + req.idempotencyKey);
            return existing;                                                             // replay: same resource, no new bank call
        }
        byId.put(fresh.id(), fresh);
        process(fresh);
        return fresh;
    }

    private void process(Payment p) {
        p.transition(PaymentStatus.PROCESSING);
        publish(p);
        List<BankConnector> capable = connectors.stream().filter(c -> c.supports(p.request().methodType)).collect(Collectors.toList());
        List<BankConnector> ordered = routing.route(p.request(), p.instrument(), capable, health);
        AuthRequest auth = new AuthRequest(p.id(), p.request().amountMinor, p.instrument());

        for (BankConnector connector : ordered) {
            ConnectorHealth h = health.of(connector.name());
            for (int attempt = 1; attempt <= retry.maxAttempts(); attempt++) {
                if (!h.allowRequest(clock)) { p.addAttempt(connector.name(), "SKIPPED_CIRCUIT_OPEN"); break; }
                if (attempt > 1) sleeper.sleep(retry.delayBefore(attempt));
                AuthResponse resp = connector.authorize(auth);
                p.addAttempt(connector.name(), resp.outcome.name());
                switch (resp.outcome) {
                    case APPROVED:
                        h.recordSuccess(); p.recordRoute(connector.name(), resp.bankRef);
                        p.transition(PaymentStatus.SUCCESS); publish(p); return;
                    case PENDING:                                   // e.g. UPI collect: final status arrives via bank callback
                        h.recordSuccess(); p.recordRoute(connector.name(), resp.bankRef);
                        p.transition(PaymentStatus.PENDING); publish(p); return;
                    case DECLINED:                                  // business decision: do NOT retry or fail over
                        h.recordSuccess(); p.recordRoute(connector.name(), null); p.recordFailure("DECLINED_" + resp.code);
                        p.transition(PaymentStatus.FAILED); publish(p); return;
                    case TECHNICAL_ERROR:                           // request not processed: safe to retry, then fail over
                        h.recordFailure(clock);
                        break;
                }
            }
        }
        p.recordFailure("ALL_ROUTES_FAILED");
        p.transition(PaymentStatus.FAILED);
        publish(p);
    }

    /** Inbound async result from the bank (UPI callback). Idempotent: duplicates/late callbacks are ignored. */
    public Payment onBankCallback(String paymentId, boolean success) {
        Payment p = find(paymentId);
        PaymentStatus target = success ? PaymentStatus.SUCCESS : PaymentStatus.FAILED;
        if (p.transitionIf(PaymentStatus.PENDING, target)) {
            if (!success) p.recordFailure("BANK_CALLBACK_FAILED");
            publish(p);
        }
        return p;
    }

    public Payment refund(String paymentId) {
        Payment p = find(paymentId);
        synchronized (p) {          // per-payment monitor: prevents double refund; the lock is scoped to ONE payment
            if (p.status() != PaymentStatus.SUCCESS) throw new GatewayException("Only SUCCESS payments can be refunded, was " + p.status());
            BankConnector c = connectors.stream().filter(x -> x.name().equals(p.connectorName())).findFirst().orElseThrow();
            if (!c.refund(p.bankRef(), p.request().amountMinor)) throw new GatewayException("Bank refused refund");
            p.transition(PaymentStatus.REFUNDED);
        }
        publish(p);
        return p;
    }

    public Payment find(String paymentId) {
        Payment p = byId.get(paymentId);
        if (p == null) throw new GatewayException("Unknown payment " + paymentId);
        return p;
    }

    private void publish(Payment p) {
        PaymentEvent e = new PaymentEvent(p);
        for (PaymentListener l : listeners) {
            try { l.onEvent(e); } catch (RuntimeException ignored) { /* listener failures never affect the payment */ }
        }
    }
}

5.7 Demo driver

// Main.java
import java.time.Clock;
import java.time.Duration;
import java.util.*;
import java.util.concurrent.*;
import java.util.concurrent.atomic.AtomicInteger;

public class Main {
    static final Map<String, String> VISA = Map.of("pan", "4111 1111 1111 1111", "cvv", "123", "expiry", "12/29");
    static int n;
    static PaymentRequest card(String key, String order, long amt) {
        return new PaymentRequest("M1", order, key, amt, PaymentMethodType.CARD, VISA);
    }

    public static void main(String[] args) throws Exception {
        Clock clock = Clock.systemUTC();
        List<Long> sleeps = new CopyOnWriteArrayList<>();
        Sleeper sleeper = sleeps::add;                                  // record backoff instead of sleeping

        ScriptedConnector hdfc = new ScriptedConnector("HDFC", Set.of(PaymentMethodType.CARD, PaymentMethodType.NET_BANKING),
                () -> AuthResponse.approved("HDFC-REF"));
        ScriptedConnector icici = new ScriptedConnector("ICICI", Set.of(PaymentMethodType.CARD, PaymentMethodType.NET_BANKING),
                () -> AuthResponse.approved("ICICI-REF"));
        ScriptedConnector npci = new ScriptedConnector("NPCI-UPI", Set.of(PaymentMethodType.UPI),
                () -> AuthResponse.pending("UPI-REF"));

        HealthRegistry health = new HealthRegistry(3, Duration.ofSeconds(30));
        PaymentService svc = new PaymentService(PaymentMethodFactory.withDefaults(), List.of(hdfc, icici, npci),
                new StaticPriorityRouting(), new RetryPolicy(2, 100), health, sleeper, clock);

        // merchant webhook endpoint that fails the first two deliveries
        AtomicInteger hookCalls = new AtomicInteger();
        WebhookDispatcher hooks = new WebhookDispatcher(sleeper, 4);
        hooks.register("M1", new WebhookDispatcher.Endpoint("https://merchant.example/hook", "whsec_demo",
                (url, body, sig) -> hookCalls.incrementAndGet() > 2));
        svc.addListener(hooks);

        System.out.println("== 1. transient error on HDFC, retry on same connector succeeds ==");
        hdfc.thenReturn(AuthResponse.technical("TIMEOUT_503"));
        System.out.println(svc.pay(card("k1", "ORD-1", 49900)));

        System.out.println("== 2. hard decline: no retry, no fallback ==");
        hdfc.thenReturn(AuthResponse.declined("51_INSUFFICIENT_FUNDS"));
        System.out.println(svc.pay(card("k2", "ORD-2", 99900)));

        System.out.println("== 3. HDFC down: retries exhausted -> fallback to ICICI ==");
        hdfc.thenReturn(AuthResponse.technical("503"), AuthResponse.technical("503"));
        System.out.println(svc.pay(card("k3", "ORD-3", 15000)));

        System.out.println("== 4. third consecutive HDFC failure (2 in step 3 + 1 now) opens the circuit ==");
        hdfc.thenReturn(AuthResponse.technical("503"));
        System.out.println(svc.pay(card("k4", "ORD-4", 1000)));
        System.out.println(svc.pay(card("k5", "ORD-5", 2000)));          // HDFC skipped entirely
        System.out.println("HDFC calls so far: " + hdfc.calls());

        System.out.println("== 5. smart routing uses live success rate ==");
        PaymentService smart = new PaymentService(PaymentMethodFactory.withDefaults(), List.of(hdfc, icici, npci),
                new SuccessRateRouting(), new RetryPolicy(2, 100), health, sleeper, clock);
        System.out.println(smart.pay(card("k6", "ORD-6", 3000)));

        System.out.println("== 6. idempotency ==");
        int before = icici.calls() + hdfc.calls();
        Payment a = svc.pay(card("k1", "ORD-1", 49900));
        System.out.println("replay returns same payment: " + a.id() + " " + a.status() + ", extra bank calls = " + (icici.calls() + hdfc.calls() - before));
        try { svc.pay(card("k1", "ORD-1", 1)); } catch (GatewayException e) { System.out.println("conflict: " + e.getMessage()); }
        try { svc.pay(new PaymentRequest("M1", "ORD-7", "k7", 100, PaymentMethodType.CARD, Map.of("pan", "4111111111111112", "cvv", "123", "expiry", "12/29"))); }
        catch (PaymentValidationException e) { System.out.println("validation: " + e.getMessage()); }

        System.out.println("== 7. UPI: PENDING then async bank callback (duplicates ignored) ==");
        Payment upi = svc.pay(new PaymentRequest("M1", "ORD-8", "k8", 25000, PaymentMethodType.UPI, Map.of("vpa", "alice@okbank")));
        System.out.println(upi);
        svc.onBankCallback(upi.id(), true);
        svc.onBankCallback(upi.id(), false);                            // late/duplicate conflicting callback: ignored
        System.out.println("after callbacks: " + upi.status());

        System.out.println("== 8. refund ==");
        System.out.println(svc.refund(upi.id()).status());
        try { svc.refund(upi.id()); } catch (GatewayException e) { System.out.println("second refund: " + e.getMessage()); }

        System.out.println("== 9. concurrency: 100 threads, same idempotency key ==");
        PaymentService svc2 = new PaymentService(PaymentMethodFactory.withDefaults(), List.of(icici), new StaticPriorityRouting(),
                new RetryPolicy(2, 100), new HealthRegistry(3, Duration.ofSeconds(30)), sleeper, clock);
        int iciciBefore = icici.calls();
        ExecutorService pool = Executors.newFixedThreadPool(16);
        Set<String> ids = ConcurrentHashMap.newKeySet();
        List<Future<?>> fs = new ArrayList<>();
        for (int i = 0; i < 100; i++) fs.add(pool.submit(() -> ids.add(svc2.pay(card("same-key", "ORD-9", 500)).id())));
        for (Future<?> f : fs) f.get();
        pool.shutdown();
        System.out.println("distinct payments=" + ids.size() + ", bank calls=" + (icici.calls() - iciciBefore));
        System.out.println("backoff delays recorded (ms, retries + webhook redeliveries): " + sleeps.subList(0, Math.min(sleeps.size(), 6)) + " ...");
    }
}

Real output (from java Main)

== 1. transient error on HDFC, retry on same connector succeeds ==
    [webhook] PAY-1 SUCCESS attempt 1 failed
    [webhook] PAY-1 SUCCESS attempt 2 failed
    [webhook] PAY-1 SUCCESS attempt 3 delivered
PAY-1 ORD-1 SUCCESS via HDFC trail=[HDFC:TECHNICAL_ERROR, HDFC:APPROVED]
== 2. hard decline: no retry, no fallback ==
    [webhook] PAY-2 FAILED attempt 1 delivered
PAY-2 ORD-2 FAILED (DECLINED_51_INSUFFICIENT_FUNDS) via HDFC trail=[HDFC:DECLINED]
== 3. HDFC down: retries exhausted -> fallback to ICICI ==
    [webhook] PAY-3 SUCCESS attempt 1 delivered
PAY-3 ORD-3 SUCCESS via ICICI trail=[HDFC:TECHNICAL_ERROR, HDFC:TECHNICAL_ERROR, ICICI:APPROVED]
== 4. third consecutive HDFC failure (2 in step 3 + 1 now) opens the circuit ==
    [webhook] PAY-4 SUCCESS attempt 1 delivered
PAY-4 ORD-4 SUCCESS via ICICI trail=[HDFC:TECHNICAL_ERROR, HDFC:SKIPPED_CIRCUIT_OPEN, ICICI:APPROVED]
    [webhook] PAY-5 SUCCESS attempt 1 delivered
PAY-5 ORD-5 SUCCESS via ICICI trail=[HDFC:SKIPPED_CIRCUIT_OPEN, ICICI:APPROVED]
HDFC calls so far: 6
== 5. smart routing uses live success rate ==
PAY-1 ORD-6 SUCCESS via ICICI trail=[ICICI:APPROVED]
== 6. idempotency ==
replay returns same payment: PAY-1 SUCCESS, extra bank calls = 0
conflict: Idempotency key reused with a different request: k1
validation: Invalid card number
== 7. UPI: PENDING then async bank callback (duplicates ignored) ==
PAY-8 ORD-8 PENDING via NPCI-UPI trail=[NPCI-UPI:PENDING]
    [webhook] PAY-8 SUCCESS attempt 1 delivered
after callbacks: SUCCESS
== 8. refund ==
    [webhook] PAY-8 REFUNDED attempt 1 delivered
REFUNDED
second refund: Only SUCCESS payments can be refunded, was REFUNDED
== 9. concurrency: 100 threads, same idempotency key ==
distinct payments=1, bank calls=1
backoff delays recorded (ms, retries + webhook redeliveries): [100, 100, 200, 100] ...

Reading the output: step 5 shows the smart service (a separate instance, so ids restart at PAY-1) choosing ICICI first because HDFC's live success rate is lower. In step 4 note HDFC:TECHNICAL_ERROR, HDFC:SKIPPED_CIRCUIT_OPEN - the breaker opened mid-payment on the third consecutive failure so the second retry was skipped.

6. Walkthrough of key flows

pay() happy path with failover
Merchant     PaymentService        PaymentMethod   Idempotency   Routing     HDFC      ICICI     Listeners
 |--pay(req)-->|                        |               |            |          |         |          |
 |             |--prepare(details)----->| validate, mask               |          |         |          |
 |             |--------------------putIfAbsent(merchant:key)-->| (null => I own it)       |          |
 |             |-transition CREATED->PROCESSING ------------------------------------------------>| event
 |             |--route(capable)------------------------->| [HDFC, ICICI]
 |             |--authorize--------------------------------------->| TECHNICAL_ERROR
 |             |-backoff 100ms, retry---------------------------->| TECHNICAL_ERROR
 |             |-(attempts exhausted: fail over)
 |             |--authorize----------------------------------------------->| APPROVED
 |             |-transition PROCESSING->SUCCESS ----------------------------------------------->| event -> webhook
 |<--Payment---|
UPI async
 pay() -> connector returns PENDING -> state PENDING (HTTP 202 to merchant)
 bank/NPCI POSTs callback -> onBankCallback(id, success)
      payment.transitionIf(PENDING, SUCCESS|FAILED)   // atomic; false => duplicate/late, ignored
      -> publish event -> webhook to merchant
Webhook delivery
 event(SUCCESS) -> body JSON + HMAC_SHA256(secret, body) in header -> POST
   fail -> sleep 100ms -> retry -> sleep 200ms -> retry ... up to N -> dead-letter queue (manual/automatic replay)
 merchant verifies signature and dedupes on paymentId+status (at-least-once delivery)

Retry/failover decision table (the core insight)

Bank answerMeaningAction
APPROVEDmoney movedSUCCESS, stop
DECLINED (insufficient funds, wrong CVV, blocked card)business decision, same on any bankFAILED, no retry, no failover (would annoy customer/risk fraud flags)
TECHNICAL_ERROR (connect refused, 503 before processing)request not processedretry w/ backoff, then fail over
PENDINGaccepted, result laterPENDING, wait for callback / poll status
TIMEOUT (not modelled)unknown whether processednever blind retry: status-inquiry/reverse first (see follow-ups)

7. Extensibility

  • New payment method (BNPL, EMI, crypto): implement PaymentMethod, factory.register(...), add enum constant, make a connector report supports(...). PaymentService unchanged.
  • New bank/acquirer: implement BankConnector (Adapter) and add to the list. Capability matching is by supports().
  • New routing rule (by card network, amount slab, MDR cost, merchant preference, A/B experiment): implement RoutingStrategy; compose with a decorator new CostAwareRouting(new SuccessRateRouting()). Instrument carries network and bank attributes for this purpose.
  • New listener (fraud analytics, ledger posting, email): addListener.
  • New retry policy (jitter, per-method budgets, retry budget per merchant): swap RetryPolicy.
  • New terminal/intermediate status (AUTHORIZED -> CAPTURED for auth/capture flows, PARTIALLY_REFUNDED): add to PaymentStatus.next(); other code only reacts to statuses it cares about.

8. Concurrency and edge cases

  • Idempotency: putIfAbsent on merchantId:key -> exactly one thread runs the payment; duplicates receive the same Payment (demo: 100 threads, 1 payment, 1 bank call). Key scoped per merchant. Fingerprint mismatch -> conflict error. In production persist the key with a UNIQUE constraint and TTL.
  • Lock granularity: per-Payment monitor for transitions (transition, transitionIf); everything else is lock-free (ConcurrentHashMap, atomics in ConnectorHealth). Unrelated payments never contend; a global lock would serialise the gateway.
  • Callbacks race with pay(): transitionIf(PENDING, ...) is a CAS-style transition: duplicates, conflicting late callbacks, or a callback arriving before PENDING was recorded (it simply does nothing; the reconciler/poller resolves).
  • Refund holds the payment's own monitor while calling the bank to prevent double refund; trade-off: IO under a lock, scoped to a single payment. Production alternative: add REFUND_INITIATED state and CAS into it, then call the bank outside the lock.
  • Listeners run outside locks and exceptions are swallowed: a dead merchant endpoint must never fail a payment. In production publish to a queue/outbox and let workers deliver (the sync loop here would block the request thread).
  • Retry storms: exponential backoff + jitter + per-connector circuit breaker + global retry budget.
  • Edge cases: invalid PAN (Luhn), expired card, amount <= 0, unsupported method, no capable connector (-> FAILED ALL_ROUTES_FAILED), all circuits open, duplicate callbacks, callback for unknown payment, merchant without webhook endpoint, clock skew on breaker (injected Clock), webhook signature replay (add timestamp to signed body).

9. Follow-up questions and model answers

  1. How do you avoid a double charge when the bank call times out? A timeout is unknown. Do not retry the authorize blindly. Mark the payment PENDING, call the bank's status inquiry API (with our reference), or send a reversal/void (idempotent by our txn id) before trying another bank; a reconciliation job closes anything still PENDING after N minutes using the bank's settlement/report files. Also send a stable paymentId to the bank as the merchant reference so the bank can dedupe.
  2. Why not fail over on a decline? A decline is the issuer's decision (insufficient funds, risk). Another acquirer gets the same answer; extra attempts increase fraud-score risk and may double-hold funds.
  3. How do you persist this? payment(id, merchant_id, order_id, idempotency_key, request_hash, amount, method, status, connector, bank_ref, version, created_at) with UNIQUE(merchant_id, idempotency_key); payment_attempt(payment_id, seq, connector, outcome, code, latency_ms, ts); webhook_delivery(id, payment_id, status, attempts, next_retry_at); outbox(event). State changes use optimistic locking: UPDATE payment SET status=? , version=version+1 WHERE id=? AND status=? (rows==1) - this is the DB form of transitionIf.
  4. Make it distributed / horizontally scalable? Stateless gateway nodes behind a load balancer; idempotency in DB/Redis (SET NX + DB unique constraint); circuit-breaker state local per node (cheap, converges) or shared via Redis for global view; success-rate metrics from a streaming aggregation (sliding window) feeding the router; callbacks routed to any node (they act through the DB CAS). Partition payments by merchantId or paymentId for hot-key isolation.
  5. Exactly-once webhooks? Impossible; provide at-least-once with a stable event id and signature; merchants dedupe. Use transactional outbox so an event exists iff the state change committed.
  6. How is the routing decision made smarter? Sliding-window success rate per (connector, method, issuer bank/BIN, amount band); cost (MDR) weighted score; exploration (epsilon-greedy / 5% traffic) so a recovering bank gets sampled; manual overrides per merchant; kill switches.
  7. Authorize vs capture? Add AUTHORIZED state and a capture operation with expiry; refunds partial amounts tracked by a refunded_amount and a cap check under CAS.
  8. Security? Never persist PAN/CVV (PCI-DSS); tokenise via a vault; TLS + mTLS to banks; HMAC-signed webhooks with timestamp; API auth per merchant; mask in logs (Instrument.maskedRef).
  9. How to test? Scripted connectors (as in the demo), injected Clock and Sleeper, property test on the state machine (random event sequences never reach illegal states), concurrency test on same key.
  10. Where do wallets/ledger fit? On SUCCESS, a listener posts a ledger entry (merchant payable, MDR fee) - the double-entry design from the wallet problem; settlement later pays the merchant.

10. Common mistakes, red flags, rubric

Red flags

  • One giant processPayment() with if (method == CARD) ... else if (UPI).
  • Retrying everything, including declines; failing over after an ambiguous timeout without status inquiry.
  • No idempotency, or idempotency with containsKey/put race; no payload-mismatch check.
  • Status as free String/boolean flags; allowing SUCCESS -> FAILED.
  • Doing webhook HTTP inside the locked section / failing the payment when the webhook fails.
  • Storing raw card number/CVV; logging full PAN.
  • Thread.sleep hard-wired (untestable); no backoff or infinite retry loops.
  • Assuming synchronous bank responses only (no PENDING/callback story for UPI).

Mini rubric (25 pts)

AreaPoints
Requirements: async, idempotency, retries vs declines clarified3
Strategy/Factory for methods; Adapter for banks4
Routing + retry + fallback semantics (decline vs technical), circuit breaker5
State machine with guarded transitions + callbacks4
Idempotency and thread-safety reasoning (lock granularity)4
Webhooks via Observer: signing, retry, DLQ2
Working demo + persistence/distributed follow-ups3