ICEGATE retries and the idempotency key that doesn't exist

If you’ve ever spent time in a customs brokerage office, you’ve noticed the second screen. There’s the broker’s primary software — a desktop application from a vendor whose UI peaked sometime around 2008 — and there’s a notebook, or a printed sheet, or an Excel file open on a second monitor. The broker types into the software, and they also update the notebook. Always both.

You can ask them why. You’ll get one of two answers. The polite version is “habit.” The honest version is “because last month the software said the filing went through, and the customs officer said it didn’t, and I had to spend three days reconstructing what happened, and now I keep a list.”

This isn’t a UX problem. It’s an idempotency problem. And it’s structural — it lives in the public ICEGATE API itself, which means every software product that builds on top of it inherits the issue. Customs Pro is no exception. The difference is whether you pretend the problem isn’t there or design the entire system around the fact that it is.

The failure mode

The classic ICEGATE filing flow, at the API level, looks roughly like this:

1. Your software constructs a Bill of Entry (BoE) or Shipping Bill payload.
2. You POST it to ICEGATE’s filing endpoint.
3. ICEGATE responds with an acknowledgement and, eventually, an ICEGATE reference number (sometimes called the BE number for imports).
4. You record the reference number against the consignment in your database. Done.

The failure isn’t in the happy path. The failure is in what happens when step 2 fails to return cleanly. The TCP connection times out. Your client never sees the response. The customs server may or may not have processed the filing.

You don’t know which.

You can’t retry blindly. If the server processed it, your retry creates a duplicate filing — and now you have two BoEs for one consignment, which is its own enforcement event when an inspector eventually notices. You also can’t pretend it didn’t happen. The consignment is sitting at the port; if no filing reaches customs, it gets stuck.

The textbook answer to this problem in distributed systems is the idempotency key. Stripe, AWS, Razorpay, every modern payment processor — they accept a client-generated unique key on writes, and the server guarantees that if the same key arrives twice, the operation is performed exactly once. Your client retries freely; the server deduplicates. You get exactly-once semantics over an at-least-once network.

ICEGATE does not accept idempotency keys.

What ICEGATE does instead

What you get is eventually consistent reconciliation. ICEGATE accepts the filing. ICEGATE assigns a server-side reference at some point after acceptance. ICEGATE exposes a separate query endpoint that returns the status of filings against your IEC (Importer-Exporter Code) and a date range.

If your filing POST times out, the recovery procedure is:

1. Wait some indeterminate amount of time (minutes to hours, depending on system load).
2. Query the status endpoint for your IEC.
3. Search the response for a filing that matches the consignment you tried to file.
4. Match it by — what, exactly? Your client never recorded an ICEGATE reference, because that’s what you didn’t receive.

The matching is the hard part. ICEGATE returns filings keyed by the reference it generated. To match a record back to your client-side intent, you have to compare on the payload fields you submitted: invoice number, port code, IEC, item descriptions. That comparison is fuzzy by design — customs systems normalize whitespace, change case, sometimes transcode characters. Two filings that look identical to a broker can differ in the bytes ICEGATE returns.

So what brokers do, in practice, is keep a paper ledger. Every filing attempt, with timestamp, with invoice number, with the consignment ID. When the software loses sync with ICEGATE, the ledger is what they reconcile against. The ledger is the idempotency record the API doesn’t provide.

How to design around a missing contract

You have two choices. You can write software that pretends ICEGATE behaves like a modern API and breaks every time the broker has a connectivity blip. Or you can build the reconciliation layer that ICEGATE doesn’t provide, into your own product.

Customs Pro takes the second path. Three design decisions follow from that:

First, every filing attempt is durable before the network call. When the broker hits “File this BoE,” Customs Pro writes a filing-attempt record to the local database with a client-side UUID, the full payload, the user, and a timestamp. Only then does it call ICEGATE. If the call fails to return, the attempt record persists. The system never has a “we tried but I don’t know what we sent” state.

Second, reconciliation is a first-class background process. A scheduled job polls ICEGATE’s status endpoint per active IEC, fetches recent filings, and attempts to match them to filing-attempt records by the fields available. Matches close the loop and stamp the ICEGATE reference onto the filing-attempt. Non-matches stay open and surface in a “needs human reconciliation” queue.

Third, the human-reconciliation queue is treated as a feature, not an edge case. Most products that integrate with messy APIs treat the “couldn’t auto-reconcile” cases as bugs. They surface them in error logs that nobody reads. Customs Pro treats them as a UI surface — a screen the broker checks every morning, with the filing-attempt on one side, the candidate ICEGATE matches on the other side, and a one-click “yes that’s the same filing” action. The product replaces the paper ledger, instead of competing with it.

Why this matters past ICEGATE

ICEGATE isn’t unique. Most regulator-facing APIs in India and across emerging markets share the same shape — they were designed by procurement-driven enterprise integrations, not by people thinking about distributed-systems failure modes. GST’s old filing API had similar issues before they shipped the GSTN reference key. CAAC’s portal for aviation operators has its own version. The DGFT EXIM portal. Every state-level commercial-tax portal.

If you’re building software that integrates with these systems, the temptation is to write thin wrappers and hope the network behaves. It will not. The wrapper-only approach is exactly why most regulatory-integration software has a “this didn’t sync, please call support” footer.

The discipline is to treat the missing idempotency contract as part of your system, not part of your bug list. Every write to a regulator-facing API needs:

• A durable client-side record before the call.
• A reconciliation worker after the call.
• A human-resolvable queue when reconciliation can’t match.
• A UI that treats unresolved attempts as data, not exceptions.

That’s a non-trivial amount of code. It’s a non-trivial amount of UX. It’s the part most products skip, because it doesn’t show up in a feature checklist and doesn’t demo well. It’s also the part that determines whether your customer keeps the second screen open.

Where we are

Customs Pro is in scoping; the ICEGATE reconciliation layer is one of the foundational pieces. We expect to learn things during the first pilot that will reshape the design — the fuzzy-matching heuristics specifically are going to need real broker workflows to calibrate against, not architecture documents.

If you operate in this space and have stories about ICEGATE reconciliation failures — or about other regulator APIs that share this shape — we’d be glad to hear them. The product gets better the more failure modes we see before we ship.