BulkForge · Case study · October 2026
A rate limit that only a faster release exposed.
A customer’s bulk email jobs started failing right after I shipped a faster agent. The speedup had removed an accidental safety margin. This is how I found the real constraint, what I got wrong along the way, and the trade-offs in the fix.
Context
BulkForge is hybrid by design: a hosted control plane coordinates jobs, while an agent deployed in the customer’s Azure tenant does the sending through their own Azure Communication Services (ACS) resource. Credentials never leave the customer’s tenant, and the customer owns the provider subscription and its quotas.
- Dashboard
- Control plane
- Customer agent polls
- Customer ACS
- Recipients
Symptom
After the customer installed the new agent, the first four jobs (30 recipients each) completed in one to three minutes. The fifth job had one accepted recipient and twenty-nine failures. The next eight jobs failed on every recipient with a 429 and the message “Not sent: rate limit reached; retry this recipient later.” In total, 269 recipients were certainly not sent.
Diagnosis, including the wrong turn
My first read was that ACS was not throttling at all, because the previous agent had never produced a 429. That was wrong, and the reason is the useful part: the old agent was so slow that it could not send fast enough to reach the limit.
ACS custom domains default to 30 messages per minute and 100 per hour per subscription. I had been reasoning about the per-minute pacing. Pulling hourly counts of confirmed sends showed the old agent never exceeded 90 in any hour, and the customer’s real pattern was about five jobs of 30, or roughly 150 per hour. The binding constraint was the hourly quota, and the old agent’s slowness had been hiding it. A faster agent was the correct improvement and it exposed a latent limit.
There was also a communication error to correct. I had told the customer that Azure rejected 269 messages. The audit rows showed the failures were my own agent’s fail-fast behavior after it received a throttling signal, not 269 separate provider rejections. I sent a correction and fixed the agent.
Decisions and trade-offs
Cap sends at 90 per hour, below the 100 per hour provider quota
Why A hard budget in the control plane means the agent stays under the limit, so nothing is rejected and nothing needs retrying. The margin below 100 leaves headroom instead of running at the edge.
Cost Throughput is deliberately lower than the customer wanted (about 150 per hour) until their quota increase is approved. I chose predictable and complete over fast and partial.
Re-send failed recipients as new jobs behind live traffic
Why Re-sends run as clone jobs that the governor schedules only into spare budget, so live customer jobs always outrank recovery work.
Cost Recovery took most of a night instead of an hour.
Do not re-drive old stuck jobs that cannot be proven undelivered
Why The re-send targeted only recipients with an audit row showing they were rejected. Older stuck jobs may have been partly delivered with no per-recipient record, so re-driving them risked emailing people twice.
Cost Some old jobs stay unresolved. A duplicate email to a real recipient is worse than a missing one that the customer can see and decide on.
Make the limits editable by the customer, with safe defaults
Why The customer owns the provider subscription, so only they know their real quota. The limits live in the dashboard (defaults 90 per hour, 25 per minute) and the governor learns a lower cap if it ever sees real throttling.
Cost A customer can set a limit too high. The learned cap steps down 25% on real rejections, with a floor at half, and ignores the agent’s own fail-fast errors so it cannot spiral.
How I verified it
I built a Docker test rig that runs the real control plane and the real agent against fake ACS, Postmark and SendGrid endpoints. It reproduces the production pattern (150 offered per hour against a cap of 100) with the governor off, then on, then adds the rate-limited re-send. The expected result, and the observed one, is that every recipient is delivered exactly once. The rig proves logic and failure handling; it uses fake providers, so it says nothing about real ACS latency.
The agent release (v3.3.0) also added retry with backoff on 429 for the Postmark and SendGrid channels, a fresh throttle budget per batch, verification of ACS delivery status instead of trusting acceptance, and smaller progress-report chunks so a killed job records partial progress.
Outcome
- 331 recipients re-sent as 12 clone jobs: all completed, none failed, and none delivered twice. I verified this directly in the database rather than from dashboards.
- The governor held at or below 90 accepted sends in every hour overnight, with no learned cap triggered and no provider rejections.
- The customer reports installing the fixed agent, and the dashboard now lets them set their own sending limits.
What is still open
The durable fix on the customer side is an ACS quota increase, which they must request. Until then the system stays under the current cap by design. I am also building automatic staged retries for provider-rejected recipients over a 24-hour window, so a future overflow recovers without manual re-sends. That work is not shipped yet.