Bulk operations

The Pivotal API doesn’t expose bulk endpoints (e.g. there is no POST /customers/bulk that takes an array). The reason: bulk endpoints either lie about partial-failure handling, or push the complexity into a job queue you can’t see into.

For now, fan out from your side. The patterns below handle 95% of bulk imports — backfills, migrations, periodic refreshes.

Sizing the work

Two numbers matter:

• Rate limit: 60 requests/min on a default key, 600/min on paid. Each bulk operation is N requests, where N is your record count.
• Worker concurrency: how many requests you have in flight at once.

A safe default for a 60/min key: concurrency of 4, with each task waiting briefly between requests. That keeps you well under the bucket while staying fast enough to finish 5,000 records in ~85 minutes.

For 600/min: concurrency of 10 finishes 5,000 records in under 10 minutes.

A bulk import script

1
import pLimit from "p-limit";
2
3
interface SourceCustomer { name: string; slug: string; domain: string; external_id: string }
4
5
const SOURCE: SourceCustomer[] = await loadFromCsv("./customers.csv");
6
const CHECKPOINT_FILE = "./bulk-import.checkpoint";
7
8
// Resume support: skip records we've already pushed.
9
const done = new Set<string>(
10
  (await Bun.file(CHECKPOINT_FILE).text().catch(() => "")).split("\n").filter(Boolean),
11
);
12
13
const limit = pLimit(4); // concurrency
14
let errors = 0;
15
16
async function importOne(c: SourceCustomer) {
17
  if (done.has(c.external_id)) return;
18
19
  const res = await fetch("https://my.pivotal.app/api/v1/customers", {
20
    method: "POST",
21
    headers: {
22
      Authorization: `Bearer ${process.env.PIVOTAL_API_KEY}`,
23
      "Content-Type": "application/json",
24
      "Idempotency-Key": `bulk-import-${c.external_id}`,
25
    },
26
    body: JSON.stringify({
27
      name: c.name,
28
      slug: c.slug,
29
      domain: c.domain,
30
      status: "onboarding",
31
      metadata: { external_id: c.external_id },
32
    }),
33
  });
34
35
  if (res.status === 429) {
36
    const wait = Number(res.headers.get("Retry-After") ?? 5);
37
    await new Promise((r) => setTimeout(r, wait * 1000));
38
    return importOne(c); // retry
39
  }
40
41
  const body = await res.json();
42
43
  if (!res.ok && body.error?.code !== "slug_taken") {
44
    errors += 1;
45
    console.error(`✗ ${c.external_id}:`, body.error?.code, body.error?.message);
46
    return;
47
  }
48
49
  done.add(c.external_id);
50
  await Bun.write(CHECKPOINT_FILE, [...done].join("\n"));
51
  console.log(`✓ ${c.external_id} → ${body.id}`);
52
}
53
54
await Promise.all(SOURCE.map((c) => limit(() => importOne(c))));
55
console.log(`Done. ${done.size} succeeded, ${errors} failed.`);

Three things to call out:

1. Idempotency-Key lets you re-run after a crash without duplicating records.
2. The checkpoint file is a flat-text list of external_ids. Crude but enough for a one-shot script. For long-running services, use Redis or a Postgres row.
3. slug_taken is accepted as a non-error — it means the customer was created in a previous run.

Why per-record Idempotency-Key

If you send the same body to POST /customers twice without a key, you get 409 slug_taken on the second attempt. With a key, you get 200 OK with the original response — which lets the script keep moving and treat it as “already done”.

Throttling against X-RateLimit-Remaining

The script above retries on 429. You can do better — slow down before you hit zero:

1
async function importOne(c: SourceCustomer) {
2
  const res = await fetch(/* ... */);
3
4
  const remaining = Number(res.headers.get("X-RateLimit-Remaining") ?? 100);
5
  if (remaining < 5) await new Promise((r) => setTimeout(r, 1000));
6
7
  // ... rest of the handler
8
}

This drops your peak throughput slightly but eliminates 429 round trips entirely.

What to do about contacts and onboardings

Bulk-importing customers is the main case. If you also need to attach contacts and onboardings, fan them out the same way after the customer pass finishes:

1
// Pass 1: customers
2
await Promise.all(SOURCE.map((c) => limit(() => importCustomer(c))));
3
4
// Pass 2: contacts (now we have Pivotal customer ids)
5
const idMap = await loadCheckpoint("./bulk-import.checkpoint.contacts");
6
await Promise.all(SOURCE.flatMap((c) =>
7
  c.contacts.map((contact) => limit(() => importContact(c.external_id, contact))),
8
));
9
10
// Pass 3: onboardings (last — they read back the customer state)
11
await Promise.all(SOURCE.map((c) => limit(() => importOnboarding(c))));

Three passes is more wall time but easier to reason about than interleaving. The total request count is the same.

Reporting

After the script finishes, eyeball:

• Checkpoint count vs source count — should match.
• Errors — anything that isn’t slug_taken, look at the error and decide.
• The Pivotal API Keys page — confirms the request count tracks what your script logged.

What NOT to do

• Don’t disable retries. Network blips are real. The script above retries automatically on 429; add retries on transient 5xx too.
• Don’t share keys across bulk jobs. One key per bulk run makes the API Keys page a useful audit trail.
• Don’t fan out without a concurrency limit. Promise.all over 5,000 records means 5,000 in-flight requests. You’ll trip the rate limit and the OS file descriptor limit before you trip anything else.
• Don’t import then mutate the same row in the same pass. Two requests against the same record concurrently is a race. Let pass 1 finish before mutating.