Production & deployment

Running the Integrations SDK in production requires attention to auth token lifecycle, webhook security, encryption key management, and tenant isolation.

OAuth in production

Signed state

Always use cryptographically random state parameters to prevent CSRF:

import { randomBytes } from "node:crypto";

function generateState(): string {
  return randomBytes(32).toString("base64url");
}

// Store in session/cache with expiry
const state = generateState();
await cache.set(`oauth:state:${state}`, { userId, redirectUrl }, { ex: 600 });

// Redirect to provider
const url = `${oauthConfig.authUrl}?client_id=${clientId}&state=${state}&...`;

Validate the state on callback:

const cached = await cache.get(`oauth:state:${callbackState}`);
if (!cached || cached.userId !== currentUserId) {
  throw new Error("Invalid or expired OAuth state");
}

PKCE

Use PKCE for all public clients (SPA, mobile) and as defense-in-depth for server-side flows:

import { randomBytes, createHash } from "node:crypto";

const codeVerifier = randomBytes(32).toString("base64url");
const codeChallenge = createHash("sha256")
  .update(codeVerifier)
  .digest("base64url");

// Send codeChallenge in auth URL
// Send codeVerifier in token exchange

Plugins that support PKCE (Airtable, etc.) set pkce: true in their OAuthConfig.

Refresh tokens

Store refresh tokens encrypted at rest. Rotate them on use if the provider supports it:

// Token response
interface TokenResponse {
  access_token: string;
  refresh_token?: string;
  expires_at?: number;
  scope?: string;
}

// Encrypt before storing
const encryptedRefresh = await encrypt(refreshToken, kek);
await db.tokens.create({
  userId,
  provider: "gmail",
  accessToken: encryptedAccess,
  refreshToken: encryptedRefresh,
  expiresAt: new Date(expiresAt * 1000),
});

The plugin's keyBuilder decrypts the token at call time:

keyBuilder: async (ctx, source) => {
  const token = await db.tokens.findFirst({
    where: { userId: ctx.userId, provider: "gmail" },
  });
  if (!token) return "";
  
  // Refresh if expired
  if (token.expiresAt < new Date()) {
    const refreshed = await refreshGmailToken(token.refreshToken);
    await db.tokens.update({ where: { id: token.id }, data: refreshed });
    return refreshed.accessToken;
  }
  
  return await decrypt(token.accessToken, kek);
},

The Integrations SDK uses envelope encryption: a Key Encryption Key (KEK) encrypts per-credential Data Encryption Keys (DEKs). Rotate the KEK on a schedule (e.g., quarterly) or on employee offboarding.

Rotation workflow

// 1. Generate new KEK
const newKek = await generateKek();

// 2. Re-encrypt all DEKs with new KEK
const credentials = await db.credentials.findMany();
for (const cred of credentials) {
  const dek = await decrypt(cred.encryptedDek, oldKek);
  const reencryptedDek = await encrypt(dek, newKek);
  await db.credentials.update({
    where: { id: cred.id },
    data: { encryptedDek: reencryptedDek },
  });
}

// 3. Update active KEK reference
await db.systemConfig.update({
  where: { key: "active_kek" },
  data: { value: newKek.id },
});

// 4. Retire old KEK after validation period

Use wrapKeyStoreWithKMS to delegate KEK management to AWS KMS, GCP Cloud KMS, or Azure Key Vault:

import { wrapKeyStoreWithKMS } from "@fabricorg/integrations/kms";

const keyStore = wrapKeyStoreWithKMS({
  provider: "aws",
  keyId: "arn:aws:kms:us-east-1:123456789:key/...",
  region: "us-east-1",
});

Webhook endpoint setup

Signature verification

Every webhook request must be verified before processing. Each plugin exports its verification function:

import { verifyStripeSignature } from "@fabricorg/integrations/plugins";
import { verifyResendSignature } from "@fabricorg/integrations/plugins";
import { verifyAsanaSignature } from "@fabricorg/integrations/plugins";

// Stripe
const valid = verifyStripeSignature({
  rawBody: req.body,
  signatureHeader: req.headers["stripe-signature"],
  secret: process.env.STRIPE_WEBHOOK_SECRET,
});

// Resend (Svix)
const valid = verifyResendSignature({
  rawBody: req.body,
  msgId: req.headers["svix-id"],
  timestamp: req.headers["svix-timestamp"],
  signatureHeader: req.headers["svix-signature"],
  secret: process.env.RESEND_WEBHOOK_SECRET,
});

Replay protection

All verification functions include timestamp tolerance (default 5 minutes). Reject requests outside the window:

if (Math.abs(now - timestamp) > 5 * 60 * 1000) {
  throw new Error("Webhook timestamp outside tolerance window");
}

For additional replay protection, maintain an idempotency cache:

// Stripe event IDs are unique per event
const eventId = payload.id;
const seen = await cache.get(`webhook:stripe:${eventId}`);
if (seen) return { status: 200, body: "Already processed" };

await cache.set(`webhook:stripe:${eventId}`, "1", { ex: 86400 });

Webhook routing

Use the plugin's pluginWebhookMatcher to route incoming webhooks:

const fabric = createFabric({
  plugins: [stripe({ apiKey, webhookSecret }), resend({ apiKey, webhookSecret })],
});

// Fast path: route by header sniffing
for (const plugin of fabric.plugins) {
  if (plugin.pluginWebhookMatcher?.(request)) {
    return await fabric.processWebhook(plugin.id, request);
  }
}

Multi-tenant deployment patterns

Tenant-isolated credentials

Store credentials per-tenant (user or organization) and resolve them in keyBuilder:

keyBuilder: async (ctx, source) => {
  // ctx.tenantId is injected by the runtime
  const creds = await db.credentials.findFirst({
    where: {
      tenantId: ctx.tenantId,
      provider: "slack",
    },
  });
  return creds?.accessToken ?? "";
},

Per-tenant permission overrides

Different tenants may need different permission modes:

slack({
  permissions: {
    mode: "strict",
    overrides: {
      "messages.send": "require_approval",
      "channels.archive": "deny",
    },
  },
});

Database-per-tenant

For strict isolation, use separate credential stores per tenant:

function getCredentialStore(tenantId: string) {
  return db.$extends({
    query: {
      credentials: {
        async findMany({ args, query }) {
          args.where = { ...args.where, tenantId };
          return query(args);
        },
      },
    },
  });
}

Error handling and retry configuration

Built-in retry strategies

The SDK provides these retry strategies:

Strategy	Behavior	Use case
`linear_1s`	Retry after 1s, 2s, 3s...	Transient network errors
`linear_2s`	Retry after 2s, 4s, 6s...	Moderate rate limits
`linear_3s`	Retry after 3s, 6s, 9s...	Aggressive rate limits
`linear_4s`	Retry after 4s, 8s, 12s...	Very aggressive limits
`exponential_backoff`	1s, 2s, 4s, 8s...	General-purpose
`exponential_backoff_jitter`	Exponential + random jitter	Best for thundering herd

Per-plugin error handlers

Configure retry behavior per error category:

import { AllErrors } from "@fabricorg/integrations";

stripe({
  errorHandlers: {
    RATE_LIMIT_ERROR: {
      match: (err) => err.message.includes("rate_limit"),
      handler: async (err, ctx) => {
        const delay = err.headers?.["retry-after"] ?? 60;
        await sleep(delay * 1000);
        return { retry: true };
      },
    },
    NETWORK_ERROR: {
      match: () => true,
      handler: async () => {
        await sleep(1000);
        return { retry: true, strategy: "exponential_backoff_jitter" };
      },
    },
    AUTH_ERROR: {
      match: (err) => err.status === 401,
      handler: async (err, ctx) => {
        // Attempt token refresh
        const newToken = await refreshToken(ctx);
        return { retry: true, newKey: newToken };
      },
    },
  },
});

Error taxonomy

Error	Status	Retry?	Typical cause
`AUTH_ERROR`	401	Sometimes	Expired token, revoked access
`RATE_LIMIT_ERROR`	429	✅	Provider rate limit
`NETWORK_ERROR`	—	✅	DNS, TCP, TLS failure
`TIMEOUT_ERROR`	—	✅	Request timeout
`PERMISSION_ERROR`	403	❌	Insufficient scopes
`NOT_FOUND_ERROR`	404	❌	Resource doesn't exist
`VALIDATION_ERROR`	400	❌	Invalid parameters
`DEFAULT`	—	❌	Unknown error

Circuit breaker pattern

For high-volume integrations, wrap Fabric calls in a circuit breaker:

import { CircuitBreaker } from "opossum";

const breaker = new CircuitBreaker(
  (args) => fabric.slack.api.messages.send(args),
  {
    timeout: 5000,
    errorThresholdPercentage: 50,
    resetTimeout: 30000,
  },
);

breaker.on("open", () => console.warn("Slack circuit opened"));
breaker.on("halfOpen", () => console.warn("Slack circuit half-open"));

// Use in tool handler
const result = await breaker.fire({ channel: "C01", text: "Hello" });

Environment checklist

Before deploying to production: