How to Run a CIAM Proof of Concept

Key takeaways

• A feature matrix tells you what a vendor claims. A POC tells you what the vendor shipped. Run tests, not demos.
• Test SCIM against both major clients. Entra and Okta send different PATCH shapes for the same intent, and supporting one is not supporting SCIM.
• Revoke the SCIM token and time how long until something alarms. The honest answer is often never, and that is a finding rather than a disqualification.
• Request your data and your password hashes back during the POC, while you still have leverage.
• Ask whether an AI agent can hold its own identity or has to reuse a user's token. The answer dates the platform's architecture.
• Score against pre-written pass criteria written before the first demo.

Why a feature matrix is not enough

The shortlist is the input, not the output. Start here and the key concepts for choosing a vendor get you to two or three candidates. This guide is what you do with them. If you have not written requirements yet, do that first: B2B authentication requirements and B2C authentication requirements are the testable lists these tests are drawn from.

Two rules before the first call.

Write the pass criteria first. Before any vendor engineer joins a shared channel, write down what each test must produce to count as a pass. Criteria written afterwards get adjusted to fit the platform you have already started liking.

Run two vendors in parallel. Sequential POCs compare a platform you have spent two weeks learning against a platform you have spent two hours learning. That is not a comparison.

Test 1: SCIM conformance against both clients

The operations, run once per client:

1. Create a user, and confirm the resulting local record and its attribute mapping.
2. Update a user's name and email, and confirm both propagate.
3. Deactivate a user using both PATCH shapes. One sends {"op":"replace","path":"active","value":false}, the other sends {"op":"replace","value":{"active":false}}. RFC 7644 permits both. Confirm each results in a deactivated local user, not a 400.
4. Reactivate the same user.
5. Add a user to a group, and confirm the local role or membership change.
6. Remove a user from a group. Removal is handled less reliably than addition in most implementations, and it is the one that matters for access revocation.
7. Rename a group on the IdP side, and confirm the mapping survives. Implementations that key on group display name rather than on the immutable identifier break here.
8. Delete a user, and confirm the local state matches your intended semantics rather than the vendor's default.

Record what each operation produced locally, not just the HTTP status. A 204 with no local effect is the failure mode you are looking for. SCIM provisioning covers what correct behaviour looks like for each.

Test 2: revoke the token and wait

Repeat the same test three more ways, because they fail differently:

• Expire a certificate on a SAML connection and see whether anything warns you before the first failed login.
• Change an attribute mapping on the IdP side so a required field stops arriving, then watch whether new users are created with missing data or rejected.
• Stop the sync entirely by disabling the provisioning app, and check whether the platform can tell you that a tenant has gone quiet.

Then ask the vendor directly: what event do we receive, on what channel, when provisioning stops working for a customer? A precise answer with an event name is a platform built to be operated. A reassurance that "you would see it in the logs" means you are building the detector, and that detector is the zero-event alarm described in the observability work on silent identity failures.

Test 3: session and revocation behaviour

Alongside the timing, establish:

• Whether sessions can be revoked per user, per device, per tenant, and globally.
• Whether refresh tokens rotate, and what happens on reuse detection.
• Whether the platform supports sender-constrained tokens, and on which flows.
• Whether continuous evaluation signals are consumed or emitted at all.

The reference behaviour is in token lifetime best practices and session management.

Test 4: the SAML edge cases

Run these five:

1. IdP-initiated login. Supported, refused, or refused in a way that produces a useful error. All three are defensible answers, and silence is not.
2. Assertion replay. Capture a valid assertion and submit it a second time. It must be rejected, and the rejection should appear in a log you can query.
3. Clock skew. Move the IdP clock a few minutes and confirm the tolerance is bounded and documented.
4. Encrypted assertions. Supported, and on which plan tier.
5. Domain collision. Two organizations claiming acme.com, which is what happens after your customer's acquisition. The routing decision here is a product decision the platform makes for you.

Test 5: agent and machine identity

Specifics to ask for:

• A distinct principal type for a non-human actor, with its own lifecycle.
• Delegation that records both the agent and the authorizing human, so attribution survives in the audit trail.
• Short-lived credentials with scopes narrower than the user's own.
• Whether that all works with your authorization model or only with a bolted-on API key.

Background is in authentication for AI agents and token management for AI agents.

Test 6: the export test

What to look for in the response:

• Hashes, specifically. Profile export is common. Credential export is not. Get the algorithm, parameters, and encoding, and test importing one hash somewhere else.
• Log export and retention. How many days on your tier, and is streaming available on that tier or gated to enterprise.
• Configuration as data. Whether connections, policies, and mappings can be exported and re-applied, or whether a rebuild means a console clicking exercise.
• The contract clause. Find the sentence in the agreement that entitles you to all of it, and confirm it says credentials and not only "customer data".

The consequences of each answer are laid out in password hash migration.

Test 7: scale, limits, and the bill

Measure: authentication latency at the 95th and 99th percentile from the regions you actually serve, rate-limit thresholds and their response codes, behaviour under a credential-stuffing pattern rather than a clean load pattern, and any per-tenant limit that a large customer could exhaust on your behalf. Then price it against CIAM pricing at scale.

Scoring, and the two-week shape

Score each test as pass, pass with caveat, or fail, against the criteria you wrote before the first call. Weight the tests by what your product actually needs: a B2B platform selling to regulated enterprises weights SCIM conformance and export far above registration conversion, and a consumer product inverts that.

A workable schedule per vendor:

Days	Work
1-2	Tenant setup, first login, SDK integration in a scratch app
3-5	SCIM against Entra and Okta, all eight operations, both PATCH shapes
6-7	SAML edge cases, domain collision, IdP-initiated
8-9	Session and revocation timing, token behaviour
8-10	Token revocation test running quietly in the background
10-11	Agent identity, authorization model, export request submitted
12-13	Load and rate-limit testing, pricing model
14	Score against the criteria, write the recommendation

Two outcomes count as success. One is a clear winner with known gaps you have already planned around. The other is the discovery that both candidates fail on something you assumed was table stakes, which is worth far more than a signature, and which is the moment to revisit build versus buy with better information than you had a fortnight ago.