Transaction Isolation
Banking flows that read state and write back based on that state — money movement, hold consumption, settlement — need stronger semantics than the PostgreSQL default ofREAD COMMITTED. CrateStack exposes the two pieces
this requires: explicit per-transaction isolation levels and a retry
loop for serialization failures. run_in_isolated_tx is the hand-rolled
form for code that owns a pool, and works in every release. The
@isolation procedure attribute does the
same from the schema since 0.14.1: through 0.14.0 it
is ignored.
run_in_isolated_tx
The helper wraps a closure in BEGIN, SET TRANSACTION ISOLATION LEVEL ...,
the closure body, and COMMIT:
cratestack_error_from_sqlx rather than |e| CratestackError::Database(e.to_string())
at sqlx call sites — it preserves the SQLSTATE code and constraint name on
the typed CratestackError::DatabaseTyped variant, so unique-violation helpers
and similar predicates can compare typed fields instead of substring-matching
the stringified detail. A missing row (sqlx::Error::RowNotFound) is mapped
to CratestackError::NotFound so the response is a 404 rather than a 500.
The closure receives the transaction and must return it back paired with
the body’s result — the wrapper owns the commit so the retry loop can
control it.
Supported isolation levels
Serializable. Lighter
“consistent snapshot” reads use RepeatableRead. The default level
(without the helper) remains PG’s READ COMMITTED.
Retry on serialization failure
UnderSerializable (SSI), Postgres can refuse to commit a transaction
that participates in a read-write dependency cycle, raising SQLSTATE
40001. Deadlock detection raises 40P01. Both are transient: the
PG docs are explicit
that the entire transaction must be retried.
The wrapper retries automatically:
- up to 3 times via
run_in_isolated_tx - up to a caller-chosen budget via
run_in_isolated_tx_with_retries(pool, level, retries, body) - when a statement inside the body fails with
40001or40P01 - when
tx.commit()itself fails with one of them: SSI can defer the conflict to commit time (write-skew)
40001 mapped with
cratestack_error_from_sqlx that is a DatabaseTyped error, which a
handler answers as 500 DATABASE_ERROR, not the 409 TRANSACTION_ABORTED
that @isolation answers with. Banks running
heavily contended workloads tune the retry budget up; CAS-style
fast-fail flows tune it down to 1.
Only database errors are retried (since 0.14.1). A typed
database error (DatabaseTyped, as cratestack_error_from_sqlx builds
it) is retried if and only if its SQLSTATE is 40001 or 40P01; its
message is never read. Only the untyped CratestackError::Database(String),
which has no SQLSTATE, is still classified by its text. In 0.14.0 and
earlier the helper also retried any error, Validation and Internal
included, whose text contained 40001, 40P01, could not serialize access or deadlock detected. A body that echoed request data into its
own error (field 'memo' length 40001 exceeds maximum 100, or a
RAISE EXCEPTION with the requested amount) therefore ran again, with
anything it did outside the transaction, and the caller got the last
attempt’s error instead of the first. This changes code that never uses
@isolation.
Body must use the supplied transaction
Every statement in the body should run through&mut *tx. Statements
that escape to the pool will not see the snapshot the wrapper opened, and
won’t roll back on retry. The closure signature pins this:
FnMut, not Fn — the retry loop calls the closure again on each
serialization-failure retry, so it must be callable more than once.
(Since 0.15.0, cratestack#1117.) The
framework’s own reads for a write the body makes through run_in_tx(&mut tx, ctx) (create policies
with their relation lookups, the @version probe, the upsert update-policy check, the @@audit
bootstrap) run on that same transaction. They see the body’s earlier writes, read its snapshot under
REPEATABLE READ and SERIALIZABLE, and take no second pooled connection. Through 0.14.2 only an
@isolation procedure read them on its transaction; every other caller read them on the pool.
When commit-time retry matters
Two scenarios surface 40001 fromtx.commit() rather than from a
statement:
- Write-skew anomaly. Two transactions read overlapping rows, write disjoint rows, and SSI detects the read-write dependency only at the commit boundary.
- Predicate-lock contention. A long-running SELECT participates in conflicts that aren’t visible until the transaction tries to land.
run_in_isolated_tx and (since 0.14.1) in @isolation
dispatch. Without commit-time retry, callers would observe a transient
40001 despite the API advertising automatic retries.
Procedure-level isolation
Procedures declare their required isolation level inline:- one
@isolationattribute per procedure - the level argument is a quoted string:
"serializable","repeatable_read", or"read_committed"(case-insensitive;"repeatable read"with a space is accepted too) - (since 0.14.1) not on a
@streamprocedure: a streamed response is produced after the procedure returns, so there is no point at which to commit, and a partly sent stream cannot be retried - (since 0.14.1) not in a
datasource { provider = "none" }schema; andinclude_server_schema!(.., db = None)refuses it with a compile error
What the attribute does
A procedure that declares@isolation(level) runs, on every path that can execute it, inside one
transaction begun with BEGIN ISOLATION LEVEL <level>: REST (/$procs/<name>), RPC
(/rpc/procedure.<name>), /rpc/batch, MCP tools/call, and the generated
<procedure>::invoke_with_db. Inside that transaction run:
- its authorization:
@allow/@denyand any@authorize(...)check - its body
- the policy checks of every write it makes (create policies with their relation lookups, the
@versionprobe, upsert update-policy checks) - the
@computedfields of its output
40001 or 40P01, from a statement or from COMMIT, the attempt is rolled back and
run again from authorization onwards. The budget is 3 retries (4 attempts), with a short jittered
backoff between them, and no transaction is held while waiting. Change it per runtime:
Ok. The same classifier as the helper applies: only database errors are
retried, and a typed SQLSTATE is authoritative, so an application error or a RAISE EXCEPTION
whose message contains 40001 is returned once, as itself.
The handle: IsolatedCratestack
The ProcedureRegistry method of an @isolation procedure takes
db: &cratestack_schema::IsolatedCratestack instead of &Cratestack. Every operation reachable
from it runs inside the procedure’s transaction:
It has no
pool(), events(), views() or queries(): those run on the pool, so their absence
is a compile error rather than a silent escape from the transaction. Operations on the handle run
one at a time: a concurrent call (tokio::join! of two operations) or a re-entrant one (a
.run(ctx) inside db.transaction(..), where tx already holds the connection) fails with
INTERNAL_ERROR instead of deadlocking. Inside db.transaction, use run_in_tx(tx, ctx).
For this schema fragment (alongside your datasource and auth blocks):
SERIALIZABLE transaction, so two concurrent
withdrawals cannot both pass the balance check:
db.transaction(..) that leaves the transaction aborted (a failed statement whose error
the closure does not return) or ends it, or that is cancelled or panics before it finishes, fails
the whole attempt with INTERNAL_ERROR even if the body then returns Ok. Never issue COMMIT,
ROLLBACK or END through tx: what ran before a raw COMMIT stays committed, and if the same
attempt had also swallowed a serialization failure it is retried rather than failed, so that
work is committed again.
Retries exhausted
When the retries run out, the response is409 with code TRANSACTION_ABORTED (RPC code
aborted), and the fixed message transaction could not be completed because of concurrent updates; retry the request. It is not CONFLICT: nothing was committed, and sending the same
request again is expected to succeed. CratestackError::TransactionAborted is the new error
variant. Every generated client knows the code: the TypeScript and Dart RPC runtimes list it, the
Rust client maps a batch frame’s aborted to 409, and so does @cratestack/link-batch.
The idempotency key is released. IdempotencyLayer (REST and RPC) and MCP’s idempotency
admission do not record this response: the reservation is released so the same Idempotency-Key
runs the call again. Every other response, errors included, is recorded as before.
Only the procedure whose own retries ran out answers TRANSACTION_ABORTED. An abort that
reaches a response any other way is answered as 500 INTERNAL_ERROR (RPC internal), recorded
under the key, with the original SQLSTATE in the operator’s log. That covers a procedure, with or
without @isolation, that calls another one and propagates its abort, a @computed resolver that
does, and a hand-written handler that returns the error of an @isolation procedure’s
invoke_with_db. Such a caller may have committed work of its own before the abort, so telling
its client “nothing was committed, retry” could apply that work twice. Serve the procedure through
the generated router, or map the error yourself, to answer it as retryable.
Side effects and retries
The body, and the
@computed resolvers of its output, must be safe to run more than once. Make
external effects idempotent or move them behind @@emit.
Resolvers of an @isolation procedure’s output receive the same &Cratestack argument as before,
but bound to the attempt: their model reads see the body’s writes. Their pool(), views(),
queries() and events() still run on the pool, outside the snapshot.
An @isolation procedure invoked from inside another’s attempt (a resolver can do this with the
attempt-bound &Cratestack it receives) joins that transaction as a savepoint: it commits
nothing on its own and is not retried separately, since the outermost attempt owns the retries. It
is refused if it declares a stricter level than the attempt it would join. Joined calls run one at
a time; starting a second one while another is still running, or cancelling one half-way, fails
the whole attempt.
@isolation does not make an RPC batch atomic: each /rpc/batch frame is its own transaction, as
it was before.
Upgrading
- Find the procedures that declare it:
grep -rn '@isolation'over your.cstackfiles. - Review each body, and the
@computedresolvers of its output, for effects that must not repeat. - Change
db: &Cratestacktodb: &IsolatedCratestackin those methods. Replacedb.pool()withdb.transaction(..), and remove any hand-writtenrun_in_isolated_tx(db.pool(), ..)wrapper: the dispatch provides it now. - A hand-written caller of
<procedure>::invoke_with_dbpassesFnOnce(IsolatedCratestack, Authorized) -> Fut + Cloneinstead ofFnOnce(Authorized) -> Fut; it is called once per attempt. - Treat
409 TRANSACTION_ABORTED(RPCaborted) as retryable, with the sameIdempotency-Key.409 CONFLICTkeeps its meaning. - Size the pool: each in-flight call holds one connection for its whole transaction.
@isolation: the helper’s retry classifier (above), and a
Postgres error from the framework’s own reads (find_unique, find_many, projections,
aggregates), the @authorize probe, create-policy relation lookups and @@audit writes is now the
typed DatabaseTyped rather than Database, with the same code, status and message. Code that
matched Database(_) for those errors must also match DatabaseTyped(_), or use code() /
db_sqlstate().
The full design, including what stays outside the guarantee, is
docs/design/procedure-isolation.md.
What this is not
- not a replacement for application-level conflict handling — some business logic genuinely needs the user to re-confirm after a stale read; the retry loop is the safety net, not the policy
- not a distributed transaction coordinator — PG isolation only applies inside one database
- not free —
Serializableadds locking overhead; benchmark before applying it to read-heavy procedures
Read Next
- Optimistic locking — row-level version checks complement transaction-level isolation
- Idempotency — duplicate-execution protection at the request boundary