Bounded Concurrency, Backpressure and Overload Protection in Java & Spring

Imagine a service with 10,000 virtual threads and a database pool with 50 connections. These are illustrative numbers, not a production incident or a benchmark. If every request needs a connection, what happens to the requests that cannot get one?

They wait somewhere. Making that wait inexpensive for the JVM does not make it free for the user, the network or the database. Requests retain state, deadlines keep advancing, and clients may retry before the original attempt finishes.

Virtual threads are useful precisely because they make blocking I/O practical at high concurrency. They do not make queries execute faster or create downstream capacity. Oracle's virtual-thread guide recommends resource-specific concurrency controls rather than pooling virtual threads to restrict their number.

The architecture decision is therefore: How much work may reach this resource, where may the rest wait, and when do we decline it? This guide develops that decision and supplies a runnable Spring lab. The principles apply equally to worker processes, async tasks and other languages.

1. Count the right things

An open request has arrived but has not completed. It may be parsing input, waiting for admission, using a connection, waiting for an external API, or writing a response. An active downstream operation currently occupies the resource we intend to protect. Here, “active” means holding one dependency slot, not necessarily executing CPU instructions. A database connection can be occupied while the database waits for a lock.

A queue contains admitted or pending work that cannot start yet. It need not be a queue object: threads parked in connection acquisition form a queue too. Throughput is completed work per unit of time, such as successful requests/second. Response time is elapsed time for a request, including its waits. Concurrency is a count; throughput is a rate; response time is a duration.

Our concurrency and parallelism guide explains execution models and virtual threads. This article focuses on the boundary between affordable request execution and scarce downstream capacity.

Suppose, illustratively, 50 independent operations each occupy one connection for an average of 100 ms. Ignoring overhead and contention, that model has a service ceiling near 50 / 0.1 = 500 operations/second. It is not a PostgreSQL prediction. If arrivals continue at 700 operations/second, a queue initially gains roughly 200 operations each second. More waiting threads cannot erase the difference. Real query duration can also rise as contention grows, reducing useful capacity further.

Many requests pass a bounded admission gate before reaching eight dependency slots.Select the image to read its full-size version.

2. The connection pool protects one boundary

In HikariCP's pool configuration, maximumPoolSize bounds database connections in that pool. When none is available, connection acquisition can wait until connectionTimeout expires. This timeout is about getting a connection, not the time allowed for a query or the entire HTTP request.

That is a valuable resource limit. It leaves other questions unanswered: how many callers can wait, how much request state they retain, which workloads get access, and whether their deadlines remain useful. Waiting may also accumulate at the ingress proxy, HTTP connector, executor, ORM, remote service or client.

Do not automatically add a semaphore in front of every pool. If acquisition wait, upstream occupancy and deadlines are already bounded appropriately, another gate may only add another queue. Add an application gate when it gives you a necessary policy: a smaller workload budget, bounded occupancy, priority isolation or earlier rejection before expensive work.

3. Six controls with different jobs

Control What it bounds or communicates What it does not establish
Concurrency limit Simultaneously running operations at a chosen boundary Requests per second; a bounded waiting population unless separately configured
Rate limit Admissions per time interval, usually with an explicit burst allowance A fixed number of in-flight calls when duration varies
Bounded queue Number of pending items or occupied admission places Maximum wait without deadlines; useful throughput
Backpressure Consumer pressure that causes a producer to reduce or pause production That arbitrary blocking or rejection makes a remote producer slow down
Load shedding Work declined early when a budget is exhausted Successful completion for declined users
Circuit breaker Calls suppressed after configured failure/slow-call observations, with recovery probes A hard concurrency cap during healthy operation

Reactive Streams is a concrete backpressure protocol: downstream demand controls upstream delivery. A local semaphore can simply park another request. A 503 response communicates overload, but pressure becomes effective upstream flow control only if the producer reacts by reducing work. Our lab's producer deliberately keeps its offered rate constant: the example demonstrates bounded local waiting and load shedding, not end-to-end backpressure.

Keep circuit breaking and capacity admission separate in design. You can combine them, but check ordering and whether deliberate admission failures should influence breaker statistics, as discussed in our circuit breaker guide. Spring Cloud's Resilience4j integration exposes distinct bulkhead configuration; this lab uses a small explicit gate so its ownership is visible.

4. Choose where excess work goes

Policy Behavior at saturation Benefit Deliberate cost
No additional admission bound Requests enter the resource-acquisition wait; underlying transport and pool limits still exist Can absorb a short burst and complete more work after arrivals stop Long waits, retained state, timeouts and delayed recovery
Bounded waiting Admit only a bounded population and wait only within a budget Absorbs limited bursts without an indefinitely growing local queue Queueing latency, implementation complexity, eventual rejection
Immediate rejection Decline when no execution place is available Small local waiting budget and fast overload signal Rejects bursts that a short queue might have served

There is no genuinely unlimited server. “Unbounded” usually means no explicit bound at the boundary under discussion, until some other, less desirable limit intervenes. Google SRE describes small queues and deliberate shedding as tools for containing overload and preventing cascading failures. These are operating choices, not free performance improvements.

Without an admission policy, requests accumulate at resource acquisition; with bounded occupancy, excess work receives an explicit rejection.Select the image to read its full-size version.

5. Put the bulkhead around the resource-consuming work

A bulkhead reserves or limits a workload's share so one failing dependency or noisy workload does not exhaust everything. A payment provider, inventory service and database can each have their own budget. A slow inventory API should not consume every payment permit.

Avoid holding a database permit during unrelated authentication, response serialization or a remote API call. Acquire immediately before the protected operation, release after it actually relinquishes the resource, and use finally or structured ownership to cover every exit. For JDBC, include the complete connection/transaction lifetime; releasing after one query while retaining the connection is too early. Acquire before starting the transaction when possible, or admission waiting itself may hold a connection.

Fairness has layers. FIFO-style acquisition can reduce barging among waiting callers, but does not give tenants equal shares or make a long operation short. Consider tenant quotas, separate interactive and batch budgets, maximum operation duration, and reserved capacity for essential work. Separate bulkheads still need a shared total budget when they consume the same database. Spare reserved capacity may be idle; borrowing improves utilization but makes isolation harder.

Avoid nested acquisition patterns that can deadlock: one task holding a connection while waiting for subtasks that also require connections is a different problem from ordinary overload. Fan-out matters too. At 100 requests/second with three downstream calls per request, the dependency sees 300 operations/second before retries. Parallel fan-out increases instantaneous demand; sequential fan-out still increases resource time per request. Limit each operation or acquire a correctly sized weighted budget, and define acquisition order when multiple budgets are needed.

Fan-out raises downstream operations per second while replicas multiply process-local concurrency limits.Select the image to read its full-size version.

Read the two calculations separately. A rate budget must account for operations per request; a concurrency budget must account for every instance that can hold permits. Adding replicas can increase admitted work even when the database is unchanged. These numbers are illustrative arithmetic, not the measured lab configuration.

6. Little's Law helps you ask better questions

For a stable system boundary with consistent long-run averages, Little's Law gives L = λ × W: average items in the system equal average arrival rate times average residence time. Use matching populations, units and observation intervals, with finite averages and no sustained unbounded backlog growth. A finite test with startup and drain is not automatically a steady-state estimate.

Choose the boundary “admitted database operations, from entry into their wait until completion.” Suppose accepted/completed throughput is 200 operations/second and mean residence time is 0.15 seconds, including acquisition waiting. Then average occupancy is 30 operations. If the boundary instead includes only connection holding, and mean holding time is 0.05 seconds, average occupied connections are 10.

Neither calculation says “configure a pool of exactly 10.” Averages conceal bursts, tails, workload mix and contention. A p95 latency target cannot be substituted for mean W. Rejected traffic belongs outside the admitted boundary; use admitted throughput, not total offered load. With a growing queue, the stable approximation no longer describes a sustainable operating point.

Little’s Law applied to two different boundaries: thirty operations including acquisition waiting, versus ten occupied connections during holding only.Select the image to read its full-size version.

The outer boundary counts operations waiting for or holding a connection; the inner boundary counts only connection holding. Changing the boundary changes both L and W. This is why a request-concurrency graph cannot be read as a connection-usage graph.

7. A small Spring implementation with explicit ownership

The download pins Spring Boot 4.1.2 and compiles with Java release 21. The current Boot system requirements list Java 17–27 compatibility; Boot's virtual-thread execution support requires Java 21+. We executed this project on OpenJDK 27+35-2325 with Maven 3.9.11. Java 21 runtime compatibility was not separately executed. No preview flags are required.

spring.threads.virtual.enabled=true enables Boot's virtual-thread execution support. The HTTP smoke check verifies virtual=true in the actual controller response. A custom executor can change Boot integration behavior, so verify the execution path in your own application rather than inferring it from a property alone.

The lab's resource has eight physical slots. Each call holds a slot during an interruptible 80 ms sleep. This represents fixed-capacity blocking work, not a database: there is no SQL, lock contention, storage, cache, transaction or network dependency.

Mode A calls that resource directly. Resource acquisition waits up to two seconds. There is no additional bound on callers waiting there. Mode B puts an eight-permit execution gate in front, with 16 total admission places and a 100 ms gate-acquisition budget. Excess occupancy is rejected immediately; expired gate waits are also returned as 503. The nominal eight extra places are a steady-saturation queue allowance: scheduling transitions can make the observed waiting counter briefly exceed eight, but total occupied places never exceed 16.

UML component diagram of the actual lab: controller, process-local gate, fixed-capacity dependency and metrics, with the mode A bypass.Select the image to read its full-size version.

This UML component view maps directly to the downloadable lab. The solid path is mode B; mode A bypasses the admission gate while still using the same finite dependency. Metrics observe both boundaries. In a JDBC adaptation, the dependency component would wrap the actual connection/transaction lifetime, not just the time spent calling a repository method.

// Exact call site in Application.java; one operation per request.
String result = mode.equals("B")
        ? gate.call(100, resource::call)
        : resource.call();

The gate first acquires an occupancy place without waiting. It then waits for an execution permit, runs the callable synchronously, and releases only permits it actually acquired. The full source is small enough to review in the download:

    public <T> T call(long waitMs, Callable<T> work) throws Exception {
        // Timed zero acquisition respects fairness, unlike untimed tryAcquire().
        if (!occupancy.tryAcquire(0, TimeUnit.MILLISECONDS)) throw new Full();
        boolean acquired = false;
        try {
            waiting.incrementAndGet();
            try { acquired = running.tryAcquire(waitMs, TimeUnit.MILLISECONDS); }
            finally { waiting.decrementAndGet(); }
            if (!acquired) throw new WaitExpired();
            return work.call();
        } finally {
            if (acquired) running.release();
            occupancy.release();
        }
    }

The Semaphore API's fairness rules distinguish timed zero acquisition, which respects fair ordering, from untimed tryAcquire(), which can barge. Fairness still does not guarantee CPU scheduling order or tenant equity. The waiting gauge includes the instant of trying acquisition, so interpret it as acquisition-in-progress.

The 100 ms budget covers waiting for the execution permit, not the body or the full HTTP response. Mode B does not impose a universal request deadline. The fixed sleep is finite; a real adapter needs its own operation/transport timeout and an end-to-end deadline budget. This gate is process-local, and only code using it participates in its policy. It does not prevent overload during request parsing or at the proxy.

UML sequence diagram showing mode B occupancy rejection, timed execution-permit acquisition, dependency cleanup and safe permit release.Select the image to read its full-size version.

Follow the sequence from top to bottom. Occupancy rejection releases nothing; an expired execution-permit wait releases only the occupancy place. After execution starts, cleanup returns both gate budgets even when the dependency throws. The simulated dependency separately releases its physical slot. Mode B ordinarily avoids resource-acquisition waiting because its execution budget equals the dependency capacity; the timeout branch remains part of the resource contract.

8. Timeouts, cancellation and retries must agree

The timeout in Future.get(timeout) stops the caller's wait. It does not stop the computation. Cancellation may request interruption, but a task or client library must cooperate. A JDBC statement has separate timeout and cancellation mechanisms whose effects depend on driver/database support. Verify them with the actual adapter.

Never release a permit merely because an HTTP deadline expired if the underlying operation is still consuming the resource. Otherwise abandoned work and replacement work can run together beyond the intended cap. This lab keeps permit ownership inside the synchronous body and tests interruption while holding or acquiring a permit. It also proves that a caller timeout leaves a still-running body holding its permit. An HTTP disconnect is not guaranteed to interrupt this synchronous Spring controller; disconnect-driven cancellation is not implemented or claimed.

UML state diagram of gate ownership: no place, occupancy only, execution plus occupancy, and safe release after the callable exits.Select the image to read its full-size version.

The states describe ownership, not HTTP status. A caller-side deadline can leave the operation in the running state. Releasing at that deadline would make the accounting say “capacity available” while work still consumes it. The conceptual timeout self-transition explains the existing caller-timeout test; it does not add an HTTP-disconnect handler to the lab.

Conceptual timeline separating the caller’s deadline from the underlying operation’s lifetime and permit ownership.Select the image to read its full-size version.

The timeline is a model, not measured timing. A successful cancellation shortens the operation row only when the adapter confirms that work has stopped or its resource has been relinquished. Until then, the permit remains owned. This keeps replacement traffic from silently exceeding the intended concurrency limit.

Our structured concurrency guide discusses related task lifetimes. The checked Java 27 API still marks StructuredTaskScope as preview; the current lab uses stable APIs instead.

Retries are new resource demand. Three attempts at each of three independently retrying layers can expand one incoming operation into as many as 27 downstream attempts. Following the AWS Builders Library's retry guidance, retry at a deliberate layer, within the caller's remaining deadline, with a capped budget and backoff with jitter. Retry only eligible failures. A Retry-After hint is not enforcement, and synchronized retries can create another burst.

For writes, “no response” does not establish “no effect.” Use an appropriate duplicate-suppression contract before retrying; see idempotency in Java and Spring. The load generator here never retries, so the experiment isolates admission behavior.

9. Run the experiment and interpret every outcome

Download the runnable bounded-concurrency lab. Requires JDK 21+ and Maven 3.6.3+. Build and test:

mvn clean verify
java -jar target/bounded-lab-1.0.0.jar --lab.mode=A

In a second terminal, from the extracted lab directory:

java Load.java 40 3 warm-A
java Load.java 160 10 run-A

Stop the server, start it with --lab.mode=B, then repeat the identical commands with output names warm-B and run-B. Keep port 8087 free and use only this loopback service. GET /metrics exposes open controller requests, active resource operations, resource/gate acquisition waiters and outcome counters. Warm-up requests are included in cumulative server counters; the retained measurement subtracts the pre-run snapshot.

The load model schedules 160 arrivals/second for ten seconds, after a 40/second three-second warm-up in each fresh server process. Every arrival launches a virtual-thread HTTP call without waiting for a previous response. This is an open-loop finite schedule rather than a fixed worker pool whose request rate falls automatically when responses slow down. Concurrency emerges from latency. The run contains 1,600 scheduled calls, not 1,600 simultaneous clients.

Client timeout is five seconds, connection timeout two seconds. The generator records dispatch lag, HTTP elapsed time and planned-arrival-to-completion time. Scheduler delay can bunch arrivals; HTTP/network/client queues can still distort load. Inspect the lag distribution and rerun if the generator cannot keep up. Metrics polling every approximately 50 ms is sampled observation, not proof of a maximum. The resource's lifetime peak counter and correctness tests check the active bound.

Measured on 16 November 2026; one run per policy, nearest-rank percentiles. HTTP latency runs from client dispatch through response body receipt. Counts cover the full cohort including drain.

Outcome / metric A: resource wait B: bounded gate
Scheduled arrivals 1600 1600
HTTP 200 1185 994
HTTP 503 0 606
HTTP 504: resource acquisition timeout 415 0
Client timeouts 0 0
Other client errors 0 0
Successful HTTP p50, ms 2069.39 152.11
Successful HTTP p95, ms 2083.4 165.5
Successful HTTP p99, ms 2085.04 169.06
Rejected HTTP p95, ms n/a 8.4
Planned-arrival success p95, ms 2084.34 166.64
Dispatch lag p99, ms 2.4 3.7
Peak active resource operations (counter) 8 8
Peak open requests (sampled) 329 17
Peak resource acquisition waiters (sampled) 321 0
Peak gate acquisition waiters (sampled) 0 8

Successful, rejected and timed-out response counts for 1,600 scheduled arrivals per policy.Select the image to read its full-size version.

Successful response latency percentiles in milliseconds for each admission policy.Select the image to read its full-size version.

Both policies keep the physical dependency at eight active operations. Mode B rejects more work sooner and bounds its local occupancy. Mode A can finish queued requests after the offered-load interval ends; its success count therefore cannot be divided by ten and called sustained throughput. The measurement waits for responses to drain. Fast 503s are failures from the requesting user's perspective, not a throughput gain. Successful latency must always be read beside successful, rejected and timed-out counts.

The retained run is one A-then-B comparison on a shared Windows 11 host, with 16 logical processors, Intel64 Family 6 Model 141 Stepping 1, OpenJDK 27 and other desktop applications running. Physical RAM and a retail CPU model were unavailable through the sandbox's CIM interface. Client and server share this host. A three-second warm-up is deliberately short and does not establish JVM steady state. These results explain this controlled setup; they provide no real database sizing recommendation or performance promise. Repeat both orders, longer runs and realistic dependency distributions before making an operating decision.

10. Choose limits through measurement

Observed bottleneck First protection to evaluate Measurement that checks the decision
Database connection holding / contention Shared connection ceiling plus workload concurrency budget Holding time, acquisition wait, successful queries/s, database saturation
External API quota Rate limit with a defined burst allowance; concurrency cap for slow calls Attempts/s, quota responses, in-flight calls, latency
Large pending workload / memory Bounded occupancy and queue age; early rejection Queue size/age, retained memory, reject and timeout counts
CPU-bound work Bounded execution near measured CPU capacity and a finite queue CPU utilization, useful throughput, tail latency
One slow or failing dependency Separate bulkhead, operation deadlines and circuit breaker where appropriate Per-dependency occupancy, timeouts, breaker state and recovery
Producer outpaces consumer Demand or acknowledgement protocol that actually slows production Arrival rate after pressure, consumer backlog and end-to-end delay

Start with a latency objective and workload definition. For example, an illustrative 300 ms end-to-end objective might leave 40 ms for admission after accounting for measured dependency and application time plus headroom. This is a budget to validate across the latency distribution, not a license to add component p95s and claim an end-to-end p95.

  1. Inventory the resource consumers: each route, batch job, fan-out branch, retry and replica. Measure resource holding time separately from acquisition time and whole-request latency.
  2. Establish a below-saturation baseline, then increase an open-loop arrival rate in controlled steps. Record offered, admitted and successful rates, rejection/timeout counts, latency distributions, queue age and occupancy, resource utilization, CPU and memory.
  3. Find where successful throughput stops improving while resource time or tail latency rises. Select a concurrency budget below that region with room for variability, essential work and failure recovery. Do not infer this knee from the lab's fixed sleep: its work duration does not model contention.
  4. Choose a finite occupancy and wait budget that fit useful deadlines. Queue size and wait time solve different problems; test both bursts and sustained overload. Try zero extra places when freshness matters more than burst absorption.
  5. Test slow dependencies, cancellation, partial outages, uneven routing and tenant competition. Confirm that rejected work does not trigger a retry storm, and that recovery drains promptly.
  6. Roll out gradually and revisit limits when query mix, replica count or dependency capacity changes. Track SLO success, not only the latency of the shrinking subset still admitted.

Three instances with eight permits can admit 24 operations. A rolling deployment or autoscaling event changes that total. Local fairness is not global fairness. Coordinate a shared budget through capacity allocation, a gateway or a distributed admission mechanism when necessary; account for coordination failure and skew. Leave capacity for migrations, administration and other services. A database-wide connection ceiling remains a separate protection.

Increasing the pool can help an underutilized dependency whose clients are needlessly constrained. It can also increase database contention and harm all clients. Lowering concurrency can reduce contention and queueing; lowering it too far wastes capacity and rejects useful work. Tune against measurements, deadlines and resource ownership rather than a universal formula.

The review question to take to your own service is concrete: when its database is already at capacity, which component decides how much more work to accept—and where is that decision measured?