Non-blocking HTTP-IO: Asynchronous Middleware Wiring

Abstract

This article is about the non-blocking HTTP-IO semantics for Java Application Server Middleware Instances.

Non-Blocking HTTP-IO in the context of Asynchronous Request Processing was introduced with the JSR 315 (Servlet 3.0) specification back in 2009 (that is also part of JAX-RS 2.0) [1,2,3].

Its purpose is to increase the internal number of concurrently processable requests within a Java Application Server.

You should not get confused with asynchronous request handling done in the client stub. Seen from the client perspective; your synchronous client request will still be synchronous, your asynchronous client request will still be asynchronous.

All clarity eliminated?

In this article I will give you solutions to three different abstraction level perspectives around JSR 315:

• Low abstraction level using Servlet 3.0+
• Medium abstraction level using JAX-RS 2.0+ (Jersey provider)
• High abstraction level using Spring MVC

Problem

If you run a Java Application Server, like Wildfly, then each of the incoming requests will usually be run in a delegated HTTP-IO Thread, freshly pulled from the HTTP-IO Thread-Pool of the server instance for the actual request.

The logic of the called service method will be processed within this request-scoped “HTTP-IO” thread by default.

This might lead to headaches due to potential deadlock issues, if your server instance can’t recover under high load with a lot of concurrent requests and thereby with a lot of pulled threads from the limited HTTP-IO thread pool.

Your server instance appears non-reacting to any further request as soon as the HTTP-IO Request Thread limit has reached. If the server can no longer recover from the high-load scenario, because threads are not returned to the pool due to deadlocks, then the server instance has actually died.

Solution

To avoid these situations you might want to give the HTTP-IO request thread immediately back to the HTTP-IO thread pool, so that more incoming requests can be simultaneously dispatched by your server instance.

You could do this with very short processing logics in your service methods, or you can use a separate JVM thread to delegate the actual processing logic. You will just decouple the process from the HTTP-IO thread. It’s a simple solution that is much more practicable and gives you the flexibility of an asynchronous pattern style.

The processing of the request itself will then not run in the HTTP-IO request thread, but will be delegated to another concurrent thread.

The Servlet 3.0 does specify how to handle these internal delegations and wirings, which makes the handling of high numbers of requests much more mundane.

Low abstraction level (Servlet 3.0+)

Servlet 3.0 introduced the AsyncContext Object as part of the HttpServletRequest object that can be used in any WebServlet context via HttpServletRequest.startAsync().

How to start the dedicated processing thread?

You can use the AsyncContext.start(Runnable) method to directly start a concurrent processing thread using the logic within the provided Runnable implementation.

How to return the result?

Within the processing thread upon completion you can return the result via the usual way, by using the Response object and a writer.

You should however notify that the processing thread has completed the processing by calling method AsyncContext.complete().

Example Code

package com.homannsoftware;


import java.io.IOException;
import java.io.PrintWriter;
import java.util.logging.Level;


import javax.servlet.AsyncContext;
import javax.servlet.AsyncEvent;
import javax.servlet.AsyncListener;
import javax.servlet.annotation.WebServlet;
import javax.servlet.http.HttpServlet;
import javax.servlet.http.HttpServletRequest;
import javax.servlet.http.HttpServletResponse;


/**
 * Simple AsyncContext demo.
 * 
 * @author Enrico Homann
 */
@WebServlet(asyncSupported = true, value = "/asyncServletExample", loadOnStartup = 1)
public class AsyncServletExample extends HttpServlet {


/**
* Serial UID.
*/
private static final long serialVersionUID = 1L;


/**
* Logger.
*/
private static final java.util.logging.Logger LOGGER = java.util.logging.Logger
        .getLogger(AsyncServletExample.class.getName());


@Override
public void doGet(HttpServletRequest request, HttpServletResponse response) throws IOException {
LOGGER.log(Level.INFO,
        String.format("Received request in HTTP-IO thread '%s'", Thread.currentThread().getName()));


// Prepare the response object writer to use for giving back the result
response.setContentType("text/html");
final PrintWriter out = response.getWriter();


// Retrieve the AsyncContext
AsyncContext asyncContext = request.startAsync(request, response);


// Add Callback Listener
asyncContext.addListener(new AsyncListener() {

@Override
public void onComplete(AsyncEvent event) throws IOException {
LOGGER.log(Level.INFO,
        String.format("[%s] onComplete in HTTP-IO thread", Thread.currentThread().getName()));
}


@Override
public void onTimeout(AsyncEvent event) throws IOException {
LOGGER.log(Level.INFO, String.format("[%s] onTimeout", Thread.currentThread().getName()));
}


@Override
public void onError(AsyncEvent event) throws IOException {
LOGGER.log(Level.INFO, String.format("[%s] onError", Thread.currentThread().getName()));
}


@Override
public void onStartAsync(AsyncEvent event) throws IOException {
LOGGER.log(Level.INFO, String.format("[%s] onStartAsync", Thread.currentThread().getName()));
}

});

// Set a timeout to one minute
asyncContext.setTimeout(60000);

// Start the processing thread
asyncContext.start(() -> {
Thread.currentThread().setName("demo-" + System.currentTimeMillis());
LOGGER.log(Level.INFO,
        String.format("Processing thread '%s' has started ...", Thread.currentThread().getName()));

// Do something overwhelming here
try {
Thread.sleep(5000);
} catch (InterruptedException e) {
LOGGER.log(Level.SEVERE, e.getMessage(), e);
// Interrupt state was cleared by try-catch. Re-interrupt.
Thread.currentThread().interrupt();
}

// Return the result
try {
out.println(String.format("[%s] asynchronous task finished at %s. ", Thread.currentThread().getName(),
        System.currentTimeMillis()));
} finally {
out.flush();
out.close();

// Mark as complete
LOGGER.log(Level.INFO,
        String.format("Processing thread '%s' has completed ...", Thread.currentThread().getName()));
asyncContext.complete();
}

});

}

}

Medium abstraction level (JAX-RS 2.0+)

For the medium abstraction level I will use the Jersey 2.30.1, even if this version is actually an implementation for the JAX-RS 2.1 specification.

The JAX-RS 2.0 (and higher) spec comes with an AsyncResponse interface and a @Suspended annotation to handle the internal asynchronous processing delegation.

Things to do:

• Inject a AsyncResponse object into the service method as parameter with a @Suspended annotation to suspend the AsyncResponse at first
• Make sure that the service method has a void return type
• Construct and start the processing thread manually (or via an ExecutorService)
• Make sure that, within your processing thread, you’ll return the result to the callee via a call of AsyncResponse.resume(Object)

How to start the dedicated processing thread?

You’ll construct and start the processing thread manually (or via an ExecutorService).

How to return the result?

You’ll return the result via a call of AsyncResponse.resume(Object) to a suspended instance of AsyncResponse.

Example Code

The Jersey team offers a very good documentation about the JAX-RS compliant “Asynchronous Server API” on their site:

https://eclipse-ee4j.github.io/jersey.github.io/documentation/latest/user-guide.html#d0e10017

I’ll just paste their example code here:

a@Path("/resource")
public class AsyncResource {
@GET
public void asyncGet(@Suspended final AsyncResponse asyncResponse) {
 
new Thread(new Runnable() {
@Override
public void run() {
String result = veryExpensiveOperation();
asyncResponse.resume(result);
}
 
private String veryExpensiveOperation() {
// ... very expensive operation
}
}).start();
}
}

High abstraction level (Spring MVC 5+)

If you use Spring MVC you can have actually the same semantics like with JAX-RS and the AsyncResponse object within a service method. With the difference that you use the Class DeferredResult as return value instead of using a suspended AsyncResponse parameter.

Things to do:

1. Use DeferredResult as return value in the service method
2. Create the processing thread
3. Return the result within the processing thread via a call to the created instance of DeferredResult and its setResult(Object) method.

How to start the dedicated processing thread?

You actually do the same like in the JAX-RS example and have to construct and start the processing thread manually, via an ExecutorService likewise the ForkJoinPool (as seen in the example code).

How to return the result?

You’ll use the method DeferredResult.setResult(Object), like you would use AsyncResponse.resume(Object) within the JAX-RS abstraction level.

Example Code

@RestController
public class AsyncExampleController {
@GetMapping("/asyncGet")
public DeferredResult<ResponseEntity<?>> asyncGet() {
DeferredResult<ResponseEntity<?>> output = new DeferredResult<>();
 
ForkJoinPool.commonPool().submit(() -> {
String awesomeResult = doSomethingExpensive();
output.setResult(ResponseEntity.ok(awesomeResult));
});
 
return output;
}
}

Conclusion

Using Non-Blocking IO semantics on the server side, your middleware can handle a much higher number of concurrent requests and is thereof much more stable and responsive under high-load scenarios, like DDoS attacks.

What’s next?

Since non-blocking HTTP-IO is just a brick to solve high load scenarios like they are quite common today, it can also be well combined with other approaches like “Reactive Programming” for event-driven architectures to create scalable distributed web applications.

There is a well written article about the “Reactive Programming” methodology on the Spring.io site [4]. And there is Spring WebFlux, a project dedicated to reactive programming using the Spring Framework.

Even if you read a lot about the topics nowadays in the mainstream press, the actual thought-framework of the Reactive Programming paradigm comes from the 1970s.

Did you know?

References

1. https://jcp.org/en/jsr/detail?id=315
2. https://jcp.org/aboutJava/communityprocess/final/jsr315/index.html
3. https://dzone.com/articles/an-overview-servlet-30
4. https://spring.io/blog/2016/06/07/notes-on-reactive-programming-part-i-the-reactive-landscape
5. https://docs.oracle.com/javaee/7/api/javax/ws/rs/container/AsyncResponse.html