Non Blocking JDBC, Non Blocking Servlet APIs and Other High Mysteries

Most everyone who works with server infrastructure has figured out how to use NIO – in one form or another – to transform the standard blocking threaded server model into a far more scalable beast. Without going into boring details (see here, for example, if you need to investigate), the executive summary of the transformation is that previously servers had to pretty much dedicate a thread per inbound socket which would basically block on the socket between requests, waiting in desperate silence for some bytes to show up from the client. The reason why this had scalability issues is that the server had to dedicate a thread per client and threads are really frickin’ expensive resources – especially if most of the time they are blocked waiting for the client to do something.
What NIO allowed server architects to do was – at the simplest level – dedicate only a single thread to handle the waiting for a client socket to become active. Once a client socket was found which needed attention, this socket could be handed off to a thread pool where a thread could be picked up to handle the incoming request. This has a quite amazing effect on the scalability of the server as most of the times clients don’t do anything at all – something captured by “think time” in simple benchmarks. And since you’re blocked doing nothing, you can’t use the valuable thread resource to actually accomplish work for some client which actually does need your attention.
Now, this is actually pretty good stuff and has enabled infrastructure to scale far beyond where the old, blocking, architecture could ever touch. However, what’s pretty much universal in these server architectures – and perhaps is a well known dirty secret – is that once you get into the actual request processing, the system is back into the old same blocking regime. You can see this trivially, for example, in the Servlet API. When you – the application code – try to read some bytes from the client using the ServletInputStream, you’ll find that if the client hasn’t sent the bits, or the network is slow in getting them, or any number of reasons, your code will block waiting for those bits to show up. Likewise, if you try to write some output bytes to the client, using your ServletOutputStream, you’ll find that your code will be blocked until the OS can buffer those bytes for later output to the network. If those buffers are full you’ll simply block until they become available, or the client disconnects.
This isn’t any big revelation, but it is surprising to listen to people wonder why we can’t use NIO to take care of these issues so we can have even more scalability in the server. I mean, if the API is blocking – as all APIs are – it’s kind of baffling to think that someone believes that we can magically make that blocking action go away and somehow eliminate the thread which is holding the execution state of the application code (and server invocation code) up until the point where the blocking read or write occurs.
Well, I’m pleased to say that I believe that I’ve discovered a way to magically make the blocking reads and writes “disappear” without having to change any of the JEE Servlet APIs. Well, “discovered” really isn’t the right word, seeing as how the technique is not really any new revelation to people “in the know”. However, doing this magic for pure and simple Java without redefining the existing APIs, nor requiring the application code to be rewritten in any way is something that I think is pretty magical and certainly at least deserves the characterization of a “discovery”. Further, this technique can be applied to JDBC or any other high value blocking API to transparently transform it into a threadless, non blocking API.

Now, it’s not any big deal to transform your blocking application code into a non-blocking version which would then take full advantage of the non blocking capabilities of the Java NIO stack. The technique is not that hard to learn and amounts to writing a state machine that is driven by the I/O events produced by the asynchnrous I/O network stack. The problem is three fold. First, there’s the fact that you have to rewrite your code. Rewriting code, no matter how trivial the rewrite is a pain in the ass and quite frankly is not even an option in the majority of cases. The code is written. It’s been debugged. It’s deployed. Why mess with it.
A further problem with a rewrite of the code to take advantage of asynchronous IO is simply that the APIs you are using – you know, the JEE APIs such as ServletInputStream and ServletOutputStream – are blocking APIs. You simply cannot use them in any NIO model without essentially abandoning the API and using a custom framework for implementing your state machine. So, one can wait until the JCP comes out with JEE 10, featuring non-blocking Servlet APIs – by which time your children will have long since put you in a retirement home to get rid of you – or you’ll find yourself back in the early nineties, relying on a proprietary API coming from in-house development or the vendor specific APIs provided by your JEE application server of choice.
Sound good so far? It gets even better. Why? Because writing your application as a state machine sucks. Sucks major rocks. While state machines are quite simple, conceptually, they seem to be an abstraction that most programmers don’t natively think in, much less really understand – at least in my experience. I mean, sure, they all talk about them. But I wager that probably well over 80% of the programmers out there – and I’m talking about application programmers, not just you infrastructure weenies – have never coded anything even resembling a state machine in their entire programming carreer. Not even in school.
So, what you have is a solution which pretty much sucks at multiple levels:

  • It requires a rewrite
  • It requires a proprietary API
  • It requires a transformation into a state machine

Now, in my experience, any one of these items is pretty much signals the death knell for any brilliant plan to take advantage of asynchrnous I/O in the application code. All three combine to create an energy barrier that no mere manager nor business development team nor perceived scalabilty problem would be capable of climbing to get to the promised land of end to end non-blocking application paradise.
So that’s pretty much where I’ve left it after I’ve built my quota of non-blocking servers over the past couple of years. I’ve laughed in the faces of those who wonder why we can’t make the rest of the stack non blocking and those who long for non-blocking APIs such as JDBC and other essential pieces of modern application server infrastructure.
But not any more. What I’ve realized is that the work I’ve been doing on the Event Drivent Simulation framework – which I call Prime Mover – can actually be used to transform your garden variety JEE code – or any Java code, really – into a threadless system which allows me to provide the same blocking APIs that are already baked into the collective psyche of the modern Java application programmer while under the covers using event driven asynchronous NIO to scale like no body’s business.
In Prime Mover, I have a complete framework for doing both the garden variety asynchrnonous event processing – i.e. if the method has a void result, you can treat it as a completely asynchronous event – and I can also provide events which “block”, using the continuation framework in the system. The upshot is that I now have a complete framework – well, certainly for the restricted domain I’ve described above – for doing byte code level transformations which will take the blocking APIs in a framework and simulate the blocking behavior while using NIO underneath the hood.
All without requiring a Java thread stack waiting on these blocking APIs.
So this means I should be able to take any Servlet application and use NIO for the ServletInput/OutputStream implementation. Further, I can also do this to any other well defined blocking API such as JDBC, allowing the application code to wait -thread free – for the JDBC call to complete, in a completely asynchronous fashion. Likewise, this can be done for file based APIs, etc. The possible applications are actually quite numerous, when you think about it. There’s a lot of blocking APIs out there, and certainly there’s the high probability that if the code is running in a controlled environment such as a modern application server, it can be transparently transformed into a non-blocking version which uses a fraction of the threads – i.e. resources – to do its work.

6 thoughts on “Non Blocking JDBC, Non Blocking Servlet APIs and Other High Mysteries

  1. So when you are waiting thread free – that means that the “freed” thread is doing something else in between, right? So this breaks all ThreadLocal based context architectures like JTA or ACEGI security (and numerous more which follow the same pattern).

  2. Well, certainly something that can trip things up, but having had to deal with this issue in our own servers when we moved from a blocking implementation to a thread pool with NIO, the issue is hardly unique. Still, remember that I control the horizontal and the vertical. Consequently, I can – and do – emit code during the transformation which will save the thread state and restore it appropriately when the continuation resumes. Alternatively – and I have to play around with this more to figure out what is the appropriate way of doing this – I can rewrite any references to ThreadLocal and subclasses to my alternate version which does the correct thing.
    Rest assured that ThreadLocal and similar things are something I’m quite aware of a a server architect. The system wouldn’t be of much use if it didn’t take care of keeping the illusion of the thread, would it?

  3. First of all, you’re doing good good stuff for the Java world.
    The work you’re doing reminds me of how an 1970’s Assembler TP Monitor (still) works on an IBM Mainframe. In fact, it’s how most operating systems work. The application programmer has no idea what blocking goes on. In fact, every application framework call put the current task in a wait-state, and pushed registers, and processor state etc into a control block relevant to that task. The TP Monitor would then put the application request onto the relevant queue for processing, and then made the processor task available for some other work, but adding it onto the “available tasks” dispatcher queue. When a blocked request came back, it was put onto another queue, so an available task could restore the application state and continue. It is entirely possible that the application jumped across many processors by the time it completed. The guy who did this got the IBM System Programmer Of The Year in 1986, about three years before IBM did the same thing with CICS (their own TP Monitor).
    Keep up the good work!
    P.S. Is it available for perusal?

  4. Thanks. Right now, I’m focusing pretty much on using the event driven simulation framework for use in the massively multiplayer online gaming domain, which is what I developed this framework for in the first place. I’ve since modified things to a more interface based system, rather than class based as this originally was, due to the requirement for distributed simulation entities. This has resulted in a bit more work at the data flow level for the continuations, but things are going okay.
    But I think I can actually build a dedicated system which purely does the transforms required to transform the blocking sockets into a non-blocking event driven system using a subset of the technology. However, it’s not my main interest (although it is my day job, oddly enough).
    If you’re interested in playing around with it, I can arrange to get you access – right now everything is closed source and under my copyright. It all depends on how serious you are about developing it.
    If you’re interested in this casually, then I can whole heartedly recommend the original JIST stuff. That code is freely available from their site. I haven’t actually played with that code base at all, so I don’t know how easy it will be to do what I’m proposing here, but they have continuations in much the same fashion as I have done, so it should be at least possible.

Comments are closed.