Most everyone who works with server infrastructure has figured out how to use NIO – in one form or another – to transform the standard blocking threaded server model into a far more scalable beast. Without going into boring details (see here, for example, if you need to investigate), the executive summary of the transformation is that previously servers had to pretty much dedicate a thread per inbound socket which would basically block on the socket between requests, waiting in desperate silence for some bytes to show up from the client. The reason why this had scalability issues is that the server had to dedicate a thread per client and threads are really frickin’ expensive resources – especially if most of the time they are blocked waiting for the client to do something.
What NIO allowed server architects to do was – at the simplest level – dedicate only a single thread to handle the waiting for a client socket to become active. Once a client socket was found which needed attention, this socket could be handed off to a thread pool where a thread could be picked up to handle the incoming request. This has a quite amazing effect on the scalability of the server as most of the times clients don’t do anything at all – something captured by “think time” in simple benchmarks. And since you’re blocked doing nothing, you can’t use the valuable thread resource to actually accomplish work for some client which actually does need your attention.
Now, this is actually pretty good stuff and has enabled infrastructure to scale far beyond where the old, blocking, architecture could ever touch. However, what’s pretty much universal in these server architectures – and perhaps is a well known dirty secret – is that once you get into the actual request processing, the system is back into the old same blocking regime. You can see this trivially, for example, in the Servlet API. When you – the application code – try to read some bytes from the client using the ServletInputStream, you’ll find that if the client hasn’t sent the bits, or the network is slow in getting them, or any number of reasons, your code will block waiting for those bits to show up. Likewise, if you try to write some output bytes to the client, using your ServletOutputStream, you’ll find that your code will be blocked until the OS can buffer those bytes for later output to the network. If those buffers are full you’ll simply block until they become available, or the client disconnects.
This isn’t any big revelation, but it is surprising to listen to people wonder why we can’t use NIO to take care of these issues so we can have even more scalability in the server. I mean, if the API is blocking – as all java.io.Stream APIs are – it’s kind of baffling to think that someone believes that we can magically make that blocking action go away and somehow eliminate the thread which is holding the execution state of the application code (and server invocation code) up until the point where the blocking read or write occurs.
Well, I’m pleased to say that I believe that I’ve discovered a way to magically make the blocking reads and writes “disappear” without having to change any of the JEE Servlet APIs. Well, “discovered” really isn’t the right word, seeing as how the technique is not really any new revelation to people “in the know”. However, doing this magic for pure and simple Java without redefining the existing APIs, nor requiring the application code to be rewritten in any way is something that I think is pretty magical and certainly at least deserves the characterization of a “discovery”. Further, this technique can be applied to JDBC or any other high value blocking API to transparently transform it into a threadless, non blocking API.
Now, it’s not any big deal to transform your blocking application code into a non-blocking version which would then take full advantage of the non blocking capabilities of the Java NIO stack. The technique is not that hard to learn and amounts to writing a state machine that is driven by the I/O events produced by the asynchnrous I/O network stack. The problem is three fold. First, there’s the fact that you have to rewrite your code. Rewriting code, no matter how trivial the rewrite is a pain in the ass and quite frankly is not even an option in the majority of cases. The code is written. It’s been debugged. It’s deployed. Why mess with it.
A further problem with a rewrite of the code to take advantage of asynchronous IO is simply that the APIs you are using – you know, the JEE APIs such as ServletInputStream and ServletOutputStream – are blocking APIs. You simply cannot use them in any NIO model without essentially abandoning the API and using a custom framework for implementing your state machine. So, one can wait until the JCP comes out with JEE 10, featuring non-blocking Servlet APIs – by which time your children will have long since put you in a retirement home to get rid of you – or you’ll find yourself back in the early nineties, relying on a proprietary API coming from in-house development or the vendor specific APIs provided by your JEE application server of choice.
Sound good so far? It gets even better. Why? Because writing your application as a state machine sucks. Sucks major rocks. While state machines are quite simple, conceptually, they seem to be an abstraction that most programmers don’t natively think in, much less really understand – at least in my experience. I mean, sure, they all talk about them. But I wager that probably well over 80% of the programmers out there – and I’m talking about application programmers, not just you infrastructure weenies – have never coded anything even resembling a state machine in their entire programming carreer. Not even in school.
So, what you have is a solution which pretty much sucks at multiple levels:
- It requires a rewrite
- It requires a proprietary API
- It requires a transformation into a state machine
Now, in my experience, any one of these items is pretty much signals the death knell for any brilliant plan to take advantage of asynchrnous I/O in the application code. All three combine to create an energy barrier that no mere manager nor business development team nor perceived scalabilty problem would be capable of climbing to get to the promised land of end to end non-blocking application paradise.
So that’s pretty much where I’ve left it after I’ve built my quota of non-blocking servers over the past couple of years. I’ve laughed in the faces of those who wonder why we can’t make the rest of the stack non blocking and those who long for non-blocking APIs such as JDBC and other essential pieces of modern application server infrastructure.
But not any more. What I’ve realized is that the work I’ve been doing on the Event Drivent Simulation framework – which I call Prime Mover – can actually be used to transform your garden variety JEE code – or any Java code, really – into a threadless system which allows me to provide the same blocking APIs that are already baked into the collective psyche of the modern Java application programmer while under the covers using event driven asynchronous NIO to scale like no body’s business.
In Prime Mover, I have a complete framework for doing both the garden variety asynchrnonous event processing – i.e. if the method has a void result, you can treat it as a completely asynchronous event – and I can also provide events which “block”, using the continuation framework in the system. The upshot is that I now have a complete framework – well, certainly for the restricted domain I’ve described above – for doing byte code level transformations which will take the blocking APIs in a framework and simulate the blocking behavior while using NIO underneath the hood.
All without requiring a Java thread stack waiting on these blocking APIs.
So this means I should be able to take any Servlet application and use NIO for the ServletInput/OutputStream implementation. Further, I can also do this to any other well defined blocking API such as JDBC, allowing the application code to wait -thread free – for the JDBC call to complete, in a completely asynchronous fashion. Likewise, this can be done for file based APIs, etc. The possible applications are actually quite numerous, when you think about it. There’s a lot of blocking APIs out there, and certainly there’s the high probability that if the code is running in a controlled environment such as a modern application server, it can be transparently transformed into a non-blocking version which uses a fraction of the threads – i.e. resources – to do its work.