All we need to do is take these lies and make them true (somehow)

Having worked in OSGi for quite a while, the most frequently asked question I get on the technology is “what’s the value proposition.”  Being a rapacious capitalist at heart, I think this is an eminently fair query from anyone looking at this OSGi stuff and scratching their head and wondering why the heck they would even want to consider using this technology in their systems.  There’s a non-zero and usually non-trivial cost associated with changing technology – a cost which usually grows in direct proportion to how “low” the technology is on whatever technology stack you are deploying.  OSGi is pretty low on that technology stack, so it has the potential to be very disruptive and hence very costly to an organization which adopts the technology.  Surely the benefit of making the switch should be at least proportional to the costs and a prudent business would like to understand what they’re going to get for their trouble, and more importantly, their hard earned money.

To answer this question, what I first ask is that people think about their current environment.  Assuming you’re not a startup – a domain which I’m not considering in this post – then you are undoubtedly dealing with a mature system which is now or quickly will resemble Frankenstein’s monster more than anything else.  If your system is successful to any degree – and if it isn’t, then we aren’t really having this conversation – what you find is that your system is a victim of its own success.  Grizzled veterans remember the “good old days” when builds would take less than an hour and everyone could sit in a room and share a common understanding of what the hell was going on in this money maker of yours.

Sadly, one day you look up and find that no one knows what the hell is going on anymore.  Build times – perhaps one of the more visceral measurements of complexity we have – have jumped dramatically.  These days, people fire off builds and then go on lunch breaks.  Worse, your projections are that in just a short time in the future, the nightly “integration” builds you kick off will still be running well after your developers have shown up for work.  It’s at this point that one panics and decides that dramatic action is required.  Something MUST be done.  Well, as long as that something doesn’t require any change to what you’re currently doing – i.e. one starts searching for a silver bullet which will slay this beast of chaos that you’ve collectively created and return your life back to the way things used to be.  Before “IT” happened.

Continue reading

Event Driven Autonomic Management – The First Cut Is the Deepest

(Part I – Preliminaries, Part II – The Long Kiss Goodnight)
first-cut.jpeg<sigh> Sorry. I tend to talk more about the surrounding atmosphere than the thing itself. In this post I hope to remain focussed and actually discuss the actual meat of the architecture and the skeleton upon which it is based. Apologies for not providing the color and fluff of life that actually surrounded the process ;)
Point 1 Humans are good at making declarative statements and woefully incompetent at micromanagement.
This is one of those points that one shouldn’t have to make, but the simple fact that bears repeating is that Humans are really good at figuring out what should be done, but really shitty at actually – you know – doing what they think should be done. In keeping with the spirit of the theme of this post, I leave it to the reader to reflect upon the profound reality that it is far easier to see the goal than the path to it. The profound insight I have (not singular, as I’m sure you’re aware) is that a management system which caters to the micro-manger who actually is competent at orchestrating a complex series of transformations under chaotic conditions are few and far between – so rare as to be non-existent to several orders of approximation (or so expensive, which amounts to the same thing).
The take away is simply that any system which depends on a human to do the actual work is simply not going to work – by definition. Seeing as how this is one of my premises, it’s not something I can really argue. It’s a premise derived by years of observations of not just other humans but also myself. Again, I’m not making the claim that there are not humans who are absolutely brilliant at micro-managing large scale distributed systems – let’s be crystal clear on that point. No. My point is simply that these people are incredibly rare and you – the actual person paying the price – will have to pay through the nose if you find such a person. And, quite frankly, the chances of you actually finding such a person is so miniscule as to be almost unmeasurable. Most likely, what you’ll do is find someone who claims to be such a person or someone whom someone you trust claims to be such a person. And the odds are overwhelmingly that you are just a complete maroon and have been hoodwinked into paying a lot of money for a cheap imitation of such a being. Get used to it. It’s just a simple fact of reality.

Continue reading

Event Driven Autonomic Management – The Long Kiss Goodnight

(Part I – Preliminaries
Part III – The First Cut is the Deepest)

From my perspective, one of the major pitfalls in any project which starts out to produce a management infrastructure is that the project almost immediately starts focussing on the API layer rather than the defining the large scale system behavior. In many ways, this is completely understandable, given that the API has the most immediate impact on the first users of the system – i.e. those hapless fools that form the brigade of developers who have to integrate their systems into your management infrastructure. Given that in most large organizations, APIs become the mechanism that groups use to mediate their interaction – not just at the Java level, but in a visceral sense that governs the actual political interaction between the groups. Somewhat because APIs are something concrete and form a nucleus around which people can argue concretely about. But mostly it’s because most people are rather ignorant of how systems actually interact, but one thing they do know is that there are APIs and consequently these concrete manifestations of handles that can be universally understood become the battleground upon which system integration takes place. Or, put another way, APIs are the lowest common denominator that even managers can understand, consequently they are the only focus of pretty much every large scale project.
But the problem with this focus is that an API doesn’t define a system, rather it’s the other way around. The way I think about it is that the APIs of a system are like the inner core of a sphere. Defining the surface of the system – i.e. what the system “looks” like – will provide enormous leverage on the internals of that system. And this leverage will simply force things into place – meaning that the reason the API exists is because it is literally the inevitable result of the forces that keep the system together.
But I digress.

Continue reading

Event Driven Autonomic Management – Preliminaries

Update: Part II – the long kiss goodnight, Part III – The First Cut is the Deepest
This is the first in a series of posts documenting the research I’ve been doing into a different way of thinking about system management infrastructure. For quite some time, I’ve been obsessed with the idea of how to simply and effectively manage large scale systems. Throughout this obsession, I’ve travelled down various roads and found myself in several box canyons along the way. I’ve tried out a lot of different strategies and have finally settled into something which provides the kind of framework I’ve been looking for which I haven’t found replicated anywhere.
Note that I’m certainly not making the claim that it is “Teh Best” management infrastructure. Rather, what I’m making the claim is that it’s the most interesting management architecture to me. As anyone who knows me can testify, I have rather peculiar tastes and I am a strange bird at times. So fair warning, eh?
In any event, what I plan to do is to provide a fairly deep dive into the architecture that I’ve come up with. In the standard tradition of all literature scientific and technical, it will be presented in precisely the opposite order in which I actually came up with things – i.e. from the top down, in a semi coherent form that makes sense. Lord knows that actual discoveries and explorations are more a matter of luck in which you discover something and then spend an inordinate amount of time tracking down why the heck you managed to stumble upon it and where it fits in the larger picture of things that you’re trying to map out. I’ve always found this cognitive dissonance amusing, myself, and hope you won’t mind to much when I veer off into seemingly irrelevant paths rather than sticking to the point at hand.
If you’re one of those people who can’t wait until the end of the story to find out what’s going on, by all means download the PDF of my talk on the subject at last year’s Spring Experience entitled Digging the Trenches on the Ninth Level. If you’re not familiar with Dante’s Divine Comedy, then you won’t get the joke. But suffice it to say that I’m a big believer in the principle that every time you solve a problem, you discover ten more problems that you didn’t know you had.
A perfect example of this sometimes perverse law is something as simple as email. Email solved a lot of problems that a modern economy and social population have, but in doing so it created a lot more. Without email, we would never have been subjected to the sublime beauty of penile extension spam nor would your grandmother be subjected to the horror of id phishing which you discover has snagged her bank account and drained all her life’s savings leaving you with a predicament that makes you wonder what all this progress was supposed to do in the first place.
Likewise, I firmly believe that in solving the problems I believe have been addressed by OSGi, Spring DM and management architectures like mine, we’ve inadvertently unleashed new levels of horror that will ensure future generations will curse our names as they suffer from the fall out and live the unspeakable abominations unleashed from these “solutions” and witness them unfold in ways that we couldn’t possibly imagine.
So, with that cheery panorama as the back drop, I’ll end this introductory post and start working on the next post, which provides high level overview and ten dollar tour of the sewers that I’ve been digging for your benefit on the ninth level of hell.
Remember. I dig because I care. After all, you do want that frozen crap to be routed somewhere and dealt with, don’t you?


C’mon in! The water’s wonderful!

Digging the Trenches on the Ninth Level of Hell

My talk at last year’s Spring Experience talk on the next generation of application server architecture is available here.
The talk is about OSGi and how the next generation of application server platforms will simply do away with the cumbersome and rather dated component models that we all know and hate in favor of the vastly superior OSGi platform. Or that’s the theory at least. Only time will tell if I’m correct or just another mad hatter sniffing too much mercury outgassing from the various toys littering his office.
In addition, I also lay out the management architecture I’ve been experimenting with for the past year. Obviously, it uses OSGi as its base, but OSGi – by itself – isn’t sufficient to provide the kind of management infrastructure you need to manage large numbers of processes. I call this management architecture – for lack of a better name – Event Driven Autonomic Management. I’ll be kicking off a series of posts going into far more detail on this architecture as a means of documenting the research I’ve been doing.
Think of it as therapy, as talking about it on this blog – posting, so to speak, to the wind about concepts and issues that no one else seems to find terribly interesting or useful. You can tell I’m a great hit at parties, can’t you?