(Part I – Preliminaries, Part II – The Long Kiss Goodnight)
<sigh> Sorry. I tend to talk more about the surrounding atmosphere than the thing itself. In this post I hope to remain focussed and actually discuss the actual meat of the architecture and the skeleton upon which it is based. Apologies for not providing the color and fluff of life that actually surrounded the process
Point 1 Humans are good at making declarative statements and woefully incompetent at micromanagement.
This is one of those points that one shouldn’t have to make, but the simple fact that bears repeating is that Humans are really good at figuring out what should be done, but really shitty at actually – you know – doing what they think should be done. In keeping with the spirit of the theme of this post, I leave it to the reader to reflect upon the profound reality that it is far easier to see the goal than the path to it. The profound insight I have (not singular, as I’m sure you’re aware) is that a management system which caters to the micro-manger who actually is competent at orchestrating a complex series of transformations under chaotic conditions are few and far between – so rare as to be non-existent to several orders of approximation (or so expensive, which amounts to the same thing).
The take away is simply that any system which depends on a human to do the actual work is simply not going to work – by definition. Seeing as how this is one of my premises, it’s not something I can really argue. It’s a premise derived by years of observations of not just other humans but also myself. Again, I’m not making the claim that there are not humans who are absolutely brilliant at micro-managing large scale distributed systems – let’s be crystal clear on that point. No. My point is simply that these people are incredibly rare and you – the actual person paying the price – will have to pay through the nose if you find such a person. And, quite frankly, the chances of you actually finding such a person is so miniscule as to be almost unmeasurable. Most likely, what you’ll do is find someone who claims to be such a person or someone whom someone you trust claims to be such a person. And the odds are overwhelmingly that you are just a complete maroon and have been hoodwinked into paying a lot of money for a cheap imitation of such a being. Get used to it. It’s just a simple fact of reality.
Point 2 Centralized systems simply suck
Again, this is simply one of things that shouldn’t have to be mentioned, but we live in the age when the Uber Web2.0 crowd can’t seem to scale their fantastic money making ideas and blame the language for their failings as architects. So it’s a fact worth repeating: anything centralized is limited in its ability to scale. Given the realities that most things don’t need to scale (i.e. no one is going to use your stupid Web2.0 application anyway), it’s simply no surprise that this seemingly basic fact of reality remains the province of high priests and those who walk in the light (fun fact: “He who walks in the light” is the literal translation of my name).
From this assertion, we can derive that any architecture which relies upon centralized control is an architecture which simply won’t scale – QED. Again, seeing as how many things don’t actually need to scale, this really isn’t a problem that comes up in the bulk of most people’s experience. However, when it actually does come up – i.e. when you have an actual scalability problem – it bites you so hard in the ass that it’s almost like a near Death experience.
And look around you. Find a single management architecture which does not reduce to a centralized, “spider in the middle of the web” architecture. Go ahead. I’ll wait. Shit, I’ll give you a year to come up with 3 of them. I doubt you can name 2. Personally, I know of at least four, but that’s just me. Two of these are academic, one is actually released into the wild as open source and then there’s mine. But hey. I live a sheltered life. Perhaps decentralized configuration management and provisioning architectures are common as fleas in the De La Pulgas land grant here in the SF Bay area…
Point 3 Management systems should be pattern driven via events rather than command driven
This really follows trivially from the first two points. A management system which is managing a large scale distributed system (or, I would claim, even a small scale distributed system) really is a goal driven system which reacts to events rather than smugly issues commands which are largely irrelevant at the time they were issued. The mistake I believe a lot – hell, all – management systems make is that they have the hubris in believing that that the commands they issue are a) relevant and b) will be followed.
Distributed systems are exceedingly complex environments in which the main goal is simply to adapt to the current environment. Not to be redundant, but this hearkens back to my very first point in that you should declare what you want to come about, not decree by fiat what commands will be followed. Think about it. The moment that the command has passed your conscious thought process and has been (successfully!) entered into the management system, the massively distributed system you’re trying to tell what to do has changed – almost always dramatically – under your feet and is no longer the same system that you were dealing with when you issued your command. Sorry, but you’re going to have to let that sink in a bit because it’s a really important point:
So these three points are the three driving points in what I call