Promises
Every two or three years we see the Next Big Thing. We had programming languages, Structured Programming, Object Orientation and all that stuff, and more recently things like Agile Software Development or SOA. Always we could see an industry developing, and always our managements have bought into it, and always things got … easier???
Yes and no. There is certainly an evolutionary process, and it is completely obvious, that today’s software development methods are ways ahead of what was done 50 years ago. But then, the industry favors what can be sold, and this brings its own share of problems.
A Look Back
20 years ago I wrote my programs in C. Now I use Java and sometimes C#, in between I have used Tcl/Tk and Perl a lot.
Scripting languages were a big step from C. Have you ever used regular expressions or hash tables in C? They are implemented as libraries and using them never feels natural, That’s very much different in modern scripting languages.
Java is a similar step forward, only in a slightly different direction. It not only releases me from the burden of keeping track of the memory allocated by my programs, it’s also strongly typed, and that makes a difference in big projects and in tool support. I use Eclipse as my Java IDE, and once you get used to that, there is no way back.
And still: although the tools have become so much better, the computers so much faster, we still struggle with the same class of problems we had 10 or 20 years ago.
RPCmagiX
In 1990 I began playing around with Remote Procedure Calls. Distributed Computing and Client/Server were big buzzwords then. It took me about seven years to come up with a solution that was also my Master Thesis. The tool, RPCmagiX, was meant to be published as free software, but in the end I failed at that. It was used internally though, and many of the programs made with it are still in production.
RPCmagiX was all about maximum simplification. Using RPC systems like DCE or ONC, you had to write a lot of boilerplate code, and you had to do it over and over again. Different RPC systems were incompatible not only on network protocol level, but also in the way programs had to be written to use them.
RPCmagiX did away with all of that. It used a nice unified model, and all you had to do, was to write a subroutine that would run in the server, along with a client that called the subroutine. You used a graphical tool to define the interface, datatypes, the procedures and their parameter lists, selected programming languages for client and server, and the tool would generate a library to be linked to the client, and a library including a main program, that could be linked to the procedure implementations, and that could be started as the server program. The only thing that client code used to refer to the service, was a first parameter in every procedure, a so-called service name, basically a structured name modeled after UNIX file names. The generated client stub and the client library used this name to look up matching server instances, connected to servers, handled automatic failover, etc. As a programmer, you did not have to write a single line of network code, and you had the additional benefit of being able to make cross-language calls. For instance we had Windows clients in Visual Basic calling UNIX servers written in Cobol.
RPCmagiX did a great job hiding the whole complexity of distributed computing inside of the interface. Using specification of the interface and code generation, it gave us lots of additional benefits as well, for instance you could generate data for statistical use, you could send notifications in case of failures, you could have load balancing, and all that in a standardized way. This was possible, because the system knew about the interface and could use all the information in the interface, but the actual application code was not at all entangled with networking concerns.
Using such a system gives you a clear boundary between what you use and what you write. If your application can do networking by knowing not more than a simple name of a service, if you can simply call a remote procedure without having to initialize network libraries, create stubs and connect them to sockets, if calling a remote procedure is a one-liner, only then you have a chance to later replace your networking system if you need to.
At that time we were using ONC (aka SUN RPC), because it was a free alternative to the then expensive DCE (OSF Distributed Computing Environment). We prepared to eventually make the switch to DCE, which we actually never did, but DCE largely failed anyway. Such is life 
Whatever. Responsible application design must try to avoid creating new systems that immediately become the legacy systems of the future. We can do this by separating application logic from infrastructure code. If we do that, we can change infrastructure providers, not entirely without effort, new technologies must still be tested and encapsulated, but at least it is possible. Doing this, we protect our investment in analysis and avoid writing the same systems every ten years, over and again.
The State Of The Industry
Frankly, I would have expected the industry to come to similar conclusions, but interestingly enough it did not. We still get library system after library system, we still have to use factories to create objects that we need to initialize in order to make them do their tasks. We see progress with technologies like the Java Enterprise Edition (JEE) or Spring, where we have Dependency Injection, but although I am very impressed with JEE and the elegant solutions they have come up with in release 5 and 6, there are plenty of other fields where it is bad as it ever was.
The answer is simple. Consensus in the industry is always the result of political struggle, very often even of outright war. It’s Betamax/VHS/Video 2000 all over again. Remember HD-DVD vs BluRay? Sometimes the better system wins (may have been the case with BluRay), sometimes it’s the most inferior (as it was certainly with VHS), but in no case is it due to technical merits.
And it gets even worse when you look at how buying decisions are made and who makes them. Wave after wave of management fashions washes over our heads, and although we are lucky and many of the latest buzzwords never leave management circles, some do, and then we are confronted with requests like to build a Service Oriented Architecture (SOA) based upon an Enterprise Service Bus. And that all, because some clever consultants sold management the idea, that such a mythical beast would allow non-programmers to click together meaningful applications from a host of useful generic services.
Of course this won’t happen and to a programmer, the whole idea looks ridiculously naive, but just having to constantly evaluate the currently fashionable paradigm and to cope with the ever immature and unreliable tools, just the pure effort to implement applications on poorly understood foundations, just this does all the damage.
The result is a succession of applications that have nothing in common, have no consistent architecture, are by necessity entangled with the underlying technology, and that is because they always begin as prototypes. Invariably these applications are typical first applications in a new technology. They are poorly structured, riddled with workarounds for features that the underlying technology does not correctly implement at that time, and even that is only under ideal conditions. Add some organizational incompetence and you have a poisonous brew.
Any Way Out?
I don’t think there is much to expect from the industry. Too many players, too much politics, and obviously a solution to these problems is not seen as a necessity. To the contrary. The industry makes much of its profit exactly by perpetuating the status quo. Thus we have to look elsewhere.
I have no readily available solution either. I had one, RPCmagiX, for a very narrow field, and even though I had another broader, but still limited solution for the field of creating web applications (I’ll come to that in another post), both are currently outdated. I didn’t get the resources to maintain and develop them further, but then, I have learned a lot in the process of creating them, and I plan to bring those experiences into a new process and toolset for application design. I will work on that on my own, in my free time, and I will document what I think about and what I do on this blog.
The results of my work, if any, will be published as Free Software, and from a certain point I will try to find people who join me in creating this work. I have no schedule though. If I were you, I would not bet on me finishing anything for you to use in a certain project. This is very much research, and due to being done in my free time, I can’t promise anything. On the other hand, we all may learn something on that way, and even if it all comes to nothing, we may at least have some fun trying it 