The Pattern Movement in Software Engineering has become very popular and has produced many books, some of them arguably extremely good, but apart from automatic refactoring in modern IDEs, the number of supporting tools is surprisingly low.

What Is A Pattern Anyway

You can get a good overview about Design Patterns at the Portland Pattern Repository. For the purpose of this article we just take the classic definition from the Gang Of Four book:

A design pattern systematically names, motivates, and explains a general design that addresses a recurring design problem in object-oriented systems. It describes the problem, the solution, when to apply the solution, and its consequences. It also gives implementation hints and examples. The solution is a general arrangement of objects and classes that solve the problem. The solution is customized and implemented to solve the problem in a particular context.

The crucial bit is the last sentence: Design patterns need to be implemented specifically for each particular problem. The classes in an object-oriented design pattern are no classes at all. They are archetypes, and for the particular situation real classes have to be crafted after those archetypes.

To put it in other words, a design pattern is an archetypical solution to an archetypical problem, and that’s also the reason why you can’t press patterns into libraries. If you compare two instances of the pattern, i.e. two cases where the pattern has been used to solve the problem, you’ll find almost no code that could be factored out. The pattern is not code, the pattern is a way the code is structured. Lexically two instances of a pattern have nothing in common, everything is only similar, nothing is the same.

In some cases it is even possible to put a pattern into a library, but when you begin to do so, you quickly find out that the result is not useful. Too much varies, the result are endless parameter lists or endlessly overloaded variations of the same methods. Those libraries are hard to implement and they don’t feel natural.

Generics (templates in C++) are a step in the right direction, but although they make creating archetypical libraries easier, those libraries are not necessarily easy to understand and use, much less to implement.

The Most Simple Patterns

Design patterns don’t come only at the level of class interaction in object-oriented systems. Just look at programming languages. In the beginning there was only assembly language. We basically wrote machine instructions, but after a while it turned out, that we did similar things over and again.

One of those things was to build loops of code. People did completely different things in the bodies of their loops, but there were only a few patterns for how the loop was entered, exited, and how often it was executed. With the advent of higher programming languages, those patterns were cast in basic language constructs such as “WHILE-DO”, “DO-UNTIL”, “FOR” or “FOREACH”, and these language constructs are still with us.

Assembly language did not have those high-level loops, but it had a conditional branch. Loops had to be implemented with conditional branches. The direct equivalent to that is the combination of the “IF” and “GOTO” statements in programming languages, and of course you can use those to implement all your loops. It’s only more verbose, the intent is less clear, and that is, because you don’t express your intent by naming it. Patterns are much about naming.

Pattern Languages

Interestingly enough, an “IF” statement is also a pattern. Just look at the infinite number of conditions. This brings us to another important fact, the fact that patterns are not all born equal. They come in hierarchies. There are base patterns and composite patterns. Just like with “IF-GOTO”, you can use patterns to build other, more complex patterns.

Now, if you look at where the pattern movement came from, being initiated by the architect Christopher Alexander, and if you read his books, you see that he clearly understood the hierarchical nature of design patterns. That’s why he called it a Pattern Language.

Well, I’ve never tried to build a house, and if I were to build one, I’d probably not apply all of his patterns. Many of them I agree with, some I don’t, and for most there are alternatives that he does not cover. But that again coincides nicely with the notion of languages.

My native language is German, I regularly use English, have some low level, basic and spotty understanding of some of the languages originating in Latin, but that’s it. No idea of Chinese, no idea of any African language, no idea of anything else. And still, all those languages, that I don’t have a clue of, can express exactly the same things that German or English can. They use only different patterns to solve the same problems.

So I guess we can agree that it is possible to build different houses, houses that don’t fit into Christopher Alexander’s language, and we may still be able to live in them and stay sane.

For the adoption of Alexander’s method, this may really have been detrimental. It is much too obvious that, although it makes sense as a system, you may not be able to use it and build your dream. It would be Alexander’s dream instead, and when it comes to our dreams, we really dislike any intrusion.

This may explain Alexander’s limited success (and I don’t want to belittle him, he is really one of my heroes, even if he has not taken over the world), but it would not be in our way building programs, would it? After all, houses are for individuals, and individuals care for their individuality, but CRUD is just CRUD, ain’t it?

Maybe not. People seem to have a tendency to willingly and knowingly reinvent the wheel. Look at me: I could be satisfied using the tools I have, using them to create the things I’m paid for, and otherwise have a life. Or else just join the development of the Eclipse Modeling tools. In a way there’s so much already out there, I can’t have much hope to make a valuable contribution. And still I do what I do, knowing that I’m bound to reinvent, willingly accepting it. And why? Because it’s fun :)

I don’t know if this explains anything, but fact is, that the Design Pattern movement in computer science has not produced anything that even remotely comes close to Alexander’s hierarchical completeness. With Alexander’s language you can build a house, a street, a village, a town, a region, and in the other direction you can go down to details like the actual building materials.

You can’t do that with design patterns in computer programming. The high-level patterns are all missing. The discipline has evolved to the point where it has become possible to talk to each other using design patterns, but it’s only at that certain level of detail where we talk about basic object interaction.

The Necessary Next Step

Looking into some very simple patterns of repeated assembly code, we found that these patterns became the loop constructs of modern programming languages. Those loop patterns describe arrangements of assembly instructions, but the important thing to note here is, that today nobody arranges those instructions themselves. We use compilers for that. This is very different from how we work with Gang-Of-Four patterns, because they have to be instantiated by hand.

A compiler can do what it does, because it has a complete model of the program. All variables, that the built-in patterns like loop constructs refer to, are also defined in the same model. The whole semantic is defined in terms of the language and its patterns, and where this is not the case, the semantics are defined by standardized libraries. So, actually what this boils down to is, that a programming language allows us to express the desired semantics by constructing a model. The compiler than applies its patterns to this model and this way constructs either code or calls into supporting libraries.

That’s exactly what code generators do. They take a model and translate it into code of a different, simpler language, and where it makes sense, they generate not code but calls into external libraries. In this context, libraries have two purposes:

  1. they avoid code duplication
  2. they allow us to express semantics that can’t be expressed in the language’s patterns

#1 is nice to have, but #2 is really important, because it means that we can generate code, even when our patterns are not a complete, self-sufficiant system. What we can’t express in our language, we simply assume implemented in libraries.

This is the way to go. We have to build higher-order languages, languages that implement recurring patterns just the way as today’s programming languages implement assembly loop patterns. Once we have such languages, we can translate them automatically into code. Just like with conventional programming languages, those things will be the more powerful, the more self-sufficient they are, but that does not mean that small and incomplete steps can’t be incredibly useful in their own right.

What I want to implement as my project are three things:

  1. an environment to specify models,
  2. a set of patterns that, when applied to such a model, turns the model into an
    application, and finally
  3. a code generator, that makes this process automatic

That’s it. Easy, huh?

 

In the last post we have seen that it will take me years to finish this project. The question is, what can I rely on, what can I build upon. What are the tools that I can use, without worrying that they are obsolete before my own tool even becomes usable?

Programming Language

In the case of RPCmagiX I have made a bad choice. I used Itcl 2.2 for implementing the GUI, and as soon as I had finished it, ITcl 3 came out and it was incompatible in some subtle but crucial ways. But even if it had not been so obvious, today Tcl is mostly dead.

Perl was my choice for WADL, and although Perl is still much more alive than Tcl ever was, it is not really dominant any more. Perl 6 has taken too much time, there was a lot of FUD about Perl’s imminent death, and at the moment Python is king and Ruby the fashionable contender.

I really hate Python. It’s personal, probably nothing that you’ll understand, but a language that relies on indentation for structure, c’mon! That’s nothing but a bad, tasteless joke. So Python can’t be it.

Ruby seems to be OK as a language, but I have no idea how it will fare. It could easily be the next Tcl. Remember: we are talking about at least three years before the shelf life of my program even begins!

COBOL would be a good candidate for a language that is guaranteed to never die. But who would want to work in COBOL? Nope, not even joking :)

And then there is another thing: I would like to use one language for the tool and for the code it creates.

Currently I work with Java. It took me a time to like it, but now I do. Yes, it’s a big ecosystem, yes it suffers from the fact that they had to reinvent the wheel and everything else that already was there, but now it is pretty mature and complete. And it has Eclipse. Working with Perl I never missed an IDE. A lot of Emacs windows on a big virtual desktop, some teminal windows, that’s all I ever needed.

Not so with Java. Java is pretty unusable without an IDE. But when you have something like Eclipse, when you have an integrated, incremental compiler, when you don’t write texts but actually manipulate parse trees, then all sorts of funny things are possible. Think of the refactoring support, and suddenly Java is a very flexible and dynamic language.

And Java has some other advantages: It won’t go away. Too much important code depends on it. Java is like Cobol, only a much more likeable language, and one with absolutely superior tools. Java is such an important language, because so much code for banks and insurances is written in it. There is a reason why there is a Java Enterprise Edition.

And for all that reasons my potential users are very, very likely to use Java. When I want to maximize the impact of my tool, Java is an ideal language.

But then, very similar things could be said about C#. Same category of language, same capabilities. A year ago I have written a nice mult-threaded system in it, and as a language I like it. Why not, it’s Java, deliberately disguised in an incompatible syntax, but there is no essential difference, and some solutions they have come up with, are really, really clever. For instance I like partial classes and miss them in Java.

On the other hand, it’s a matter of principle. I don’t like a language that is so tightly controlled by an entity like Microsoft. I don’t like working on Windows either. Yes, there is Mono, but when you go the Microsoft way, you really want Visual Studio.

Apart from that, it can have actual advantages when your tools are open source. Remember my ill-inspired decision for Itcl? Well, it is open source after all. For more than 10 years we have compiled Itcl 2.2 on all our AIX and Linux systems, and I see no reason why this can’t go on. A closed source core component would long since have been unsupported. Think of Visual Basic. Sure, Microsoft offered a migration path into .NET, but that’s only the core language. People often had augmented it with third-party components, and they often were not ported. We sure do have some of those cases.

Libraries And Frameworks

And then there’s all the stuff other people do. After all, what I’ll do is basically Model Driven Design. Eclipse has a whole big group of projects under that title. Shall I use them. Can I?

They play a different game. They play a numbers game. The Eclipse Modeling Project has seven people only on its Project Management Council. Look at the list of projects on that page. Everything on that list looks interesting and like something that I could need, but I would have to spend all my time, work, free, sleeping, just to follow all that.

And then think of all that constantly changing. Try imagining the hassle with keeping all that in sync. It’s impossible, and even if I could do that, it would not help me, it would only slow me down.

No. I won’t do it. It may be possible to catch up with these things later though, for instance at a time when I have some people helping me. If they are interested in the field at all, they would probably have worked with those tools and frameworks. They would probably know how to bridge the gap between EMF and my own core abstractions. After all, whatever I come up with, it must be semantically equivalent to EMF, because if it were not, it couldn’t be as expressive.

Yes, I know, in a way it’s a waste of time to re-invent such things, but I simply can’t tie my own core abstractions to something that is completely out of my control. This would be irresponsible. On the other hand, when my own modeling structures are largely equivalent, when they simply have to, we can always bridge that single gap later, and by doing so, we will gain access to the whole infrastructure. At least that’s the theory :D

So What’s Left?

I consider Java to be a good choice as a language, and I consider the Enterprise Edition in release 6 sufficiently mature. I will use all of that (well, probably not JSF, but who knows), and I won’t use any other framework.

The whole design will be very abstract, most of it completely oblivious of the environment it runs in. In the end there will be some channels where input comes from, the models will have to be stored, there will be code generators, but all that can be abstracted away, done by primitive stubs in the beginning, done by plugins later. The only things that I have to worry about now are my code abstractions and how expressive they are.

I can and should program away in my simple sandbox, not caring about slick user interfaces and high performance. While I do that, all sorts of things will happen. Computers will get faster, new frameworks, new GUI systems may come up and get mature, and I’ll connect to them when I need to. Doing it from the beginning would only be a waste of time.

 

So far I have not even said what exactly that upcoming project will be, and reality is, that I don’t know exactly yet, thus of course I can’t tell how long it will take me to create it. But still, it may be a good idea to think about time. It will take me the longer, the higher I aim. But how high shall I aim? And how much time do I have?

RPCmagiX, the subject of my Master’s Thesis, took me extraordinarily long, two implementations over the course of five or six years. But then, at that time I had not a clue of the whole business, yes I even lacked a clear understanding of where I was heading. I worked in a lab situation, I noticed that some work got repetitive, and I felt the need to do something about it.

WADL, my past project that I wrote about in the previous post, was created in a completely different situation. It was much bigger, much more advanced, but I had much more experience as well, and generating code already felt natural. Although I needed to implement a lot of infrastructure that today would be supplied by an application server, I was pretty fast. The base architecture and a first implementation took me less than half a year, almost all of the rest was done in the first two years, and all the while I myself was using it in actual projects, and in the second year I was also giving support to other developers.

The current situation is very different again. I have some very precise ideas, I think I mostly know what I need to know, but this time it is not a full-time job, this time it is something that I do in my free time. I may get support, but first I’ll have to provide something useful, something that people can play with, get excited about.

Well, that can easily put’s us in the multi-year category again, at least when I aim a little higher.

Let’s for the moment assume that it will take me a year to produce something that actually can be used. This will be an unstable implementation, no compatibility promises would be made, but the idea is, to have something that could be used as a shortcut in real projects. You would specify something, it would generate something, you would save a lot of time, but you could not hope to use any future version of the tool on that project. It would still be useful though, especially when you are in a hurry.

In order to be at that point in a year, I’ll have to have a rough implementation sometime next fall, I’d say October is a good time. One of the problems with code generators is, that you have to use them in order to iron out the bugs. I’ll have to make an actual project with my tool, and this has the potential to severely slow me down. I think I have a nice solution though. More about that later.

This is not a project schedule, there are no hard limits, it may easily take me even longer, but I think a year is a milestone. If I don’t have anything substantial in a year, I’ll cancel the project and don’t bother you any further.

So, I guess we’ll see nothing usable this year. I don’t have all time in the world though. Within a short time I will have committed to a core architecture and a rough feature set. If I want this program to be used (and of course I do, why else would I bother creating it?), I need to create something that is advanced enough to be useful at the time it begins to be usable. It’s the same problem that game developers face: you can’t make a new game for current technology, you have to design it with the future ecosystem in mind.

For me that means I should aim rather higher than lower. Not too high though, because if it takes me ten years to finish it, chances are, that other people have come up with pretty clever ideas in the meantime, and my ten year old design will show its age before it’s even done. I think five years would already be dangerously long, three years wouldn’t worry me though.

OK, without knowing what exactly I do, I have at least a rough time frame. An architecture and initial code in October, something useable for producing prototypes within a year, a commitment to compatibility in maybe two years, a first release in maybe three years, but not later than in five. Let’s see how this works out :)

 

Promises

Every two or three years we see the Next Big Thing. We had programming languages, Structured Programming, Object Orientation and all that stuff, and more recently things like Agile Software Development or SOA. Always we could see an industry developing, and always our managements have bought into it, and always things got … easier???

Yes and no. There is certainly an evolutionary process, and it is completely obvious, that today’s software development methods are ways ahead of what was done 50 years ago. But then, the industry favors what can be sold, and this brings its own share of problems.

A Look Back

20 years ago I wrote my programs in C. Now I use Java and sometimes C#, in between I have used Tcl/Tk and Perl a lot.

Scripting languages were a big step from C. Have you ever used regular expressions or hash tables in C? They are implemented as libraries and using them never feels natural, That’s very much different in modern scripting languages.

Java is a similar step forward, only in a slightly different direction. It not only releases me from the burden of keeping track of the memory allocated by my programs, it’s also strongly typed, and that makes a difference in big projects and in tool support. I use Eclipse as my Java IDE, and once you get used to that, there is no way back.

And still: although the tools have become so much better, the computers so much faster, we still struggle with the same class of problems we had 10 or 20 years ago.

RPCmagiX

In 1990 I began playing around with Remote Procedure Calls. Distributed Computing and Client/Server were big buzzwords then. It took me about seven years to come up with a solution that was also my Master Thesis. The tool, RPCmagiX, was meant to be published as free software, but in the end I failed at that. It was used internally though, and many of the programs made with it are still in production.

RPCmagiX was all about maximum simplification. Using RPC systems like DCE or ONC, you had to write a lot of boilerplate code, and you had to do it over and over again. Different RPC systems were incompatible not only on network protocol level, but also in the way programs had to be written to use them.

RPCmagiX did away with all of that. It used a nice unified model, and all you had to do, was to write a subroutine that would run in the server, along with a client that called the subroutine. You used a graphical tool to define the interface, datatypes, the procedures and their parameter lists, selected programming languages for client and server, and the tool would generate a library to be linked to the client, and a library including a main program, that could be linked to the procedure implementations, and that could be started as the server program. The only thing that client code used to refer to the service, was a first parameter in every procedure, a so-called service name, basically a structured name modeled after UNIX file names. The generated client stub and the client library used this name to look up matching server instances, connected to servers, handled automatic failover, etc. As a programmer, you did not have to write a single line of network code, and you had the additional benefit of being able to make cross-language calls. For instance we had Windows clients in Visual Basic calling UNIX servers written in Cobol.

RPCmagiX did a great job hiding the whole complexity of distributed computing inside of the interface. Using specification of the interface and code generation, it gave us lots of additional benefits as well, for instance you could generate data for statistical use, you could send notifications in case of failures, you could have load balancing, and all that in a standardized way. This was possible, because the system knew about the interface and could use all the information in the interface, but the actual application code was not at all entangled with networking concerns.

Using such a system gives you a clear boundary between what you use and what you write. If your application can do networking by knowing not more than a simple name of a service, if you can simply call a remote procedure without having to initialize network libraries, create stubs and connect them to sockets, if calling a remote procedure is a one-liner, only then you have a chance to later replace your networking system if you need to.

At that time we were using ONC (aka SUN RPC), because it was a free alternative to the then expensive DCE (OSF Distributed Computing Environment). We prepared to eventually make the switch to DCE, which we actually never did, but DCE largely failed anyway. Such is life :)

Whatever. Responsible application design must try to avoid creating new systems that immediately become the legacy systems of the future. We can do this by separating application logic from infrastructure code. If we do that, we can change infrastructure providers, not entirely without effort, new technologies must still be tested and encapsulated, but at least it is possible. Doing this, we protect our investment in analysis and avoid writing the same systems every ten years, over and again.

The State Of The Industry

Frankly, I would have expected the industry to come to similar conclusions, but interestingly enough it did not. We still get library system after library system, we still have to use factories to create objects that we need to initialize in order to make them do their tasks. We see progress with technologies like the Java Enterprise Edition (JEE) or Spring, where we have Dependency Injection, but although I am very impressed with JEE and the elegant solutions they have come up with in release 5 and 6, there are plenty of other fields where it is bad as it ever was.

The answer is simple. Consensus in the industry is always the result of political struggle, very often even of outright war. It’s Betamax/VHS/Video 2000 all over again. Remember HD-DVD vs BluRay? Sometimes the better system wins (may have been the case with BluRay), sometimes it’s the most inferior (as it was certainly with VHS), but in no case is it due to technical merits.

And it gets even worse when you look at how buying decisions are made and who makes them. Wave after wave of management fashions washes over our heads, and although we are lucky and many of the latest buzzwords never leave management circles, some do, and then we are confronted with requests like to build a Service Oriented Architecture (SOA) based upon an Enterprise Service Bus. And that all, because some clever consultants sold management the idea, that such a mythical beast would allow non-programmers to click together meaningful applications from a host of useful generic services.

Of course this won’t happen and to a programmer, the whole idea looks ridiculously naive, but just having to constantly evaluate the currently fashionable paradigm and to cope with the ever immature and unreliable tools, just the pure effort to implement applications on poorly understood foundations, just this does all the damage.

The result is a succession of applications that have nothing in common, have no consistent architecture, are by necessity entangled with the underlying technology, and that is because they always begin as prototypes. Invariably these applications are typical first applications in a new technology. They are poorly structured, riddled with workarounds for features that the underlying technology does not correctly implement at that time, and even that is only under ideal conditions. Add some organizational incompetence and you have a poisonous brew.

Any Way Out?

I don’t think there is much to expect from the industry. Too many players, too much politics, and obviously a solution to these problems is not seen as a necessity. To the contrary. The industry makes much of its profit exactly by perpetuating the status quo. Thus we have to look elsewhere.

I have no readily available solution either. I had one, RPCmagiX, for a very narrow field, and even though I had another broader, but still limited solution for the field of creating web applications (I’ll come to that in another post), both are currently outdated. I didn’t get the resources to maintain and develop them further, but then, I have learned a lot in the process of creating them, and I plan to bring those experiences into a new process and toolset for application design. I will work on that on my own, in my free time, and I will document what I think about and what I do on this blog.

The results of my work, if any, will be published as Free Software, and from a certain point I will try to find people who join me in creating this work. I have no schedule though. If I were you, I would not bet on me finishing anything for you to use in a certain project. This is very much research, and due to being done in my free time, I can’t promise anything. On the other hand, we all may learn something on that way, and even if it all comes to nothing, we may at least have some fun trying it :D

© 2010 Andreas Manessinger Suffusion theme by Sayontan Sinha