Inappropriate Use of a System Engineering Principle

It is obvious that current systems are more and more complex because they have to solve increasingly complex problems.
In enterprise computing I often observe a dangerous misinterpretation of the systems engineering principles of requirements partitioning and allocation.

A standard enterprise setting of large-scale software projects running on large server machines ("heavy-metal") often leads to the assumption that the non-functional requirements concerning performance, scalability, dependability (comprising the concepts of high-availability, reliability, security), and "operateability" (operational "fitness"; concerns like logging, monitoring, automation) have all to be realized by the system platform comprising of hardware, system software, and middleware. Thus, the developer "allocates" these quality attributes to the system platform. However, the allocation is mostly implicit, i.e., not explicitly decided, and most importantly not in accordance with the platform guys.

In my opinion this is a very dangerous setting. The system platform by itself is by no means able to provide these -ilities. Of course, there are many technological approaches and tools to improve the basic levels of provisioning these non-functional requirements (quality attributes!).
Nevertheless, without a complete view including also the application software all means of providing high-quality from the system-level up can be rendered ineffective.

As an example, just imagine a multi-location cluster of JEE application servers. Each member of the cluster is running the same enterprise application. The application itself uses stateful session beans and relies on the middleware to prvide it with high-availability. If now one site fails and, therefore, all application server cluster members on that site are busted. Due to the fact that the application did not take care for replication of the states all transactions in-flight on the failing site are dead and the users are themselves busted (they have to do the work again).

Another example would be an external service tied to the application without taking into account any fault model. In other words, the developer assumes the network connection to be absolutely available. Now if the connection fails the application also fails. So far so good. But we are going another step further. In a typical application server setting ressources are pooled. Imagine further that there is no handling in the application that takes into account that the external provider can fail and, therefore, all remaining mechanisms to contain the problem are middleware and operating system timeouts. In cases where you have a modest arrival rate of user requests and relatively long timeouts of several seconds (which is a typical necessity) you can easily find yourself in a situation where application server threads are eaten up bei requests waiting for the dead external connection.

I have observed outages due to such constellations in various situations. Now imagine yourself having the responsibility of reporting to top management and/or the customer that such outages "work as designed"!

So wouldn't it be better to include some sort of a failure model into the application so that these failures can be handled without causing a full user catastrophe? Definitely. More so, it is the responsibiltiy of every IT professional to address such problems. In the end it could even be an ethical question depending on the system's nature...

In my opinion there have to be people responsible for this kind of system-level thinking. One may call such people architects or technical leads or even senior designer. It is not that important how they are called. The important thing here is that they understand their responsibility and lead others to incorporate such thinking in their daily lives. Only then large and complex enterprise systems can yield to ever increasing customer expectations.