Most aggravating developer induced headaches for Middleware Administrators

In recent months, I have had a lot of dealings with middleware administrators, responsible for the management of WebLogic Server and other Fusion Middleware components such as SOA Suite, Oracle Service Bus, WebCenter and UCM. My role is frequently one that allows me to step back a little and observe. These observations made it very clear that not only is middleware administration a challenging task – one that is underestimated in many organizations – it is also one that is frequently made much harder than necessary by actions that developers take or do not take. Developers create the artifacts that administrators will deploy and manage on the middleware infrastructure. These developers can make life easier for the administrators if they adhere to certain best practices in creating and handing over these artifacts. However, out of ignorance, disinterest or lack of time it is unfortunately common for administrators to experience severe frustrations over the work of developers.

I am trying to compile a list of various points of frustration for middleware administrators caused by developer ignorance or carelessness. Below is the list if have compiled so far – with some help from middleware experts Jacco Landlust, Edwin Biemond and Simon Haslam. If you – yes dear reader, you! – can help me in naming other frequent frictions between development team and administrator, please write a comment on this article. Your help is of course much appreciated.

Note: the purpose of this article – apart from encouraging middleware administrators – is to try to help make clear to middleware developers how they can contribute to the run time success of the software they create – by the way the only success that really counts.

Common frustrations for middleware administrators contributed to by developers include:

• (harmless, innocent) exceptions and stack traces showing up in log files (developer comment: “oh, that is not a problem, we see that one all the time, you can just ignore it”)
• programmatic JDBC connection management (instead of proper Data Source usage)
• other inappropriate use or even lack of use of application server facilities – such as custom authentication mechanisms with user tables accessed from the application instead of using OPSS (Platform Security Services on WebLogic and FMW)
• hard coded ‘configuration stuff’ such as end points, directory names, log-files, communication protocols and other addresses, making it virtually impossible to script deployment and optimize the run time environment;
• only slightly better is use of configuration files and properties files (such as oscache.properties or log4j.properties) that are subsequently packaged inside the JAR and EAR files – making them inaccessible to administrators without manual effort; also: configuration files referred to through hard coded paths pointing to the binaries of the middleware infrastructure
• unserializable sessions (limiting options for clustering and fail over)
• lack of cache headers on static resources – causing every static resource to be retrieved through the app server
• frequent requery from the database of essentially static data (no middle tier memory cache for shared reference data and slowly changing data); also: no service result caching to prevent unnecessary load on enterprise back ends in case of frequent access of (cacheable) services
• lack of meaningful logging, lack of logging configurability (such as the ability to dynamically change log level or a log area of interest); lack of the ability to group log-messages per client session or conversation; endless Mb-s and more Mb-s of unstructured logfiles that require endless sifting in text editors; no proper use of log-file rotations.
• inappropriate packaging up of the application (typically one big ear or lots of smaller ones without any obvious logic to the division – a big pain in attempts to minimize redeployment effort and increase availability during redeployment)
• bundling of 3rd party libraries when they either are already installed or could be shared across multiple apps.
• missing, incomplete or incorrect installation/configuration instructions with new deployments – meaning the administrator does not know what and how to deploy
• and the number one sign (according to one of the experts) of a weak development team: they have a different –  i.e. specially configured – set of ear/war files for production as compared to test. (rather than configurable settings managed through environment specific deployment plans)
• a non-representative acceptance test: an acceptance test in an environment that differs from the production environment (in meaningful ways) that is therefore quite useless with regard to non-functional aspects
• Lack of error handling: exceptions that could be handled automatically are not dealt with – for unrecoverable errors, there may not be proper instructions for administrators on how to handle them and/or there may not be enough or readily accessible information required for dealing with these exceptions
• Lack of documentation on which (versions of which) components – both the custom built artifacts, the 3rd party libraries and the standard FMW components – should be deployed; also lack of documentation on the interdependencies between the deployed artifacts
• Lack of instructions on monitoring: what should administrators be checking in order to verify the correct functioning of the system and to preempt possible failure; also: lack of test-scripts that administrators can run to verify correct installation and operation of artifacts
• introducing complexity – such as clustering or use of SSL – in a very late stage (Production Acceptance or even Production environment)
• inappropriate use of version labels: either no version labels at all or reusing the same version label for different artifacts
• instructions for use of XA Data Sources when applications really are fine with non-XA Data Sources (XA can cause serious headaches, especially in RAC environments)
• roll out software with development settings in place (for example debug automatically on)
• use of the database as a dumb data store – leading to inappropriate operations in the middleware artifacts (including validation, aggregation, sorting causing massive, unnecessary data traffic from database to middle tier)
• inability to dynamically refresh resources – such as reference data, resource bundles, configuration settings – forcing the administrator to completely restart the application in order to apply required changes

Surely this list is required reading for any development team producing software artifacts that are to be deployed on middleware infrastructures. And perhaps middleware administrators can unite – to come up with clear guidelines that deliverables offered for deployment should adhere to.

As stated above – let me know if you have entries to add to this list.