Saturday, June 07, 2008

Hell is Other People's Code

My current assignment has me analyzing code for an embedded system that been under development for 15 years.

There is a lot of money invested in the development of this software.

After 15 years of blood, sweat, and tears, the embedded system now needs modern hardware. But what to do about the very expensive software? Start over or devise some way to migrate the code base so that it may execute on the new hardware?

The idea of throwing away 15 years of software development is hard to swallow - so my task is to come up with a reuse/migration strategy.

But wait, there is more.
1. The software is very brittle and cannot accommodate modification.
2. Even basic maintenance activities are becoming a risky proposition.
3. The software is monolithic with no obvious means of splitting functionality into subsystems.

A secondary goal of the system modernization is to partition the software to increase robustness, maintainability and modifiability.

So I begin to look at the code. And to be honest, I was humbled by its cleanliness. The code was well commented, and consistent naming conventions and styles were applied throughout. It was the code of experienced software developers.

So let us take a step back. The tool I am employing in my analysis is to judge the competence of the engineers that created this troubled software. I think it is fair to carefully judge software developer competence as an analysis tool - but often there is more to the story than what meets the analyst's eye. For example most software systems are developed under the duress of schedule constraints - "trying to pack ten pounds of crap into a five pound bag". But I digress - back to my story.

The "coding in the small" looks good but I had already suspected that the problem existed in the design, or more specifically, how the software modules interact with one another. Let's put the software designer under the microscope.

Now the problem became very evident, many software modules where highly dependent on many other software modules. To be specific, it was easy to find software modules that were dependent on 50 other modules. It was impossible to create a informative diagram showing the dependencies in any type of meaningful way. I also searched for a more specific way to classify or present this problem to management but this also escaped me. This software simply was a rats nest of overly coupled modules that would continue to challenge our efforts to meet our goals.

So now I have indicted the software designer, hopefully he has made a clean get-a-way after 15 years. But now I need a motive - how could a software engineer let this happen? Lets travel back in time to 15 years ago and see what was going on. I remember those days and I remember that object-oriented techniques where still a little esoteric to the average developer. Many developers failed to make the leap to OO and many projects suffered through initial forays into OO. The main problem with OO in those days was in the development of frameworks and architectures. OO gave us the ability to define code modules that mapped to objects in our domain but the ability to create the executive sections of our software were often not understood and under-designed.

None the less, our indicted software designer still should have used the most basic and powerful software design technique, the idea of layering. But no, under analysis, the software has no structure.

The trial of the software designer continues. In defense of the designer, the software has been segmented into a multitude of modules that model the domain - but somehow this act of decomposition has created a mess. Let me introduce exhibit A, the programming language.

The system was programmed in the original version of Ada, the programming language developed for the U.S. Department of Defense in the late 1970's. Ada has been upgraded several times with the most notable upgrade coming in 1995 and termed Ada95, the original Ada is usually called Ada83.

Ada was specifically created to solve the so-called software crisis that plagued the development of large software systems. Yet here was a large software system in crisis, developed using Ada.

So now I will turn the glare of the spotlight away from the software designer and place it on this unpopular programming language. Now I must confess that I am not an "Ada man". I have worked on Ada projects for short terms but I have never become enamored with the language, although I know a few who have.

I looked-up a few of my (with all due respect) "grey haired" colleagues to borrow some manuals on Ada83, not Ada95 because I wanted to know what Ada developers were thinking back during the early days of the system's development. My Ada-savvy co-workers were eager to supply me with the manuals and to share a few thoughts with me.

Now Ada83 is not object-oriented but instead is object-based. Actually Ada83 is a procedural programming language similar to C. One main difference is that Ada includes the concept of a "package". The Ada package abstracts the "file module" and has some very powerful and useful capabilities. It is important to note that the Ada package is not a type like a Java or C++ class is a type. A package is used to define types, functions and variables - much like a class, but with the distinction that only one instance of a package exists in the program. A variable declared within a package is equivalent to a class-wide variable in Java or C++ - i.e a static variable.

My object-oriented mind now saw the embedded software system as ~500 singleton software classes - with public variables - mostly interconnected to one another. Could it be that the system's engineers did not really understand the object-based, procedural nature of the Ada language and used the Ada package as some type of brain-dead object-oriented class instead of a very powerful file module abstraction?

So let's be clear, in the object-based Ada language, the object is data defined, instantiated and processed by functions in the package, and not the package itself.

Interestingly, it all goes back to some of my previous musing of how State Stinks! Certainly the most stinkiest form of state would be static, globally accessible state.

I could go on but I have said all that I need to say about this matter. And I release the incarcerated designer from his cell, we all have skeletons in our closet and besides my job is not to find out what and why but how. How to fix this software and move it into the future. So far I am the one who is falling short of his goals.

But such is life in the hell of other people's code.

5 comments:

Dominic said...

When you say the modules are tightly coupled, you mean that each module is accessing the global variables, and arbitrary functions of other modules? If so, it should be possible to determine which functions and global variables are used in external modules, and these can then be marked up as such, and considered to be the public interface; you could use some grep or awk scripts to do this if Ada doesn't provide enough dynamism/introspection.

Then, when you want to refactor, you can, provided that you don't change the public interface, or if you do need to change it, have a little script to see which places it is used from, and change those at the same time.

Mike Petry said...

Dominic
I did use Perl scripts to conduct my analysis. I think the problem with the system lies in the incorrect definition of the interfaces with regards to public/private access and the lack of layering/subsystems which would have helped to localize things. I think the global, static variables are what makes the system so brittle. Your commment did get me thinking and it may be worth-while to post a blog discussing some ideas. I think the theme would be to use the programming paradigms supported by the particular language in use and to carefully manage and protect program data.
Anonymous
I tried to clean-up the grammar a bit. I appreciate the review.

Shawn Hartsock said...

I think I may have had to maintain that software. I had a project in Ada that while we tried to port it to Linux we found that the package dependencies were circular and somehow the cycle had been broken at one time on the Irix system but the binaries for Irix were being copied around so that a particular module was never actually re-compiled.

It meant that module (missing its source after all these years) had to be rewritten in ASM to get the million lines of Ada to compile on Linux. I left shortly after we figured that out so I have no idea what they did to solve the problem.

However, GNAT from AdaCore helped along way toward figuring things out.

Mike Petry said...

shawn,
I was amazed to find hundreds of circular references in this code. I was suprised that the code would even compile, but from what you are saying, changing compilers could be a problem.
I worked on another large Ada system and it also suffered from dependency proliferation but it was more in the form of type definition dependencies.

Victoryperfect said...

Thanks for sharing, I will bookmark and be back again

Embedded System Development