Python, NumPy and matplotlib - Visualize Matrix Rotations

I am using Python, NumPy and matplotlib to experiment with, and visualize 3-D graphics techniques.  The first step was figuring out how to generate polar plots with matplotlib and to determine the domain and range of spherical to Cartesian coordinate conversions.

The next step is to rotate a point in space using matrices, and then to explore the concept of gimbal lock.  I will probably tackle gimbal lock in a future posting.  Before delving into 3-D rotations, I needed to figure out how to extend my plotting knowledge.

I cleaned up the code from the previous posting, moving the spherical to Cartesian coordinates conversion code to a separate module - sphcoords. sphcoords also includes a function that takes spherical coordinates and returns a 1x3 NumPy matrix that represents a point in space.

import numpy as np
from numpy import matrix

def x_cart(incl_sph_coords, azim_sph_coords):
   return np.sin(incl_sph_coords) * np.cos(azim_sph_coords)

def y_cart(incl_sph_coords, azim_sph_coords):
   return np.sin(incl_sph_coords) * np.sin(azim_sph_coords)

def z_cart(incl_sph_coords):
   return np.cos(incl_sph_coords)

def sph2cart(incl_sph_coord, azim_sph_coord):
   return np.matrix([
   x_cart(incl_sph_coord, azim_sph_coord),
   y_cart(incl_sph_coord, azim_sph_coord),
   z_cart(incl_sph_coord)])

Below is an example of sphcoords function, sph2cart which generates a 1x3 matrix containing Cartesian coordinates. In this case, spherical coordinates of 30 degrees inclination and 180 degrees azimuth are converted.

poi = sphcoords.sph2cart(np.pi/6, -np.pi+(10*azi_incr))

By using matplotlib's subplot capability along with spherical to Cartesian coordinate conversions, I was able to depict a 3-D spherical model of space.  A single red "point of interest" will be rotated using rotational matrices.

The initial position of the red point is arbitrary.  The red point will be initially rotated about the z axis.  The rotated position of the red point will remain along the point's original circle of inclination.  This is what I was hoping  to illustrate with the cyan-colored annotations.

Before going further, let me apologize for my inability to generate circles. The elliptical, squashed figures in my plots will be circular after I figure out how to set fixed aspect ratios.

The Initial Position of the Red Point of Interest

Here are the rotational matrices that were implemented in module rotmat using the NumPy Matrix. 

import numpy as np
from numpy import matrix

def x_rot(rads):
  return np.matrix([
  [1,0,0], 
  [0, np.cos(rads), -np.sin(rads)],
  [0, np.sin(rads), np.cos(rads)]])

def y_rot(rads):
  return np.matrix([
  [np.cos(rads), 0, np.sin(rads)],
  [0, 1, 0], 
  [-np.sin(rads), 0, np.cos(rads)]])

def z_rot(rads):
  return np.matrix([
  [np.cos(rads), -np.sin(rads), 0],
  [np.sin(rads), np.cos(rads), 0],
  [0,0,1]]) 

The 1x3 matrix containing the point of interest's (POI) Cartesian coordinates is rotated about the z axis thusly:

poi = poi*rotmat.z_rot(-5*azi_incr)

The plot below shows the POI rotated 90 degrees about the z axis.  The cyan annotations show the range of the next rotation which is about the x axis.  The POI will be rotated 30 degrees about the x axis.  The most interesting update will be on the upper right plot that will show the the POI jumping from the yellow 30 degrees circle to the black circle at 60 degrees.

After Rotating the POI about the Z axis.

After Rotating the POI about the X axis.

 

The final position of the POI after rotating about each Euler Axis.

 


import numpy as np
import matplotlib.pylab as plt
from numpy import matrix
import rotmat
import sphcoords


def generate_plot(inclin, azi, poi):
# plot x-y plane
plt.subplot(221)
plt.plot(
   sphcoords.x_cart(inclin[0],azi[0]),
   sphcoords.y_cart(inclin[0],azi[0]), "g-o",
   sphcoords.x_cart(inclin[1],azi[1]),
   sphcoords.y_cart(inclin[1],azi[1]), "y-o",
   sphcoords.x_cart(inclin[2],azi[2]),
   sphcoords.y_cart(inclin[2],azi[2]), "y-o",
   sphcoords.x_cart(inclin[3],azi[3]),
   sphcoords.y_cart(inclin[3],azi[3]), "k-o",
   sphcoords.x_cart(inclin[4],azi[4]),
   sphcoords.y_cart(inclin[4],azi[4]), "k-o",
   sphcoords.x_cart(inclin[5],azi[5]),
   sphcoords.y_cart(inclin[5],azi[5]), "b-o",
   sphcoords.x_cart(inclin[6],azi[6]),
   sphcoords.y_cart(inclin[6],azi[6]), "b-o",
   poi[0,0], poi[0,1], "r-o")
plt.grid(True)
plt.ylabel('Y Axis')
plt.xlabel('X Axis')

# plot y-z plane
plt.subplot(222)
plt.plot(
   sphcoords.z_cart(inclin[0]),
   sphcoords.y_cart(inclin[0],azi[0]), "g-o",
   sphcoords.z_cart(inclin[1]),
   sphcoords.y_cart(inclin[1],azi[1]), "y-o",
   sphcoords.z_cart(inclin[2]),
   sphcoords.y_cart(inclin[2],azi[2]), "y-o",
   sphcoords.z_cart(inclin[3]),
   sphcoords.y_cart(inclin[3],azi[3]), "k-o",
   sphcoords.z_cart(inclin[4]),
   sphcoords.y_cart(inclin[4],azi[4]), "k-o",
   sphcoords.z_cart(inclin[5]),
   sphcoords.y_cart(inclin[5],azi[6]), "b-o",
   sphcoords.z_cart(inclin[6]),
   sphcoords.y_cart(inclin[6],azi[6]), "b-o",
   poi[0,2], poi[0,1], "r-o")
plt.xlabel('Z Axis')
plt.grid(True)

#plot x-z plane
plt.subplot(223)
plt.plot(
   sphcoords.x_cart(inclin[0],azi[0]),
   sphcoords.z_cart(inclin[0]),"g-o",
   sphcoords.x_cart(inclin[1],azi[1]),
   sphcoords.z_cart(inclin[1]),"y-o",
   sphcoords.x_cart(inclin[2],azi[2]),
   sphcoords.z_cart(inclin[2]),"y-o",
   sphcoords.x_cart(inclin[3],azi[3]),
   sphcoords.z_cart(inclin[3]),"k-o",
   sphcoords.x_cart(inclin[4],azi[4]),
   sphcoords.z_cart(inclin[4]),"k-o",
   sphcoords.x_cart(inclin[5],azi[5]),
   sphcoords.z_cart(inclin[5]),"b-o",
   sphcoords.x_cart(inclin[6],azi[6]),
   sphcoords.z_cart(inclin[6]),"b-o",
   poi[0,0], poi[0,2], "r-o")
plt.ylabel('Z Axis')
plt.grid(True)
plt.show()

if __name__ == '__main__':
   inclin = []
   azi = []

   inclin_points = range(0,20)
   azi_incr = (2 * np.pi)/float(20)
   azi_points = np.arange(-np.pi, np.pi, azi_incr)

   #90 deg inclination
   inclin.append([np.pi/2 for i in inclin_points])
   #30,150 deg inclinations
   inclin.append([np.pi/6 for i in inclin_points])
   inclin.append([np.pi-np.pi/6 for i in inclin_points])
   #60,120 deg inclinations
   inclin.append([np.pi/3 for i in inclin_points])
   inclin.append([np.pi-np.pi/3 for i in inclin_points])
   # poles
   inclin.append([0])
   inclin.append([np.pi])

   azi.append(azi_points)
   azi.append(azi_points)
   azi.append(azi_points)
   azi.append(azi_points)
   azi.append(azi_points)
   azi.append([np.pi])
   azi.append([np.pi])

   poi = sphcoords.sph2cart(np.pi/6, -np.pi+(10*azi_incr))
   poi = poi*rotmat.z_rot(-5*azi_incr)
   poi = poi*rotmat.x_rot(np.pi/6)
   poi = poi*rotmat.y_rot(np.pi/2)
   generate_plot(inclin, azi, poi)

Spherical to Cartesian Coords - Thanks Python!

Game and graphics programming requires the modeling of three-dimensional geometries in software.  This stuff is relatively simple in theory but easy to screw up and difficult to debug. 

The math used in 3-D geometrical programming has its limitations and special cases.  One example is the gimbal lock problem that occurs when rotating a body using matrices and Euler angles.

Another problem that I recently stumbled across occurred when trying to find the intersection point of a line and plane.  If the line is pointing away from the plane, the intersection point ends up being behind the line's start point.  Essentially, if we shoot an arrow directly away from a target, it will still hit the target, but with the back of the arrow.

The problem I describe above with the intersection point, line and plane is not really a problem at all.  The special case is easily detected while computing the result and the code can respond as required.

The real issue is that we must have a complete understanding of the mathematical techniques we use in our software.

There are two ways to understand math, one is to use analysis, the other is intuition/visualization.  Analysis is the stronger of the two methods and includes things such as proofs.  Intuition, visualization and experimentation is the approach that I take.

My latest interest is converting spherical coordinates into Cartesian coordinates.  I can easily plug and chug using the standard formulas, but after learning a few hard lessons, I know I had better gain some understanding.

When it comes to three-dimensional programming, it is not as easy to visualize 3-D space as one would imagine.  I needed some help to visualize this stuff.

Locations on earth are identified using spherical coordinates, using the intersection of two angles, an angle of inclination (latitude) and an angle of azimuth (longitude).  In my investigation, I would like to consider inclination angles from 0 to 180 degrees, azimuths from 0 to 360 degrees and a spherical radius of one.

The plot simplifies the problem by reducing three-dimensions to a two-dimensional display with circles of common inclination.  This is essentially a polar-plot. Each circle of common inclination (latitude) represents two inclinations, the inclination (northern hemisphere) and 180 - inclination (southern hemisphere).

As a two-dimensional polar plot, the problem is solved using the azimuth angle, and the distance from the origin to the proper circle of common inclination.

It appears from this work, that spherical to cartesian coordinate conversions is a simple and safe process (with the exception of when the inclination angle is 0 or 180 degrees, of course).


import numpy as np
import matplotlib.pylab as plt

def x(inclin, azi):
""" compute the x cartesian coord """
   return np.sin(inclin) * np.cos(azi)

def y(inclin, azi):
""" compute the y cartesian coord """
   return np.sin(inclin) * np.sin(azi)

def z(inclin):
""" compute the z cartesian coord """
   return np.cos(inclin)


inclin = []
inclin.append([np.pi/2 for i in range(0,20)])
inclin.append([np.pi/3 for i in range(0,20)])
inclin.append([np.pi/4 for i in range(0,20)])
inclin.append([np.pi/6 for i in range(0,20)])
inclin.append([np.pi/8 for i in range(0,20)])
inclin.append([np.pi/12 for i in range(0,20)])
inclin.append([np.pi/24 for i in range(0,20)])
inclin.append([0 for i in range(0,20)])
azi = np.arange(-np.pi, np.pi, (2 * np.pi)/float(20))

plt.plot(
   x(inclin[0],azi), y(inclin[0],azi), "g-o",
   x(inclin[1],azi), y(inclin[1],azi), "r-o",
   x(inclin[2],azi), y(inclin[2],azi), "c-o",
   x(inclin[3],azi), y(inclin[3],azi), "m-o",
   x(inclin[4],azi), y(inclin[4],azi), "y-o",
   x(inclin[5],azi), y(inclin[5],azi), "k-o",
   x(inclin[6],azi), y(inclin[6],azi), "b-o",
   x(inclin[7],azi), y(inclin[7],azi), "w-o")
plt.legend(
   ('90', '60, 120', '45, 135',
   '30, 150', '23, 157',
   '15, 165', '8, 172', '0, 180'))
plt.grid(True)
plt.ylabel('Y Cartesian Coord')
plt.title('Cartesian from Spherical')
plt.xlabel('X Cartesian Coord')
plt.show()



The Cult of Software Engineering Academia

This morning I was online checking-out a software architecture conference that I had attended before and was considering to attend this year. I was looking over the keynote speakers and found the bio of one particular speaker to be interesting.

The speaker, a PhD and college professor is a leading authority on modern software development methodologies.  Her focus is guiding organizations to institute the cultural changes required to adopt modern software development methods.

Her bio included a quotable-quote or catch-phrase that went like this: "You can take a man out of the Stone Age, but you can't take the Stone Age out of the man".

I was rocked-back by the arrogance of that statement.  I am sure that she and I would be on the same page regarding software methods and the need for cultural change, but to publicly blast those who are not ready to accept your opinions as "less-evolved" is arrogant and absurd.

Perhaps we are seeing the deficiencies in her culture. The culture of academia. A culture accustomed to feeding bright, fresh, hungry minds. A culture accustomed to unquestionable omnipotence over its audience.

This culture of education has attempted to recast itself into a "culture of change", targeting the software industry. But in the process of recasting itself, it has done little in the way of introspection.

Perhaps the most important skills fostered by working software professionals include communication, collaboration and negotiation. Influence does not come easy and neither does experience. Both have to be earned.

Veteran software professionals have learned to listen as well as to speak, to accept criticism and to consider the ideas and opinions of others. Veteran software professionals would feel negatively about a one-way monologue that targets topics so close to the practice of their art.

I am not suggesting that "change agents" engage in conversation as a way to "handle" an audience, using dialog merely as a soft-skill. Instead I would hope that any discussion would be an honest, open, two-way exchange of ideas.

Of course, this may require the "change agent" to closely engage with her audience. In effect, to lose some of her elite status in order to gain the acceptance of her ideas.

To be a participant instead of a prescriber.

Write Software Requirements For The Love Of It

Yeah sure. Right. Who am I kidding, writing software requirements kind of stinks. But right now, I cannot imagine a better use of my time.

Writing Good Software Requirements is Hard

Writing good requirements is a least as hard as writing good code. One key attribute of a good software requirement is that it must be unambiguous. The English language is notoriously ambiguous. The legal profession, political debate and religious holy wars are based on differing interpretations of the written word. Conversely, software programming languages are designed to be unambiguous because they have to be interpreted or parsed by a machine. Perhaps this is why programmers love to code - the machine detects and points out errors, or better yet, validates our efforts by correctly performing the operations that we have specified.

This brings takes us to another key attribute of a good software requirement, the requirement must be correct. This is obvious but the critical point is that the requirement must be perceived as correct by all of the software system's stakeholders. The point of the SRS is to capture a commonly agreed upon view of how the software system will function. The act of generating the SRS can end up being a process of socialization and team building. (Right now most programmers are saying "Have fun, I will be in the lab writing code".)

Looking at some job postings, they all seem to seek programming language expertise and experience. Bah! Programming is natural, joyous and simple compared to writing requirements. Writing requirements is like being a salmon, battling its way up stream to spawn and die. Well, maybe its not that bad, but its not as gratifying as writing code.

The Greatest Software Requirements Pitfall

Specifying software requirements that includes design and implementation details will lead you and your project to your doom.

It is actually difficult, unnatural, and odd to specify system requirements in way that is bereft of design and implementation. For one thing, as software developers, we are primarily problem solvers and solution creators. We are chomping on the bit to design, and implement. Secondly, we are comfortable talking in terms of the software system itself. Specifying software functions without including implementation details is not easy.

Writing software requirements that includes design and implementation is a trap. Essentially, if the implementation is used to define requirements, the implementation itself must be controlled. This leads to an explosion of requirements documentation. It becomes difficult to make even simple changes to the system without stakeholder consensus and documentation maintenance.

A Software Requirements Fallacy

There is an old software engineering joke, one developer says to the other, "You start coding while I go and get the requirements". At this point all learned software engineers slap their knees and laugh because its funny to think that some developers would start building a system before they knew what the system was required to do.

I believe this is line of thought is incorrect. Software requirements are not necessary to start software system development. Software requirements are paramount when trying to figure out if software development is complete.

Initial development can begin with some basic knowledge of the intended system. Initial development, proof-of-concepts, and prototypes will foster an understanding of the domain, encourage communications and team-building.

Essentially, I am suggesting that the act of software requirements specification does not have to imply the use of the waterfall methodology. I am suggesting that software requirements specification works well with iterative and incremental methods.

Unfortunately I cannot speak to how Agile methodologies factor in software requirements. I would bet that most include some form of functional description document. Perhaps the system is developed in increments that leads to some collective consensus of the intended systems functions.

A Software Requirements Truth

An SRS is simply a checklist of functions that a software system must perform.

Ultimately, the effectiveness and usability of the software system, as perceived by its end-users, is primarily driven by the talent, dedication and expertise of the software development team. This is because the one, unwritten requirement levied on all software systems should be:

The software system shall not stink.

Friqing Out with Python Closures

You have to love snow days. After spending an hour shoveling snow this morning, I fired up my lap-top for a guilt-free afternoon of tinkering around with some code. I opened up some Python code that I had coded last year when I was discovering functional programming with Erlang and relating it to languages that I was familiar with.

Python is a great language for exploration. Back in the old days I did a lot of programming using C++ and Microsoft's Component Object Model (COM). C++ COM is a fairly complex technology so I found it preferable to prototype object-oriented designs using Java. The Java language's interface mechanism was a good analog for COM's interface-based programming paradigm. Java was a sexy thing compared to COM and Java was the lingua-franca of object oriented design during this period.

Then I discovered Python and found that it featured a familar approach to object orientation and it looked like it would support rapid prototyping and concept exploration.

Python proved to be a excellent language, but it did not serve well as a prototypical language for C++ applications. In the C++ language, data types are everything. Most of the advanced features of the language, such as templates and classes, provide ways to manage functionality across data types. The C++ compiler is the ultimate master and the programmer is subjected to its rule of type.

Python cannot be used to prototype solutions to C++ problems because the problems do not appear in Python.

Most circa 2000 C++ programmer were ill-equipped to absorb Python. To absorb Python meant to be influenced and changed by Python - to refute idioms previously held sacrosanct.

To close the loop, a next logical step would be to gain perspective by relating Python to functional programming languages and not just to Java and C++.

My Python Data Structures Project

I go through a cycle every now and again that involves Python and data structures. A good data structure to code is the Priority Queue. The binary heap-based Priority Queue features a tree data structure, implemented using an array and clever array index manipulations. A binary min heap is shown below. Essential a binary heap must always be correctly ordered, children must always be greater than its parents (in the case of the binary min heap).



The binary heap is stored in a simple array. Algorithms maintain the relations between array elements as shown below. One exception is that it is easier to implement the array-based binary heap if the first element is stored at index 1 in the array.

One reason to code up a Priority Queue is that it is a building-block data structure with multiple uses. The Priority Queue conceptually fits well into an abstract data type (ADT) such as a C++/Java/Python class. In fact there are Priority Queue ADTs in all three languages. My recent investigations into functional programming led me to create a Priority Queue using lexical closures. My original class-based Priority Queue was named priq. Thus the functional-closure based version became friq. And indeed it is a friq.

The friq

The screen shot above shows the definition of friq which is function with a single parameter lt. Parameter lt is a comparison "less-than" function which is easier demonstrated than explained. It will be demonstrated in the next section. Note that friq is a function definition that encloses other function definitions, and a rather strange nested list heef. The functionality exported by friq is done with its return statement (see below).



The functions enqueue and dequeue shown above are the two functions, enclosed by friq, that are exported to the outside world using friq's return statement.

Functions down_heap and up_heap implement the heavy lifting algorithms for the priority queue and are not exported to the outside world.

Finally we have the enclosing function friq returning a tuple containing the functions we wish to export.

Demonstrating the friq

The following code demonstrates usage of the friq. A list d is loaded with some random integers. The friq is invoked returning a tuple containing a function to enqueue values and another for dequeueing. List d is iterated and its values are enqueued. Then the friq is drained by dequeueing.

The usage of friq appears reasonably simple and straighforward. Note friq's lt parameter being passed a Python lambda function. The results are seen below.

Next we have a slightly more complex demonstration in which a tuple containing a number and a string is enqueued. Note that the code that handles dequeueing has to do extra work to expand the tuple. Also note that the lambda function is slightly more complex in order to handle the tuple. The index 0 signifies that the sort will be on the tuple's number.

The next demonstration is identifical to the previous with the exception that the lamda function uses an index of 1 which signifies that the sort will be on the tuple's string.



And finally, a display of the results of both demonstrations that involve the (number, str) tuple.




Here is the complete listing of friq.py

friq.py

#! /usr/local/bin/python

def friq(lt):
    """ 
    function friq that returns lexical closure function objects.    
    param lt - a 'less than' function or lambda
    """
    heef = [[None]]

    def enqueue(x):
        """
        enqueue - visible function 
        Add to the priority queue
        """
        heap = heef[0]
        heap.append(x)
        up_heap(heap)
        
    def dequeue():
        """ 
        dequeue - visible function 
        Get the highest priority elemement
        """
        value = None
        heap = heef[0]
        if lenf() > 0:
            # The dequeue value is taken from the 2nd element in the array.
            # The first element is not used to ease index arithmetic.
            value = heap[1]
            if lenf() > 1:
                # The last element value is copied to the first position.
                heap[1] = heap[-1]
            # The last value is deleted to reduce the size of the pq by one.
            del heap[-1]
            if lenf() > 1:
                # The val at heap[1] is moved down heap to maintain order.
    down_heap(heap)
        return value

    def heap():
        """ 
        heap - non-visible function 
        Returns the underlying array-based binary heap.
        Can be made visible for debug.    
        """
        return heef[0]

    def lenf():
        """ 
        lenf - non-visible function
        Returns the size of the underlying array-based binary heap.
        Can be made visible for debug.    
        """
        return len(heef[0]) - 1

    def down_heap(heap):
        """ 
        down-heap - non-visible function 
        The first element is moved down heap as required
        to satisfy heap order.
        """
        # px is an index to a bi heap parent
        # cx is an index to a bi heap child 
        px = 1
        v = heap[px]
        while px <= (len(heap)-1)//2:
            # calc the index of the child
            cx = px + px
            # find the index of the min of the two children 
            # (if there are two).
            if cx < len(heap) - 1 and \
                lt(heap[cx + 1], heap[cx]):
                cx = cx + 1
            # make the comparison - if v is higher pri - we are done.
            if lt(v, heap[cx]):
                break;
            else:
                # move the childs value to the parent
                heap[px] = heap[cx]
                # make the parent index that of the child.
                px  = cx  
        # finally the v is set to the current parent
        heap[px] = v

    def up_heap(heap):
        """ 
        up-heap - non-visible function
        The last element is moved up heap to satisfy the heap order.
        """
        # cx//2 idenifies cx's parent in the bi heap.
        cx = len(heap) - 1
        v = heap[cx]
        while cx//2 > 0 and lt(v, heap[cx//2]): 
            heap[cx] = heap[cx//2]
            cx = cx//2
        heap[cx] = v

    # the enclosing function returning functions that are to be visible
    return (enqueue, dequeue)

Hell is Other People's Code

My current assignment has me analyzing code for an embedded system that been under development for 15 years.

There is a lot of money invested in the development of this software.

After 15 years of blood, sweat, and tears, the embedded system now needs modern hardware. But what to do about the very expensive software? Start over or devise some way to migrate the code base so that it may execute on the new hardware?

The idea of throwing away 15 years of software development is hard to swallow - so my task is to come up with a reuse/migration strategy.

But wait, there is more.
1. The software is very brittle and cannot accommodate modification.
2. Even basic maintenance activities are becoming a risky proposition.
3. The software is monolithic with no obvious means of splitting functionality into subsystems.

A secondary goal of the system modernization is to partition the software to increase robustness, maintainability and modifiability.

So I begin to look at the code. And to be honest, I was humbled by its cleanliness. The code was well commented, and consistent naming conventions and styles were applied throughout. It was the code of experienced software developers.

So let us take a step back. The tool I am employing in my analysis is to judge the competence of the engineers that created this troubled software. I think it is fair to carefully judge software developer competence as an analysis tool - but often there is more to the story than what meets the analyst's eye. For example most software systems are developed under the duress of schedule constraints - "trying to pack ten pounds of crap into a five pound bag". But I digress - back to my story.

The "coding in the small" looks good but I had already suspected that the problem existed in the design, or more specifically, how the software modules interact with one another. Let's put the software designer under the microscope.

Now the problem became very evident, many software modules where highly dependent on many other software modules. To be specific, it was easy to find software modules that were dependent on 50 other modules. It was impossible to create a informative diagram showing the dependencies in any type of meaningful way. I also searched for a more specific way to classify or present this problem to management but this also escaped me. This software simply was a rats nest of overly coupled modules that would continue to challenge our efforts to meet our goals.

So now I have indicted the software designer, hopefully he has made a clean get-a-way after 15 years. But now I need a motive - how could a software engineer let this happen? Lets travel back in time to 15 years ago and see what was going on. I remember those days and I remember that object-oriented techniques where still a little esoteric to the average developer. Many developers failed to make the leap to OO and many projects suffered through initial forays into OO. The main problem with OO in those days was in the development of frameworks and architectures. OO gave us the ability to define code modules that mapped to objects in our domain but the ability to create the executive sections of our software were often not understood and under-designed.

None the less, our indicted software designer still should have used the most basic and powerful software design technique, the idea of layering. But no, under analysis, the software has no structure.

The trial of the software designer continues. In defense of the designer, the software has been segmented into a multitude of modules that model the domain - but somehow this act of decomposition has created a mess. Let me introduce exhibit A, the programming language.

The system was programmed in the original version of Ada, the programming language developed for the U.S. Department of Defense in the late 1970's. Ada has been upgraded several times with the most notable upgrade coming in 1995 and termed Ada95, the original Ada is usually called Ada83.

Ada was specifically created to solve the so-called software crisis that plagued the development of large software systems. Yet here was a large software system in crisis, developed using Ada.

So now I will turn the glare of the spotlight away from the software designer and place it on this unpopular programming language. Now I must confess that I am not an "Ada man". I have worked on Ada projects for short terms but I have never become enamored with the language, although I know a few who have.

I looked-up a few of my (with all due respect) "grey haired" colleagues to borrow some manuals on Ada83, not Ada95 because I wanted to know what Ada developers were thinking back during the early days of the system's development. My Ada-savvy co-workers were eager to supply me with the manuals and to share a few thoughts with me.

Now Ada83 is not object-oriented but instead is object-based. Actually Ada83 is a procedural programming language similar to C. One main difference is that Ada includes the concept of a "package". The Ada package abstracts the "file module" and has some very powerful and useful capabilities. It is important to note that the Ada package is not a type like a Java or C++ class is a type. A package is used to define types, functions and variables - much like a class, but with the distinction that only one instance of a package exists in the program. A variable declared within a package is equivalent to a class-wide variable in Java or C++ - i.e a static variable.

My object-oriented mind now saw the embedded software system as ~500 singleton software classes - with public variables - mostly interconnected to one another. Could it be that the system's engineers did not really understand the object-based, procedural nature of the Ada language and used the Ada package as some type of brain-dead object-oriented class instead of a very powerful file module abstraction?

So let's be clear, in the object-based Ada language, the object is data defined, instantiated and processed by functions in the package, and not the package itself.

Interestingly, it all goes back to some of my previous musing of how State Stinks! Certainly the most stinkiest form of state would be static, globally accessible state.

I could go on but I have said all that I need to say about this matter. And I release the incarcerated designer from his cell, we all have skeletons in our closet and besides my job is not to find out what and why but how. How to fix this software and move it into the future. So far I am the one who is falling short of his goals.

But such is life in the hell of other people's code.

Bottom-Up Software Design is for Sheep

Considering the devisive issues of our day: Vim vs Emacs, Ruby vs Python, .Net vs Java, Mac vs PC, and REST vs SOAP, there are plenty of opportunities to spark a holy war. Lets consider another polarizing issue:

Is it better to design and develop software from the top-down or from the bottom-up?

I am fortunate to have the opportunity to work across a wide variety projects and I am always curious to discover how my newly met colleagues are inclined regarding design. This evening I came across a compelling blog, by Gustavo Duarte, that took a stance opposed to my own. Here is my take on the issue.

Design by decomposition is the quintessential technique of the engineer

One of Gustavo's main points is that Feynman believed that software engineering has much in common with the other engineering disciplines. Considering this point, the most important concept in engineering is modularization. Most products and devices are made up of replaceable parts. Modularization is achieved by decomposing a system into sub-systems. System decomposition is top-down design.

Certainly we can agree that some of the first decisions we might make include the selection of a platform (Linux, Windows), technology stack (LAMP, Java, .NET), or language/framework (Ruby on Rails, Django).

Our choices will determine the composition of our system, how functionality is divided among system components, and usage of key patterns (e.g Model-View-Controller).

Do we even realize that we are doing top-down design?

Perhaps the availability of powerful platforms and frameworks is removing the importance of the top-down design perspective from our collective conscience.

None-the-less, software system design by decomposition into sub-systems implies a top-down approach to design.

The Space Shuttle Challenger was not a victim of top-down design.

Gustavo's blog walks us through some evidence that suggests that Dr. Feynman believed that top-down design doomed Challenger. The Challenger Disaster illustrates that a likely point of system failure is at sub-system interfaces. I don't see this as an indictment against top-down design.

For a analysis of the mulitude of events that led to the Challenger disaster, I recommend the book NO DOWNLINK. Essentially, the Challenger and her crew where the victims of pork-barrel politics, post-space race budget cuts, and the Challenger Syndrome.

Poor design choices such as the use of Solid Rocket Boosters (SRBs) and then the segmenting of the SRBs to aid in transportability where driven more by budgetary and political concerns then by faulty engineering practices.

Defending up-front software design

Another point made by Gustavo is that "Big up-front design is foolish". The word "Big" implies a non-incremental approach. Up-front design is not a foolish idea. However developing software in a non-incremental fashion (i.e. the waterfall method) is a bad idea.

Top-down software design leads to effectivly modularized code that fosters many good things such as reuse, testability, and maintainability.

Gustavo drags UML into this discussion and I agree with some of his points. The idea of designers churning out UML blue-prints to throw "over the wall" to "implementors" has no appeal to me and in fact, simple does not work.

UML is not implicitly evil and the use of UML does not have to imply some type of bureaucratic software factory where coder slaves toil away, mindlessly "implementing" designs.

UML is just a tool that lets you work on your system at a high level. UML is just a white board.

Using UML to create diagrams (which are essentially pictures) to foster communication is a beautiful thing.

To summarize, UML should be used to open a new communication channel, never to replace face-to-face discussion.

Risk

The main problem with up-front design from a developer's perspective is that it doesn't address many of the underlying risks to the success of the project. The term "risk" is a nice way to articulate that feeling you get in your stomach when you know you are doomed.

The best way to handle "risk" is to start working on the risky parts of the system as soon as possible, so that you have plenty of time to handle all the foo bar.

Risk is what leads developers to favor the bottom-up approach. This is because the risky sections of a system exist deep within the system and at sub-system interfaces such as APIs.

Up-front design may add risk because some design decisions are based on assumptions that can only be validated with working code.

Managing Risk

All forms of engineering require lab testing, research and development and prototyping to determine how elements of a system will actually perform and software is no different.

Risk-reduction prototyping is a technique used to mitigate risks, validate design assumptions, and to develop techniques that will be used during the production of a software system.

Conclusion

As software developers, we are also software users. We use frameworks and tool-sets that allow us to focus on the specific application that we are creating. If we don't maintain the ability to design software from the top down, we function more as users and less as developers. We become in effect, sheep walking the well-worn path instead of being being the pioneering trail-blazers we had hoped to be.

I believe that a pragmatic combination of both top-down (up-front design) and bottom-up (risk-reduction prototyping) techniques will greatly increase a software project's chances for success.

Dr. Feynman believed that software engineer has much in common with the other engineering disciplines. This is certaintly true, but 'how much' is less certain. Thinking of software engineering as just another engineering discipline comes with its own pitfalls.

What is your take on this topic?