Growing a Tree versus Building a House

When we say that we want to build "good" software, we tend to use terms that come from other engineering fields: foundation, framework, scafolding, architecture.  One of the things that the agile software movement has shown us is that good solutions can come from evolutionary models as well as construction models.  The difference comes from the fact that code is far easy to manipulate that physical raw materials.

When building a data warehouse, we often draw traditional, stacked-tier pictures of the data architecture: data warehouse tables, semantic layer, data marts, etc.  If we start our design discussions with an assumption that anything we "build on top of" has to be "solid" then we quickly drive the overall solution away from agility.  "Solid" conjures an image of a concrete foundation that has to be built to withstand floods and earthquakes.  If we find a crack in our foundation, it has to be patched so the things on top don't come crumbling down.

If, instead, we try to imagine a conceptual architecture that has in mind goals of adaptability to purpose (rather than firmness) and loose coupling (rather than high contact), you can begin to imagine a higher level of agility.  Look at the picture from the start of this post (from webecoist).  The trees are being shaped and molded into a purpose built structure.  If, part-way through the growth process, the structure needed to change to be another 6 inches higher or hold a second story of some kind, the necessary changes could be interwoven into the growth already complete.  If we were constructing a new art museum and decided, half way through, that we wanted a library instead, we'd have to make some major changes or compromised to account for the fact that the foundation was only designed to hold the weight of portraits, not stacks of books.

This conceptual discussion about growing something organically rather than building it from the ground up is directly related to the sashimi discussion from yesterday.  A legacy build approach says data model, build ETL, build semantic layer, build reports.  There aren't any opportunities in that model to create meaningful yet consumable vertical slices. 

I hear some agile BI conversations only go halfway toward the mind shift that I think is necessary.  These "think big, act small" solutions sound like a model where the only change is that you poor some of the concrete foundation at a time.  Building a house using this semi-agile approach:

Iteration One:

1. Pour foundation for the kitchen only.
2. Build kitchen walls.
3. Wire up kitchen outlets.
4. Install kitchen plumbing.

Iteration Two:

1. Pour foundation for the family room only.
2. Build family room walls.
3. Realize you need to tear out a kitchen wall to open to family room.
4. Reroute electricity in that wall.
5. Rerun wiring to kitchen
6. Run new wiring to family room

In this approach to agile BI, you might well deliver value to customers more quickly than if you took a monolithic waterfall approach.  Since you aren't requiring yourself to plan everything up front, you run a high risk of having to do rework later, though.  In a physical construction mindset, rework is very expensive (rip out wall, rewire, etc).

An organic build approach says plant a seed, watch it grow.  First, a sprout appears, with roots a stem and leaves.  The stem gets thicker, the roots grow deeper, and more leaves sprout.  Branches grow.  Flowers bud and fruit appears.  When requirements change some pruning and grafting is required, but you don't have to tear down the tree and plan a new one from scratch or start a new tree on the side.  The tree will grow around power lines and rocks and other trees as needed.

There's the mindset.  I don't think it's easy to shift from a constructionist perspective to an organic one.  Success in agile BI requires this change in thinking, though.  If you're still laying foundations and screwing sheetrock onto studs, your attempt at agile BI will not be optimal.


Good luck with that.