Yet another clueless manager

Nov 17, 2008 9:33 AM by Ilya Sterin

First, methodologies are only as good as the people that apply them. I hate the thought of process over people. Intelligent people will find a way to produce good software, agile or not. Morons will fail even with the process. Software development is more art than it is science, though I wish the people that never made it as software developers would stop trying to pile process on top of process and think that engineers are code monkeys that can develop good software by following some process. Process is good, but smart people are better.
Also, it’s not that agile is failing, software projects are failing and have been failing before agile and will be failing after. Again, this is art and creativity is required not process.

Thanks Ilya for the comment!

Having worked with several Object-Relational mapping frameworks in the last few years, I got to a point where I couldn’t justify their complexity in my project. We often talk about the mismatch between the database and the object worlds, and that is where ORMs are often stated and referenced for “bridging the gap”!

Well I prefer to call it lifting the gap, or highering the gap, to have it now between DAOs and the rest of the code than having it between database and code.But I wouldn’t call this in any way reducing the gap.

First thing to mention here, is that data structures are not evil. We often deal with XML data structure but we absolutely refuse to deal with relational data structures. The problem might be related to time, XML came with a whole arsenal of managed APIs that represent its model in a way that makes it more comfortable for you to manipulate these documents. All together with transformation APIs that made it a lot easier to fill your object model out of an XML. Relational data structure on the other hand did not have the luxury of standard APIs and representation models in mainstream languages, and that made dealing with a database request result so arbitrary (Datasets existed in .Net but they have been cursed heavily and has been forced out of the cool toolbox, do not know yet why!) . 

In some way, I guess that the “mismatch” between Relational data structures and objects exists much similarly between XML and objects, however we didn’t think for so long of a doing a mapping between XML and object OXM? (no we are starting to do so, writing some XML to map XML to objects!).

With the experience I had with mapping technologies, I can say that they do not solve the problem for me. They merely add yet another useless level of abstraction where your domain types look nothing like relational data structures, but yet nothing like your target domain model! They have gone a half way, and now its your turn to poison your application with a lot of absurd ORM code trying to stretch this bridge to reduce the gap…

Geeks love talking about intrusivty of an ORMs and compare level of intrusivty in different terms. Intrusivty is much more than the lack of an interface or methods to implement in your domain objects. I guess I am ok to deal with that kind of intrusivity. However all ORMs I tried do not give me the opportunity to express my domain model in the way I would like to. I have to deal with notions of Lazy Loading, Attached/Detached entities, Entity sets, ids, back fields and a lot of other things that I don’t want to appear in my model and that make the resulting code so ugly and complex that I don’t wish to visit soon.

As Bob Martin said before, we are missing the opportunity to do oop, I would enforce and say we are missing the opportunity to model at all. And I guess we have gone too far with ORMs and it is time to go back to the basics, in the same way we have done after the EJB great abstraction.

It is very contradictory, while we are so excited about DSLs and the goodness they can potentially bring to enterprise development that results in a declarative semanticful style of programming, we can’t help but to engage all of our means in trying to bury SQL (and effective, semanticful, declarative powerful domain specific language) behind a dumb API to “abstract database access”! I guess we got to chose one strategy, either we keep SQL and we  keep marketing DSLs or we hope for a new advanced GPL compiler that will hide all of this in the same way modern languages did with garbage collection, concurrency and synchronization.

Since I am not yet a compiler writer, and I aint got around an effective way of integrating database in mainstream languages (there is actually a very productive in memory database on Haskell that worth trying), I decide to keep the DSL, and to work with them without the need to hide them, but rather to get a better tooling to help converting between difference data structures.

I guess you’ve guessed it by now. For me LinQ is the mid-term solution for the data structure mismatch. No not LinQ to sql, nor Ado.Net entities (they are helpful and we might use them but lets focus on the solution). LinQ is a very productive tool that halps effectively transform anything to anything in few lines of code. I mean that I can still use powerful SQL to get data effeciently and then convert them with one line of LinQ into the destination data structure. No abstraction leak, less complexity: Use the right tool where it applies.

So what I would actually do is to use a framework that will execute my SQL or SQL like query on the database and get me the records in much the same way they are in the database,Much in the same way Active Record gets you records as they are in the database. No useless efforts from the framework to bend the data to make it look untruthfully like the destination model.  I guess I know better my domain model, and with this better tooling I can get it done with no problem.

http://www.infoq.com/interviews/coplien-martin-tdd

Thanks Cope, thanks Bob.

Many efforts in today’s software community are targeted at bridging the gap between software professionals and business people. Several blogspace authors look at the issue from a slightly different perspective, highlighting the gap between developers and the software they are building.

According to Jeff Attwood, Amazon’s experience of regularly involving its developers with costumer service is a valuable means for improving quality and usability of software.  He believes indeed that “all too often, software developers are merely tourists in their own codebase” because they lack a basic understanding of software users, their problems and concerns. This is what he has earlier referred to as  “ivory tower software development”:

In the absence of any other compelling evidence, developers assume everyone else is a developer. [...] The more isolated the developers, the worse the resulting end product. It doesn’t help that most teams have a layer of business analysts who feel it is their job it to shield developers from users [...] It’s dangerous to create an environment where developers have no idea who the users are.

However, today’s industry is characterized, according to Abhijit Nadgouda, by hierarchy, existence of multiple layers and information shielding among those layers. He highlights that this simplifies management and makes business safer, but has rather negative implications on software quality:

We create a hierarchy in our projects, and every level shields the lower levels from some information. How many members of the software development team know value of the software they are working on, or its importance to the client’s business? How many of them even know about pieces other the one they are working on?

[...]

There seems to be a disconnect between better business and better software development. Which is why, I believe, most of us are doing good business, but we as an industry still stink at software development.

In his attempt to identify the reasons why “we still stink at software development”, Reg Braithwaite also raises this issue arguing that it might be wrong to divide up work on a project in a way “that the people with the least experience are protected from damaging the code”.

Architecture based on such approach tends to simplify developers work through abstaction. If this is pushed to the extreme, developers tasks are taken out of functional context to become purely technical, which can potentially create a gap between developers and the software they are working on.

What is your opinion? Is such a protectionist architecture an impediment for software usability? Can architecture with developers ignorant of the full picture be effective? Does it deliver sofware and value? 

An interesting debate has occurred in the blogspace around the issue of team sizes. It started with Reg Braithwaite’s post questioning the relevance of powerful languages in a large team environment:

What if programming conventions and closures and meta-programming and expressive type systems and annotations and all of the other tools that give us the power to build powerful abstractions actually don’t scale to larger teams?

Reg argues that "productivity drops off a cliff as team size increases" and this assertion is rather consensual. Bob Warfield stresses in his earlier post that "small talented teams crush big teams" not only with regard to productivity but also in quality terms. According to Guillaume who comments on Braithwaite’s post, this performance gap is due to the fact that with large teams there are only two choices: "either you code for the lowest common denominator or you end up with a lot of programmers who don’t understand the code base." As a remedy, Reg Braithwaite advocates for scaling teams up in productivity rather than in personnel and investing "in tools and processes that are specifically tuned to maximizing the productivity of small teams". This way, it would become possible to build teams around "tools and processes proven to produce working software" rather than "dumbing the tools down to the level of whomever we hired last Tuesday".

Many bloggers argue however that even if very sophisticated software – such as Linux – can be developed by a small team, eventually team growth is inevitable. Chris Winters asserts, for instance, that there is "a huge distinction between creating something and maintaining/improving it" especially since any successful software results in increasing number of costumers and requests coming from them. Nevertheless, according to Warfield, team growth trend is much more related to cultural reasons. Many decision makers are not familiar with the specificity of IT projects and tend to believe that increasing personnel would result in better/faster outcomes. Moreover, career growth aspirations of team members result in hiring more junior stuff, thus contributing to team growth.

If larger teams are inevitable, how can quality and productivity level be maintained? Oftentimes, the solution consists in multiplying the layers of review, which many consider as waste. Commenting on Braithwaite’s post, Mike Kucera, who actually does not believe that "having a large team automatically means that the software will be crappy and bloated", insists on the role of abstraction and code readability that help to "figure out the code quickly and get productive maintaining it". Moreover, he highlights that having correct abstraction allows breaking the project down into distinct components that can be developed by several small teams:

With the correct abstractions you can break software down into loosely coupled components, then have a small team focus on each component.

Warfield argues that it is not always possible to break the project into separate components. Hence he suggests cutting the scope of the project because he believes that much of it can "be done away with" without significant impact on costumer satisfaction. This is what according to Warfield accounts for the "art of great software" and allows "a small team [keeping] a product fresh and current for years"

James O. Coplien has recently argued that today’s industry is based on buzzwords and checklists. The use of some techniques and methodologies, TDD for instance, has become “a religious issue”. This prevents from inspecting possible tradeoffs and focusing on finding solutions that would be the most appropriate and the most cost-effective .

“For some reason, we have invented and are following religions […].” This is how James O. Coplien describes today’s industry which, he believes, is based on buzzwords and checklists rather than on thinking, inspection and efforts to find solutions that would be the most appropriate and the most cost-effective for a given project:

How much thought did you put into the tradeoffs of the last technique you brought into your organization: Ajax, TDD, On-Site Customer, or other buzzwords? Did you research its track record? Or did you go to the buzzword yellow pages?

James uses the example of the debate around TDD which has occurred in the blog space in the wake of Jaoo 2007 conference. When the discussion focuses on what approach to testing is the best (i.e. TDD or acceptance testing), it essentially addresses the wrong question. It obfuscates the real objective which is “to deliver what the customer wanted while optimizing quality” and to create value for the client. Testing is what we have on today’s checklist for quality but it is far from covering all the issues that actually affect quality:

Quality suffers if you misunderstand what the customer wanted, or if the code has internal interactions that are difficult to foresee, or if local code violates array boundaries or uses untethered pointers. It also suffers if the interface makes it possible for the user to commit errors, or if it takes too many keystrokes to enter a small amount of common information.

Focusing on quality requires considering every weapon in the arsenal:

That means using Use Cases (which are an Agile way of engaging customers […]) instead of XP-style user stories (so you understand feature interactions up-front), doing pair programming or desk-check code inspections, doing design by contract, driving the process with usability experts and doing good usability testing, and a host of other things.

Coplien emphasizes that it is impossible to do everything and that it is “a matter of cost effectiveness”. But being focused on techniques and driven by buzzwords we get stuck in practices that might not be optimal. The use of some techniques and methodologies, e.g. TDD, has become “a religious issue”, according to James:

We are told that you have to do TDD to be a professional […] and yet are given no reason to believe this, no substantiation, and no evidence. Just believe.

Coplien argues, for instance, that “integration and system testing have long been demonstrated to be the least efficient way of finding bugs”, that TDD “deteriorates the architecture”, and that “acceptance testing is orders of magnitude less efficient than good old-fashioned code inspections, extremely expensive, and comes too late to keep the design clean.” But he stresses how difficult it is to engage a challenging discussion on such “religious” topics because any criticism is met with a lot of emotion.

People have tied their Agile success to their introduction of TDD practices into their work. It’s the one thing they can do as individuals rather than as an enterprise.

Coplien advocates for focusing on quality rather than testing and, more generally speaking, on thinking rather than checklists. He highlights however that “sorting these things out requires a system view”, which often times lacks in today’s industry. He believes that “New Age Agile movements consistently move away from” systems engineering. According to Coplien, “one root of the problem lies in modern education, which has increased its focus on technique and reduced its focus on thinking.” Today’s students “fewer and fewer understand software history” and “have stronger memories […]on exactly when to use --v and when to use v-- than they recall anything about logical design.”

Close links which exist between academia and industry are also a part of the problem. Universities tend to adapt curricula to industry’s needs in order to facilitate students’ placement. Hence, it becomes difficult for academics to challenge predominate views, probably “as difficult as it was for Newton” to prevail “with new insights on the working of the universe”:

In fighting the broad misunderstandings that industry has chosen to adopt from selected interpretations of Agile, SOA, and programming language buzzwords, we face no less a religion, and no less threat to our cozy industry-financed academic positions, than he did.