Recently, I’ve been working on a Web Api system where, thanks to support of @jeanlaurent http://morlhon.net/blog/, I used Scala applying DCI architecture in a real world project. This post is about reporting benefits of using this approach. Other posts will follow that will be more focused on the use of Scala and Functional Programming in that project. Code included is a bit simplified and parts of the system that are not of interest are omitted.

The concerned part of the system is an API for logging a Web API usage and using it for showing interesting statistics to users and for implementing a Web 2.0 business model authorization.

The story starts with an API key passed as a parameter in the requested URI something like product/1234?api_key=AFFSEFFVFE2344

I create an object that represents the API user containing the key: (did I tell I like one liners?)

This is a class of an object that holds the apiKey, now I need a method that I will call to log the use of product/123 for user represented by the apiKey AFFSEFFVFE2344. What could be quite convenient and OOP-iesh is to call a record method on the apiKey instance, something like apiUser.record(…) . The problem with this, as explained in the original DCI article, is that adding these methods to the domain objects adds noise in other contexts for which recording is not relevant such as authorization and statistics, as we will see later. I would rather use here the DCI’s role concept (or context). Doing so I would say (apiUser in recording role).record(…) . This pseudo code tends to be very close to mixins implemented in Scala language and will look like following in real world Scala code:

Here the ApiUser object doesn’t contain a record method by default and so it can be shared in different contexts. In other words we kept our domain objects pure yet not scarifying OOP convenience. Now the Recorder is a trait that minds only business in the context of recording. The definition looks like the following:

Another view of the system is the end user’s statistics. Statistics should be represented in a meaningful way to the user and the user can’t care less about how the recording is done. So, it is quite desirable not to have the noise of recording when talking about the statistics view context. Here is how the call looks like in our example in Scala:

Again what we are doing here looks merely like saying: I want to use the ApiUser in a statistics context and I choose to call method getRequestsPerMonthForLast12Months. I guess the convenience brought by using this approach is obvious. This is how the Statistics trait looks like:

Note that as in the Recording trait, the ApiUser object is not passed as an explicit parameter but rather as an implicit receiver parameter (something like this for an object). To access it we use Self that refers to it.

The third view of the system is what allows us to provide an authorization mechanism based on user’s activities for a business plan they choose (be it a quota per day, per month or other interesting combinations). It could be quite convenient to have the call looking like apiUser.isAuthorized . We get pretty close with DCI without polluting our domain object with a specific context logic:

Like the previous, the Authorizer trait looks straightforward :

Now that we looked at the three views of our system introducing three examples of “Context” from DCI pattern we can go beyond that to discover how we use nested contexts (or roles).

Looking at the Authorizer trait, the ApiUser referred to by Self, doesn’t tell us much about how many requests the user did today or this month. And for authorizing user one needs a richer ApiUser that can give the necessary information that live themselves in a file or a database. But once again, we don’t want to pollute our domain object with Data Access code that varies on a different axis than the domain object itself.  Introducing a nested context/role that adds the Data Access logic would pretty convenient here:

Here Self is a richer ApiUser that carries Requests Data Access logic defined in the RequestsDataAccessor trait:

And the implementation of the Authorizer trait defined in terms of ApiUser with RequestsDataAccessor:

In the same way we implement the Statistics and Recorder traits using ApiUser with RequestsDataAccessor:

If you are not familiar with generics ( getRequestsPer[Day] ) you can ignore them since they are not central to the subject of this post and simply use their readability to understand the purpose of the method. The use of generics in this project will be the subject of a future post.

Encapsulation Test!

Now in the first version of the system, the ApiKey was passed in clear in the service URI. This is not very secured as it is quite easy to get the key and use it for using the API. We might prefer to encrypt the key in some way. This fortunately changes nothing for any of our traits implementations since the decryption of the key is the responsiblity of the ApiUSer object and all what our traits need is to get the concerned ApiKey from that object which can be easily done with pattern matching. Same thing if we want to replace database logging with file system in RequestsDataAccessor. Modularity and separation of concerns are apparent in this DCI architecture implementation.

Conclusion

This is a refreshing approach to OOP. The Domain Context Interaction pattern allowed us to restore interesting properties of real OOP approach yet not sacrificing modularity and separation of concerns. This simple view of “an object in several contexts” helps quite much in thinking explicitly about and understanding context boundaries. Note that an object(instance) is not defined by a single class but each instance curries the necessary context. This is essential to OOP which is NOT COP (for Class Oriented Programming ;) )

Feel free to ask questions and/or comment about this implementation. Refer to original papers for more in-depth explanation of the pattern.

Thanks to JetBrains’ IDEA and the Scala plug-in that helped me programming in Scala.

One thing that annoys me though about most programming language books is how raw they often are.

  Probably constrained by time, authors and book teams seem to invest less on editorial stuff and teaching methodology and more on explaining details of technical and behind the scenes stuff. This often results in more manual-like books where you have a great wealth of material that you are not able to exploit anyway because of, well, lack of time. I am not sure I can invest time in reading 500+ manual of a new, or at least non mainstream, language that I might decide not to use by the end. 

I am not saying here that technical and behind the scene stuff does not matter, quite the opposite. I just think that manual, important it is, should not be the introduction to the programming language. A rather 100 to 200 pages brief book that is carefully edited to introduce paradigm, concepts, and strengths of the programming language is a more attractive choice , accompanied with a wordy but well organized reference book with examples and more in-depth explanation of language features. I guess this is a more pragmatic approach to programming language learning in the presence of WWW and Google search.

PS: I personally find Paul Hudak’s Haskell book a very good example of an enjoyable, brief, nicely and carefully edited programming language book. Also LUA’s programming book is nicely done.

posted originally on http://www.infoq.com/news/2008/10/measure-programmers-productivity

In the field of software development, just as in any other area, managers need to appreciate performance of their programmers and the progress of their projects. However, defining appropriate metrics to this end appears to be a tricky task.

Measuring source lines of code (SLOC) is one of commonly used approaches that presents however a number of important limitations highlighted recently by Shahar Yair and Steve McConnell. First of all, the amount of lines of code does not allow effectively measuring the progress of a project because it focuses on activity rather than results. LOC do not have any value as such: the value of the final product depends on its performance and quality, not on the amount of code. Hence, focusing on the latter is actually a very limited approach to productivity.

SLOC doesn’t tell anything about the complexity of the issue at hand or about the quality of the final product in terms of maintainability, flexibility, extensibility, etc. With regard to quality, it can actually be negatively correlated. Refactoring as well as some design patterns result in reducing LOC while improving code’s quality. Larger code base may mean more noise, more unnecessary complexity, less readability and flexibility.

What is particularly risky about adopting such a reductive view of programmers’ performance is that it creates wrong incentives. Rather than optimizing their work in terms of final product, developers may be encouraged to favor the quantity of code to the detriment of quality and even intentionally write more verbose code. “What gets measured gets done”, recalls Steve McConnell.

He points out that some of these issues can be solved by using function points as measuring metric. The program size is then determined by the number of inputs, outputs, queries and files. Nevertheless, this approach also has its downsides. McConnell mentions some practical issues like the necessity of having a certified function point counter and the difficulties with mapping each function point back to individual programmers. A certified function point specialist Daniel Yokomizo highlights in his comment other limitations of such approach: lack of tools to measure function points’ complexity and to take into consideration things such as code sharing, frameworks, libraries, etc., that affect the time needed to create a feature.

Even though many commentators involved in the discussion about measuring approaches agree on their limitations, they do not necessarily dismiss the need for measuring programmers’ performance. Many insist on the fact that SLOC, for instance, can be used as a baseline for a more complex analysis combining different factors. This goes along the lines of four principles outlined by McConnell that should guide any analysis of programmers’ productivity:

1. Don’t expect any single dimensional measure of productivity to give you a very good picture of individual productivity […]

2. Don’t expect any measures […] to support fine-grain discrimination in productivity among individuals. [They] give you questions to ask but they don’t give you the answers […]

3. Remember that trends are usually more important than single-point measures […]

4. Ask why you need to measure individual productivity at all […]

In which context measuring programmers’ productivity is actually meaningful? What criteria can be used for it? How can they be combined? Many questions are still open for discussion and if your experience has brought you some answers, don’t hesitate to share.

http://www.infoq.com/interviews/coplien-martin-tdd

Thanks Cope, thanks Bob.

Many efforts in today’s software community are targeted at bridging the gap between software professionals and business people. Several blogspace authors look at the issue from a slightly different perspective, highlighting the gap between developers and the software they are building.

According to Jeff Attwood, Amazon’s experience of regularly involving its developers with costumer service is a valuable means for improving quality and usability of software.  He believes indeed that “all too often, software developers are merely tourists in their own codebase” because they lack a basic understanding of software users, their problems and concerns. This is what he has earlier referred to as  “ivory tower software development”:

In the absence of any other compelling evidence, developers assume everyone else is a developer. [...] The more isolated the developers, the worse the resulting end product. It doesn’t help that most teams have a layer of business analysts who feel it is their job it to shield developers from users [...] It’s dangerous to create an environment where developers have no idea who the users are.

However, today’s industry is characterized, according to Abhijit Nadgouda, by hierarchy, existence of multiple layers and information shielding among those layers. He highlights that this simplifies management and makes business safer, but has rather negative implications on software quality:

We create a hierarchy in our projects, and every level shields the lower levels from some information. How many members of the software development team know value of the software they are working on, or its importance to the client’s business? How many of them even know about pieces other the one they are working on?

[...]

There seems to be a disconnect between better business and better software development. Which is why, I believe, most of us are doing good business, but we as an industry still stink at software development.

In his attempt to identify the reasons why “we still stink at software development”, Reg Braithwaite also raises this issue arguing that it might be wrong to divide up work on a project in a way “that the people with the least experience are protected from damaging the code”.

Architecture based on such approach tends to simplify developers work through abstaction. If this is pushed to the extreme, developers tasks are taken out of functional context to become purely technical, which can potentially create a gap between developers and the software they are working on.

What is your opinion? Is such a protectionist architecture an impediment for software usability? Can architecture with developers ignorant of the full picture be effective? Does it deliver sofware and value? 

James O. Coplien has recently argued that today’s industry is based on buzzwords and checklists. The use of some techniques and methodologies, TDD for instance, has become “a religious issue”. This prevents from inspecting possible tradeoffs and focusing on finding solutions that would be the most appropriate and the most cost-effective .

“For some reason, we have invented and are following religions […].” This is how James O. Coplien describes today’s industry which, he believes, is based on buzzwords and checklists rather than on thinking, inspection and efforts to find solutions that would be the most appropriate and the most cost-effective for a given project:

How much thought did you put into the tradeoffs of the last technique you brought into your organization: Ajax, TDD, On-Site Customer, or other buzzwords? Did you research its track record? Or did you go to the buzzword yellow pages?

James uses the example of the debate around TDD which has occurred in the blog space in the wake of Jaoo 2007 conference. When the discussion focuses on what approach to testing is the best (i.e. TDD or acceptance testing), it essentially addresses the wrong question. It obfuscates the real objective which is “to deliver what the customer wanted while optimizing quality” and to create value for the client. Testing is what we have on today’s checklist for quality but it is far from covering all the issues that actually affect quality:

Quality suffers if you misunderstand what the customer wanted, or if the code has internal interactions that are difficult to foresee, or if local code violates array boundaries or uses untethered pointers. It also suffers if the interface makes it possible for the user to commit errors, or if it takes too many keystrokes to enter a small amount of common information.

Focusing on quality requires considering every weapon in the arsenal:

That means using Use Cases (which are an Agile way of engaging customers […]) instead of XP-style user stories (so you understand feature interactions up-front), doing pair programming or desk-check code inspections, doing design by contract, driving the process with usability experts and doing good usability testing, and a host of other things.

Coplien emphasizes that it is impossible to do everything and that it is “a matter of cost effectiveness”. But being focused on techniques and driven by buzzwords we get stuck in practices that might not be optimal. The use of some techniques and methodologies, e.g. TDD, has become “a religious issue”, according to James:

We are told that you have to do TDD to be a professional […] and yet are given no reason to believe this, no substantiation, and no evidence. Just believe.

Coplien argues, for instance, that “integration and system testing have long been demonstrated to be the least efficient way of finding bugs”, that TDD “deteriorates the architecture”, and that “acceptance testing is orders of magnitude less efficient than good old-fashioned code inspections, extremely expensive, and comes too late to keep the design clean.” But he stresses how difficult it is to engage a challenging discussion on such “religious” topics because any criticism is met with a lot of emotion.

People have tied their Agile success to their introduction of TDD practices into their work. It’s the one thing they can do as individuals rather than as an enterprise.

Coplien advocates for focusing on quality rather than testing and, more generally speaking, on thinking rather than checklists. He highlights however that “sorting these things out requires a system view”, which often times lacks in today’s industry. He believes that “New Age Agile movements consistently move away from” systems engineering. According to Coplien, “one root of the problem lies in modern education, which has increased its focus on technique and reduced its focus on thinking.” Today’s students “fewer and fewer understand software history” and “have stronger memories […]on exactly when to use --v and when to use v-- than they recall anything about logical design.”

Close links which exist between academia and industry are also a part of the problem. Universities tend to adapt curricula to industry’s needs in order to facilitate students’ placement. Hence, it becomes difficult for academics to challenge predominate views, probably “as difficult as it was for Newton” to prevail “with new insights on the working of the universe”:

In fighting the broad misunderstandings that industry has chosen to adopt from selected interpretations of Agile, SOA, and programming language buzzwords, we face no less a religion, and no less threat to our cozy industry-financed academic positions, than he did.