Recently, I’ve been working on a Web Api system where, thanks to support of @jeanlaurent http://morlhon.net/blog/, I used Scala applying DCI architecture in a real world project. This post is about reporting benefits of using this approach. Other posts will follow that will be more focused on the use of Scala and Functional Programming in that project. Code included is a bit simplified and parts of the system that are not of interest are omitted.

The concerned part of the system is an API for logging a Web API usage and using it for showing interesting statistics to users and for implementing a Web 2.0 business model authorization.

The story starts with an API key passed as a parameter in the requested URI something like product/1234?api_key=AFFSEFFVFE2344

I create an object that represents the API user containing the key: (did I tell I like one liners?)

This is a class of an object that holds the apiKey, now I need a method that I will call to log the use of product/123 for user represented by the apiKey AFFSEFFVFE2344. What could be quite convenient and OOP-iesh is to call a record method on the apiKey instance, something like apiUser.record(…) . The problem with this, as explained in the original DCI article, is that adding these methods to the domain objects adds noise in other contexts for which recording is not relevant such as authorization and statistics, as we will see later. I would rather use here the DCI’s role concept (or context). Doing so I would say (apiUser in recording role).record(…) . This pseudo code tends to be very close to mixins implemented in Scala language and will look like following in real world Scala code:

Here the ApiUser object doesn’t contain a record method by default and so it can be shared in different contexts. In other words we kept our domain objects pure yet not scarifying OOP convenience. Now the Recorder is a trait that minds only business in the context of recording. The definition looks like the following:

Another view of the system is the end user’s statistics. Statistics should be represented in a meaningful way to the user and the user can’t care less about how the recording is done. So, it is quite desirable not to have the noise of recording when talking about the statistics view context. Here is how the call looks like in our example in Scala:

Again what we are doing here looks merely like saying: I want to use the ApiUser in a statistics context and I choose to call method getRequestsPerMonthForLast12Months. I guess the convenience brought by using this approach is obvious. This is how the Statistics trait looks like:

Note that as in the Recording trait, the ApiUser object is not passed as an explicit parameter but rather as an implicit receiver parameter (something like this for an object). To access it we use Self that refers to it.

The third view of the system is what allows us to provide an authorization mechanism based on user’s activities for a business plan they choose (be it a quota per day, per month or other interesting combinations). It could be quite convenient to have the call looking like apiUser.isAuthorized . We get pretty close with DCI without polluting our domain object with a specific context logic:

Like the previous, the Authorizer trait looks straightforward :

Now that we looked at the three views of our system introducing three examples of “Context” from DCI pattern we can go beyond that to discover how we use nested contexts (or roles).

Looking at the Authorizer trait, the ApiUser referred to by Self, doesn’t tell us much about how many requests the user did today or this month. And for authorizing user one needs a richer ApiUser that can give the necessary information that live themselves in a file or a database. But once again, we don’t want to pollute our domain object with Data Access code that varies on a different axis than the domain object itself.  Introducing a nested context/role that adds the Data Access logic would pretty convenient here:

Here Self is a richer ApiUser that carries Requests Data Access logic defined in the RequestsDataAccessor trait:

And the implementation of the Authorizer trait defined in terms of ApiUser with RequestsDataAccessor:

In the same way we implement the Statistics and Recorder traits using ApiUser with RequestsDataAccessor:

If you are not familiar with generics ( getRequestsPer[Day] ) you can ignore them since they are not central to the subject of this post and simply use their readability to understand the purpose of the method. The use of generics in this project will be the subject of a future post.

Encapsulation Test!

Now in the first version of the system, the ApiKey was passed in clear in the service URI. This is not very secured as it is quite easy to get the key and use it for using the API. We might prefer to encrypt the key in some way. This fortunately changes nothing for any of our traits implementations since the decryption of the key is the responsiblity of the ApiUSer object and all what our traits need is to get the concerned ApiKey from that object which can be easily done with pattern matching. Same thing if we want to replace database logging with file system in RequestsDataAccessor. Modularity and separation of concerns are apparent in this DCI architecture implementation.

Conclusion

This is a refreshing approach to OOP. The Domain Context Interaction pattern allowed us to restore interesting properties of real OOP approach yet not sacrificing modularity and separation of concerns. This simple view of “an object in several contexts” helps quite much in thinking explicitly about and understanding context boundaries. Note that an object(instance) is not defined by a single class but each instance curries the necessary context. This is essential to OOP which is NOT COP (for Class Oriented Programming ;) )

Feel free to ask questions and/or comment about this implementation. Refer to original papers for more in-depth explanation of the pattern.

Thanks to JetBrains’ IDEA and the Scala plug-in that helped me programming in Scala.

Having worked with several Object-Relational mapping frameworks in the last few years, I got to a point where I couldn’t justify their complexity in my project. We often talk about the mismatch between the database and the object worlds, and that is where ORMs are often stated and referenced for “bridging the gap”!

Well I prefer to call it lifting the gap, or highering the gap, to have it now between DAOs and the rest of the code than having it between database and code.But I wouldn’t call this in any way reducing the gap.

First thing to mention here, is that data structures are not evil. We often deal with XML data structure but we absolutely refuse to deal with relational data structures. The problem might be related to time, XML came with a whole arsenal of managed APIs that represent its model in a way that makes it more comfortable for you to manipulate these documents. All together with transformation APIs that made it a lot easier to fill your object model out of an XML. Relational data structure on the other hand did not have the luxury of standard APIs and representation models in mainstream languages, and that made dealing with a database request result so arbitrary (Datasets existed in .Net but they have been cursed heavily and has been forced out of the cool toolbox, do not know yet why!) . 

In some way, I guess that the “mismatch” between Relational data structures and objects exists much similarly between XML and objects, however we didn’t think for so long of a doing a mapping between XML and object OXM? (no we are starting to do so, writing some XML to map XML to objects!).

With the experience I had with mapping technologies, I can say that they do not solve the problem for me. They merely add yet another useless level of abstraction where your domain types look nothing like relational data structures, but yet nothing like your target domain model! They have gone a half way, and now its your turn to poison your application with a lot of absurd ORM code trying to stretch this bridge to reduce the gap…

Geeks love talking about intrusivty of an ORMs and compare level of intrusivty in different terms. Intrusivty is much more than the lack of an interface or methods to implement in your domain objects. I guess I am ok to deal with that kind of intrusivity. However all ORMs I tried do not give me the opportunity to express my domain model in the way I would like to. I have to deal with notions of Lazy Loading, Attached/Detached entities, Entity sets, ids, back fields and a lot of other things that I don’t want to appear in my model and that make the resulting code so ugly and complex that I don’t wish to visit soon.

As Bob Martin said before, we are missing the opportunity to do oop, I would enforce and say we are missing the opportunity to model at all. And I guess we have gone too far with ORMs and it is time to go back to the basics, in the same way we have done after the EJB great abstraction.

It is very contradictory, while we are so excited about DSLs and the goodness they can potentially bring to enterprise development that results in a declarative semanticful style of programming, we can’t help but to engage all of our means in trying to bury SQL (and effective, semanticful, declarative powerful domain specific language) behind a dumb API to “abstract database access”! I guess we got to chose one strategy, either we keep SQL and we  keep marketing DSLs or we hope for a new advanced GPL compiler that will hide all of this in the same way modern languages did with garbage collection, concurrency and synchronization.

Since I am not yet a compiler writer, and I aint got around an effective way of integrating database in mainstream languages (there is actually a very productive in memory database on Haskell that worth trying), I decide to keep the DSL, and to work with them without the need to hide them, but rather to get a better tooling to help converting between difference data structures.

I guess you’ve guessed it by now. For me LinQ is the mid-term solution for the data structure mismatch. No not LinQ to sql, nor Ado.Net entities (they are helpful and we might use them but lets focus on the solution). LinQ is a very productive tool that halps effectively transform anything to anything in few lines of code. I mean that I can still use powerful SQL to get data effeciently and then convert them with one line of LinQ into the destination data structure. No abstraction leak, less complexity: Use the right tool where it applies.

So what I would actually do is to use a framework that will execute my SQL or SQL like query on the database and get me the records in much the same way they are in the database,Much in the same way Active Record gets you records as they are in the database. No useless efforts from the framework to bend the data to make it look untruthfully like the destination model.  I guess I know better my domain model, and with this better tooling I can get it done with no problem.

How many languages are you using on the same project? If you go counting you will see that they are many. I mean XML, Java, XSLT, HTML, CSS… etc. But the reason why you are using almost all of them is that they happen to be mainstream and, oftentimes, they are the only language choice for a needed framework. You are actually almost obliged to use them. The choice is done for you. Style? CSS. Configuration? Often XML. Web interface description? Html. However, if you want to adopt true polyglot programming, you will have to face inevitable decision of language choice.

Read my article on Paradigm based Polyglot Programming at InfoQ.com

http://www.infoq.com/news/2008/03/dsls-are-not-natural-languages

There is a widespread opinion that a good DSL has to be English-like in order to be readable for non-programmers. Dave Thomas advocates against such approach asserting that DSL are not about getting as close as possible to natural languages. Moreover, he argues that having this as a guiding principle of DSL design can be rather detrimental. He also highlights what he believes is important in DSL design and provides some examples of successful DSL that do not necessarily reed like English.

According to Dave, DSL don’t need to be close to English or any other natural language because they targets a very specific category or users – domain experts – who actually don’t speak a natural language

Domain experts […] are speaking jargon, a specialized language that they’ve invented as a shorthand for communicating effectively with their peers. Jargon may use English words, but these words have been warped into having very different meanings—meanings that you only learn through experience in the field.

Hence, DSL should reflect this jargon and express the expertise of domain specialists in a concise way. Make for dependency management, Groovy builders for expressing data in code and Active record declaration for data modeling in Ruby are a few successful examples of such DSL that respond to domain experts needs without necessarily being English-like. Even though some statements in Active record declaration may look like English, e.g. has_many or belongs_to, they actually are not: “they are jargon from the world of modeling” and “they have a specific meaning in that context.”

Another important point raised by Dave is that, in his opinion, “domain experts” should not be understood as business users but rather as people who are writing specs. These people are programmers. They do not really need an English-like language. Dave actually believes that the notion of fluent interface is often misunderstood: “the fluency here is programmer fluency, not English fluency. It’s writing succinct, expressive code”.

Dave Thomas argues that not only isn’t it necessary trying to get closer to a natural language, but it can also be detrimental. Natural languages are imprecise. This makes their power in the real world but this cannot apply to programming. This is why, “whenever we try to create a DSL that looks like a natural language, we fall short”. However hard one tries, syntax tends to remain “very unEnglish like”. And this gap is rather confusing:

There’s a major cognitive dissonance—I have to take ideas expressed in a natural language (the problem), then map them into an artificial language (the AppleScript programming model), but then write something that is a kind of faux natural language.

To illustrate the possible confusion, Dave gives an example of piece code from a test written using the test/spec framework and analyses one expression:

@result.should.be.a.kind.of String

It reads like English. But it isn’t. The words are separated by periods, except the last two, where we have a space. As a programmer, I know why. But as a user, I worry about it. In the first example, we write @result.should.be.a.kind_of. Why not kind.of? If I want to test that floats are roughly equal, I’d have said @result.should.be.close value. Why not close.to value?

Trivial details, but it means that I can’t just write tests using my knowledge of English—I have to look things up. And if I have to do that, why not just use a language/API that is closer to the domain of specifications and testing?

It is true that English-like DSL may be more readable, but Dave argues that “the attempt to create a natural language feel in the DSL leads to all sorts of leaks in the abstraction”. It might add to readability of code but it would “be taking away from its writability” and “adding uncertainty and ambiguity”:

The second you find yourself writing

  def a
     self
  end

so that you can use "a" as a connector in

add.a.diary.entry.for("Lunch").for(August.10.at(3.pm))

you know you’ve crossed a line. This is not longer a DSL. It’s broken English.

One of commentators, Has, also believes that trying to make a language readable to non-programmers one risks to end up with a "read-only language”. He takes the example of AppleScript. To improve its readability, it was necessary to remove “most of the usual symbolic cues that describe a language’s semantics”. As a result, “the syntax effectively obfuscates, not clarifies, the language semantics”. If “it’s very easy to read an AppleScript and understand _what_ it does, it’s damnably hard to figure out exactly _how_ it does it”.

Has highlights another issue that may result from using an English-like DSL: users might assume that “because it _looks_ like English, it will also _behave_ like it” and “form all sorts of very strong associations and conclusions about its nature, which then have to be undone the long, hard way”. Hence, according to Has, English-like appearance “accidentally encourages unrealistic user assumptions”

If DSL readability and expressiveness are of interest for you, find more examples and comments on Dave’s blog post.

The first time I got introduced to mutation, I had a bad feeling. How can x:=x+1 be logical at all?

It felt so unnatural, scary, or maybe like a hack. Then, because of society constrains, I got to forget the bad feeling about that. Well, my mind started to tolerate with counter-logic logic. And that is how I became an enterprise developer. I am not sure how proud I am with this title anyway. I feel that tolerating and accepting the counter-logic logic is one, and most important one, of the prerequisite to this title.

Nowadays, I am working on a project that has to process a huge number of data to do calculations on it and produce results that make sense to the user. Performance is a constraint. And I decided this time to do it the right way, the way that I believe is right. The logical way.

I decided to the calculations with no mutable state. I wanted immutability to be my guarantee to safety. Mutability breaks logic, and so produces bugs. But I couldn’t imagine how hard this choice can be. You tend all the time to feel that it is easier to introduce a mutable object, mutable flags, temporary mutable containers inside loops. Imperative programming is so tempting. Just like drugs. It feels so strange and weird in the first time, but then, it becomes a habit, a bad one, and you can’t help but to break into it. Mutation and imperative programming is the simplest thing that work in some way.

Today, I am happy I almost finished the whole calculation logic I had to model ( and it is quite a complex one). And I am proud that I could resist introducing mutability in the algorithm. Yet, I did some experiments along the way to introduce just a bit of state, kind of just what I need to introduce faster one kind of calculation. This very small mutation just breaks it all. It just breaks the whole modularity of the algorithm, and just makes the whole calculation much less generic, and more importantly less maintainable and evolutive. Such a small mutation, like introducing a mutable flag, or a global variable, just breaks the whole logic of my calculation, and drives me far away from a good modular calculation, with a good design based on plain old principles of separation of concerns. It just makes impossible to go further with the implementation! It is like a cancer of code…

Now, after the discipline and the resistance to a mutable state, we are so happy with the resulting code that yet needs polishing, makes it very easy to parameter more calculations and so add more desired functionality to the program. And just for fun, we compared it to the same calculation done with imperative programming. It is much more concise, intention concealing, and as a result, surprisingly we paid no performance cost for a much more declarative code.

I owe a lot to Erik Meijer, Paul Hudak, Linq, and Haskell :)

I just got back from JAOO2007. One session of the conference was James Coplien’s “Scrum Architecture”. After this session, a discussion took place about Coplien’s statement that “TDD harms your architecture”! This discussion continued afterwards as an Open Spaces debate. I had the chance then, with the help of Floyd Marinescu, to organize a filmed debate at InfoQ’s studio between Bob Martin and James Coplien on the same topic. I really enjoyed this debate which really showed a high quality spirit of discussion from both sides. Both had their arguments. I will not tell my opinion but rather let you judge yourself once the video is out on www.InfoQ.com .

Recently, I posted about the fact that I always write tests before code, but I don’t do Test Driven Development where I discussed the way I view testing. Yet I don’t guess I’ve been clear enough, so I decided to write few lines trying to better reflect the way I view a testing approach.

When I have a domain problem at hand, first what I think about is partitioning and abstraction, which means dividing the domain into sub-domains. This partitioning procedure is not driven by any tests, but rather by domain knowledge acquired from domain experts. This is extremely important; the architecture/design should reflect the domain/sub-domain problem it is trying to solve. This helps the software be flexible adopting business changes, as it was designed from a business perspective. A book that explains very well this approach is Eric Evans’s Domain Driven Design.

This Up-Front analysis is not defining the full up-front architecture, far from it, rather defining good bases for an architecture that maps to business. It shouldn’t take much time, just enough to help going forward with an iterative approach. This should define, either in code or in a tinny UML sketch, an object structure that forms the basic sub-domain’s components.

After defining the main structure of the sub-domain’s solution, I follow an iterative process as follows:

1: Driven by the business knowledge crunching, I define a component’s contract with its invariants and conditions

2: I write a test that forms the specification of this contract and its invariants

3: I start the implementation to satisfy the specifications I defined in the test

A very important note here is that the contract specifications should be readable, very clear and explicit. This explicitness helps to have a better vision of the whole system while reading the contracts-invariants of its defined components. This also helps spot bugs without the painful debugging: this approach of Design by Contract reflects developers understanding of the business domain in a more explicit way, so it makes it easier to spot misjudgments developers might have done in their implementation. This intention revealing method of development highlights important business rules, rather than letting them hidden behind an abstraction. Abstractions should not hide any important business concern as an implementation details. Explicit contracts and invariants allow business experts to have a look on what is going on inside the software. For an example of bugs that result from developer’s misjudgment see Reganwald’s recent post .

Of course unit tests are not the best tools for this approach. They work fine if we emphasize on methods and class naming for readability and visibility of invariants and rules. Still I prefer tools like BDD or Eiffel/Spec# Design by Contract. Hope we will have some more support for such tools in the industry soon.

A metaphor James Coplien used to illustrate the problem in Test Driven Development approach, which made sense to me, is a test driven course in a school. It’s about a teacher that finds that most of his students fail in the final test. So to make things work better, he decided to ask the students to write themselves their tests by the beginning of the semester, and then to let them pass these tests by the end of that semester! Hence, they might get better grades, but that doesn’t mean they got better knowledge at all.

 

Floyd Marinescu did a good summary of Simonyi’s presentation on JAOO and posted it on IinfoQ

http://www.infoq.com/news/2007/09/intentional-at-jaoo

It seems that guys at Intentional software are having something that works seriously, in real world business.

I am not driven by tests, my code neither. Actually my code is rather driven by design. I do use a test first approach, but I use for a contract first discipline in order to test that my decoupling works. Yes, I think about decoupling first, and then I write tests. I design, and then I write tests. Of course I don’t take a lot of time in my design reflection. But I am not driven by any  force other than my experience and the context at hand. I do abstractions and I validate them with tests. Actually, I use test first approach to specify my design. I don’t write any piece of code without a failed test waiting for it, but still I prefer to call that a test-first approach than the confusing “test-driven development”!

More about this approach to testing in my newer post Design Driven Testing: Using tests as specifications of system’s invariants and contracts design…

One of the most important properties of Domain Specific Languages is that it is an abstraction which is technology-agnostic. We know how often technology changes, adopting new frameworks, discarding old used ones. DSLs keep evolving to be a more and more mature abstraction of the domain. And integration of the new technologies and frameworks becomes nothing more than implementation details behind this abstraction. It forms a kind of specification of the sub domains.

http://www.infoq.com/news/2007/08/multi-lingual-programming