Monday, May 26, 2008

An internal API exposed as SOAP is not a webservice

As the system we build at work grows, more other systems are to be connected to our system. So is the case with a system that handles domain registration and hosting products for our smaller customers.

This system is originally a self-contained system. It's a web-application that has all the work-flow, provisioning and billing build in. And now some architects have decided that it should be connected to our system, so the big commercial systems can all offer these products without having to resort to manual steps performed by human operators.

There was resistance. The product managers and designers didn't like that idea. I assume the builders don't like it either, because the development is out-sourced and that would mean less income for that company. So after some political 'nudging' it was accepted that our system should be able to talk to theirs. And they we're going to give us webservices that enabled our system to offer the same products and functionality as their web interface does.

When I got the documentation on the webservice I was unpleasantly surprised. There are a lot of methods. And they're all small methods. They all have a very, small, singular purpose. As far as I can tell, they took the public properties of their Business Logic Layer and their DataAccess Layer and made those all into SOAP methods and called it a nice webservice interface. It's going to be a challenge to get that working smoothly...

Saturday, May 17, 2008

Your reference is not my reference

This week we had a discussion at work. It was about reference IDs. When a system calls the webservice on another system, sometimes you need to specify a reference ID. One such case is with asynchronous webservices. The system does the call, but does not get an immediate answer (other than: "Yeah, we're working on it"). And at some point in the future the system gets a call from the webservice (a callback) to return the result of the webservice. One way (and in my opinion the easiest way) to deal with this is to have the system send a reference ID with the original request and have the webservice put the reference in the asynchronous result. This way the system can correlate the result with the request and process it.

But as we found out today, that's thinking in "functions". Meaning that calling a webservice function and getting it's results is an operation that's completely separate of anything else. Any other thing can occur between calling and receiving a result and after you get the results, if there is a process, that ends that part of the result. But for some calls we need to think in "processes".

Our system is like a webservice bridge. We expose a generic interface to commercial systems and we consume very specific interfaces from technical back-end systems. So when a commercial system sends us a request, we map it into a specific request to a back-end system. This way the commercial system doesn't need to know all the gritty details of the technical system. Also all of our calls are atomic....except for two. Normally the commercial system deals with the process and it has never had any effect on our system. Until we needed to support a certain function that did. Now in my opinion the biggest problem was caused by a faulty design of the technical back-end, but it did expose a pit-trap we could fall in ourselves now that we are designing a new interface for the commercial systems. Sometimes you not only need to know what request was responsible for the incoming result, but also which commercial process it's part of. This is required if you need to perform certain steps in order and the technical back-end system also needs to follow-up on an earlier request with a follow-up request.

So we had a discussion about this. Do we take the reference form the commercial system and pass it on to the technical system. Out-sourcing the generation of references as it were? I was opposed, because I think a reference should only have meaning to the system that generated it. If we use the reference from the commercial system, we give it a meaning it shouldn't have. Also, if some time in the future we add more commercial systems there is a chance (and knowing Murphy it will happen) that more then one commercial system will come up with the same reference and then we have a problem.

My take on the whole situation is that we apparently need both a "function-reference" and a "process-reference". We generate our own function reference and use the process reference of the system that guards the process. This could be our system, in which case we generate the process reference, or it could be a commercial system, in which case we just take the reference and pass it on.

Tuesday, May 6, 2008

Some Ajax/WCF things I learned

I'm reworking a small ad-hoc web application I wrote because like all ad-hoc tools, it got a rather permanent status. It's like they say: "Nothing is as permanent as a temporal solution".
I wrote the first version using ASP.NET MVC. The only reason I did so was because I wanted to experiment with MVC and they told me to create the application in any way I saw fit. So it was a great way to experiment a little. Now the web application will be used on a more permanent basis and we will have to actively maintain it I decided to rewrite it. Also an additional requirement made the whole MVC thing a little harder then I ought to be. So I rewrote the web application, but this time I decided to use a WCF JSON webservice and go all-out Ajax. Still experimental, but less CTP-al.

So my first order of business was to write a WCF webservice that did the things I needed it to do. This was easy. No hurdles there.
Did you know you can put a [DataMember] attribute on a private member? It will even show up in the WCF service while still being private in your code. I used this because I had some data that needed to be exposed in the webservice as a string, but as a different data type internally. So I made a private property that made a string out of that data and put the [DataMember] attribute on it. And then I had a public property without the attribute for use inside the code. Very handy.

Next up after getting the WCF webservice done was consuming it through JavaScript. Nothing really difficult here either. Just Google for JavaScript and WCF and you'll find lots of examples. That's what I did. The only thing I did ran into was that at first I tried to have two separate web projects. One for the WCF webservice and one for the web application. For some reason I never got the JavaScript to work with the WCF webservice. I don't think it's impossible, but after fiddling with the WCF configuration for an hour I gave up and just made it into one project and it worked.

After I got the WCF/JavaScript communication to work I started to work on the JavaScript logic. And in doing so I learned some stuff about JavaScript.

JavaScript isn't really object oriented, but you can fake it.
You can create a new 'object' using the new keyword and a function declaration:

var myObject = new MyObject(){}

If you leave out the 'new' keyword the variable "myObject" would contain a reference to a function named "MyObject". A function that doesn't do anything. But add the 'new' keyword and now it's something like an object.
Unfortunately I have no idea how to proceed any further. It's possible to add properties and methods to the object. but I'm not 100% sure how.Also I had the site working in Firefox and Internet Explorer 8 beta, but when I tried it with Internet Explorer 7 it didn't work. It turns out that doing something like:

var element = document.getElementById("someId"); element.setAttribute("class", "cssClass"); element.setAttribute("onclick", "alert('Yay!');");

Works perfectly in Firefox and IE8, but not in IE7. The annoying part is that if you use the Developer Toolbar in IE7 and inspect the DOM properties of the element they look good. I created a similar element in straight HTML and compared it to the element I added and setup dynamically through JavaScript, but I didn't see any difference. Yet it didn't work.
Turns out you can't do this in IE7 and you need to use the proper properties of the element to set stuff up. So the above needs to be:

var element = document.getElementById("someId"); element.className = "cssClass"; element.onclick = function() { alert('Yay!'); };

Luckily this also works in Firefox and IE8.

More as I learn more...

Sunday, April 27, 2008

Unit-testing private methods

Suppose you have a class that has some private methods and you want to make sure these methods function correctly, so you want to write unit-tests for them. The problem is that you can't call these methods, because they are private.

The class we want to test:

public class ClassWithPrivateMethods
{
     public int Add(int firstNumber, int secondNumber)
     {
         return firstNumber + secondNumber;
     }

     private int Substract(int initialNumber, int numberToSubstract)
     {
         return initialNumber - numberToSubstract;
     }
}

Testing the "Add" function is simple. It's public, so we can just call it from a unit test:

[TestMethod]
public void TestAddition()
{
     ClassWithPrivateMethods classToTest = new ClassWithPrivateMethods();
     int result = classToTest.Add(5, 5);
     Assert.AreEqual(10, result);
}

Testing the "Substract" function is a problem. You can't call the method from the unit-test. One way around this is creating a new class, specific for the unit-test, that inherits from the class you want to test and use the "new" keyword to make the function available as a public function. The problem with this approach is that the method can't be private. It has to be protected at the minimum:

public class ClassWithPrivateMethods
{
    public int Add(int firstNumber, int secondNumber)
    {
        return firstNumber + secondNumber;
    }

    protected int Substract(int initialNumber, int numberToSubstract)
    {
        return initialNumber - numberToSubstract;
    }
}

Which means you'll have a test class like this:

private class ClassWithPrivateMethods_TestWrapper : ClassWithPrivateMethods
{
    public new int Substract(int initialNumber, int numberToSubstract)
    {
        return base.Substract(initialNumber, numberToSubstract);
    }
}

And a unit-test like this:

[TestMethod]
public void TestSubstractionInt()
{
    ClassWithPrivateMethods_TestWrapper classToTest = new ClassWithPrivateMethods_TestWrapper();
    int result = classToTest.Substract(10, 5);
    Assert.AreEqual(5, result);
}

I've done this before and it works well. But this way you still can't have private functions.
Enter reflection.
With reflection you can do anything you want to the class. Even invoke private methods! So we'll change the helper class into this:

private class ClassWithPrivateMethods_TestWrapper : ClassWithPrivateMethods
{
    public int Substract(int initialNumber, int numberToSubstract)
    {
       Type t = typeof(ClassWithPrivateMethods);
       return (int)t.InvokeMember("Substract", BindingFlags.NonPublic | BindingFlags.Instance | BindingFlags.InvokeMethod,
       null, this, new object[] { initialNumber, numberToSubstract });
    }
}

Ofcourse you don't really need the wrapper class, you could just as easy put these two lines of reflection stuff inside the unit-test if you wanted. But if you do use the wrapper class, you can just instantiate it and call the "Substract" method for your unit-testing purposes, just like the unit-test above.
This also works for methods that have been overloaded for multiple data types. Suppose we have the following class we want to test:

public class ClassWithPrivateMethods
{
    public int Add(int firstNumber, int secondNumber)
    {
        return firstNumber + secondNumber;
    }

    private int Substract(int initialNumber, int numberToSubstract)
    {
         return initialNumber - numberToSubstract;
    }

    private float Substract(float initialNumber, float numberToSubstract)
    {
        return initialNumber - numberToSubstract;
    }
}

Our wrapper class would look like this:

private class ClassWithPrivateMethods_TestWrapper : ClassWithPrivateMethods
{
    public int Substract(int initialNumber, int numberToSubstract)
    {
        Type t = typeof(ClassWithPrivateMethods);
        return (int)t.InvokeMember("Substract", BindingFlags.NonPublic | BindingFlags.Instance | BindingFlags.InvokeMethod,
        null, this, new object[] { initialNumber, numberToSubstract });
    }

    public float Substract(float initialNumber, float numberToSubstract)
    {
        Type t = typeof(ClassWithPrivateMethods);
        return (float)t.InvokeMember("Substract", BindingFlags.NonPublic | BindingFlags.Instance | BindingFlags.InvokeMethod,
        null, this, new object[] { initialNumber, numberToSubstract });
    }
}

And you can unit-test like this:

[TestMethod]
public void TestAddition()
{
    ClassWithPrivateMethods classToTest = new ClassWithPrivateMethods();
    int result = classToTest.Add(5, 5);
    Assert.AreEqual(10, result);
}

[TestMethod]
public void TestSubstractionInt()
{
    ClassWithPrivateMethods_TestWrapper classToTest = new ClassWithPrivateMethods_TestWrapper();
    int result = classToTest.Substract(10, 5);
    Assert.AreEqual(5, result);
}

[TestMethod]
public void TestSubstractionFloat()
{
    ClassWithPrivateMethods_TestWrapper classToTest = new ClassWithPrivateMethods_TestWrapper();
    float result = classToTest.Substract(10.0f, 2.5f);
    Assert.AreEqual(7.5f, result);
}

Wednesday, April 23, 2008

The blame game, or: It works on our end.

The system I'm working on is in the end-to-end testing phase. It's being tested for the past 2 weeks or so and today the testers ran into an issue where request that they sent from the first system  timed-out. Their view on the issue: It's our problem.

So we did some problem analyses. We checked some log files, but we never even saw a request coming into the webserver. Then I sent the same request they we're trying to do and it got accepted, it got processed and they even got the callback to notify that the request had been completed. So we established that our system worked. Even the engineer from that other system agreed to that.

So the engineer went to check on stuff. And at the end of the day I got an e-mail that basically stated:
"We checked everything. The request is correct and we have the correct URL configured. So the only conclusion in that is must be a problem in [the system I work on]. And we'd like them to solve the problem ASAP."

This reminded me on a joke I read in David Platts excellent Why Software Sucks...and What You Can Do About It.  It's a joke about Microsoft in  a chapter about Microsoft. The short version is something like this:

A Microsoft engineer goes into the army and needs to shoot at the shooting range, but he misses every time. The drill-sergeant  gives the Microsoft engineer craps and tells him to do better. At which point the engineer holds his finger in front of the nozzle of the gun and shoots his finger off. Then he points the bloody stump towards the target and says: "Well, it comes out here alright, so the problem must be on their side".

This is exactly what I thought about when I read that e-mail and it's this kind of shortsightedness  that really pissed me off. A good engineer, in my opinion, always assumes the problem is in his code. Only after very careful examination and testing should you state it's probably a problem on the other side. And even then the engineer should fully cooperate in finding out the problem. After all, it's not like we're competitors. We're all building parts for the same product and work in the same company on the same goals.
With this kind of thinking we'll never deliver a good system.

Tuesday, April 15, 2008

LINQ means working fast

Now granted, I have not yet done a really big project with LINQ and I have really limited experience with LINQ so far, but let's not kill the hype here, okay? :-)

At my office we have almost finished our part in a whole connected chain of systems to provision products that my company offers to its end-users. The system we build is an abstraction on top of the technical provisioning systems and through our system they are integrated for use by the commercial systems at the top of the chain. In the beginning of last week they told me that one of the big commercial system would need more time to get done. "More time" in this case meant around a month. Knowing how good estimates by programmers are, I'm positive it will be more then a month, but I digress. Even with that delay, they still want to start the pilot project and start using parts of the provisioning chain. So they asked me to build a kind off web-front-end to our system. So that they could start handling orders manually at the very least. And while the big commercial system was delayed for (at least) a month. They asked me to do it in 2 or 3 days. And because they were desperate they told me to build it no matter what. The front-end would contain next to no logic and would only need to communicate with out system, which I know very well, so I decided to build it with some new technology I was itching to try out, so I chose to build it with ASP.NET MVC and LINQ.

I had played with ASP.NET MVC before, but they updated the preview bits and I found some welcome changes. Aside from spending some extra time in getting the URLs to look nice it wasn't really that challenging. No, the fun part was in using LINQ.

In my team we have defined a set of "best practices" or some development "rules" and one of the rules is that we always use Stored Procedures (sprocs). So when I write a Data Access Layer (DAL) for any program I always create a data-model (the database tables), then I create sprocs for all the things I need to do to the data that will be stored in the database (reads, inserts, updates, etc) and then I will write some C# code to access these sprocs and to create C# objects out of the returned data (and of course code to be able to input data to the sprocs). This time I chose to go with LINQ and I've never written my DAL in such a short time. Even if I still had to find out some basic things on how to use LINQ.

I think that I wrote my DAL with LINQ in less then 20% of the time it would have taken me if I had to use our 'conventional' way. It's so simple. Now, be aware that I did not do any complicated stuff here. Just you basic selects, inserts and left outer joins. Nothing more. But I think that's exactly where the strength of LINQ lies. All the simple stuff is trivial to do in LINQ, but takes more effort to do as sprocs and SqlDataReaders. Why waste your valuable development time on that?

I encourage every developer out there to try LINQ. Give it a good whirl and don't judge the LINQ book by its cover. Yes, it generates dynamic SQL, but I've some tests (some very unscientific tests) and found that LINQ is faster then using SqlDataReader with sprocs on simple queries. I just took a large table (500,000+ items) with some 12 columns and made an sproc and a LINQ query (that resulted in the exact same SQL as the sproc) and did some loop testing. When doing the same query as fast as possible in a loop, LINQ will be faster then the SqlDataReader/sproc method until around 300 queries. After that LINQ seems to be getting slower. Personally I think this has to do with the fact that for the sproc SQL Server only needs to see a single word (the name of the sproc) to recognize what you want and re-use it's caching, whereas with the LINQ query it will need to compare the whole query to notice it already exists in the cache. And maybe network traffic also plays a role. Network traffic is less with an sproc. So with these test results in mind, sprocs would only be faster if you have a really high-traffic site (but even then you should measure before you start to optimise). And if you want to be Agile. You'll release early and often. So you can start cramming out features while using LINQ for your DAL so you won't waste precious time on sprocs and such, and then replace the LINQ DAL with an sproc DAL when you start to notice the performance hit. Or you combine them, which is also entirely possible.

Sunday, April 6, 2008

About being a good programmer

I consider myself to be a good programmer. I am my own worst critic, I always question methods and don't use methods "just because I did last time". I always assess if they are appropriate for the new challenge.
I also try to keep up with new technologies and trends in development and although I can be a bit too enthusiastic about certain things, I never really use new technology just because it's new.

At the office my team is currently at the end of a very big (for us) project. As always, the time was short and the deadline was set in stone, but no one really knew exactly what we had to build. So me and 6 colleagues started working on the project. It was jump-started by a Software Architect, who created the very basic architecture and then I took over.
The system is basically a routing and handling system for product provisioning and we envisioned a program where we would just write a plug-in for a specific product to handle the product specific tasks. That way we can easily extend the system for new products.
In the initial planning we had the system to handle 4 products, of course this soon turned into 5 products and then 6 products. So me and two other colleagues set off, each assigned his own product(s), to build the plug-ins.
One of the colleagues copied the basic architecture of his plug-ins from my plug-ins, but the other colleague took a whole different route.

After initial programming was done and we were in the test phase, we finally had time for code reviews. So as I was reviewing the code of the colleague who took a very different route then I did, I found it hard to determine what he was doing. I didn't get it and his solution seemed overly complex to me. Then he explained his architecture....

I was shocked. His architecture was much better then mine. I was so set in the mind-set of my architecture that I failed to see the beauty of his decisions. I was shocked and ashamed. Ashamed because I thought my design was better then his before he explained it to me.

This reiterated a very important lesson to me. I am not perfect, I won't always do the best thing. I might not even be close from time to time.
One of the things I consider add to me being a good programmer, is that I will admit if I am wrong and that I will accept a better solution, even if my solution was the inferior one. I'm not too proud to admit I am not perfect, but somehow, this time, it kind of sneaked up on me.

The lesson I take away from this? Don't judge too soon and don't think you've got the best solution. There will always be a better solution.