Logging in web services context – Part 2

In my previous blog posting, I discussed the relevence of logging in an enterprise. In this posting,I would like go into the technology perspective behind it. So let’s discuss about – How do we do logging in an application, in a sub system, in a department and in the enterprise? Do we affect the application in any way by doing logging? What are best practices in logging? Build or buy decisions that we need to take?

How do we do logging in an application, in a sub system, in a department and in the enterprise?


All right, we know that logging is good. But, let’s think about this scenario…

Consider an enterprise with multiple departments, and each with multiple applications. Just for fun, let’s picture it as below.



These is a good need to log each application in the enterprise, because each application is written with a business purpose. So, when each application logs into it's own proprietary data store then there will be “n” data stores for “n” applications. Imagine what would happen in a very large enterprise.. The production support team(s) need to maintain “n” more data stores in production. In order to get a single report these department managers need to run behind these “n” applications, collect the statistics of the key performance indicators, aggregate them and summarize them. This is a major problem.

After going through this, the smart architects of individual departments start thinking of a common logging framework. I deliberately chose the words “smart architects of individual departments”. I have seen logging frameworks developed for single department not sharing it with other deprtments (Don’t mean to be derogatory, it does happen in real life). Now it looks like this..



“Ideally”, it needs to look like this:



Can this “ideal” situation ever happen. Remember unless you are starting a brand new company and thought about this when you release the first application. It almost never happens. The enterprises are started even before IT ever existed, and all the new appliocations back then are now legacy systems which everyone is afraid of touching.

So in reality, we have some applications in production which cannot be changed, there are some packaged applications which have their own logging, some applications do have some funding and they can be changed in the upcoming versions; and finally, there are some brand new applications. This is the real scenario, and we have to accept it. Now, tell me about your logging framework again. … :)

So the answer to this problem my friend is this… think carefully and strategically.Try to exploit every opportunity to put the applications together. The problem of logging is much bigger than it appears. The right person to solve this problem is the “Enterprise architect” who has the visibility to the enterprise development, not the application architects nor the project-level managers.

Do we affect the application in any way by doing logging?

Yes of course.

It affects every application that it touches:

Let’s take a web service to illustrate the situation - “Customer web service” which provides a very critical and a comprehensive informtion about customer’s address, bank accounts, credit history, and driving history. While processing this request, “Customer web service” would call few systems. Say mainframe gives customer’s address and Finance application gives bank accounts and credit history, and driving history comes from some other system. Say the enterprise portal makes a call to this web service then, the call needs to be logged in all the these applications - Front-end portal, in Customer web service, in mainframe ,in financial package and in any other system/application. Unfortunately these applications are maintained by different departments. Now you see that, enterprise scale logging not a simple problem to solve.

Performance Vs logging:

Also, if you start logging every web method call, then there will be some negative consequences too. The first one that logging affects is the performance. Now, while serving the client web service needs to do two things – perform the needed action and also log the call in a central logging system. The central logging could be slow or too many routers away from the current web service hosting machine. It is even worse in the peak loads.

Oh god! Should we implement logging at all after all this. I still say yes. It is the necessary evil that we have to deal with no matter what. It is like a bad tasting pill that you need to take when you have fever.

Now that, we have to do logging.... Let’s figure out how.