How Logging helps better ROI in Test-Automation

Test-Automation solution has been in high demand now, where most of the projects are eagerly looking for it with a better ROI. Though better ROI depends on many factors where Logging and tracing is one of the most important one - unfortunately the most neglected too.

The big purpose of a strong logging mechanism is to let the tester distinguish between a bug and test issue without re-running the test.

If for every test-failure we are required to re-run the test then the purpose of Test-Automation seems loosing. As most of the Test-Automation scripts are supposed to be running independently, without manual supervision and in batch with other scripts, it become really hard for the tester to analyse the test-failure without a strong logging mechanism.

                                   

Logging enables a tester to make a logical inference from the error at the end of execution.

Moreover, there are situations where re-running test script(s) did not reproduce the same failure that happened in the batch run (among other scripts) which apparently makes a strong logging mechanism a must to analyse and prove the issue and file a bug (if necessary).

1. Invest Early

It is very important to include it in the Framework architecture and design. Once the architecture is freeze and the framework has been coded (or in midway) it’s a real pain in buttock to add a logging mechanism.

Planning early make us log each execution path otherwise it’s almost impossible to dry run our complete code and cover each execution path with the logging.

2. Log Levels

Different Test-Execution situations demands different levels of log. Sometimes it become imperative to include everything that we can and sometime it’s only required to have minimum of effective messages in the log.

Log-levels allow us to control what we really want to be log and allow us to filter best depending on the situation. Few most important log-levels that should be considered for any Test-automation:

·         OFF: No Log will be saved - a quiet execution.

·         DEBUG: Most verbose level. It used to put lower level information that is very helpful to diagnose a test-failure e.g. print parameters value with which a function is called, print important variables etc

·         INFO: Used to show the progress and chosen state information.

·         WARNING: Used to show un-expected events that are non-fatal and do not halt the progress.

·         ERROR: Used to show information about serious error that need to be handle immediately. It halts the progress.

It’s advisable to maintain a hierarchy in the log-levels (if custom made, as most of the logging libraries maintain it already) that said, the higher level always include all of the lower level messages.

3. What to Log?

Too less of Log is in-efficient and Too much of Log is noise.

Once the Architecture including Logging mechanism, log-levels are defined the next big question that came is what to Log, what log-level to use for each message e.g. “If a Thread is locked, whether it should be DEBUG or WARN?”

1.       The least expected conditions should always be logged

2.       Avoid redundancy in logs e.g. if you have printed parameters value in INFO then no need to print them in DEBUG/ERROR as INFO is(should) always included in DEBUD/INFO

3.       Put relevant information in debug/error messages e.g. Thread number, Browser name(if relevant) or any other relevant information

4.       In case of an error, consider putting module name, code line number and other relevant information that can directly hint you about code where error was generated.

2014-05-15 14:57:36,867 [INFO ] - Starting Execution on Firefox

2014-05-15 14:57:36,943 [DEBUG] since:56051542 thread:Thread-36 NDC: (com.common.util.Execution(Execution.java:110)) - Function open called

2014-05-15 14:58:19,235 [WARN ] [Firefox:Thread-36 ] - Not exist=0 => Sign_in

2014-05-15 14:59:00,818 [ERROR] [Firefox:Thread-36] since:56135417 | (com.common.util.Execution (Execution.java:91)) - Function: verifyTextPresent execution Failed || java.lang.reflect.InvocationTargetException

4. Exception Handling

Most of the Test Automation solutions are built on object-oriented paradigm or at-least the chosen platform has a facility to handle the exceptions (e.g. in Perl you can choose only the exception handling while relaxing with modular programming style)

It’s very important to handle individual Exceptions instead of generic Exceptions for all of the suspicious code as it allows test-script automation engineer to put much relevant messages in log.

Consider having custom made Exceptions that allow us putting valuable messages in-case of Exceptions along with the trace.

5. All relevant logs under one roof

To make a conclusion about the test-failure only test-logs are usually not enough, but there are other logs that are relevant too e.g. Application Under Test logs, framework logs (e.g. Selenium logs), System log (if they got some relevant information from Application too).

It’s always helpful to have all of the relevant logs for any execution under one folder in many ways. At the end of execution make sure the Framework is copying all the relevant logs in the log folder of current execution.

6. Do not re-invent the wheel

For most of the platforms that we use for test-automation, open source Libraries are already available with rich logging features that make implementing logging a cake walk. They are very flexible to fit to our configuration setting (e.g. handling log-levels through configuration files etc). Exploring these libraries to confirm our framework requirements against there features before writing our own could save us a lot of our time.

Log4j for Java, Log4Net for .Net, Log4Perl for Perl to name a few.

7. Screenshots

Screenshots (if applicable) have been proved to be very helpful in proving a defect, especially in case of intermittent or hard-to-reproduce bugs. Consider having a good screenshot utility in the framework and keep them in the log folder.

Having screenshots only for failures are generally enough.

8. Summary file

Logs have different end-users with different expectations e.g. Test-Engineer (running test and analysing test-failure), Test Lead/Manager, Project Manager/Lead. There requirements are generally not meet with a single log file.

I have seen cases where having a summary (text) log file generated for each execution is very handy. Noting only the test-failures in a summary file or the Test-case Pass/Fail status gives an overall confidence on the current build of AUT.

Conclusion

Giving logging a deserving place in architecture design makes the life of a tester easy and help in a better ROI.

“A Test-automation solution always comes with an extra cost (generally not counted in project budget) and in these scenarios it becomes very important to make sure the ROI is good. A good logging mechanism can save a lot of valuable time of a tester that could be utilized in other high value tasks and hence a better ROI.”