Test-Automation solution has been in high demand now, where
most of the projects are eagerly looking for it with a better ROI. Though better
ROI depends on many factors where Logging and tracing is one of the most important
one - unfortunately the most neglected too.
The big purpose of a strong logging mechanism is to let the tester distinguish between a bug and test issue without re-running the test.
 |
If for every test-failure we are required to re-run the test then the
purpose of Test-Automation seems loosing. As most of the Test-Automation
scripts are supposed to be running independently, without manual supervision
and in batch with other scripts, it become really hard for the tester to
analyse the test-failure without a strong logging mechanism.
|
Logging enables a tester to make a logical inference from
the error at the end of execution.
Moreover, there are situations where re-running test script(s)
did not reproduce the same failure that happened in the batch run (among other
scripts) which apparently makes a strong logging mechanism a must to analyse
and prove the issue and file a bug (if necessary).
1. Invest Early
It is very important to include
it in the Framework architecture and design. Once the architecture is freeze
and the framework has been coded (or in midway) it’s a real pain in buttock to
add a logging mechanism.
Planning early make us log each
execution path otherwise it’s almost impossible to dry run our complete code
and cover each execution path with the logging.
2. Log Levels
Different Test-Execution
situations demands different levels of log. Sometimes it become imperative to
include everything that we can and sometime it’s only required to have minimum
of effective messages in the log.
 |
Log-levels allow us to control what we really want to be log and
allow us to filter best depending on the situation. Few most important
log-levels that should be considered for any Test-automation:
|
·
OFF: No Log will be
saved - a quiet execution.
·
DEBUG: Most verbose
level. It used to put lower level information that is very helpful to diagnose a
test-failure e.g. print parameters value with which a function is called, print
important variables etc
·
INFO: Used to show the
progress and chosen state information.
·
WARNING: Used to show
un-expected events that are non-fatal and do not halt the progress.
·
ERROR: Used to show
information about serious error that need to be handle immediately. It halts
the progress.
It’s advisable to maintain a
hierarchy in the log-levels (if custom made, as most of the logging libraries
maintain it already) that said, the higher level always include all of the
lower level messages.
3. What to Log?
Too less of Log is in-efficient and Too much of Log is noise.
 |
Once the Architecture including Logging mechanism, log-levels are
defined the next big question that came is what to Log, what log-level to use
for each message e.g. “If a Thread is locked, whether it should be DEBUG or
WARN?”
|
1. The least expected conditions should always
be logged
2. Avoid redundancy in logs e.g. if you have
printed parameters value in INFO then no need to print them in DEBUG/ERROR as
INFO is(should) always included in DEBUD/INFO
3. Put relevant information in debug/error
messages e.g. Thread number, Browser name(if relevant) or any other relevant
information
4. In case of an error, consider putting
module name, code line number and other relevant information that can directly hint
you about code where error was generated.
2014-05-15 14:57:36,867 [INFO ] - Starting Execution on Firefox
2014-05-15 14:57:36,943 [DEBUG] since:56051542 thread:Thread-36 NDC:
(com.common.util.Execution(Execution.java:110)) - Function open called
2014-05-15 14:58:19,235 [WARN ] [Firefox:Thread-36 ] - Not exist=0
=> Sign_in
2014-05-15 14:59:00,818 [ERROR] [Firefox:Thread-36] since:56135417 |
(com.common.util.Execution (Execution.java:91)) - Function: verifyTextPresent
execution Failed || java.lang.reflect.InvocationTargetException
4. Exception Handling
Most of the Test Automation solutions are built on object-oriented paradigm or at-least the chosen platform has a facility to handle the exceptions (e.g. in Perl you can choose only the exception handling while relaxing with modular programming style)
 |
It’s very
important to handle individual Exceptions instead of generic Exceptions for
all of the suspicious code as it allows test-script automation engineer to
put much relevant messages in log.
|
Consider having custom made Exceptions that allow us putting valuable messages in-case of Exceptions along with the trace.
5. All relevant logs under one roof
To make a conclusion about the test-failure only test-logs are usually not enough, but there are other logs that are relevant too e.g. Application Under Test logs, framework logs (e.g. Selenium logs), System log (if they got some relevant information from Application too).
It’s always helpful to have all of the relevant logs for any execution under one folder in many ways. At the end of execution make sure the Framework is copying all the relevant logs in the log folder of current execution.
6. Do not re-invent the wheel
For most of the platforms that we use for test-automation, open source Libraries are already available with rich logging features that make implementing logging a cake walk. They are very flexible to fit to our configuration setting (e.g. handling log-levels through configuration files etc). Exploring these libraries to confirm our framework requirements against there features before writing our own could save us a lot of our time.
Log4j for Java, Log4Net for .Net, Log4Perl for Perl to name a few.
7. Screenshots
Screenshots (if applicable) have been proved to be very helpful in proving a defect, especially in case of intermittent or hard-to-reproduce bugs. Consider having a good screenshot utility in the framework and keep them in the log folder.
Having screenshots only for failures are generally enough.
8. Summary file
Logs have different end-users with different expectations e.g. Test-Engineer (running test and analysing test-failure), Test Lead/Manager, Project Manager/Lead. There requirements are generally not meet with a single log file.
I have seen cases where having a summary (text) log file generated for each execution is very handy. Noting only the test-failures in a summary file or the Test-case Pass/Fail status gives an overall confidence on the current build of AUT.
Conclusion
Giving logging a deserving place
in architecture design makes the life of a tester easy and help in a better
ROI.
“A Test-automation solution always comes with an extra cost (generally not counted in project budget) and in these scenarios it becomes very important to make sure the ROI is good. A good logging mechanism can save a lot of valuable time of a tester that could be utilized in other high value tasks and hence a better ROI.”
