From 917d6d61eb3a2c0ef075e51ad5803d72602a2b56 Mon Sep 17 00:00:00 2001 From: Siddharth Ravikumar Date: Fri, 22 Jan 2016 16:22:08 -0500 Subject: Started fleshing out chapter 5. --- report/chapters/5-ucases-benchmarks.tex | 73 ++++++++++++++++++++++++++++++++- 1 file changed, 72 insertions(+), 1 deletion(-) (limited to 'report/chapters/5-ucases-benchmarks.tex') diff --git a/report/chapters/5-ucases-benchmarks.tex b/report/chapters/5-ucases-benchmarks.tex index 78e9e66..08421a3 100644 --- a/report/chapters/5-ucases-benchmarks.tex +++ b/report/chapters/5-ucases-benchmarks.tex @@ -1,4 +1,75 @@ -\chapter{Use Cases, Benchmarks, and Demonstration} +\chapter{Testing} + +\section{Unit testing} + +The \verb+nose+\cite{pylib:nose} testing framework was used to +write unit tests for the functions and classes part of the +\verb+combox.config+, \verb+combox.crypto+, \verb+combox.events+, +\verb+combox.file+, \verb+combox.silo+ \verb+combox._version+ +modules. Unit tests were not written for \verb+combox.cbox+, +\verb+combox.gui+, \verb+combox.combox.log+ modules. + +Unit tests for combox become reality by pure serendipity. During the +time, when I started working on combox, I was learning to use the +\verb+nose+ library to unit test python code. Since, \verb+combox+ was +being written in python, I started making it a norm to write unit +tests for functions and classes in combox modules. + +As mentioned before, unit tests were not written for some modules +either because it would make no sense to write one (for the +\verb+combox.cbox+ module, for instance, which basically uses +functions and classes defined in other modules to run combox) or it +was not clear how to write unit tests it (the \verb+combox.gui+ +contains just the \verb+ComboxConfigDialog+ a graphical front-end +which uses the configuration function defined in the +\verb+combox.config+ module to complete the combox configuration based +on the user input). + +It must be noted here that pure Test Driven Development (TDD) was not +observed -- most of the time the function/class was written before the +its corresponding test was written. + +\subsection{Benefits} + +While writing unit tests definitely increased the time to write a +particular feature, it enabled me to immediately check if a feature +worked as it should for the given use case or given set of inputs. + +With the benefit of hindsight, unit tests greatly helped in testing +the compatibility of combox on OSX. Before the \verb+v0.1.0+ release, +combox's node directory monitor always assumed that a file's first +shard (\verb+shard0+) is always available; while this assumption did +not create any problems on GNU/Linux, on OS X, this assumption made +the node directory monitor to behave erraticly -- this issue (bug +\#4\cite{combox-issue-tracker} was immediately found when the unit +tests were run for the first time on OS X. Another instance where unit +tests helped was just before the \verb+v0.2.0+ release; major changes, +including the introduction of file locks in the +\verb+ComboxDirMonitor+, were made to the \verb+combox.events+. When +the unit tests were run OS X, two tests failed, revealing a difference +in behaviour of watchdog\cite{pylib:watchdog} on GNU/Linux and OS X on +file creation\cite{combox-wd-fix}; without unit tests, there is a high +probability that this bug would never have been found by now. + +\subsection{Caveats} + +Unit tests are helpful in testing the correctness of a feature for +\verb+N+ number of use cases but it does not necessarily mean the +written feature correctly behaves for use cases that the author of the +feature did not consider or did not think about while writing the +respective feature. As Dijkstra correctly observed\cite{dijkstra69}: + +\begin{quote} +Testing shows the presence, not the absence of bugs +\end{quote} + +Unit tests failed reveal bugs \#4, \#5 \#6 \#7 \#5 \#10 +\#11\cite{combox-issue-tracker}; these bugs were found when manually +testing combox. + +\section{Manual testing} + +\section{Stress testing} \begin{figure}[h] \centering -- cgit v1.2.3