1 files changed, 96 insertions, 0 deletions
diff --git a/report/chapters/5-testing.tex b/report/chapters/5-testing.tex
new file mode 100644
index 0000000..08421a3
--- /dev/null
+++ b/report/chapters/5-testing.tex
@@ -0,0 +1,96 @@
+\chapter{Testing}
+
+\section{Unit testing}
+
+The \verb+nose+\cite{pylib:nose} testing framework was used to
+write unit tests for the functions and classes part of the
+\verb+combox.config+, \verb+combox.crypto+, \verb+combox.events+,
+\verb+combox.file+, \verb+combox.silo+ \verb+combox._version+
+modules. Unit tests were not written for \verb+combox.cbox+,
+\verb+combox.gui+, \verb+combox.combox.log+ modules.
+
+Unit tests for combox become reality by pure serendipity. During the
+time, when I started working on combox, I was learning to use the
+\verb+nose+ library to unit test python code. Since, \verb+combox+ was
+being written in python, I started making it a norm to write unit
+tests for functions and classes in combox modules.
+
+As mentioned before, unit tests were not written for some modules
+either because it would make no sense to write one (for the
+\verb+combox.cbox+ module, for instance, which basically uses
+functions and classes defined in other modules to run combox) or it
+was not clear how to write unit tests it (the \verb+combox.gui+
+contains just the \verb+ComboxConfigDialog+ a graphical front-end
+which uses the configuration function defined in the
+\verb+combox.config+ module to complete the combox configuration based
+on the user input).
+
+It must be noted here that pure Test Driven Development (TDD) was not
+observed -- most of the time the function/class was written before the
+its corresponding test was written.
+
+\subsection{Benefits}
+
+While writing unit tests definitely increased the time to write a
+particular feature, it enabled me to immediately check if a feature
+worked as it should for the given use case or given set of inputs.
+
+With the benefit of hindsight, unit tests greatly helped in testing
+the compatibility of combox on OSX. Before the \verb+v0.1.0+ release,
+combox's node directory monitor always assumed that a file's first
+shard (\verb+shard0+) is always available; while this assumption did
+not create any problems on GNU/Linux, on OS X, this assumption made
+the node directory monitor to behave erraticly -- this issue (bug
+\#4\cite{combox-issue-tracker} was immediately found when the unit
+tests were run for the first time on OS X. Another instance where unit
+tests helped was just before the \verb+v0.2.0+ release; major changes,
+including the introduction of file locks in the
+\verb+ComboxDirMonitor+, were made to the \verb+combox.events+. When
+the unit tests were run OS X, two tests failed, revealing a difference
+in behaviour of watchdog\cite{pylib:watchdog} on GNU/Linux and OS X on
+file creation\cite{combox-wd-fix}; without unit tests, there is a high
+probability that this bug would never have been found by now.
+
+\subsection{Caveats}
+
+Unit tests are helpful in testing the correctness of a feature for
+\verb+N+ number of use cases but it does not necessarily mean the
+written feature correctly behaves for use cases that the author of the
+feature did not consider or did not think about while writing the
+respective feature. As Dijkstra correctly observed\cite{dijkstra69}:
+
+\begin{quote}
+Testing shows the presence, not the absence of bugs
+\end{quote}
+
+Unit tests failed reveal bugs \#4, \#5 \#6 \#7 \#5 \#10
+\#11\cite{combox-issue-tracker}; these bugs were found when manually
+testing combox.
+
+\section{Manual testing}
+
+\section{Stress testing}
+
+\begin{figure}[h]
+\centering
+\input{graphs/tot-time.tex}
+\caption{time to process all files}
+\end{figure}
+
+\begin{figure}[h]
+\centering
+\input{graphs/avg-time-sae.tex}
+\caption{avg. time to split and encrypt}
+\end{figure}
+
+\begin{figure}[h]
+\centering
+\input{graphs/tot-time-diff.tex}
+\caption{time to process all files - difference between 2015 and 2016}
+\end{figure}
+
+\begin{figure}[h]
+\centering
+\input{graphs/avg-time-sae-diff.tex}
+\caption{avg. time to split and encrypt - difference between 2015 and 2016}
+\end{figure}
+\ No newline at end of file