Moved around chapters.

Chapter 3 -> Chapter 2
Chapter 4 -> Chapter 3
Chapter 5 -> Chapter 4

1 files changed, 668 insertions, 0 deletions

diff --git a/report/chapters/4-testing.tex b/report/chapters/4-testing.tex
new file mode 100644
index 0000000..e6376f1
--- /dev/null
+++ b/report/chapters/4-testing.tex
@@ -0,0 +1,668 @@
+\chapter{Testing}\label{ch:5}
+
+\epigraph{Testing shows the presence, not the absence of
+  bugs.}{\textit{Dijkstra}\cite{dijkstra69}}
+
+\section{Unit testing}\label{sec:5-unit-testing}
+
+The \verb+nose+\cite{pylib:nose} testing framework was used to
+write unit tests for the functions and classes part of the
+\verb+combox.config+, \verb+combox.crypto+, \verb+combox.events+,
+\verb+combox.file+, \verb+combox.silo+ \verb+combox._version+
+modules. Unit tests were not written for \verb+combox.cbox+,
+\verb+combox.gui+, \verb+combox.combox.log+ modules.
+
+Unit tests for combox become reality by pure serendipity. During the
+time, when I started working on combox, I was learning to use the
+\verb+nose+ library to unit test python code. Since, \verb+combox+ was
+being written in python, I started making it a norm to write unit
+tests for functions and classes in combox modules.
+
+As mentioned before, unit tests were not written for some modules
+either because it would make no sense to write one (for the
+\verb+combox.cbox+ module, for instance, which basically uses
+functions and classes defined in other modules to run combox) or it
+was not clear how to write unit tests it (the \verb+combox.gui+
+contains just the \verb+ComboxConfigDialog+ a graphical front-end
+which uses the configuration function defined in the
+\verb+combox.config+ module to complete the combox configuration based
+on the user input).
+
+It must be noted here that pure Test Driven Development (TDD) was not
+observed -- most of the time the function/class was written before the
+its corresponding test was written.
+
+\subsection{Benefits}
+
+While writing unit tests definitely increased the time to write a
+particular feature, it enabled me to immediately check if a feature
+worked as it should for the given use case or given set of inputs.
+
+With the benefit of hindsight, unit tests greatly helped in testing
+the compatibility of combox on OSX. Before the \verb+v0.1.0+ release,
+combox's node directory monitor always assumed that a file's first
+shard (\verb+shard0+) is always available; while this assumption did
+not create any problems on GNU/Linux, on OS X, this assumption made
+the node directory monitor to behave erratically -- this issue (bug
+\#4\cite{combox-issue-tracker} was immediately found when the unit
+tests were run for the first time on OS X. Another instance where unit
+tests helped was just before the \verb+v0.2.0+ release; major changes,
+including the introduction of file locks in the
+\verb+ComboxDirMonitor+, were made to the \verb+combox.events+. When
+the unit tests were run OS X, two tests failed, revealing a difference
+in behavior of watchdog\cite{pylib:watchdog} on GNU/Linux and OS X on
+file creation\cite{combox-wd-fix}; without unit tests, there is a high
+probability that this bug would never have been found by now.
+
+\subsection{Caveats}
+
+Unit tests are helpful in testing the correctness of a feature for
+\verb+N+ number of use cases but it does not necessarily mean the
+written feature correctly behaves for use cases that the author of the
+feature did not consider or did not think about while writing the
+respective feature. As Dijkstra correctly observed:
+
+Unit tests failed to reveal bugs \#4, \#5 \#6 \#7 \#5 \#10
+\#11\cite{combox-issue-tracker}; these bugs were found when manually
+testing combox.
+
+\section{Manual testing}\label{sec:5-manual-testing}
+
+The unit tests for the \verb+combox.events+ module test the
+correctness of the \verb+ComboxDirMonitor+ and \verb+NodeDirMonitor+
+independently; in order to comprehensively test the correctness of
+both \verb+ComboxDirMonitor+ and \verb+NodeDirMonitor+, it was
+required to manually test combox running on more than one computer. As
+you'll see in the following subsections, several bugs were found and
+fixed while doing manual testing.
+
+Three different types of setups were used to test combox. The first
+kind of setup has two GNU/Linux machines each using combox to sync
+files between each other with Dropbox and Google Drive being the
+nodes; the second kind of setup has a GNU/Linux machine and a OS X
+machine each using combox to sync files between each other with
+Dropbox and Google Drive being the nodes; the third kind of setup has
+a GNU/Linux machine and OS X machine each using combox to sync files
+between each other with Dropbox, Google Drive and a USB stick as
+nodes.
+
+\subsection{General setup and notes}
+
+\begin{itemize}
+\item On the GNU/Linux machines, the official Dropbox client was used
+  to sync the Dropbox node directory to Dropbox'
+  servers. \verb+rclone+\cite{program:rclone} was used to sync the
+  Google Drive node directory to Google Drive' servers;At the time of
+  testing, Google Drive did not have client for GNU/Linux.
+\item On OS X, the official Dropbox client was used to sync the
+  Dropbox node directory to Dropbox's servers; the official Google
+  Drive client was used to sync the Google Drive node directory to
+  Google Driver' servers.
+\item Since combox is extremely event-driven, combox must be started
+  before the Dropbox and Google Drive clients start syncing their
+  respective directories (nodes).
+\end{itemize}
+
+\subsection{Testing on two GNU/Linux machines}
+
+combox was run to two GNU/Linux machines and a file was alternatively
+created/modified/renamed/deleted on an of the GNU/Linux machine and it
+was verified if the respective file was also
+created/modified/renamed/deleted on the other GNU/Linux machine. One
+of the GNU/Linux machine (\verb+lyra)+ was a virtual machine running
+Debian GNU/Linux stable (version 8.x); the other GNU/Linux machine
+(\verb+grus+) was a physical machine running Debian GNU/Linux
+testing. The node directories to scatter the files' shards were the
+Dropbox directory and Google Drive directory. The official Dropbox
+client was used to automatically sync files from the Dropbox directory
+to the Dropbox' server; \verb+rclone+\cite{program:rclone} was used to
+sync files from Google Drive directory to Google Drive' server.
+
+\subsubsection{Issues found}\label{ch-5-2gnus-issues}
+
+\begin{itemize}
+\item Some editors, especially on POSIX complaint systems, create
+  backup version of the file being edited. combox was detecting this
+  backup file as a ``new file'' and it split it into shards, encrypted
+  the shards and scattered the shards across the node directories. The
+  right thing for combox to do was to ignore these backup files and do
+  nothing about them. This issue was fixed on
+  \verb+2015-09-29+\cite{combox-issue-tracker}. Now the
+  \verb+ComboxDirMonitor+, on a ``file created'' or ``file modified''
+  event, returns from the \verb+on_created+ or \verb+on_modified+
+  callback when it finds that the file is a backup/temporary file.
+\item Dropbox client maintains the \verb+.dropbox.cache+ directory
+  under the root of the Dropbox directory.
+
+  \begin{itemize}
+  \item When a file (shard) was created on another computer, the
+    Dropbox client pulls the new file (shard) to this computer into
+    \verb+.dropbox.cache+ as a temporary file and then moves the new
+    file (shard) to its respective location with the appropriate name.
+  \item When a file (shard) was modified on another computer, the
+    Dropbox client pulls the modified file (shard) to this computer
+    into the \verb+.dropbox.cache+ as a temporary file; moves the old
+    version of the file (shard) under the Dropbox directory into the
+    \verb+.dropbox.cache+; finally moves the updated copy of the file,
+    stored as a temporary file, into the Dropbox directory to its
+    respective location with the appropriate name.
+  \item When a file (shard) was deleted on another computer, the
+    Dropbox client moves the delete file into the
+    \verb+.dropbox.cache+ directory on this computer.
+  \end{itemize}
+
+  All of the above behavior of the Dropbox client epically broke
+  combox. Commits \verb+3d714c5+ to
+  \verb+6e1133f+\cite{git:dropbox-fix} fixed combox by making it aware
+  of Dropbox's client behavior.
+\end{itemize}
+
+\subsubsection{Demo}
+
+Demo of combox being used on two GNU/Linux machines can be viewed at
+\url{https://ricketyspace.net/combox/combox-2-gnus.webm}.
+
+\verb+lyra+ (virtual machine) and \verb+grus+ (bare-metal) are the two
+GNU/Linux machines being used for the demo.
+
+Description of what happens in the demo follows:
+
+ - (lyra) install combox.
+
+ - (lyra) run combox (test mode).
+
+ - (lyra) create file \verb+walden.pond+ with content ``It must be
+ beautiful there''.
+
+ - (lyra) sync Google Drive using \verb+rclone+.
+
+ - (grus) sync Google Drive using \verb+rclone+.
+
+ - (grus) git pull latest copy of combox.
+
+ - (grus) install combox 
+
+ - (grus) run combox (testing mode).
+
+ - (grus) verify that \verb+walden.pond+ was create on this machine.
+
+ - (grus) append 'Peaceful too.' to \verb+walden.pond+.
+
+ - (grus) sync Google Drive using \verb+rclone+.
+
+ - (lyra) sync Google Drive using \verb+rclone+.
+
+ - (lyra) verify that the latest copy of \verb+walden.pond+ is there
+ in the combox directory; it should contain 'Peaceful too.' in the
+ last line.
+
+ - (lyra) append ``I've a dream'' to \verb+walden.pond+.
+
+ - (lyra) sync Google Drive using \verb+rclone+.
+
+ - (grus) sync Google Drive using \verb+rclone+.
+
+ - (grus) verify that the latest copy of \verb+walden.pond+ is there
+ in the combox directory; it should contain ``I've a dream'' in the
+ last line.
+
+ - (grus) remove \verb+walden.pond+ from combox directory.
+
+ - (grus) sync Google Drive using \verb+rclone+.
+
+ - (lyra) sync Google Drive using \verb+rclone+.
+
+ - (lyra) verify that \verb+walden.pond+ is removed from the combox
+ directory.
+
+ - (grus) open dropbox and Google drive accounts from the web browser.
+
+ - (lyra) create file \verb+manufacturing.consent.+ with content ``Chomsky stuff?''.
+
+ - (lyra) sync Google Drive using \verb+rclone+.
+
+ - (grus) sync Google Drive using \verb+rclone+.
+
+ - (grus) verify that \verb+manufacturing.consent+ was created in the
+ combox directory.
+
+ - (grus) verify that the shards of \verb+manufacturing.consent+ were
+ created on Dropbox and Google Drive through the web browser.
+
+\subsection{Testing on a GNU/Linux and an OS X machine}
+
+combox was run on a GNU/Linux machine and an OS X machine and a file
+was alternatively created/modified/renamed/deleted on one of the
+machine and it was verified if the respective file was also
+created/modified/renamed/deleted on the other machine. The GNU/Linux
+machine was a virtual machine (\verb+lyra+) running Debian GNU/Linux
+stable; the OS X machine was on Mavericks (10.9) during the initial
+stage of testing, later it was upgraded to Yosemite (10.10). The node
+directories to scatter files' shards were the Dropbox directory and
+the Google Drive directory. The official Dropbox client was used to
+automatically sync files from the Dropbox directory to the Dropbox'
+server on both the GNU/Linux machine and the OS X machine; the
+official Google Drive client was used to automatically sync files from
+the Google Drive directory to Google Drive' server on OS X and
+\verb+rclone+\cite{program:rclone} was used to sync files from the
+Google Drive directory to Google Drive's server on GNU/Linux.
+
+\subsubsection{Issues found}
+
+\begin{itemize}
+\item When a file was modified on another computer, on this computer
+  combox assumed that first shard (shard0) will be updated first and
+  also counted on the existence of the first shard (shard0). It was
+  observed that the order in which the shards were updated were
+  unpredictable on this computer and if the first shard (shard0) was
+  stored in the Dropbox directory, it will momentarily disappear
+  before the most updated shard becomes available in the Dropbox
+  directory; this broke combox. This issue was fixed on
+  2015-08-25\cite{git:bug-four-fix}. This issue is not got to do with
+  the nature of the setup but it is related to the Dropbox's behavior
+  elaborated in section \ref{ch-5-2gnus-issues}.
+\item The official Google Drive client when it pulls an updated
+  version of the file from Google Drive' server, instead directly
+  updating the respective file on the computer, it deletes the older
+  version of the file and creates the latest version of the file at
+  the respective location in the Google Drive directory; this behavior
+  of the Google Drive confused and broke combox. This issue was fixed
+  2015-09-06 by making combox under the official Google Client's
+  behavior\cite{git:bug-googledc-fix}.
+\item When a non-empty directory was move/renamed on another computer,
+  the old directory was not getting properly deleted on this computer;
+  this was happening because the files under the directory being
+  renamed were not deleted when it was time for \verb+NodeDirMonitor+
+  to \verb+rmdir+ the old directory. This issue again is not specific
+  to the nature of the setup but was found while testing combox on
+  this setup. This issue was fixed on
+  2015-09-12\cite{git:bug-six-fix}.
+\item It was found that \verb+combox.file.rm_path+ function failed
+  when it was given a non-existent path to remove; this issue was
+  fixed on 2015-09-12\cite{git:bug-seven-fix}.
+\end{itemize}
+
+\subsubsection{Demo}
+
+Demo of combox being used on a GNU/Linux machine and OS X machine can
+be viewed at \url{https://ricketyspace.net/combox/combox-gnu-osx.webm}
+
+\verb+lyra+ is the GNU/Linux (virtual) machine and
+\verb+dhcp-129-1-66-1+ is the OS X machine that is being used for the
+demo. The OS X machine is accessed through VNC\cite{article:vnc}.
+
+Description of what happens in the demo follows:
+
+  - (\verb+lyra+) create file \verb+cat.stevens+ with content ``peace train''.
+
+  - (\verb+lyra+) sync Google Drive using \verb+rclone+.
+
+  - (\verb+dhcp-129-1-66-1+) verify that file \verb+cat.stevens+ is
+  created with content ``peace train''.
+
+  - (\verb+dhcp-129-1-66-1+) append string ``moonshadow'' to file
+  \verb+cat.stevens+.
+
+  - (\verb+lyra+) sync Google Drive using \verb+rclone+.
+
+  - (\verb+lyra+) verify that the file \verb+cat.stevens+ was updated
+  (modified); last line must have the string ``moonshadow''.
+
+  - (\verb+lyra+) append string ``father and son'' to the file
+  \verb+cat.stevens+.
+
+  - (\verb+lyra+) sync Google Drive using \verb+rclone+.
+
+  - (\verb+dhcp-129-1-66-1+) verify that the file \verb+cat.stevens+
+  was updated (modified); last line must have the string ``father and
+  son''.
+
+  - (\verb+dhcp-129-1-66-1+) rename file \verb+cat.stevens+ to
+  \verb+yusuf.islam+
+
+  - (\verb+lyra+) sync Google Drive using \verb+rclone+.
+
+  - (\verb+lyra+) verify that the file \verb+cat.stevens+ was renamed
+  to \verb+yusuf.islam+.
+
+\subsection{Testing with a USB stick as a node}
+
+combox was run on a GNU/Linux machine and an OS X machine and a file
+was alternatively created/modified/deleted on one of the machine and
+it was verified if the repsective file was also
+create/modified/deleted on the other machine. The GNU/Linux machine
+was a physical machine (\verb+grus+) running Debian GNU/Linux stable;
+The OS X machine was on Mavericks (10.9). The node directories to
+scatter files' shards were the Dropbox directory, Google Drive
+directory and the USB stick (\verb+ZAPHOD+, FAT filesystem). The
+official Dropbox client was used to automatically sync files from
+Dropbox directory to Dropbox' server on both the GNU/Linux machine and
+OS X machine; the official Google Drive client was used to
+automatically sync files from the Google Drive directory to Google
+Drive' server on OS X and \verb+rclone+\cite{program:rclone} was used
+to sync files from the Google Drive directory to Google Drive's server
+on GNU/Linux; the same USB stick (\verb+ZAPHOD+) was used on bothe
+GNU/Linux and Dropbox to store the third shard (shard2) of a file.
+
+\subsubsection{Caveats}
+
+\begin{itemize}
+\item When a removable USB disk is used as a node, combox must be
+  turned off before ejecting/unmounting the USB disk; combox does not
+  expect a node directory to disappear when it is running, if the USB
+  disk is removed when combox is running, then combox goes to a
+  undefined state.
+
+\item When a file modified on machine A is synced to machine B, combox
+  must be turned on first before turning on Dropbox and Google Drive
+  clients and the shard in the USB disk needs to be ``touched'' for
+  combox to detect that the file was modified on the remote computer
+  and update the file locally on this machine.
+
+\item File rename/move does not work. To make it work, core
+  functionality of combox must be re-written.
+\end{itemize}
+
+\subsubsection{Demo}
+
+Demo of combox being used with a USB stick as the third node can be
+view at \url{https://ricketyspace.net/combox/combox-usb-node-demo.webm}
+
+\verb+grus+ is the GNU/Linux machine and \verb+dhcp-129-1-66-1+ is the
+OS X machine that is being used for the demo. \verb+ZAPHOD+ is the
+FAT32 USB stick used as the third node.
+
+Description of what happens in the demo follows:
+
+  - (\verb+grus+) start combox.
+
+  - (\verb+grus+) create a file called \verb+simon.and.garfunkel+ with
+  content ``the boxer''.
+
+  - (\verb+grus+) sync Google Drive using \verb+rclone+.
+
+  - (\verb+grus+) stop combox.
+
+  - (\verb+grus+) unmount USB stick (\verb+ZAPHOD+) from \verb+grus+.
+
+  - (\verb+dhcp-129-1-66-1+) mount USB stick (\verb+ZAPHOD+) to
+  (\verb+dhcp-129-1-66-1+).
+
+  - (\verb+dhcp-129-1-66-1+) start Dropbox client.
+
+  - (\verb+dhcp-129-1-66-1+) start Google Drive client.
+
+  - (\verb+dhcp-129-1-66-1+) start combox.
+
+  - (\verb+dhcp-129-1-66-1+) verify that the file
+  \verb+simon.and.garfunkel+ with content ``the boxer'' was created.
+
+  - (\verb+dhcp-129-1-66-1+) append string ``mrs. robinson'' to file
+  \verb+simon.and.garfunkel+.
+
+  - (\verb+dhcp-129-1-66-1+) stop combox.
+
+  - (\verb+dhcp-129-1-66-1+) stop Google Drive client.
+
+  - (\verb+dhcp-129-1-66-1+) stop Dropbox client.
+
+  - (\verb+dhcp-129-1-66-1+) unmount the USB stick (\verb+ZAPHOD+)
+  from (\verb+dhcp-129-1-66-1+).
+
+  - (\verb+grus+) mount the USB stick (\verb+ZAPHOD+) to
+  (\verb+grus+).
+
+  - (\verb+grus+) start combox.
+
+  - (\verb+grus+) start Dropbox client.
+
+  - (\verb+grus+) sync Google Drive using \verb+rclone+.
+
+  - (\verb+grus+) touch \verb+simon.and.garfunkel.shard2+ in the USB
+  stick (\verb+ZAPHOD+).
+
+  - (\verb+grus+) verify that the file \verb+simon.and.garfunkel+ is
+  updated; the last line must contain the string ``mrs. robinson''.
+
+  - (\verb+grus+) remove the file \verb+simon.and.garfunkel+.
+
+  - (\verb+grus+) sync Google Drive using \verb+rclone+.
+
+  - (\verb+grus+) unmount the USB stick (\verb+ZAPHOD+) from
+  (\verb+grus+).
+
+  - (\verb+grus+) stop Dropbox client.
+
+  - (\verb+dhcp-129-1-66-1+) mount the USB stick (\verb+ZAPHOD+) to
+  (\verb+dhcp-129-1-66-1+).
+
+  - (\verb+dhcp-129-1-66-1+) start Google Drive client.
+
+  - (\verb+dhcp-129-1-66-1+) start Dropbox client.
+
+  - (\verb+dhcp-129-1-66-1+) start combox.
+
+  - (\verb+dhcp-129-1-66-1+) verify that the file
+  \verb+simon.and.garfunkel+ was deleted.
+
+
+\section{Stress testing}
+
+Large number of files of different sizes were dumped to the combox
+directory between an one second interval to see how combox responds to
+high load. The file dump size was varied from \verb+424.798190MiB+ (27
+files) to \verb+10800.000000MiB+ (180 files); the average time taken
+to split a file and the total time to process all files were
+calculated for each dump.
+
+Stress testing was first done on \verb+2015-11-08+. In mid November
+the \verb+ComboxDirMonitor+ was drastically modified to make it use
+the file Lock shared the instances of
+\verb+NodeDirMonitor+\cite{git:bug-eleven-fix}; my hunch was that this
+change in \verb+ComboxDirMonitor+ directly affected the performance of
+combox and therefore the results that were got from stress testing on
+\verb+2015-11-08+ would no longer be valid. Stress testing was again
+done on \verb+2016-01-16+; the results of this stress test are in
+sections \ref{5-st-424} to \ref{5-st-10800}, section \ref{5-st-tu}
+gives information about the tools used for stress testing, section
+\ref{5-st-o} contains the observations and comparisons between this
+stress test and the one done on \verb+2015-11-08+, lastly section
+\ref{5-st-if} reveals the issues that were found with combox by virtue
+of doing the stress tests.
+
+\subsection{flac dump (27 files - 424.798190MiB)}\label{5-st-424}
+
+\begin{center}
+\begin{tabular}{ll}
+field & value\\
+\hline
+delay between a file dump & 1s\\
+start time of processing & 11:00:54\\
+end time of processing & 11:01:38\\
+total time taken to process all files & 00:00:44\\
+no. of files & 27\\
+total size of all files & 445433187.000000 bytes (424.798190MiB)\\
+avg. file size & 16497525.000000 bytes (15.733266MiB)\\
+avg. time to split and encrypt a file & 352.583370 ms\\
+\end{tabular}
+\end{center}
+
+\subsubsection{Differences from previous stress test (2015-11-08)}
+
+\begin{itemize}
+\item Total time to process all files was faster by 1min3secs.
+\item Average time to split and encrypt a file reduced by
+  28.337963000000002ms.
+\end{itemize}
+
+\subsection{20MiB - 90MiB dump (27 files - 1620.000000MiB)}\label{5-st-1620}
+
+\begin{center}
+\begin{tabular}{ll}
+field & value\\
+\hline
+delay between a file dump & 1s\\
+start time of processing & 12:26:45\\
+end time of processing & 12:29:07\\
+total time taken to process all files & 00:02:22\\
+no. of files & 27\\
+total size of all files & 1698693120.000000 bytes (1620.000000MiB)\\
+avg. file size & 62914560.000000 bytes (60.000000MiB)\\
+avg. time to split and encrypt a file & 2670.596556ms\\
+\end{tabular}
+\end{center}
+
+\subsubsection{Differences from previous stress test (2015-11-08)}
+
+\begin{itemize}
+\item Total time to process all files was slower by 4secs.
+\item Average time to split and encrypt a file reduced by
+   25.52536999999984ms.
+\end{itemize}
+
+\subsection{20MiB - 90MiB dump (99 files - 5940.000000MiB)}\label{5-st-5940}
+
+\begin{center}
+\begin{tabular}{ll}
+field & value\\
+\hline
+delay between a file dump & 1s\\
+start time of processing & 13:10:16\\
+end time of processing & 13:19:26\\
+total time taken to process all files & 00:09:10\\
+no. of files & 99\\
+total size of all files & 6228541440.000000 bytes (5940.000000MiB)\\
+avg. file size & 62914560.000000 bytes (60.000000MiB)\\
+avg. time to split and encrypt a file & 2979.647586ms\\
+\end{tabular}
+\end{center}
+
+\subsubsection{Differences from previous stress test (2015-11-08)}
+
+\begin{itemize}
+\item Total time to process all files was faster by 59secs.
+\item Average time to split and encrypt a file increased by
+  206.20906100000002ms.
+\end{itemize}
+
+\subsection{20MiB - 90MiB dump (180 files - 10800.000000MiB)}\label{5-st-10800}
+
+\begin{center}
+\begin{tabular}{ll}
+field & value\\
+\hline
+delay between a file dump & 1s\\
+start time of processing & 13:42:06\\
+end time of processing & 14:00:10\\
+total time taken to process all files & 00:18:04\\
+no. of files & 180\\
+total size of all files & 11324620800.000000 bytes (10800.000000MiB)\\
+avg. file size & 62914560.000000 bytes (60.000000MiB)\\
+avg. time to split and encrypt a file & 3423.087539ms\\
+\end{tabular}
+\end{center}
+
+\subsubsection{Differences from previous stress test (2015-11-08)}
+
+\begin{itemize}
+\item Total time to process all files was slower by 1min2secs
+\item Average time to split and encrypt a file increased by
+  399.87623299999996ms.
+\end{itemize}
+
+\subsection{Tools used}\label{5-st-tu}
+
+The \verb+dump+ script\cite{program:dump} was used to dump files to
+the combox directory between one second intervals; a night of Emacs
+Lisp indulgence made it possible to quickly slurp the required data
+from the combox output and calculate the average time to split and
+encrypt a file and the total amount of time taken to process the files
+for a given dump\cite{program:dumps.el}; lastly \verb+org-mode+ was
+used to document all data gathered during stress
+testing\cite{doc:benchmarks.org}.
+
+\subsection{Observations}\label{5-st-o}
+
+\begin{figure}[h]
+\centering
+\input{graphs/tot-time.tex}
+\caption{time to process all files}
+\label{fig:5-st-tt}
+\end{figure}
+
+\begin{figure}[h]
+\centering
+\input{graphs/avg-time-sae.tex}
+\caption{avg. time to split and encrypt}
+\label{fig:5-st-atsae}
+\end{figure}
+
+
+\begin{itemize}
+\item Figure \ref{fig:5-st-tt} shows the time it takes combox to
+  process files for a given file dump\footnote{A ``file dump'' here
+    means a bunch of files copied to the combox directory between 1
+    sec intervals.}. As can be observed from the graph, the total time
+  taken to process all the files tends almost linearly increase with
+  the increase in the size of the file dump\footnote{The ``size of the
+    file dump'' is the total size of all files in a given file dump.}.
+\item Figure \ref{fig:5-st-atsae} show the average time it takes
+  combox to split and encrypt a file for a given file dump. There is a
+  steep increase in the average time from the \verb+424.798190MiB+
+  dump and the \verb+1620.000000MiB+ dump, after which the average
+  time to split and encrypt a file seems to almost linearly increase;
+  The main reason for this is that the average file size for dumps
+  from \verb+1620.000000MiB+ to \verb+10800.000000MiB+ are the same.
+\end{itemize}
+
+\begin{figure}[h]
+\centering
+\input{graphs/tot-time-diff.tex}
+\caption{time to process all files - difference between 2015 and 2016}
+\label{fig:5-st-tt-diff}
+\end{figure}
+
+\begin{figure}[h]
+\centering
+\input{graphs/avg-time-sae-diff.tex}
+\caption{avg. time to split and encrypt - difference between 2015 and 2016}
+\label{fig:5-st-atsae-diff}
+\end{figure}
+
+\begin{itemize}
+\item Figure \ref{fig:5-st-tt-diff} shows the graphs for the total
+  amount of time taken to process all files for a given file dump in
+  the \verb+2016-01-16+ and \verb+2015-11-8+ stress test. The amount
+  of time needed to process all fills seems to be reduced for the
+  \verb+5940.000000MiB+ file dump when compared to the \verb+2015+
+  stress test results and it seems to be slightly higher for the
+  \verb+10800.000000MiB+ file dump when compared to the \verb+2015+
+  stress test.
+\item Similarly, figure \ref{fig:5-st-atsae-diff} shows the graphs for
+  the average time to split and encrypt for a given file dump in the
+  \verb+2016-01-16+ and the \verb+2015-11-8+ stress test. The average
+  time taken seems to able almost the same for the
+  \verb+424.798190MiB+ and the \verb+1620.000000+ dump, but for the
+  \verb+5940.000000MiB+ and the \verb+10800.000000MiB+ dump the
+  average time taken seems to higher for the \verb+2016+ stress test
+  when compared to the \verb+2015+ stress test.
+\end{itemize}
+
+\subsection{Issues found}\label{5-st-if}
+
+\begin{itemize}
+\item Initially when combox was stress tested with huge files, combox
+  would get overwhelmed leading to the computer running out of memory
+  and the load average sometimes peaking at \verb+8+. At first, it was
+  assumed that there was a bug in combox which caused this to happen,
+  but later it was found that \verb+watchdog+\cite{pylib:watchdog} was
+  generating a large number ``file modified'' events when a huge file
+  (\verb+~500MiB+ was modified). To prevent \verb+watchdog+ from
+  generating a large number ``file modified'' events for a single
+  modification of a huge file, a delay proportional to the size of the
+  file was created in the \verb+on_modified+ callback methods in both
+  \verb+ComboxDirMonitor+ and
+  \verb+NodeDirMonitor+\cite{git:bug-ten-fix}, this fixed the
+  issue. Also, this it might be useful to note here that this was
+  ``the'' hardest issue I dealt with in working on combox.
+\end{itemize}
\ No newline at end of file