\chapter{Testing}\label{ch:4}

\epigraph{Testing shows the presence, not the absence of
  bugs.}{\textit{Dijkstra}\cite{dijkstra69}}

\section{Unit testing}\label{sec:4-unit-testing}

The \verb+nose+\cite{pylib:nose} testing framework was used to write
unit tests for the functions and classes part of the
\verb+combox.config+, \verb+combox.crypto+, \verb+combox.events+,
\verb+combox.file+, \verb+combox.silo+ and \verb+combox._version+
modules. Unit tests were not written for \verb+combox.cbox+,
\verb+combox.gui+ and \verb+combox.combox.log+ modules either because
it did not sense to write one -- for instance, the \verb+combox.cbox+
module, which uses functions and classes defined in other modules
which are unit tested -- or it was not clear how to write unit tests
for it (the \verb+combox.gui+ module).

It must be noted here that pure Test Driven Development (TDD) was not
observed -- most of the time the function/class was written before the
its corresponding test was written.

\subsection{Benefits}

While writing unit tests definitely increased the time to write a
particular feature, it made it possible to immediately check if a
feature worked as it should for a given set of use cases or given set
of inputs.

Unit tests greatly helped in testing the compatibility of combox on OS
X. Before the \verb+v0.1.0+ release, combox's node directory monitor
always assumed that a file's first shard (\verb+shard0+) is always
available; while this assumption did not create any problems on
GNU/Linux, on OS X, this assumption made the node directory monitor to
behave erratically -- this issue (bug \#4 was immediately found when
the unit tests were run for the first time on OS X. Another instance
where unit tests helped was just before the \verb+v0.2.0+ release;
major changes, including the introduction of file locks in the
\verb+ComboxDirMonitor+, were made to the \verb+combox.events+. When
the unit tests were run OS X, two tests failed, revealing a difference
in behavior of watchdog\cite{pylib:watchdog} on GNU/Linux and OS X on
file
creation\footnote{https://git.ricketyspace.net/combox/commit/?id=8c86e7c28738c66c0e04ae7886b44dbcdfc6369exo};
without unit tests, there is a high probability that this bug would
never have been found by now.

\subsection{Caveats}

Unit tests are helpful in testing the correctness of a feature for
\verb+N+ number of use cases but it does not necessarily mean the
written feature correctly behaves for use cases that the author of the
feature did not consider or did not think about while writing the
respective feature.

Unit tests failed to reveal bugs \#4, \#5 \#6 \#7 \#5 \#10
\#11\footnote{https://git.ricketyspace.net/combox/plain/TODO.org}; these bugs were found when manually
testing combox.

\section{Manual testing}\label{sec:4-manual-testing}

The unit tests for the \verb+combox.events+ module tested the
correctness of the \verb+ComboxDirMonitor+ and \verb+NodeDirMonitor+
independently; in order to comprehensively test the correctness of
both \verb+ComboxDirMonitor+ and \verb+NodeDirMonitor+, it was
required to manually test combox running on more than one
computer. Several bugs were found and fixed while doing manual
testing.

Three different types of setups were used to manually test combox. The
first kind of setup has two GNU/Linux machines each using combox to
sync files between each other with Dropbox and Google Drive being the
nodes; the second kind of setup has a GNU/Linux machine and a OS X
machine each using combox to sync files between each other with
Dropbox and Google Drive being the nodes; the third kind of setup has
a GNU/Linux machine and OS X machine each using combox to sync files
between each other with Dropbox, Google Drive and a USB stick as
nodes.

\subsection{General setup and notes}

\begin{itemize}
\item On the GNU/Linux machines, the official Dropbox client was used
  to sync the Dropbox node directory to Dropbox' data
  store. \verb+rclone+\cite{program:rclone} was used to sync the
  Google Drive node directory to Google Drive' data store; at the time
  of testing, Google Drive does not have a client program for
  GNU/Linux which can sync to Google Drive's data store.
\item On OS X, the official Dropbox client was used to sync the
  Dropbox node directory to Dropbox's data store; the official Google
  Drive client was used to sync the Google Drive node directory to
  Google Driver' data store.
\item Since combox is extremely event-driven, combox must be started
  before the Dropbox and Google Drive clients start syncing their
  respective directories.
\end{itemize}

\subsection{Testing on two GNU/Linux machines}

combox was run on two GNU/Linux machines and a file was alternatively
created/modified/renamed/deleted on one of the GNU/Linux machine and
it was verified if the respective file was also
created/modified/renamed/deleted on the other GNU/Linux machine. One
of the GNU/Linux machine (\verb+lyra)+ was a virtual machine running
Debian GNU/Linux stable (version 8.x); the other GNU/Linux machine
(\verb+grus+) was a physical machine running Debian GNU/Linux
testing. The node directories to scatter the files' shards were the
Dropbox directory and Google Drive directory. The official Dropbox
client was used to automatically sync files from the Dropbox directory
to the Dropbox' data store; \verb+rclone+\cite{program:rclone} was
used to sync files from Google Drive directory to Google Drive'
data store.

\subsubsection{Issues found}\label{ch-4-2gnus-issues}

\begin{itemize}
\item Some editors, especially on POSIX complaint systems, create a
  backup version of the file being edited. combox was detecting this
  backup file as a ``new file'' and it split it into shards, encrypted
  the shards and scattered the shards across the node directories. The
  right thing for combox to do was to ignore these backup files and do
  nothing about them. This issue was fixed on
  \verb+2015-09-29+\footnote{https://git.ricketyspace.net/combox/plain/TODO.org}. Now
  the \verb+ComboxDirMonitor+, on a ``file created'' or ``file
  modified'' event, returns from the \verb+on_created+ or
  \verb+on_modified+ callback when it finds that the file is a
  backup/temporary file.
\item Dropbox client maintains the \verb+.dropbox.cache+ directory
  under the root of the Dropbox directory.

  \begin{itemize}
  \item When a file (shard) was created on another computer, the
    Dropbox client pulls the new file (shard) to this computer into
    \verb+.dropbox.cache+ as a temporary file and then moves the new
    file (shard) to its respective location with the appropriate name.
  \item When a file (shard) was modified on another computer, the
    Dropbox client pulls the modified file (shard) to this computer
    into the \verb+.dropbox.cache+ as a temporary file; moves the old
    version of the file (shard) under the Dropbox directory into the
    \verb+.dropbox.cache+; finally moves the updated copy of the file,
    stored as a temporary file, into the Dropbox directory to its
    respective location with the appropriate name.
  \item When a file (shard) was deleted on another computer, the
    Dropbox client moves the deleted file into the
    \verb+.dropbox.cache+ directory on this computer.
  \end{itemize}

  All of the above behavior of the Dropbox client royally broke
  combox. Commits between \verb+3d714c5+ to
  \verb+6e1133f+\footnote{https://git.ricketyspace.net/combox/log/?qt=range\&q=3d714c5..6e1133f}
  fixed combox by making it aware of Dropbox's client behavior.
\end{itemize}

\subsubsection{Demo}

Demo of combox being used on two GNU/Linux machines can be viewed at
\url{https://ricketyspace.net/combox/combox-2-gnus.webm}.

\verb+lyra+ (virtual machine) and \verb+grus+ (bare-metal) are the two
GNU/Linux machines being used for the demo.

Description of what happens in the demo follows:

 - (lyra) install combox.

 - (lyra) run combox (test mode).

 - (lyra) create file \verb+walden.pond+ with content ``It must be
 beautiful there''.

 - (lyra) sync Google Drive using \verb+rclone+.

 - (grus) sync Google Drive using \verb+rclone+.

 - (grus) git pull latest copy of combox.

 - (grus) install combox 

 - (grus) run combox (testing mode).

 - (grus) verify that \verb+walden.pond+ was create on this machine.

 - (grus) append 'Peaceful too.' to \verb+walden.pond+.

 - (grus) sync Google Drive using \verb+rclone+.

 - (lyra) sync Google Drive using \verb+rclone+.

 - (lyra) verify that the latest copy of \verb+walden.pond+ is there
 in the combox directory; it should contain 'Peaceful too.' in the
 last line.

 - (lyra) append ``I've a dream'' to \verb+walden.pond+.

 - (lyra) sync Google Drive using \verb+rclone+.

 - (grus) sync Google Drive using \verb+rclone+.

 - (grus) verify that the latest copy of \verb+walden.pond+ is there
 in the combox directory; it should contain ``I've a dream'' in the
 last line.

 - (grus) remove \verb+walden.pond+ from combox directory.

 - (grus) sync Google Drive using \verb+rclone+.

 - (lyra) sync Google Drive using \verb+rclone+.

 - (lyra) verify that \verb+walden.pond+ is removed from the combox
 directory.

 - (grus) open Dropbox and Google drive accounts from the web browser.

 - (lyra) create file \verb+manufacturing.consent.+ with content ``Chomsky stuff?''.

 - (lyra) sync Google Drive using \verb+rclone+.

 - (grus) sync Google Drive using \verb+rclone+.

 - (grus) verify that \verb+manufacturing.consent+ was created in the
 combox directory.

 - (grus) verify that the shards of \verb+manufacturing.consent+ were
 created on Dropbox and Google Drive through the web browser.

\subsection{Testing on a GNU/Linux and an OS X machine}

combox was run on a GNU/Linux machine and an OS X machine and a file
was alternatively created/modified/renamed/deleted on one of the
machine and it was verified if the respective file was also
created/modified/renamed/deleted on the other machine. The GNU/Linux
machine was a virtual machine (\verb+lyra+) running Debian GNU/Linux
stable; the OS X machine was on Mavericks (10.9) during the initial
stage of testing, later it was upgraded to Yosemite (10.10). The node
directories to scatter files' shards were the Dropbox directory and
the Google Drive directory. The official Dropbox client was used to
automatically sync files from the Dropbox directory to the Dropbox'
data store on both the GNU/Linux machine and the OS X machine; the
official Google Drive client was used to automatically sync files from
the Google Drive directory to Google Drive' data store on OS X and
\verb+rclone+\cite{program:rclone} was used to sync files from the
Google Drive directory to Google Drive's data store on GNU/Linux.

\subsubsection{Issues found}

\begin{itemize}
\item When a file was modified on another computer, on this computer
  combox assumed that first shard (shard0) will be updated first and
  also counted on the existence of the first shard (shard0). It was
  observed that the order in which the shards were updated were
  unpredictable on this computer and if the first shard (shard0) was
  stored in the Dropbox directory, it will momentarily disappear
  before the most updated shard becomes available in the Dropbox
  directory; this broke combox. This issue was fixed on
  2015-08-25\footnote{https://git.ricketyspace.net/combox/commit/?id=d5b52030348d40600b4c9256f76e5183a85fbb17}. This issue is not got to do with
  the nature of the setup but it is related to the Dropbox's behavior
  elaborated in section \ref{ch-4-2gnus-issues}.
\item When the official Google Drive client pulls an updated version
  of the file from Google Drive' data store, instead directly updating
  the respective file on the computer, it deletes the older version of
  the file and creates the latest version of the file at the
  respective location in the Google Drive directory; this behavior of
  the Google Drive client confused and broke combox. This issue was
  fixed 2015-09-06 by making combox aware of the official Google
  Client's
  behavior\footnote{https://git.ricketyspace.net/combox/commit/?id=37385a90f90cb9d4dfd13d9d2e3cbcace8011e9e}.
\item When a non-empty directory was move/renamed on another computer,
  the old directory was not getting properly deleted on this computer;
  this was happening because, sometimes, the files under the
  directory being renamed were not deleted when it was time for
  \verb+NodeDirMonitor+ to \verb+rmdir+ the old directory. This issue
  was fixed on
  2015-09-12\footnote{https://git.ricketyspace.net/combox/commit/?id=9d14db03da5d10d5ab0d7cc76b20e7b1ed5523bf}.
\item It was found that \verb+combox.file.rm_path+ function failed
  when it was given a non-existent path to remove; this issue was
  fixed on 2015-09-12\footnote{https://git.ricketyspace.net/combox/commit/?id=422238eb4904de14842221fa09a2b4028801afb1}.
\end{itemize}

\subsubsection{Demo}

Demo of combox being used on a GNU/Linux machine and OS X machine can
be viewed at \url{https://ricketyspace.net/combox/combox-gnu-osx.webm}

\verb+lyra+ is the GNU/Linux (virtual) machine and
\verb+dhcp-129-1-66-1+ is the OS X machine that is being used for the
demo. The OS X machine is accessed through VNC\cite{article:vnc}.

Description of what happens in the demo follows:

  - (\verb+lyra+) create file \verb+cat.stevens+ with content ``peace train''.

  - (\verb+lyra+) sync Google Drive using \verb+rclone+.

  - (\verb+dhcp-129-1-66-1+) verify that file \verb+cat.stevens+ is
  created with content ``peace train''.

  - (\verb+dhcp-129-1-66-1+) append string ``moonshadow'' to file
  \verb+cat.stevens+.

  - (\verb+lyra+) sync Google Drive using \verb+rclone+.

  - (\verb+lyra+) verify that the file \verb+cat.stevens+ was updated
  (modified); last line must have the string ``moonshadow''.

  - (\verb+lyra+) append string ``father and son'' to the file
  \verb+cat.stevens+.

  - (\verb+lyra+) sync Google Drive using \verb+rclone+.

  - (\verb+dhcp-129-1-66-1+) verify that the file \verb+cat.stevens+
  was updated (modified); last line must have the string ``father and
  son''.

  - (\verb+dhcp-129-1-66-1+) rename file \verb+cat.stevens+ to
  \verb+yusuf.islam+

  - (\verb+lyra+) sync Google Drive using \verb+rclone+.

  - (\verb+lyra+) verify that the file \verb+cat.stevens+ was renamed
  to \verb+yusuf.islam+.

\subsection{Testing with a USB stick as a node}

combox was run on a GNU/Linux machine and an OS X machine and a file
was alternatively created/modified/deleted on one of the machine and
it was verified if the respective file was also
create/modified/deleted on the other machine. The GNU/Linux machine
was a physical machine (\verb+grus+) running Debian GNU/Linux testing;
The OS X machine was on Mavericks (10.9). The node directories to
scatter files' shards were the Dropbox directory, Google Drive
directory and the USB stick (\verb+ZAPHOD+, FAT filesystem). The
official Dropbox client was used to automatically sync files from
Dropbox directory to Dropbox' data store on both the GNU/Linux machine
and the OS X machine; the official Google Drive client was used to
automatically sync files from the Google Drive directory to Google
Drive' data store on OS X and \verb+rclone+\cite{program:rclone} was
used to sync files from the Google Drive directory to Google Drive's
data store on GNU/Linux; the same USB stick (\verb+ZAPHOD+) was used
on both GNU/Linux and Dropbox to store the third shard (shard2) of the
files stored in combox directory.

\subsubsection{Caveats}

\begin{itemize}
\item When a removable USB disk is used as a node, combox must be
  turned off before ejecting/unmounting the USB disk; combox does not
  expect a node directory to disappear when it is running, if the USB
  disk is removed when combox is running, then combox goes to an
  undefined state.

\item When a file modified on machine A is synced to machine B, combox
  must be turned on first before turning on Dropbox and Google Drive
  clients and the shard in the USB disk needs to be ``touched'' for
  combox to detect that the file was modified on the remote computer
  and update the file locally on this machine.

\item File rename/move does not work. To make it work, core
  functionality of combox must be re-written.
\end{itemize}

\subsubsection{Demo}

Demo of combox being used with a USB stick as the third node can be
viewed at \url{https://ricketyspace.net/combox/combox-usb-node-demo.webm}

\verb+grus+ is the GNU/Linux machine and \verb+dhcp-129-1-66-1+ is the
OS X machine that is being used for the demo. \verb+ZAPHOD+ is the
FAT32 USB stick used as the third node.

Description of what happens in the demo follows:

  - (\verb+grus+) start combox.

  - (\verb+grus+) create a file called \verb+simon.and.garfunkel+ with
  content ``the boxer''.

  - (\verb+grus+) sync Google Drive using \verb+rclone+.

  - (\verb+grus+) stop combox.

  - (\verb+grus+) unmount USB stick (\verb+ZAPHOD+) from \verb+grus+.

  - (\verb+dhcp-129-1-66-1+) mount USB stick (\verb+ZAPHOD+) to
  (\verb+dhcp-129-1-66-1+).

  - (\verb+dhcp-129-1-66-1+) start Dropbox client.

  - (\verb+dhcp-129-1-66-1+) start Google Drive client.

  - (\verb+dhcp-129-1-66-1+) start combox.

  - (\verb+dhcp-129-1-66-1+) verify that the file
  \verb+simon.and.garfunkel+ with content ``the boxer'' was created.

  - (\verb+dhcp-129-1-66-1+) append string ``mrs. robinson'' to file
  \verb+simon.and.garfunkel+.

  - (\verb+dhcp-129-1-66-1+) stop combox.

  - (\verb+dhcp-129-1-66-1+) stop Google Drive client.

  - (\verb+dhcp-129-1-66-1+) stop Dropbox client.

  - (\verb+dhcp-129-1-66-1+) unmount the USB stick (\verb+ZAPHOD+)
  from (\verb+dhcp-129-1-66-1+).

  - (\verb+grus+) mount the USB stick (\verb+ZAPHOD+) to
  (\verb+grus+).

  - (\verb+grus+) start combox.

  - (\verb+grus+) start Dropbox client.

  - (\verb+grus+) sync Google Drive using \verb+rclone+.

  - (\verb+grus+) touch \verb+simon.and.garfunkel.shard2+ in the USB
  stick (\verb+ZAPHOD+).

  - (\verb+grus+) verify that the file \verb+simon.and.garfunkel+ is
  updated; the last line must contain the string ``mrs. robinson''.

  - (\verb+grus+) remove the file \verb+simon.and.garfunkel+.

  - (\verb+grus+) sync Google Drive using \verb+rclone+.

  - (\verb+grus+) unmount the USB stick (\verb+ZAPHOD+) from
  (\verb+grus+).

  - (\verb+grus+) stop Dropbox client.

  - (\verb+dhcp-129-1-66-1+) mount the USB stick (\verb+ZAPHOD+) to
  (\verb+dhcp-129-1-66-1+).

  - (\verb+dhcp-129-1-66-1+) start Google Drive client.

  - (\verb+dhcp-129-1-66-1+) start Dropbox client.

  - (\verb+dhcp-129-1-66-1+) start combox.

  - (\verb+dhcp-129-1-66-1+) verify that the file
  \verb+simon.and.garfunkel+ was deleted.


\section{Stress testing}

Large number of files of different sizes were dumped to the combox
directory between an one second interval to see how combox responds to
high load. The file dump size was varied from \verb+424.798190MiB+ (27
files) to \verb+10800.000000MiB+ (180 files); the average time taken
to split a file and the total time to process all files were
calculated for each dump.

Stress testing was first done on \verb+2015-11-08+. In mid November
2015, the \verb+ComboxDirMonitor+ was drastically modified to make it
use the file Lock shared by the instances of
\verb+NodeDirMonitor+\footnote{https://git.ricketyspace.net/combox/commit/?id=5aa1ba0c1dcad62931ba27bb66bf115233086d6c};
the hunch was that this change in \verb+ComboxDirMonitor+ directly
affected the performance of combox and therefore the results that were
got from stress testing on \verb+2015-11-08+ would no longer be
valid. Stress testing was again done on \verb+2016-01-16+; the results
of this stress test are in sections \ref{4-st-424} to
\ref{4-st-10800}, section \ref{4-st-tu} gives information about the
tools used for stress testing, section \ref{4-st-o} contains the
observations and comparisons between this stress test and the one done
on \verb+2015-11-08+, lastly section \ref{4-st-if} reveals the issues
that were found with combox by virtue of doing the stress tests.

\subsection{flac dump (27 files - 424.798190MiB)}\label{4-st-424}

\begin{center}
\begin{table}[h]
\begin{tabular}{ll}
field & value\\
\hline
delay between a file dump & 1s\\
start time of processing & 11:00:54\\
end time of processing & 11:01:38\\
total time taken to process all files & 00:00:44\\
no. of files & 27\\
total size of all files & 445433187.000000 bytes (424.798190MiB)\\
avg. file size & 16497525.000000 bytes (15.733266MiB)\\
avg. time to split and encrypt a file & 352.583370 ms\\
\end{tabular}
\caption{Stress Testing combox - flac dump (27 files - 424.798190MiB) to combox directory}
\end{table}
\end{center}

\subsubsection{Differences from previous stress test (2015-11-08)}

\begin{itemize}
\item Total time to process all files was faster by 1min3secs.
\item Average time to split and encrypt a file reduced by
  28.337963000000002ms.
\end{itemize}

\subsection{20MiB - 90MiB dump (27 files - 1620.000000MiB)}\label{4-st-1620}

\begin{center}
\begin{table}[h]
\begin{tabular}{ll}
field & value\\
\hline
delay between a file dump & 1s\\
start time of processing & 12:26:45\\
end time of processing & 12:29:07\\
total time taken to process all files & 00:02:22\\
no. of files & 27\\
total size of all files & 1698693120.000000 bytes (1620.000000MiB)\\
avg. file size & 62914560.000000 bytes (60.000000MiB)\\
avg. time to split and encrypt a file & 2670.596556ms\\
\end{tabular}
\caption{Stress Testing combox - 20MiB - 90MiB dump (27 files - 1620.000000MiB) to combox directory}
\end{table}
\end{center}

\subsubsection{Differences from previous stress test (2015-11-08)}

\begin{itemize}
\item Total time to process all files was slower by 4secs.
\item Average time to split and encrypt a file reduced by
   25.52536999999984ms.
\end{itemize}

\subsection{20MiB - 90MiB dump (99 files - 5940.000000MiB)}\label{4-st-5940}

\begin{center}
\begin{table}[h]
\begin{tabular}{ll}
field & value\\
\hline
delay between a file dump & 1s\\
start time of processing & 13:10:16\\
end time of processing & 13:19:26\\
total time taken to process all files & 00:09:10\\
no. of files & 99\\
total size of all files & 6228541440.000000 bytes (5940.000000MiB)\\
avg. file size & 62914560.000000 bytes (60.000000MiB)\\
avg. time to split and encrypt a file & 2979.647586ms\\
\end{tabular}
\caption{Stress Testing combox - 20MiB - 90MiB dump (99 files - 5940.000000MiB) - to combox directory}
\end{table}
\end{center}

\subsubsection{Differences from previous stress test (2015-11-08)}

\begin{itemize}
\item Total time to process all files was faster by 59secs.
\item Average time to split and encrypt a file increased by
  206.20906100000002ms.
\end{itemize}

\subsection{20MiB - 90MiB dump (180 files - 10800.000000MiB)}\label{4-st-10800}

\begin{center}
\begin{table}[h]
\begin{tabular}{ll}
field & value\\
\hline
delay between a file dump & 1s\\
start time of processing & 13:42:06\\
end time of processing & 14:00:10\\
total time taken to process all files & 00:18:04\\
no. of files & 180\\
total size of all files & 11324620800.000000 bytes (10800.000000MiB)\\
avg. file size & 62914560.000000 bytes (60.000000MiB)\\
avg. time to split and encrypt a file & 3423.087539ms\\
\end{tabular}
\caption{Stress Testing combox - 20MiB - 90MiB dump (180 files - 10800.000000MiB) to combox directory}
\end{table}
\end{center}

\subsubsection{Differences from previous stress test (2015-11-08)}

\begin{itemize}
\item Total time to process all files was slower by 1min2secs
\item Average time to split and encrypt a file increased by
  399.87623299999996ms.
\end{itemize}

\subsection{Tools used}\label{4-st-tu}

The \verb+dump+ script\footnote{https://git.ricketyspace.net/combox-paper/plain/dumper/dump} was used to dump files to
the combox directory between one second intervals; a night of Emacs
Lisp indulgence made it possible to quickly slurp the required data
from the combox output and calculate the average time to split and
encrypt a file and the total amount of time taken to process the files
for a given dump\footnote{https://git.ricketyspace.net/combox-paper/plain/scripts/dumps.el}; lastly \verb+org-mode+ was
used to document all data gathered during stress
testing\footnote{https://git.ricketyspace.net/combox-paper/plain/notes/benchmarks.org}.

\subsection{Observations}\label{4-st-o}

\begin{figure}[h]
\centering
\input{graphs/tot-time.tex}
\caption{Stress testing combox - Observations - Time taken to process
  all files in a given file dump.}
\label{fig:4-st-tt}
\end{figure}

\begin{figure}[h]
\centering
\input{graphs/avg-time-sae.tex}
\caption{Stress testing combox - Observations - Avg. time to split and
  encrypt a file in a given file dump.}
\label{fig:4-st-atsae}
\end{figure}


\begin{itemize}
\item Figure \ref{fig:4-st-tt} shows the time it takes combox to
  process files for a given file dump\footnote{A ``file dump'' here
    means a bunch of files copied to the combox directory between 1
    sec intervals.}. As can be observed from the graph, the total time
  taken to process all the files tends almost linearly increase with
  the increase in the size of the file dump\footnote{The ``size of the
    file dump'' is the total size of all files in a given file dump.}.
\item Figure \ref{fig:4-st-atsae} show the average time it takes
  combox to split and encrypt a file for a given file dump. There is a
  steep increase in the average time from the \verb+424.798190MiB+
  dump and the \verb+1620.000000MiB+ dump, after which the average
  time to split and encrypt a file seems to almost linearly increase;
  The main reason for this is that the average file size for dumps
  from \verb+1620.000000MiB+ to \verb+10800.000000MiB+ are the same.
\end{itemize}

\begin{figure}[h]
\centering
\input{graphs/tot-time-diff.tex}
\caption{Stress testing combox - Difference between 2015 and 2016 tests - time taken to process all files in a given file dump.}
\label{fig:4-st-tt-diff}
\end{figure}

\begin{figure}[h]
\centering
\input{graphs/avg-time-sae-diff.tex}
\caption{Stress testing combox - Difference between 2015 and 2016 tests - Avg. time to split and encrypt a file in a given file dump.}
\label{fig:4-st-atsae-diff}
\end{figure}

\begin{itemize}
\item Figure \ref{fig:4-st-tt-diff} shows the graphs for the total
  amount of time taken to process all files for a given file dump in
  the \verb+2016-01-16+ and \verb+2015-11-8+ stress test. The amount
  of time needed to process all fills seems to be reduced for the
  \verb+5940.000000MiB+ file dump when compared to the \verb+2015+
  stress test results and it seems to be slightly higher for the
  \verb+10800.000000MiB+ file dump when compared to the \verb+2015+
  stress test.
\item Similarly, figure \ref{fig:4-st-atsae-diff} shows the graphs for
  the average time to split and encrypt for a given file dump in the
  \verb+2016-01-16+ and the \verb+2015-11-8+ stress test. The average
  time taken seems to be almost the same for the \verb+424.798190MiB+
  and the \verb+1620.000000+ dump, but for the \verb+5940.000000MiB+
  and the \verb+10800.000000MiB+ dump the average time taken seems to
  higher for the \verb+2016+ stress test when compared to the
  \verb+2015+ stress test.
\end{itemize}

\subsection{Issues found}\label{4-st-if}

\begin{itemize}
\item Initially when combox was stress tested with huge files, combox
  would get overwhelmed leading to the computer running out of memory
  and the load average sometimes peaking at \verb+8+. At first, it was
  assumed that there was a bug in combox which caused this to happen,
  but later it was found that \verb+watchdog+\cite{pylib:watchdog} was
  generating a large number ``file modified'' events when a huge file
  (\verb+~500MiB+) was modified. To prevent \verb+watchdog+ from
  generating a large number ``file modified'' events for a single
  modification of a huge file, a delay proportional to the size of the
  file was created in the \verb+on_modified+ callback methods in both
  \verb+ComboxDirMonitor+ and
  \verb+NodeDirMonitor+\footnote{https://git.ricketyspace.net/combox/commit?id=7ed3c9cbe6e56223b043a23408474f9df08f119e},
  this fixed the issue.
\end{itemize}