path: root/report/chapters/5-testing.tex

\chapter{Testing}

\epigraph{Testing shows the presence, not the absence of
  bugs.}{\textit{Dijkstra}\cite{dijkstra69}}

\section{Unit testing}

The \verb+nose+\cite{pylib:nose} testing framework was used to
write unit tests for the functions and classes part of the
\verb+combox.config+, \verb+combox.crypto+, \verb+combox.events+,
\verb+combox.file+, \verb+combox.silo+ \verb+combox._version+
modules. Unit tests were not written for \verb+combox.cbox+,
\verb+combox.gui+, \verb+combox.combox.log+ modules.

Unit tests for combox become reality by pure serendipity. During the
time, when I started working on combox, I was learning to use the
\verb+nose+ library to unit test python code. Since, \verb+combox+ was
being written in python, I started making it a norm to write unit
tests for functions and classes in combox modules.

As mentioned before, unit tests were not written for some modules
either because it would make no sense to write one (for the
\verb+combox.cbox+ module, for instance, which basically uses
functions and classes defined in other modules to run combox) or it
was not clear how to write unit tests it (the \verb+combox.gui+
contains just the \verb+ComboxConfigDialog+ a graphical front-end
which uses the configuration function defined in the
\verb+combox.config+ module to complete the combox configuration based
on the user input).

It must be noted here that pure Test Driven Development (TDD) was not
observed -- most of the time the function/class was written before the
its corresponding test was written.

\subsection{Benefits}

While writing unit tests definitely increased the time to write a
particular feature, it enabled me to immediately check if a feature
worked as it should for the given use case or given set of inputs.

With the benefit of hindsight, unit tests greatly helped in testing
the compatibility of combox on OSX. Before the \verb+v0.1.0+ release,
combox's node directory monitor always assumed that a file's first
shard (\verb+shard0+) is always available; while this assumption did
not create any problems on GNU/Linux, on OS X, this assumption made
the node directory monitor to behave erratically -- this issue (bug
\#4\cite{combox-issue-tracker} was immediately found when the unit
tests were run for the first time on OS X. Another instance where unit
tests helped was just before the \verb+v0.2.0+ release; major changes,
including the introduction of file locks in the
\verb+ComboxDirMonitor+, were made to the \verb+combox.events+. When
the unit tests were run OS X, two tests failed, revealing a difference
in behavior of watchdog\cite{pylib:watchdog} on GNU/Linux and OS X on
file creation\cite{combox-wd-fix}; without unit tests, there is a high
probability that this bug would never have been found by now.

\subsection{Caveats}

Unit tests are helpful in testing the correctness of a feature for
\verb+N+ number of use cases but it does not necessarily mean the
written feature correctly behaves for use cases that the author of the
feature did not consider or did not think about while writing the
respective feature. As Dijkstra correctly observed:

Unit tests failed to reveal bugs \#4, \#5 \#6 \#7 \#5 \#10
\#11\cite{combox-issue-tracker}; these bugs were found when manually
testing combox.

\section{Manual testing}

The unit tests for the \verb+combox.events+ module test the
correctness of the \verb+ComboxDirMonitor+ and \verb+NodeDirMonitor+
independently; in order to comprehensively test the correctness of
both \verb+ComboxDirMonitor+ and \verb+NodeDirMonitor+, it was
required to manually test combox running on more than one computer. As
you'll see in the following subsections, several bugs were found and
fixed while doing manual testing.

Three different types of setups were used to test combox. The first
kind of setup has two GNU/Linux machines each using combox to sync
files between each other with Dropbox and Google Drive being the
nodes; the second kind of setup has a GNU/Linux machine and a OS X
machine each using combox to sync files between each other with
Dropbox and Google Drive being the nodes; the third kind of setup has
a GNU/Linux machine and OS X machine each using combox to sync files
between each other with Dropbox, Google Drive and a USB stick as
nodes.

\subsection{General setup and notes}

\begin{itemize}
\item On the GNU/Linux machines, the official Dropbox client was used
  to sync the Dropbox node directory to Dropbox'
  servers. \verb+rclone+\cite{program:rclone} was used to sync the
  Google Drive node directory to Google Drive' servers;At the time of
  testing, Google Drive did not have client for GNU/Linux.
\item On OS X, the official Dropbox client was used to sync the
  Dropbox node directory to Dropbox's servers; the official Google
  Drive client was used to sync the Google Drive node directory to
  Google Driver' servers.
\item Since combox is extremely event-driven, combox must be started
  before the Dropbox and Google Drive clients start syncing their
  respective directories (nodes).
\end{itemize}

\subsection{Testing on two GNU/Linux machines}

combox was run to two GNU/Linux machines and a file was alternatively
created/modified/renamed/deleted on an of the GNU/Linux machine and it
was verified if the respective file was also
created/modified/renamed/deleted on the other GNU/Linux machine. One
of the GNU/Linux machine (\verb+lyra)+ was a virtual machine running
Debian GNU/Linux stable (version 8.x); the other GNU/Linux machine
(\verb+grus+) was a physical machine running Debian GNU/Linux
testing. The node directories to scatter the files' shards were the
Dropbox directory and Google Drive directory. The official Dropbox
client was used to automatically sync files from the Dropbox directory
to the Dropbox' server; \verb+rclone+\cite{program:rclone} was used to
sync files from Google Drive directory to Google Drive' server.

\subsubsection{Issues found}\label{ch-5-2gnus-issues}

\begin{itemize}
\item Some editors, especially on POSIX complaint systems, create
  backup version of the file being edited. combox was detecting this
  backup file as a ``new file'' and it split it into shards, encrypted
  the shards and scattered the shards across the node directories. The
  right thing for combox to do was to ignore these backup files and do
  nothing about them. This issue was fixed on
  \verb+2015-09-29+\cite{combox-issue-tracker}. Now the
  \verb+ComboxDirMonitor+, on a ``file created'' or ``file modified''
  event, returns from the \verb+on_created+ or \verb+on_modified+
  callback when it finds that the file is a backup/temporary file.
\item Dropbox client maintains the \verb+.dropbox.cache+ directory
  under the root of the Dropbox directory.

  \begin{itemize}
  \item When a file (shard) was created on another computer, the
    Dropbox client pulls the new file (shard) to this computer into
    \verb+.dropbox.cache+ as a temporary file and then moves the new
    file (shard) to its respective location with the appropriate name.
  \item When a file (shard) was modified on another computer, the
    Dropbox client pulls the modified file (shard) to this computer
    into the \verb+.dropbox.cache+ as a temporary file; moves the old
    version of the file (shard) under the Dropbox directory into the
    \verb+.dropbox.cache+; finally moves the updated copy of the file,
    stored as a temporary file, into the Dropbox directory to its
    respective location with the appropriate name.
  \item When a file (shard) was deleted on another computer, the
    Dropbox client moves the delete file into the
    \verb+.dropbox.cache+ directory on this computer.
  \end{itemize}

  All of the above behavior of the Dropbox client epically broke
  combox. Commits \verb+3d714c5+ to
  \verb+6e1133f+\cite{git:dropbox-fix} fixed combox by making it aware
  of Dropbox's client behavior.
\end{itemize}

\subsubsection{Demo}

Demo of combox being used on two GNU/Linux machines can be viewed at
\url{https://ricketyspace.net/combox/combox-2-gnus.webm}.

\verb+lyra+ (virtual machine) and \verb+grus+ (bare-metal) are the two
GNU/Linux machines being used for the demo.

Description of what happens in the demo follows:

 - (lyra) install combox.

 - (lyra) run combox (test mode).

 - (lyra) create file \verb+walden.pond+ with content ``It must be
 beautiful there''.

 - (lyra) sync Google Drive using \verb+rclone+.

 - (grus) sync Google Drive using \verb+rclone+.

 - (grus) git pull latest copy of combox.

 - (grus) install combox 

 - (grus) run combox (testing mode).

 - (grus) verify that \verb+walden.pond+ was create on this machine.

 - (grus) append 'Peaceful too.' to \verb+walden.pond+.

 - (grus) sync Google Drive using \verb+rclone+.

 - (lyra) sync Google Drive using \verb+rclone+.

 - (lyra) verify that the latest copy of \verb+walden.pond+ is there
 in the combox directory; it should contain 'Peaceful too.' in the
 last line.

 - (lyra) append ``I've a dream'' to \verb+walden.pond+.

 - (lyra) sync Google Drive using \verb+rclone+.

 - (grus) sync Google Drive using \verb+rclone+.

 - (grus) verify that the latest copy of \verb+walden.pond+ is there
 in the combox directory; it should contain ``I've a dream'' in the
 last line.

 - (grus) remove \verb+walden.pond+ from combox directory.

 - (grus) sync Google Drive using \verb+rclone+.

 - (lyra) sync Google Drive using \verb+rclone+.

 - (lyra) verify that \verb+walden.pond+ is removed from the combox
 directory.

 - (grus) open dropbox and Google drive accounts from the web browser.

 - (lyra) create file \verb+manufacturing.consent.+ with content ``Chomsky stuff?''.

 - (lyra) sync Google Drive using \verb+rclone+.

 - (grus) sync Google Drive using \verb+rclone+.

 - (grus) verify that \verb+manufacturing.consent+ was created in the
 combox directory.

 - (grus) verify that the shards of \verb+manufacturing.consent+ were
 created on Dropbox and Google Drive through the web browser.

\subsection{Testing on a GNU/Linux and an OS X machine}

combox was run on a GNU/Linux machine and an OS X machine and a file
was alternatively created/modified/renamed/deleted on one of the
machine and it was verified if the respective file was also
created/modified/renamed/deleted on the other machine. The GNU/Linux
machine was a virtual machine (\verb+lyra+) running Debian GNU/Linux
stable; the OS X machine was on Mavericks (10.9) during the initial
stage of testing, later it was upgraded to Yosemite (10.10). The node
directories to scatter files' shards were the Dropbox directory and
the Google Drive directory. The official Dropbox client was used to
automatically sync files from the Dropbox directory to the Dropbox'
server on both the GNU/Linux machine and the OS X machine; the
official Google Drive client was used to automatically sync files from
the Google Drive directory to Google Drive' server on OS X and
\verb+rclone+\cite{program:rclone} was used to sync files from the
Google Drive directory to Google Drive's server on GNU/Linux.

\subsubsection{Issues found}

\begin{itemize}
\item When a file was modified on another computer, on this computer
  combox assumed that first shard (shard0) will be updated first and
  also counted on the existence of the first shard (shard0). It was
  observed that the order in which the shards were updated were
  unpredictable on this computer and if the first shard (shard0) was
  stored in the Dropbox directory, it will momentarily disappear
  before the most updated shard becomes available in the Dropbox
  directory; this broke combox. This issue was fixed on
  2015-08-25\cite{git:bug-four-fix}. This issue is not got to do with
  the nature of the setup but it is related to the Dropbox's behavior
  elaborated in section \ref{ch-5-2gnus-issues}.
\item The official Google Drive client when it pulls an updated
  version of the file from Google Drive' server, instead directly
  updating the respective file on the computer, it deletes the older
  version of the file and creates the latest version of the file at
  the respective location in the Google Drive directory; this behavior
  of the Google Drive confused and broke combox. This issue was fixed
  2015-09-06 by making combox under the official Google Client's
  behavior\cite{git:bug-googledc-fix}.
\item When a non-empty directory was move/renamed on another computer,
  the old directory was not getting properly deleted on this computer;
  this was happening because the files under the directory being
  renamed were not deleted when it was time for \verb+NodeDirMonitor+
  to \verb+rmdir+ the old directory. This issue again is not specific
  to the nature of the setup but was found while testing combox on
  this setup. This issue was fixed on
  2015-09-12\cite{git:bug-six-fix}.
\item It was found that \verb+combox.file.rm_path+ function failed
  when it was given a non-existent path to remove; this issue was
  fixed on 2015-09-12\cite{git:bug-seven-fix}.
\end{itemize}

\subsubsection{Demo}

Demo of combox being used on a GNU/Linux machine and OS X machine can
be viewed at \url{https://ricketyspace.net/combox/combox-gnu-osx.webm}

\verb+lyra+ is the GNU/Linux (virtual) machine and
\verb+dhcp-129-1-66-1+ is the OS X machine that is being used for the
demo. The OS X machine is accessed through VNC\cite{article:vnc}.

Description of what happens in the demo follows:

  - (\verb+lyra+) create file \verb+cat.stevens+ with content ``peace train''.

  - (\verb+lyra+) sync Google Drive using \verb+rclone+.

  - (\verb+dhcp-129-1-66-1+) verify that file \verb+cat.stevens+ is
  created with content ``peace train''.

  - (\verb+dhcp-129-1-66-1+) append string ``moonshadow'' to file
  \verb+cat.stevens+.

  - (\verb+lyra+) sync Google Drive using \verb+rclone+.

  - (\verb+lyra+) verify that the file \verb+cat.stevens+ was updated
  (modified); last line must have the string ``moonshadow''.

  - (\verb+lyra+) append string ``father and son'' to the file
  \verb+cat.stevens+.

  - (\verb+lyra+) sync Google Drive using \verb+rclone+.

  - (\verb+dhcp-129-1-66-1+) verify that the file \verb+cat.stevens+
  was updated (modified); last line must have the string ``father and
  son''.

  - (\verb+dhcp-129-1-66-1+) rename file \verb+cat.stevens+ to
  \verb+yusuf.islam+

  - (\verb+lyra+) sync Google Drive using \verb+rclone+.

  - (\verb+lyra+) verify that the file \verb+cat.stevens+ was renamed
  to \verb+yusuf.islam+.

\subsection{Testing with a USB stick as a node}

combox was run on a GNU/Linux machine and an OS X machine and a file
was alternatively created/modified/deleted on one of the machine and
it was verified if the repsective file was also
create/modified/deleted on the other machine. The GNU/Linux machine
was a physical machine (\verb+grus+) running Debian GNU/Linux stable;
The OS X machine was on Mavericks (10.9). The node directories to
scatter files' shards were the Dropbox directory, Google Drive
directory and the USB stick (\verb+ZAPHOD+, FAT filesystem). The
official Dropbox client was used to automatically sync files from
Dropbox directory to Dropbox' server on both the GNU/Linux machine and
OS X machine; the official Google Drive client was used to
automatically sync files from the Google Drive directory to Google
Drive' server on OS X and \verb+rclone+\cite{program:rclone} was used
to sync files from the Google Drive directory to Google Drive's server
on GNU/Linux; the same USB stick (\verb+ZAPHOD+) was used on bothe
GNU/Linux and Dropbox to store the third shard (shard2) of a file.

\subsubsection{Caveats}

\begin{itemize}
\item When a removable USB disk is used as a node, combox must be
  turned off before ejecting/unmounting the USB disk; combox does not
  expect a node directory to disappear when it is running, if the USB
  disk is removed when combox is running, then combox goes to a
  undefined state.

\item When a file modified on machine A is synced to machine B, combox
  must be turned on first before turning on Dropbox and Google Drive
  clients and the shard in the USB disk needs to be ``touched'' for
  combox to detect that the file was modified on the remote computer
  and update the file locally on this machine.

\item File rename/move does not work. To make it work, core
  functionality of combox must be re-written.
\end{itemize}

\subsubsection{Demo}

Demo of combox being used with a USB stick as the third node can be
view at \url{https://ricketyspace.net/combox/combox-usb-node-demo.webm}

\verb+grus+ is the GNU/Linux machine and \verb+dhcp-129-1-66-1+ is the
OS X machine that is being used for the demo. \verb+ZAPHOD+ is the
FAT32 USB stick used as the third node.

Description of what happens in the demo follows:

  - (\verb+grus+) start combox.

  - (\verb+grus+) create a file called \verb+simon.and.garfunkel+ with
  content ``the boxer''.

  - (\verb+grus+) sync Google Drive using \verb+rclone+.

  - (\verb+grus+) stop combox.

  - (\verb+grus+) unmount USB stick (\verb+ZAPHOD+) from \verb+grus+.

  - (\verb+dhcp-129-1-66-1+) mount USB stick (\verb+ZAPHOD+) to
  (\verb+dhcp-129-1-66-1+).

  - (\verb+dhcp-129-1-66-1+) start Dropbox client.

  - (\verb+dhcp-129-1-66-1+) start Google Drive client.

  - (\verb+dhcp-129-1-66-1+) start combox.

  - (\verb+dhcp-129-1-66-1+) verify that the file
  \verb+simon.and.garfunkel+ with content ``the boxer'' was created.

  - (\verb+dhcp-129-1-66-1+) append string ``mrs. robinson'' to file
  \verb+simon.and.garfunkel+.

  - (\verb+dhcp-129-1-66-1+) stop combox.

  - (\verb+dhcp-129-1-66-1+) stop Google Drive client.

  - (\verb+dhcp-129-1-66-1+) stop Dropbox client.

  - (\verb+dhcp-129-1-66-1+) unmount the USB stick (\verb+ZAPHOD+)
  from (\verb+dhcp-129-1-66-1+).

  - (\verb+grus+) mount the USB stick (\verb+ZAPHOD+) to
  (\verb+grus+).

  - (\verb+grus+) start combox.

  - (\verb+grus+) start Dropbox client.

  - (\verb+grus+) sync Google Drive using \verb+rclone+.

  - (\verb+grus+) touch \verb+simon.and.garfunkel.shard2+ in the USB
  stick (\verb+ZAPHOD+).

  - (\verb+grus+) verify that the file \verb+simon.and.garfunkel+ is
  updated; the last line must contain the string ``mrs. robinson''.

  - (\verb+grus+) remove the file \verb+simon.and.garfunkel+.

  - (\verb+grus+) sync Google Drive using \verb+rclone+.

  - (\verb+grus+) unmount the USB stick (\verb+ZAPHOD+) from
  (\verb+grus+).

  - (\verb+grus+) stop Dropbox client.

  - (\verb+dhcp-129-1-66-1+) mount the USB stick (\verb+ZAPHOD+) to
  (\verb+dhcp-129-1-66-1+).

  - (\verb+dhcp-129-1-66-1+) start Google Drive client.

  - (\verb+dhcp-129-1-66-1+) start Dropbox client.

  - (\verb+dhcp-129-1-66-1+) start combox.

  - (\verb+dhcp-129-1-66-1+) verify that the file
  \verb+simon.and.garfunkel+ was deleted.


\section{Stress testing}


\begin{figure}[h]
\centering
\input{graphs/tot-time.tex}
\caption{time to process all files}
\end{figure}

\begin{figure}[h]
\centering
\input{graphs/avg-time-sae.tex}
\caption{avg. time to split and encrypt}
\end{figure}

\begin{figure}[h]
\centering
\input{graphs/tot-time-diff.tex}
\caption{time to process all files - difference between 2015 and 2016}
\end{figure}

\begin{figure}[h]
\centering
\input{graphs/avg-time-sae-diff.tex}
\caption{avg. time to split and encrypt - difference between 2015 and 2016}
\end{figure}