summaryrefslogtreecommitdiffstats
path: root/report/chapters/4-testing.tex
blob: fb6577b006616d6d8c7eb19b831dd687107ba67d (plain) (blame)
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
300
301
302
303
304
305
306
307
308
309
310
311
312
313
314
315
316
317
318
319
320
321
322
323
324
325
326
327
328
329
330
331
332
333
334
335
336
337
338
339
340
341
342
343
344
345
346
347
348
349
350
351
352
353
354
355
356
357
358
359
360
361
362
363
364
365
366
367
368
369
370
371
372
373
374
375
376
377
378
379
380
381
382
383
384
385
386
387
388
389
390
391
392
393
394
395
396
397
398
399
400
401
402
403
404
405
406
407
408
409
410
411
412
413
414
415
416
417
418
419
420
421
422
423
424
425
426
427
428
429
430
431
432
433
434
435
436
437
438
439
440
441
442
443
444
445
446
447
448
449
450
451
452
453
454
455
456
457
458
459
460
461
462
463
464
465
466
467
468
469
470
471
472
473
474
475
476
477
478
479
480
481
482
483
484
485
486
487
488
489
490
491
492
493
494
495
496
497
498
499
500
501
502
503
504
505
506
507
508
509
510
511
512
513
514
515
516
517
518
519
520
521
522
523
524
525
526
527
528
529
530
531
532
533
534
535
536
537
538
539
540
541
542
543
544
545
546
547
548
549
550
551
552
553
554
555
556
557
558
559
560
561
562
563
564
565
566
567
568
569
570
571
572
573
574
575
576
577
578
579
580
581
582
583
584
585
586
587
588
589
590
591
592
593
594
595
596
597
598
599
600
601
602
603
604
605
606
607
608
609
610
611
612
613
614
615
616
617
618
619
620
621
622
623
624
625
626
627
628
629
630
631
632
633
634
635
636
637
638
639
640
641
642
643
644
645
646
647
648
649
650
651
652
653
654
655
656
657
658
659
660
661
662
663
664
665
666
667
668
669
670
671
672
673
674
675
676
677
678
679
680
\chapter{Testing}\label{ch:4}

\epigraph{Testing shows the presence, not the absence of
  bugs.}{\textit{Dijkstra}\cite{dijkstra69}}

\section{Unit testing}\label{sec:4-unit-testing}

The \verb+nose+ \cite{pylib:nose} testing framework was used to write
unit tests for the functions and classes that are part of the
\verb+combox.config+, \verb+combox.crypto+, \verb+combox.events+,
\verb+combox.file+, \verb+combox.silo+ and \verb+combox._version+
modules. Unit tests were not written for \verb+combox.cbox+,
\verb+combox.gui+ and \verb+combox.combox.log+ modules either because
it did not sense to write one -- for instance, the \verb+combox.cbox+
module, which uses functions and classes defined in other modules
which are unit tested -- or it was not clear how to write unit tests
for it (the \verb+combox.gui+ module).

It must be noted here that pure Test Driven Development (TDD) was not
observed -- most of the time the function/class was written before the
its corresponding test was written.

\subsection{Benefits}

While writing unit tests definitely increased the time to write a
particular feature, it made it possible to immediately check if a
feature worked as it should for a given set of use cases or given set
of inputs.

Unit tests greatly helped in testing the compatibility of combox on OS
X. Before the \verb+v0.1.0+ release, combox's node directory monitor
always assumed that a file's first shard (\verb+shard0+) is always
available. While this assumption did not create any problems on
GNU/Linux, on OS X this assumption made the node directory monitor to
behave erratically. This issue (bug \#4) was immediately found when
the unit tests were run for the first time on OS X. Another instance
where unit tests helped was just before the \verb+v0.2.0+ release.
Major changes, including the introduction of file locks in the
\verb+ComboxDirMonitor+, were made to the \verb+combox.events+. When
the unit tests were run on OS X, two tests failed, revealing a
difference in behavior of watchdog \cite{pylib:watchdog} on GNU/Linux
and OS X on file creation
\footnote{https://git.ricketyspace.net/combox/commit/?id=8c86e7c28738c66c0e04ae7886b44dbcdfc6369exo}.
Without unit tests, there is a high probability that this bug would
never have been found by now.

\subsection{Caveats}

Unit tests are helpful in testing the correctness of a feature for
\verb+N+ number of use cases but it does not necessarily mean the
written feature correctly behaves for use cases that the author of the
feature did not consider or did not think about while writing the
respective feature.

Unit tests failed to reveal bugs \#5, \#6, \#7, \#10 and \#11
\footnote{https://git.ricketyspace.net/combox/plain/TODO.org}. These
bugs were found when manually testing combox.

\section{Manual testing}\label{sec:4-manual-testing}

The unit tests for the \verb+combox.events+ module tested the
correctness of the \\ \verb+ComboxDirMonitor+ and
\verb+NodeDirMonitor+ independently. In order to comprehensively test
the correctness of both \verb+ComboxDirMonitor+ and
\verb+NodeDirMonitor+, it was required to manually test combox running
on more than one computer. Several bugs were found and fixed while
doing manual testing.

Three different types of setups were used to manually test combox. The
first kind of setup has two GNU/Linux machines each using combox to
sync files between each other with Dropbox and Google Drive being the
nodes. The second kind of setup has a GNU/Linux machine and a OS X
machine each using combox to sync files between each other with
Dropbox and Google Drive being the nodes. The third kind of setup has
a GNU/Linux machine and OS X machine each using combox to sync files
between each other with Dropbox, Google Drive and a USB stick as
nodes.

\subsection{General setup and notes}

\begin{itemize}
\item On the GNU/Linux machines, the official Dropbox client was used
  to sync the Dropbox node directory to Dropbox' data
  store. \verb+rclone+ \cite{program:rclone} was used to sync the
  Google Drive node directory to Google Drive' data store. At the time
  of testing, Google Drive did not have a client program for
  GNU/Linux which can sync to Google Drive's data store.
\item On OS X, the official Dropbox client was used to sync the
  Dropbox node directory to Dropbox's data store. The official Google
  Drive client was used to sync the Google Drive node directory to
  Google Driver' data store.
\item Since combox is extremely event-driven, combox must be started
  before the Dropbox and Google Drive clients start syncing their
  respective directories.
\end{itemize}

\subsection{Testing on two GNU/Linux machines}

combox was run on two GNU/Linux machines and a file was alternatively
created/modified/renamed/deleted on one of the GNU/Linux machine and
it was verified if the respective file was also
created/modified/renamed/deleted on the other GNU/Linux machine. One
of the GNU/Linux machines, (\verb+lyra)+, was a virtual machine
running Debian GNU/Linux stable (version 8.x). The other GNU/Linux
machine (\verb+grus+) was a physical machine running Debian GNU/Linux
testing. The node directories to scatter the files' shards were the
Dropbox directory and Google Drive directory. The official Dropbox
client was used to automatically sync files from the Dropbox directory
to the Dropbox' data store. \verb+rclone+ \cite{program:rclone} was
used to sync files from Google Drive directory to Google Drive' data
store.

\subsubsection{Issues found}\label{ch-4-2gnus-issues}

\begin{itemize}
\item Some editors, especially on POSIX complaint systems, create a
  backup version of the file being edited. combox was detecting this
  backup file as a ``new file'' and it split it into shards, encrypted
  the shards and scattered the shards across the node directories. The
  right thing for combox to do was to ignore these backup files and do
  nothing about them. This issue was fixed on \verb+2015-09-29+
  \footnote{https://git.ricketyspace.net/combox/plain/TODO.org}. Now
  the \verb+ComboxDirMonitor+, on a ``file created'' or ``file
  modified'' event, returns from the \verb+on_created+ or
  \verb+on_modified+ callback when it finds that the file is a
  backup/temporary file.
\item Dropbox client maintains the \verb+.dropbox.cache+ directory
  under the root of the Dropbox directory.

  \begin{itemize}
  \item When a file (shard) was created on another computer, the
    Dropbox client pulls the new file (shard) to this computer into
    \verb+.dropbox.cache+ as a temporary file and then moves the new
    file (shard) to its respective location with the appropriate name.
  \item When a file (shard) was modified on another computer, the
    Dropbox client pulls the modified file (shard) to this computer
    into the \verb+.dropbox.cache+ as a temporary file, moves the old
    version of the file (shard) under the Dropbox directory into the
    \verb+.dropbox.cache+, and finally moves the updated copy of the
    file, stored as a temporary file, into the Dropbox directory to
    its respective location with the appropriate name.
  \item When a file (shard) was deleted on another computer, the
    Dropbox client moves the deleted file into the
    \verb+.dropbox.cache+ directory on this computer.
  \end{itemize}

  All of the above behavior of the Dropbox client broke
  combox. Commits between \verb+3d714c5+ to \verb+6e1133f+
  \footnote{https://git.ricketyspace.net/combox/log/?qt=range\&q=3d714c5..6e1133f}
  fixed combox by making it aware of Dropbox's client behavior.
\end{itemize}

\subsubsection{Demo}

A demo of combox being used on two GNU/Linux machines can be viewed at
\url{https://ricketyspace.net/combox/combox-2-gnus.webm}. \verb+lyra+
(virtual machine) and \verb+grus+ (bare-metal) are the two GNU/Linux
machines being used for the demo.

Description of what happens in the demo follows:

- (lyra) install combox.

- (lyra) run combox (test mode).

- (lyra) create file \verb+walden.pond+ with content ``It must be
beautiful there''.

- (lyra) sync Google Drive using \verb+rclone+.

- (grus) sync Google Drive using \verb+rclone+.

- (grus) git pull latest copy of combox.

- (grus) install combox

- (grus) run combox (testing mode).

- (grus) verify that \verb+walden.pond+ was create on this machine.

- (grus) append 'Peaceful too.' to \verb+walden.pond+.

- (grus) sync Google Drive using \verb+rclone+.

- (lyra) sync Google Drive using \verb+rclone+.

- (lyra) verify that the latest copy of \verb+walden.pond+ is there in
the combox directory -- it should contain 'Peaceful too.' in the last
line.

- (lyra) append ``I've a dream'' to \verb+walden.pond+.

- (lyra) sync Google Drive using \verb+rclone+.

- (grus) sync Google Drive using \verb+rclone+.

- (grus) verify that the latest copy of \verb+walden.pond+ is there in
the combox directory -- it should contain ``I've a dream'' in the last
line.

- (grus) remove \verb+walden.pond+ from combox directory.

- (grus) sync Google Drive using \verb+rclone+.

- (lyra) sync Google Drive using \verb+rclone+.

- (lyra) verify that \verb+walden.pond+ is removed from the combox
directory.

- (grus) open Dropbox and Google drive accounts from the web browser.

- (lyra) create file \verb+manufacturing.consent.+ with content
``Chomsky stuff?''.

- (lyra) sync Google Drive using \verb+rclone+.

- (grus) sync Google Drive using \verb+rclone+.

- (grus) verify that \verb+manufacturing.consent+ was created in the
combox directory.

- (grus) verify that the shards of \verb+manufacturing.consent+ were
created on Dropbox and Google Drive through the web browser.

\subsection{Testing on a GNU/Linux and an OS X machine}

combox was run on a GNU/Linux machine and an OS X machine and a file
was alternatively created/modified/renamed/deleted on one of the
machine and it was verified if the respective file was also
created/modified/renamed/deleted on the other machine. The GNU/Linux
machine was a virtual machine (\verb+lyra+) running Debian GNU/Linux
stable, the OS X machine was on Mavericks (10.9) during the initial
stage of testing, later it was upgraded to Yosemite (10.10). The node
directories to scatter files' shards were the Dropbox directory and
the Google Drive directory. The official Dropbox client was used to
automatically sync files from the Dropbox directory to the Dropbox'
data store on both the GNU/Linux machine and the OS X machine, the
official Google Drive client was used to automatically sync files from
the Google Drive directory to Google Drive' data store on OS X and
\verb+rclone+ \cite{program:rclone} was used to sync files from the
Google Drive directory to Google Drive's data store on GNU/Linux.

\subsubsection{Issues found}

\begin{itemize}
\item When a file was modified on another computer, on this computer
  combox assumed that first shard (shard0) will be updated first and
  also counted on the existence of the first shard (shard0). It was
  observed that the order in which the shards were updated were
  unpredictable on this computer and if the first shard (shard0) was
  stored in the Dropbox directory, it will momentarily disappear
  before the most updated shard becomes available in the Dropbox
  directory, this broke combox. This issue was fixed on 2015-08-25
  \footnote{https://git.ricketyspace.net/combox/commit/?id=d5b52030348d40600b4c9256f76e5183a85fbb17}. This
  issue is not got to do with the nature of the setup but it is
  related to the Dropbox's behavior elaborated in section
  \ref{ch-4-2gnus-issues}.
\item When the official Google Drive client pulls an updated version
  of the file from Google Drive' data store, instead directly updating
  the respective file on the computer, it deletes the older version of
  the file and creates the latest version of the file at the
  respective location in the Google Drive directory, this behavior of
  the Google Drive client confused and broke combox. This issue was
  fixed 2015-09-06 by making combox aware of the official Google
  Client's behavior
  \footnote{https://git.ricketyspace.net/combox/commit/?id=37385a90f90cb9d4dfd13d9d2e3cbcace8011e9e}.
\item When a non-empty directory was move/renamed on another computer,
  the old directory was not getting properly deleted on this computer.
  This was happening because, sometimes, the files under the directory
  being renamed were not deleted when it was time for
  \verb+NodeDirMonitor+ to \verb+rmdir+ the old directory. This issue
  was fixed on 2015-09-12
  \footnote{https://git.ricketyspace.net/combox/commit/?id=9d14db03da5d10d5ab0d7cc76b20e7b1ed5523bf}.
\item It was found that \verb+combox.file.rm_path+ function failed
  when it was given a non-existent path to remove, this issue was
  fixed on 2015-09-12
  \footnote{https://git.ricketyspace.net/combox/commit/?id=422238eb4904de14842221fa09a2b4028801afb1}.
\end{itemize}

\subsubsection{Demo}

A demo of combox being used on a GNU/Linux machine and OS X machine
can be viewed at
\url{https://ricketyspace.net/combox/combox-gnu-osx.webm}

\verb+lyra+ is the GNU/Linux (virtual) machine and
\verb+dhcp-129-1-66-1+ is the OS X machine that is being used for the
demo. The OS X machine is accessed through VNC\cite{article:vnc}.

Description of what happens in the demo follows:

- (\verb+lyra+) create file \verb+cat.stevens+ with content ``peace
train''.

- (\verb+lyra+) sync Google Drive using \verb+rclone+.

- (\verb+dhcp-129-1-66-1+) verify that file \verb+cat.stevens+ is
created with content ``peace train''.

- (\verb+dhcp-129-1-66-1+) append string ``moonshadow'' to file
\verb+cat.stevens+.

- (\verb+lyra+) sync Google Drive using \verb+rclone+.

- (\verb+lyra+) verify that the file \verb+cat.stevens+ was updated
(modified) -- last line must have the string ``moonshadow''.

- (\verb+lyra+) append string ``father and son'' to the file
\verb+cat.stevens+.

- (\verb+lyra+) sync Google Drive using \verb+rclone+.

- (\verb+dhcp-129-1-66-1+) verify that the file \verb+cat.stevens+ was
updated (modified) -- last line must have the string ``father and son''.

- (\verb+dhcp-129-1-66-1+) rename file \verb+cat.stevens+ to
\verb+yusuf.islam+

- (\verb+lyra+) sync Google Drive using \verb+rclone+.

- (\verb+lyra+) verify that the file \verb+cat.stevens+ was renamed to
\verb+yusuf.islam+.

\subsection{Testing with a USB stick as a node}

combox was run on a GNU/Linux machine and an OS X machine and a file
was alternatively created/modified/deleted on one of the machine and
it was verified if the respective file was also
create/modified/deleted on the other machine. The GNU/Linux machine
was a physical machine (\verb+grus+) running Debian GNU/Linux testing,
The OS X machine was on Mavericks (10.9). The node directories to
scatter files' shards were the Dropbox directory, Google Drive
directory and the USB stick (\verb+ZAPHOD+, FAT filesystem). The
official Dropbox client was used to automatically sync files from
Dropbox directory to Dropbox' data store on both the GNU/Linux machine
and the OS X machine, the official Google Drive client was used to
automatically sync files from the Google Drive directory to Google
Drive' data store on OS X and \verb+rclone+ \cite{program:rclone} was
used to sync files from the Google Drive directory to Google Drive's
data store on GNU/Linux, the same USB stick (\verb+ZAPHOD+) was used
on both GNU/Linux and Dropbox to store the third shard (shard2) of the
files stored in combox directory.

\subsubsection{Caveats}

\begin{itemize}
\item When a removable USB disk is used as a node, combox must be
  turned off before ejecting/unmounting the USB disk, combox does not
  expect a node directory to disappear when it is running, if the USB
  disk is removed when combox is running, then combox goes to an
  undefined state.

\item When a file modified on machine A is synced to machine B, combox
  must be turned on first before turning on Dropbox and Google Drive
  clients and the shard in the USB disk needs to be ``touched'' for
  combox to detect that the file was modified on the remote computer
  and update the file locally on this machine.

\item File rename/move does not work. To make it work, core
  functionality of combox must be re-written.
\end{itemize}

\subsubsection{Demo}

A demo of combox being used with a USB stick as the third node can be
viewed at
\url{https://ricketyspace.net/combox/combox-usb-node-demo.webm}

\verb+grus+ is the GNU/Linux machine and \verb+dhcp-129-1-66-1+ is the
OS X machine that is being used for the demo. \verb+ZAPHOD+ is the
FAT32 USB stick used as the third node.

Description of what happens in the demo follows:

- (\verb+grus+) start combox.

- (\verb+grus+) create a file called \verb+simon.and.garfunkel+ with
content ``the boxer''.

- (\verb+grus+) sync Google Drive using \verb+rclone+.

- (\verb+grus+) stop combox.

- (\verb+grus+) unmount USB stick (\verb+ZAPHOD+) from \verb+grus+.

- (\verb+dhcp-129-1-66-1+) mount USB stick (\verb+ZAPHOD+) to
(\verb+dhcp-129-1-66-1+).

- (\verb+dhcp-129-1-66-1+) start Dropbox client.

- (\verb+dhcp-129-1-66-1+) start Google Drive client.

- (\verb+dhcp-129-1-66-1+) start combox.

- (\verb+dhcp-129-1-66-1+) verify that the file
\verb+simon.and.garfunkel+ with content ``the boxer'' was created.

- (\verb+dhcp-129-1-66-1+) append string ``mrs. robinson'' to file
\verb+simon.and.garfunkel+.

- (\verb+dhcp-129-1-66-1+) stop combox.

- (\verb+dhcp-129-1-66-1+) stop Google Drive client.

- (\verb+dhcp-129-1-66-1+) stop Dropbox client.

- (\verb+dhcp-129-1-66-1+) unmount the USB stick (\verb+ZAPHOD+) from
(\verb+dhcp-129-1-66-1+).

- (\verb+grus+) mount the USB stick (\verb+ZAPHOD+) to (\verb+grus+).

- (\verb+grus+) start combox.

- (\verb+grus+) start Dropbox client.

- (\verb+grus+) sync Google Drive using \verb+rclone+.

- (\verb+grus+) touch \verb+simon.and.garfunkel.shard2+ in the USB
stick (\verb+ZAPHOD+).

- (\verb+grus+) verify that the file \verb+simon.and.garfunkel+ is
updated -- the last line must contain the string ``mrs. robinson''.

- (\verb+grus+) remove the file \verb+simon.and.garfunkel+.

- (\verb+grus+) sync Google Drive using \verb+rclone+.

- (\verb+grus+) unmount the USB stick (\verb+ZAPHOD+) from
(\verb+grus+).

- (\verb+grus+) stop Dropbox client.

- (\verb+dhcp-129-1-66-1+) mount the USB stick (\verb+ZAPHOD+) to
(\verb+dhcp-129-1-66-1+).

- (\verb+dhcp-129-1-66-1+) start Google Drive client.

- (\verb+dhcp-129-1-66-1+) start Dropbox client.

- (\verb+dhcp-129-1-66-1+) start combox.

- (\verb+dhcp-129-1-66-1+) verify that the file
\verb+simon.and.garfunkel+ was deleted.


\section{Stress testing}

A large number of files of different sizes were dumped to the combox
directory between an one second interval to see how combox responds to
high load. The file dump size was varied from \verb+424.80MiB+ (27
files) to \verb+10,800.00MiB+ (180 files). The average time taken to
split a file and the total time to process all files were calculated
for each dump.

Stress testing was first done on \verb+2015-11-08+. In mid November
2015, the \\ \verb+ComboxDirMonitor+ was drastically modified to make
it use the file Lock shared by the instances of \verb+NodeDirMonitor+
\footnote{https://git.ricketyspace.net/combox/commit/?id=5aa1ba0c1dcad62931ba27bb66bf115233086d6c}.
The hypothesis was that this change in \verb+ComboxDirMonitor+
directly affected the performance of combox and therefore the results
that were got from stress testing on \verb+2015-11-08+ would no longer
be valid. Stress testing was again done on \verb+2016-01-16+. The
results of this stress test are in sections \ref{4-st-424} to
\ref{4-st-10800}. Section \ref{4-st-tu} gives information about the
tools used for stress testing, section \ref{4-st-o} contains the
observations and comparisons between this stress test and the one done
on \verb+2015-11-08+, and, lastly section \ref{4-st-if} reveals the
issues that were found with combox by virtue of doing the stress
tests.

\subsection{flac dump (27 files - 424.80MiB)}\label{4-st-424}

\begin{center}
  \begin{table}[h]
    \begin{tabular}{ll}
      field & value\\
      \hline
      delay between a file dump & 1s\\
      start time of processing & 11:00:54\\
      end time of processing & 11:01:38\\
      total time taken to process all files & 00:00:44\\
      no. of files & 27\\
      total size of all files & 445433187.00 bytes (424.79MiB)\\
      avg. file size & 16497525.00 bytes (15.73MiB)\\
      avg. time to split and encrypt a file & 352.58 ms\\
    \end{tabular}
    \caption{Stress Testing combox - flac dump (424.79MiB)}
  \end{table}
\end{center}

\subsubsection{Differences from previous stress test (2015-11-08)}

\begin{itemize}
\item Total time to process all files was faster by 1min3secs.
\item Average time to split and encrypt a file reduced by 28.33ms.
\end{itemize}

\subsection{20MiB - 90MiB dump (27 files -
  1620.00MiB)}\label{4-st-1620}

\begin{center}
  \begin{table}[h]
    \begin{tabular}{ll}
      field & value\\
      \hline
      delay between a file dump & 1s\\
      start time of processing & 12:26:45\\
      end time of processing & 12:29:07\\
      total time taken to process all files & 00:02:22\\
      no. of files & 27\\
      total size of all files & 1698693120.00 bytes (1620.00MiB)\\
      avg. file size & 62914560.00 bytes (60.00iB)\\
      avg. time to split and encrypt a file & 2670.59ms\\
    \end{tabular}
    \caption{Stress Testing combox - 20MiB - 90MiB dump (1620.00MiB)}
  \end{table}
\end{center}

\subsubsection{Differences from previous stress test (2015-11-08)}

\begin{itemize}
\item Total time to process all files was slower by 4secs.
\item Average time to split and encrypt a file reduced by 25.52ms.
\end{itemize}

\subsection{20MiB - 90MiB dump (99 files -
  5940.00MiB)}\label{4-st-5940}

\begin{center}
  \begin{table}[h]
    \begin{tabular}{ll}
      field & value\\
      \hline
      delay between a file dump & 1s\\
      start time of processing & 13:10:16\\
      end time of processing & 13:19:26\\
      total time taken to process all files & 00:09:10\\
      no. of files & 99\\
      total size of all files & 6228541440.00 bytes (5940.00MiB)\\
      avg. file size & 62914560.00 bytes (60.00MiB)\\
      avg. time to split and encrypt a file & 2979.64ms\\
    \end{tabular}
    \caption{Stress Testing combox - 20MiB - 90MiB dump (5940.00MiB)}
  \end{table}
\end{center}

\subsubsection{Differences from previous stress test (2015-11-08)}

\begin{itemize}
\item Total time to process all files was faster by 59secs.
\item Average time to split and encrypt a file increased by 206.20ms.
\end{itemize}

\subsection{20MiB - 90MiB dump (180 files -
  10800.00MiB)}\label{4-st-10800}

\begin{center}
  \begin{table}[h]
    \begin{tabular}{ll}
      field & value\\
      \hline
      delay between a file dump & 1s\\
      start time of processing & 13:42:06\\
      end time of processing & 14:00:10\\
      total time taken to process all files & 00:18:04\\
      no. of files & 180\\
      total size of all files & 11324620800.00 bytes (10800.00MiB)\\
      avg. file size & 62914560.00 bytes (60.00MiB)\\
      avg. time to split and encrypt a file & 3423.08ms\\
    \end{tabular}
    \caption{Stress Testing combox - 20MiB - 90MiB dump (10800.00MiB)}
  \end{table}
\end{center}

\subsubsection{Differences from previous stress test (2015-11-08)}

\begin{itemize}
\item Total time to process all files was slower by 1min2secs
\item Average time to split and encrypt a file increased by 399.87ms.
\end{itemize}

\subsection{Tools used}\label{4-st-tu}

The \verb+dump+ script
\footnote{https://git.ricketyspace.net/combox-paper/plain/dumper/dump}
was used to dump files to the combox directory between one second
intervals. A night of Emacs Lisp indulgence made it possible to
quickly slurp the required data from the combox output and calculate
the average time to split and encrypt a file and the total amount of
time taken to process the files for a given dump
\footnote{https://git.ricketyspace.net/combox-paper/plain/scripts/dumps.el}.
Lastly \verb+org-mode+ was used to document all data gathered during
stress testing
\footnote{https://git.ricketyspace.net/combox-paper/plain/notes/benchmarks.org}.

\subsection{Observations}\label{4-st-o}

\begin{figure}[h]
  \centering \input{graphs/tot-time.tex}
  \caption{Stress testing combox - Observations - Time taken to
    process all files.}
  \label{fig:4-st-tt}
\end{figure}

\begin{figure}[h]
  \centering \input{graphs/avg-time-sae.tex}
  \caption{Stress testing combox - Observations - Avg. time to split
    and encrypt a file.}
  \label{fig:4-st-atsae}
\end{figure}


\begin{itemize}
\item Fig. \ref{fig:4-st-tt} shows the time it takes combox to process
  files for a given file dump \footnote{A ``file dump'' here means a
    bunch of files copied to the combox directory between 1 sec
    intervals.}. As can be observed from the graph, the total time
  taken to process all the files tends almost linearly increase with
  the increase in the size of the file dump \footnote{The ``size of
    the file dump'' is the total size of all files in a given file
    dump.}.
\item Fig. \ref{fig:4-st-atsae} show the average time it takes combox
  to split and encrypt a file for a given file dump. There is a steep
  increase in the average time from the \verb+424.79MiB+ dump and the
  \verb+1620.00MiB+ dump, after which the average time to split and
  encrypt a file seems to almost linearly increase. The main reason
  for this is that the average file size for dumps from
  \verb+1620.00MiB+ to \verb+10800.00MiB+ are the same.
\end{itemize}

\begin{figure}[h]
  \centering \input{graphs/tot-time-diff.tex}
  \caption{Difference between 2015 and 2016 tests - time taken to
    process all files.}
  \label{fig:4-st-tt-diff}
\end{figure}

\begin{figure}[h]
  \centering \input{graphs/avg-time-sae-diff.tex}
  \caption{Difference between 2015 and 2016 tests - Avg. time to split
    and encrypt a file.}
  \label{fig:4-st-atsae-diff}
\end{figure}

\begin{itemize}
\item Fig. \ref{fig:4-st-tt-diff} shows the graphs for the total
  amount of time taken to process all files for a given file dump in
  the \verb+2016-01-16+ and \verb+2015-11-8+ stress test. The amount
  of time needed to process all fills seems to be reduced for the
  \verb+5940.00MiB+ file dump when compared to the \verb+2015+ stress
  test results and it seems to be slightly higher for the
  \verb+10800.00MiB+ file dump when compared to the \verb+2015+ stress
  test.
\item Similarly, Fig. \ref{fig:4-st-atsae-diff} shows the graphs for
  the average time to split and encrypt for a given file dump in the
  \verb+2016-01-16+ and the \verb+2015-11-8+ stress test. The average
  time taken seems to be almost the same for the \verb+424.79MiB+ and
  the \verb+1620.00+ dump, but for the \verb+5940.00MiB+ and the
  \verb+10800.00MiB+ dump the average time taken seems to higher for
  the \verb+2016+ stress test when compared to the \verb+2015+ stress
  test.
\end{itemize}

\subsection{Issues found}\label{4-st-if}

\begin{itemize}
\item Initially when combox was stress tested with huge files, combox
  would get overwhelmed leading to the computer running out of memory
  and the load average sometimes peaking at \verb+8+. At first, it was
  assumed that there was a bug in combox which caused this to happen,
  but later it was found that \verb+watchdog+ \cite{pylib:watchdog}
  was generating a large number ``file modified'' events when a huge
  file (\verb+~500MiB+) was modified. To prevent \verb+watchdog+ from
  generating a large number ``file modified'' events for a single
  modification of a huge file, a delay proportional to the size of the
  file was created in the \verb+on_modified+ callback methods in both
  \verb+ComboxDirMonitor+ and \verb+NodeDirMonitor+
  \footnote{https://git.ricketyspace.net/combox/commit?id=7ed3c9cbe6e56223b043a23408474f9df08f119e},
  this fixed the issue.
\end{itemize}