Experiment Data

Experiment Data

The foraging server is a one-shot server process that should be restarted for each experiment run. Each time the foraging server is run it will generate a collection of raw data files in the experiment-data directory - each raw data file corresponds to a given round in the experiment. These files are stored relative to the experiment's server process, for a typical installation this is the root of the downloaded foraging codebase. The raw data files are stored in directories following a month-day-year/time-of-experiment-run format, e.g., foraging/experiment-data/02-16-2018/13.47.14/round-3.save

Raw Data

The raw data comes in two forms, serialized .save files, and an equivalent XStream XML file for sanity checking (NOTE: XStream support may go away soon due to deprecated security / serialization issues). The serialized .save file uses Java's native serialization format to persist two Java objects: the RoundConfiguration representing the configuration parameters for the given experiment, and a ServerDataModel that contains all other experiment data, namely a List<PersistableEvent> that represents all participant actions within the game and the mapping between participants and their randomized groups.

Processing Data

Use the ant convert target to convert the raw data into plaintext csv files and statistics for further analysis. By default the convert target looks in a raw-data directory relative to the root of the codebase. You can change this by passing in a savefile.dir property which should point at a directory containing the month-day-year/time-of-experiment-run subdirectories. A typical data processing workflow is:

• Run an experiment
• After the experiment has concluded, copy experiment data from experiment-data to a location with scheduled redundancy and backups, e.g.,

% cp -rp experiment-data/* /data/foraging/raw
% l /data/foraging/raw
total 40
drwxrwsr-x 10 user user 4096 2010-04-15 16:06 ./
drwxrwsr-x  7 user user 4096 2010-08-05 17:41 ../
drwxrwsr-x  4 user user 4096 2010-03-22 12:44 03-10-2010/
drwxrwsr-x  4 user user 4096 2010-03-22 12:45 03-11-2010/
drwxrwx---  5 user user 4096 2010-03-31 15:19 03-31-2010/
drwxrwx---  3 user user 4096 2010-04-01 11:34 04-01-2010/
drwxrwxr-x  4 user user 4096 2010-04-06 15:51 04-06-2010/
drwxrwx---  4 user user 4096 2010-04-15 14:28 04-07-2010/
drwxrwx---  4 user user 4096 2010-04-14 13:32 04-14-2010/
drwxrwx---  5 user user 4096 2010-04-15 14:25 04-15-2010/
% ant convert -Dsavefile.dir=/data/foraging/raw # converts all the data at /data/foraging/raw
% rsync -avzP /data/foraging/raw --exclude "*.save" --exclude "*.xml" /data/foraging/processed # copy all the generated CSV text files into the /data/foraging/processed directory
% tar zcvf processed-data.tar.gz /data/foraging/processed # create a tarball of all the data files to be analyzed by intrepid data analysts

TODO: describe each converted savefile and statistics.