Experiment Data

Experiment Data

The foraging server is a one-shot server process that should be restarted for each experiment run. Each time the foraging server is run it will generate a collection of raw data files in the experiment-data directory, relative to where the experiment's server process was run on the filesystem (for a typical installation this is the root of the codebase you've downloaded). The raw data files themselves are stored in directories following a month-day-year/time-of-experiment-run format, e.g., foraging/experiment-data/02-16-2018/13.47.14/round-3.save

Raw Data

The raw data comes in two forms, a serialized .save file, and an equivalent XStream XML file (NOTE: XStream support may go away soon due to deprecated security / serialization issues). The serialized .save file uses Java's native serialization format to persist two Java objects: the RoundConfiguration representing the configuration parameters for the given experiment, and a ServerDataModel that contains all other experiment data, namely a List<PersistableEvent> that represents all participant actions within the game and the mapping of participants to their randomized groups.

Processing Data

Use the ant convert target to convert the raw data into plaintext csv files and statistics for further analysis. By default the convert target looks in a raw-data directory relative to the root of the codebase. You can change this by passing in a savefile.dir property which should point at a directory containing the month-day-year/time-of-experiment-run subdirectories. A typical data processing workflow is:

• Run an experiment
• After the experiment has concluded, copy experiment data from experiment-data to a location with scheduled redundancy and backups, e.g.,

% cp -rp experiment-data/* /data/foraging/raw
% l /data/foraging/raw
total 40
drwxrwsr-x 10 user user 4096 2010-04-15 16:06 ./
drwxrwsr-x  7 user user 4096 2010-08-05 17:41 ../
drwxrwsr-x  4 user user 4096 2010-03-22 12:44 03-10-2010/
drwxrwsr-x  4 user user 4096 2010-03-22 12:45 03-11-2010/
drwxrwx---  5 user user 4096 2010-03-31 15:19 03-31-2010/
drwxrwx---  3 user user 4096 2010-04-01 11:34 04-01-2010/
drwxrwxr-x  4 user user 4096 2010-04-06 15:51 04-06-2010/
drwxrwx---  4 user user 4096 2010-04-15 14:28 04-07-2010/
drwxrwx---  4 user user 4096 2010-04-14 13:32 04-14-2010/
drwxrwx---  5 user user 4096 2010-04-15 14:25 04-15-2010/
% ant convert -Dsavefile.dir=/data/foraging/raw # converts all the data at /data/foraging/raw
% rsync -avzP /data/foraging/raw --exclude "*.save" --exclude "*.xml" /data/foraging/processed # copy all the generated CSV text files into the /data/foraging/processed directory
% tar zcvf processed-data.tar.gz /data/foraging/processed # create a tarball of all the data files to be analyzed by intrepid data analysts