README.devel-tests

<!-- To convert to html: pandoc -o foo.html README.devel-test -->

About This README
=================

This document describes how to add, extend, or fix individual
rpmgrill test plugins. It is targeted to commando developers:
those who simply want to jump in, write some test code, dash
back out, nobody gets hurt. You must be familiar with Fedora
build processes, with RPM and specfile development (to a degree;
you don't need to be Panu), and be willing to dabble in Perl.
No Perl expertise needed, just the ability to learn by example.

You will need some basic familiarity with rpmgrill and its
invocation; see README.AAA_FIRST.

Overview
========

All plugins live under the `lib/RPM/Grill/Plugin/` subdirectory.
All plugins work the same way:

1. Someone (rpmgrill) invokes the plugin's **->analyze()** method. No args.
   `$self` will be an `RPM::Grill::Plugin::Xxxxx` object, where `Xxxxx`
   is the plugin package name.
2. `analyze()` examines RPMs and invokes **`->gripe()`** when it
   detects a problem.

That's it. `analyze()` then returns; as of this writing (Sept 2012)
its return values is ignored. `analyze()` can even die: the calling code
will trap it and mark the test as failed.

Zeroth Decision: Can You Piggyback?
-----------------------------------

Before you start writing code: does your test fit into any of the
existing plugins? You can save development effort as well as run time
by adding your logic to one of the existing tests. Keep reading, but
keep in mind the possibility of adding on to an existing test.

First Decision: What to Analyze
-------------------------------

To date, most plugins work in one of two ways:

a) check something in the SRPM, e.g. the specfile
b) check something in all (or possibly a subset of) built RPMs.

(there are some exceptions: `BuildLog.pm`, which examines log files
outside of the built RPMs; `VirusCheck.pm` which simply checks files
without knowing or caring about their RPMness. If that's what you need,
go peruse them and skip this section.)

If you're checking the specfile, you'll probably want a loop like:

    for my $line ($self->specfile->lines) { ... }

See `Patches.pm` and `SpecFileEncoding.pm` for examples.

For almost anything else, you'll want to start with a loop such as:

     for my $rpm ($self->rpms) {
         for my $f ($rpm->files) {
             $self->_do_something_with( $f );
         }
     }

See `Setxid.pm` for a simple example of this type of test;
or `Manifest.pm` for a more complex one.

Second Decision: How/What to Gripe
----------------------------------

The heart of rpmgrill is the gripe messages. These are collected
by rpmgrill, and preserved in such a way that another tool can store
into a database for presenting to developers. A minimal call looks like:

      $self->gripe({
          code => 'MyGripeCode',
          diag => "This is a human-readable message complaining about $file",
      });

Those are the crucial elements: `code` is a StudlyCaps error code
unique to this gripe (or, in some special cases, shared among a
very few gripes); `diag` is a human-readable string explaining
the warning and perhaps giving context (e.g. filename).

When applicable, you might want to include `arch` and/or `subpackage`.
These will be included automatically if you use `$rpm->gripe()` or
`$f->gripe()` (instead of `$self`).

Context ... well, I didn't really get it right. I'm sorry. The gist
is that you can use it to specify a filename, or an excerpt from a file
(such as a log file or a bad config entry), and/or a line number:

    $self->gripe({
                  code    => '...', diag => '...',
                  context => {
                       path => '/path/to/file',
                       excerpt => $excerpt,
                  } });

The problem is that sometimes 'path' is a specfile; and that
sometimes you're complaining *about* a path (e.g. 'Fedora in Filename')
and sometimes about something *in* the file. And sometimes you
want to aggregate files together into one message ('The following
files have such-and-such a problem: ...'), sometimes you don't.

Testing
=======

The test scaffolding is clunky. I'm sorry. But you're going to write
regression tests anyway, because that's the kind of person you are.

Each plugin `Xxxxx` has a corresponding subdirectory `t/RPM/Grill/Plugin/Xxxxx`
containing one or more `NNfoo.t` scripts. Each of those does some setup,
invokes the plugin (or perhaps a function inside the plugin), then
checks results. This is complicated because of all the setup needed
by each plugin (populated RPM::Grill object) and because of the
structure of the gripes.

I recommend starting with `t/RPM/Grill/Plugin/ManPages/10basic.t`.
At this writing (September 2012) it's the cleanest test script and
the only one using `t/lib/FakeTree.pm`, a reusable library for
doing the RPM::Grill setup work.