Stroom Proxy Failed Forwarding

[stroomdev00]

If/when stroom-proxy fails to send data to its onward destination after the configured retry limit, the files are stored within either
/stroomdata-local/stroom-proxy-repo/50_forwarding/stroomViaALB/03_failure or
/stroomdata-local/stroom-proxy-home/failures

There appears to be no sensible way of resubmitting these files when whatever caused the initial failure has been resolved. To allow manual resubmission we need the following to occur:

1. Files to be stored in feed-specific directories for ease of sorting.
2. Files/directories should be able to be moved/copied to another directory manually either individually or in bulk so that they can be resubmitted to the proxy.
3. If file forwarding starts to fail, the retry holdoff period should be automatically increased so that files do not appear in the failed directories too soon.
4. There must be an ability to switch off automatic forwarding attempts for situations such as a planned backend outage or known fault.

[at055612]

Also, if the proxy gets a REJECTED response from the destination then it should delete the sent file and NOT mark it for retry.

The default for retrying should be infinite retries.

[stroomdev66]

stroom.proxy.app.handler.HttpSender is the code responsible for proxy sending data to Stroom.

The rejected response is specific and relates to this:

    FEED_IS_NOT_SET_TO_RECEIVED_DATA(HttpServletResponse.SC_NOT_ACCEPTABLE, 110, "Feed is not set to receive data",
            "The feed you have provided has not been setup to receive data"),

This is an HTTP response code of 406 and a Stroom response code of 110. We need to be sure that we are only deleting data if we get this error response. We also need to be sure that Stroom doesn't provide this error respoonse just because a feed isn't defined.

Note that a feed not defined error should be returned as:

    FEED_IS_NOT_DEFINED(HttpServletResponse.SC_NOT_ACCEPTABLE, 101, "Feed is not defined",
            "The feed you have provided is not known within Stroom"),

Which is the same HTTP response code but a Stroom response code of 101.