DDoS mitigation testing

[vankoven]

There are some DDoS simulation tools available to evaluate system behaviour under DDoS attack:

• https://github.com/MatrixTM/MHDDoS (seems like quite an advanced suite)
• https://github.com/shekyan/slowhttptest
• https://github.com/markus-go/bonesi
• https://sourceforge.net/projects/ddosim/
• https://github.com/jseidl/GoldenEye
• https://github.com/palahsu/DDoS-Ripper

It would be nice to test DDoS mitigation effectiveness as a part of stress tests.

[krizhanovsky]

Tempesta emphasises L7 DDoS, which requires large (at least 100k) TCP connections with different source IP addresses. So ideally massive number of virtual IP interfaces are required.

Basically, we can just check correctness of ratelimits with #673 and do similar tests with wrk without rate limiting to estimate the system under the stress.

[krizhanovsky]

DDoS mitigation is a crucial feature, so the task is crucial as well. At least we must be sure that 100-1000 valid clients work just fine with 100-1000 bots sending malicious requests concurrently. We have tests for cookie and JS challenges, but we do not verify them in real life scenario:

• relatively complex web site, e.g. tempesta-tech.com
• concurrent realistic innocent clients with full browser emulation and performance analytics to catch not fully loaded or too slowly loaded pages
• many bot agents and correspondingly many filtering rules on Tempesta FW side
• a comprehensive HTTP rate limit test (linked with Functional test for Frang tempesta#673)

At least these DDoS attacks must be emulated with appropriate HTTP limits in both the modes to block malicious IPs and reply:

• HTTP flood (basic GET / HTTP/1.1\nHost: foo.com\n\n, the same for HTTP/1.0 w/ Host)
• random URLs bypassing cache
• sloworis / slow connections
• many concurrent connections

[krizhanovsky]

To emulate a more or less close to a real life DDoS scenario, we need to run the DDoS agents on multiple network interfaces or interface aliases to assign each of them a designated IP. It seems Route the traffic over specific interface for a process in linux provides a good solution with the script. The script contains a lot of configuration and error handling code, so I'd propose to use it just as a reference.

We can use the loopback network 127.0.0.0/8 or create any big enough local network (I think 1024 addresses would be enough for the testing). Next you need to create the network interfaces or interface aliases. Lastly use network namespaces, netfilter marks and the routing tables as in the script.

[krizhanovsky]

Blocked by tempesta-tech/tempesta#1464

[krizhanovsky]

Now is blocked again on tempesta-tech/tempesta#2000

[RomanBelozerov]

I used the mhddos utility for testing. It is implemented using python3 and has been reworked to match our requirements. Unsupported http versions removed, host, if-none-match headers fixed, etc.

The current work principle:
create multiple interfaces for IP addresses -> set the rpc, duration, threads parameters -> create HttpFlood objects (according to the number of threads) -> each thread chooses (randomly) an attack method (from a predefined list) -> each thread opens a connection with a random IP address -> sends requests according to the logic of a method -> repeating.

Setup for testing:

• TempestaFW is on a separate VM. 8 GB memory and 6 cores;
• The attacker and tempesta-tech.com website (wordpress) in lxc container are on yet one VM;

I checked frang with block_action directives. Results:

1. block_action attack drop and ip_block on are the most effective combination for protection.

   • if the attacker exceeds the limits, then we block this IP and the memory consumption will be small (I got about 100 MB) for both small (5-20 rate) and large limits (100+ rate);
   • if the client does not exceed the limits, then we block IP addresses through other directives. For example: http_resp_code_blocks, http_strict_host_checking, http_ct_vals etc. The memory consumption is higher, but less than 500 MB for me

2. block_action attack reply\drop and ip_block off are not effective for protection, because a lot of memory is consumed. If the attacker exceeds the limits, TempestaFW eventually consumes all available memory and my VM was shutdown. Perhaps this problem will be solved after closing issues 2284 and 2286. It took me 2-5 minutes to use all available resources (depends on the methods).

Getting a response from the cache\upstream:

• responses from the cache were returned without delay under any load and configuration (if there are enough resources on TempestaFW VM)
• upstream responses without ip_block on require a very long time. Our website with wordpress and lxc was used as the upstream. The response was returned up to 30 seconds. Probably lxc did not have enough resources, because the attacker used all available resources.

I added base L7 tests using https://github.com/MatrixTM/MHDDoS in #438. What can be improved:

• L4 tests from this tool;
• checks in L7 for the legitimate client (after fix 2284 and 2286);
• add a compare with Nginx. I used default config and with request/connection limit, but I didn't get a lot of resource consumption. Probably, this tool requires to update for that.
• probably upstream should move to a separate VM. The lack of resources should be excluded
• test on SUT+generator