Feature Request: Support External Lists for Blacklist/Whitelist Settings

[jg1vxg]

Hi, thanks to maintain this excellent tool.

For blacklist / whitelist setings, is there any way to loading / updating blacklist from external list?
For example, I want to use like this:

blacklist:
    - value1
    - value2
    - "!file /tmp/blacklist1.txt"
    - "!file /tmp/blacklist2.txt"
    - "!external https://raw.githubusercontent.com/Neo23x0/signature-base/master/iocs/c2-iocs.txt"

I can write with another script and cron to make flat txt files, but I want to get rid of number of scripts and summrize in one setting file cause a lot of list is opened.

Thankyou

[jertel]

This isn't supported today but it sounds like an interesting idea. You are welcome to submit a PR following the contribution guidelines. You'd want to consider if the remote file should only be checked on startup, or on a recurring basis. Also consider other validation logic, such as making sure the remote file is valid.

[jg1vxg]

Thanks reply, I thought try to make PRs.
Can I have some comments for these specification?

I'm grad to check remote files both of start up and periodically recurring.

blacklist:
    - "!file /tmp/blacklist2.txt"
    - "!external https://raw.githubusercontent.com/Neo23x0/signature-base/master/iocs/c2-iocs.txt"
        - frequency: 1h

To make code for recurring period, is it good way to use BackgroundScheduler in class ElastAlerter? or somewhere more suitable ?

And validation, I thought it useful to "weak" validation cause It is not surely to indexed correctly Domains/Ipaddrs like RFC1035. So, I think it suitable to aim for validation with the goal to ensuring safety of ES, Elastalert2, and other notify systems e.g., mail.
This is a proposed condition for validation: ^[a-zA-Z0-9./:-]{1,253}$

• [a-zA-Z0-9]
• Ignore symbols except . : / -.
  • These excepted symbols for IPv4, v6, FQDN and port specification.
• Maximum length of each line <= 253
  • RFC1035

Thankyou for your comment.

[jertel]

I think you'll run into complications with tucking the frequency: 1h param under the list item. It would be simpler to introduce a top-level option that applies to all files and remote URLs . Ex:

blacklist_file_refresh: 
  hours: 1

Regarding validation, I was talking about validating the content of the file. Suppose your blacklist URL is critical to be accurate 100% of the time. But if you remotely query the blacklist URL every 1h and occasionally the network fetch fails, what will you do?

[jg1vxg]

Options "blacklist_file_refresh" is so nice, I like it.

If network failed when fetching, I think its robust to keep latest contents of external lists.
So, flow of code,

• if responce.code is not 200 or something error in GET requests
• trying again to fetch several times imediately
• if nevertheless occured errors
• then keep last contents you success fetching, logging, and skipping this pireodically fetchnig.

How does that sound to you?

[jertel]

That sounds good to me.