Version 4.5.0

.n6-version

@@ -1 +1 @@
-4.4.0.post3
+4.5.0

CHANGELOG.md

@@ -1,7 +1,7 @@
 # Changelog
 
 The *[n6](https://n6.readthedocs.io/)* project uses a versioning scheme
-***distinct from** Semantic Versioning*. Each *n6* version's identifier
+**distinct from** *Semantic Versioning*. Each *n6* version's identifier
 consists of three integer numbers, separated with `.` (e.g.: `4.11.2`).
 We can say it is in the `<FOREMOST>.<MAJOR>.<MINOR>` format -- where:
 
@@ -24,6 +24,43 @@ Some features of this document's layout were inspired by
 [Keep a Changelog](https://keepachangelog.com/).
 
 
+## [4.5.0] - 2023-11-29
+
+### Features and Notable Changes
+
+#### System/Configuration/Programming-Only
+
+- [data sources, setup, config, etc/docker, tests] Globally renamed the
+  parser class `SpamhausEdrop202303Parser` to `SpamhausEdropParser`
+  (defined in `n6datasources.parsers.spamhaus` and referred to in a few
+  other places -- in particular, being the name of the-parser-dedicated
+  configuration section!) and the executable script
+  `n6parser_spamhausedrop202303` to `n6parser_spamhausedrop`; also, fixed
+  `n6datasources.tests.parsers.test_spamhaus.TestSpamhausEdropParser` by
+  removing its attribute `PARSER_RAW_FORMAT_VERSION_TAG`. The rationale
+  for these changes is that the actual parser has never had any *raw
+  format version tag* assigned.
+
+### Less Notable Changes and Fixes
+
+- [data pipeline, lib] `n6filter`: fixed an
+  *event-ownership-criteria*-related bug (in
+  `n6lib.auth_api.InsideCriteriaResolver`'s machinery...) regarding
+  handling the (practically hardly probable yet not impossible) corner
+  case of the `0.0.0.0/32` IP network being set in Auth DB as such a
+  criterion... The bug might make `n6filter` reject all incoming data
+  (because of raised exceptions).
+
+- [tests, docs] Made non-major enhancements and fixes regarding some unit
+  tests and documentation.
+
+#### Programming-Only
+
+- [tests] `n6datasources.tests.parsers._parser_test_mixin`: enhanced
+  certain `ParserTestMixin`-provided checks related to *raw format
+  version tags*.
+
+
 ## [4.4.0] - 2023-11-23
 
 ### Features and Notable Changes
@@ -168,7 +205,7 @@ Some features of this document's layout were inspired by
 
 #### System/Configuration/Programming-Only
 
-- [data sources, data pipeline, config, docker/etc] Added, fixed, changed
+- [data sources, data pipeline, config, etc/docker] Added, fixed, changed
   and removed several config prototype (`*.conf`) files in the
   directories: `N6DataSources/n6datasources/data/conf/`,
   `N6DataPipeline/n6datapipeline/data/conf/` and `etc/n6/`. *Note:* for
@@ -208,7 +245,7 @@ Some features of this document's layout were inspired by
   behavior...
 
 - [tests] `n6datasources.tests.parsers._parser_test_mixin.ParserTestMixin`
-  (and all inheriting *parser* test classes): added checking that if the
+  (and all inheriting *parser test* classes): added checking that if the
   *parser*'s `default_binding_key` includes the *raw format version tag*
   segment then that segment matches the test class's attribute
   `PARSER_RAW_FORMAT_VERSION_TAG`.
@@ -228,10 +265,11 @@ Some features of this document's layout were inspired by
   the hostname being a domain name (to become `fqdn`) or an IP address (to
   become `ip` in `address`...) from `url`. Those bugs caused that, for
   certain (rather uncommon) cases of malformed or untypical URLs, whole
-  events were rejected, or (*only* for some cases and *only* if the
-  Python's *assertion-removal optimization* mode was in effect) the
-  resultant event's `enriched` field erroneously included the `"fqdn"`
-  marker whereas `fqdn` was *not* successfully extracted from `url`.
+  events were rejected (because of an exception), or (*only* for some
+  cases and *only* if the Python's *assertion-removal optimization* mode
+  was in effect) the resultant event's `enriched` field erroneously
+  included the `"fqdn"` marker whereas `fqdn` was *not* successfully
+  extracted from `url`.
 
 - [data pipeline] Fixed `n6anonymizer`: now output bodies
   produced by the `_get_result_dicts_and_output_body()` method
@@ -255,10 +293,10 @@ Some features of this document's layout were inspired by
   *organization-specific* results into the existing data set (broadening
   the overall search capabilities).
 
-- [docker/etc] Replaced expired test/example certificates.
+- [etc/docker] Replaced expired test/example certificates.
 
 - [data sources, data pipeline, portal, setup, config, cli, lib, tests,
-  docker/etc, docs] Various additions, fixes, changes, enhancements as
+  etc/docker, docs] Various additions, fixes, changes, enhancements as
   well as some cleanups and code modernization/refactoring.
 
 #### Programming-Only

N6DataPipeline/n6datapipeline/tests/test_filter.py

@@ -579,7 +579,7 @@ def test__get_client_and_urls_matched__empty_fileds_asn_ip_cc_fqdn_address(self)
                                 'cc_seq': ["RU", "DE", "US"],
                                 'asn_seq': [4235],
                                 'fqdn_seq': [u"nask.pl", u"cert.pl"],
-                                'ip_min_max_seq': [(0, 4194303),
+                                'ip_min_max_seq': [(1, 4194303),
                                                    (4294901760, 4294901792)], }]
 
         body = {"category": "bots", "restriction": "public", "confidence": "medium",

N6DataSources/console_scripts

@@ -184,7 +184,7 @@ n6parser_shadowserverxdmcp201412 = n6datasources.parsers.shadowserver:Shadowserv
 ## spamhaus.*
 n6parser_spamhausbots = n6datasources.parsers.spamhaus:SpamhausBotsParser_main
 n6parser_spamhausdrop = n6datasources.parsers.spamhaus:SpamhausDropParser_main
-n6parser_spamhausedrop202303 = n6datasources.parsers.spamhaus:SpamhausEdrop202303Parser_main
+n6parser_spamhausedrop = n6datasources.parsers.spamhaus:SpamhausEdropParser_main
 
 ## spam404_com.*
 n6parser_spam404comscamlist = n6datasources.parsers.spam404_com:Spam404ComScamListParser_main

N6DataSources/n6datasources/data/conf/60_spamhaus.conf

@@ -25,5 +25,5 @@ prefetch_count = 1
 [SpamhausDropParser]
 prefetch_count = 1
 
-[SpamhausEdrop202303Parser]
+[SpamhausEdropParser]
 prefetch_count = 1

N6DataSources/n6datasources/parsers/spamhaus.py

@@ -79,7 +79,7 @@ class SpamhausDropParser(_BaseSpamhausBlacklistParser):
     default_binding_key = 'spamhaus.drop'
 
 
-class SpamhausEdrop202303Parser(_BaseSpamhausBlacklistParser):
+class SpamhausEdropParser(_BaseSpamhausBlacklistParser):
 
     default_binding_key = 'spamhaus.edrop'
 

N6DataSources/n6datasources/tests/parsers/_parser_test_mixin.py

@@ -73,20 +73,24 @@ class ParserTestMixin(TestCaseMixin):
 
     def test_basics(self):
         self.assertIn(self.PARSER_BASE_CLASS, self.PARSER_CLASS.__bases__)
-        splitted_default_binding_key = self.PARSER_CLASS.default_binding_key.split(".")
-        source_from_default_binding_key = '.'.join(splitted_default_binding_key[:2])
+        split_default_binding_key = self.PARSER_CLASS.default_binding_key.split('.')
+        source_from_default_binding_key = '.'.join(split_default_binding_key[:2])
         self.assertEqual(source_from_default_binding_key,
                          self.PARSER_SOURCE)
         self.assertEqual(self.PARSER_CLASS.constant_items,
                          self.PARSER_CONSTANT_ITEMS)
-        # Check if default_binding_key have version tag in it
-        # default binding key looks like this:
-        # - <source provider>.<source channel>.<RAW_FORMAT_VERSION_TAG> e.g. 'abuse-ch.feodotracker.202110'
-        # - with RAW_FORMAT_VERSION_TAG only present when there is more than one version of given parser
-        if len(splitted_default_binding_key) == 3:
-            # assert that PARSER_RAW_FORMAT_VERSION_TAG is correct
-            self.assertEqual(self.PARSER_RAW_FORMAT_VERSION_TAG,
-                             splitted_default_binding_key[2])
+        if self.PARSER_RAW_FORMAT_VERSION_TAG is sentinel.not_set:
+            # Parser's `default_binding_key`
+            # is just equal to `PARSER_SOURCE`
+            # (e.g., "abuse-ch.feodotracker")
+            self.assertEqual(len(split_default_binding_key), 2)
+        else:
+            assert re.fullmatch(r'[0-9]{6}', self.PARSER_RAW_FORMAT_VERSION_TAG)  # (e.g. "202110")
+            # Parser's `default_binding_key` consists of
+            # `PARSER_SOURCE` and `PARSER_RAW_FORMAT_VERSION_TAG`
+            # (e.g., "abuse-ch.feodotracker.202110")
+            self.assertEqual(len(split_default_binding_key), 3)
+            self.assertEqual(split_default_binding_key[2], self.PARSER_RAW_FORMAT_VERSION_TAG)
 
     def test__input_callback(self):
         # (We want to make `LegacyQueuedBase.__new__()`'s stuff isolated

N6DataSources/n6datasources/tests/parsers/test_spamhaus.py

@@ -8,7 +8,7 @@
 from n6datasources.parsers.spamhaus import (
     SpamhausBotsParser,
     SpamhausDropParser,
-    SpamhausEdrop202303Parser,
+    SpamhausEdropParser,
     _BaseSpamhausBlacklistParser,
 )
 from n6lib.datetime_helpers import parse_iso_datetime_to_utc
@@ -60,13 +60,12 @@ def cases(self):
         )
 
 
-class TestSpamhausEdrop202303Parser(ParserTestMixin, unittest.TestCase):
+class TestSpamhausEdropParser(ParserTestMixin, unittest.TestCase):
 
     RECORD_DICT_CLASS = BLRecordDict
 
     PARSER_SOURCE = 'spamhaus.edrop'
-    PARSER_RAW_FORMAT_VERSION_TAG = '202303'
-    PARSER_CLASS = SpamhausEdrop202303Parser
+    PARSER_CLASS = SpamhausEdropParser
     PARSER_BASE_CLASS = _BaseSpamhausBlacklistParser
     PARSER_CONSTANT_ITEMS = {
         'restriction': 'public',

N6Lib/n6lib/auth_api.py

@@ -1003,6 +1003,10 @@ def __init__(self, inside_criteria):
 
             # IPs
             for min_ip, max_ip in cri.get('ip_min_max_seq', ()):
+                assert min_ip >= 1
+                if (min_ip, max_ip) == (1, 0):
+                    # (corner case related to exclusion of 0, see: #8861...)
+                    continue
                 assert min_ip <= max_ip
                 ip_to_id_endpoints[min_ip].append((org_id, True))
                 ip_to_id_endpoints[max_ip + 1].append((org_id, False))

N6Lib/n6lib/tests/auth_related_quicktest.py

@@ -585,6 +585,7 @@ def data_maker_for____TestAuthAPI___get_inside_criteria(session):
                          models.InsideFilterFQDN(fqdn='example.com'),
                          models.InsideFilterFQDN(fqdn='xyz.example.net')],
                      inside_filter_ip_networks=[
+                         models.InsideFilterIPNetwork(ip_network='0.0.0.0/32'),
                          models.InsideFilterIPNetwork(ip_network='0.10.20.30/8'),
                          models.InsideFilterIPNetwork(ip_network='1.2.3.4/16'),
                          models.InsideFilterIPNetwork(ip_network='101.102.103.104/32')],

N6Lib/n6lib/tests/test_auth_api.py

@@ -850,7 +850,12 @@ class TestAuthAPI___get_inside_criteria(_AuthAPILdapDataBasedMethodTestMixIn,
                     'n6asn': ['12', '34'],
                     'n6cc': ['PL', 'US'],
                     'n6fqdn': ['example.com', 'xyz.example.net'],
-                    'n6ip-network': ['0.10.20.30/8', '1.2.3.4/16', '101.102.103.104/32'],
+                    'n6ip-network': [
+                        '0.0.0.0/32',
+                        '0.10.20.30/8',
+                        '1.2.3.4/16',
+                        '101.102.103.104/32',
+                    ],
                     'n6url': ['example.info', 'institution.example.pl/auth.php', 'Łódź'],
                 }),
                 ('o=o2,ou=orgs,dc=n6,dc=cert,dc=pl', {
@@ -871,6 +876,7 @@ class TestAuthAPI___get_inside_criteria(_AuthAPILdapDataBasedMethodTestMixIn,
                     'cc_seq': ['PL', 'US'],
                     'fqdn_seq': ['example.com', 'xyz.example.net'],
                     'ip_min_max_seq': [
+                        (1, 0),         # <- Note: here the minimum IP is 1, not 0 (see: #8861).
                         (1, 16777215),  # <- Note: here the minimum IP is 1, not 0 (see: #8861).
                         (16908288, 16973823),
                         (1701209960, 1701209960),
@@ -2210,6 +2216,12 @@ def test(self):
             (1, 1),
         ],
     },
+    {
+        'org_id': 'o43',
+        'ip_min_max_seq': [
+            (1, 0),  # (<- corner case: result of exclusion of 0, see: #8861...)
+        ],
+    },
 ]
 
 EXPECTED_RESULTS_AND_GIVEN_IP_SETS_FOR_EXTREME_IP_CRITERIA = [
@@ -2981,6 +2993,27 @@ def test___ids_and_urls(self, inside_criteria, expected_content):
             ),
         ),
 
+        param(
+            inside_criteria=[
+                {
+                    'org_id': 'o43',
+                    'ip_min_max_seq': [
+                        (1, 0),  # (<- corner case: result of exclusion of 0, see: #8861...)
+                    ],
+                },
+            ],
+            expected_content=(
+                [
+                    -1,           # guard item
+                    2 ** 32,      # guard item
+                ],
+                [
+                    frozenset(),  # guard item
+                    frozenset(),  # guard item
+                ],
+            ),
+        ),
+
         param(
             inside_criteria=EXTREME_IP_CRITERIA,
             expected_content=(

N6Lib/n6lib/tests/test_auth_db_validation.py

@@ -31,6 +31,7 @@
     EntityContactPointPhone,
     EntityExtraId,
     EntityExtraIdType,
+    EntityIPNetwork,
     EntitySector,
     InsideFilterASN,
     InsideFilterCC,
@@ -352,6 +353,7 @@ def test_illegal_type_notification_time(self):
         OrgConfigUpdateRequestIPNetwork,
         RegistrationRequestIPNetwork,
         CriteriaIPNetwork,
+        EntityIPNetwork,
     )
     def test_ip_network(self, model, val):
         self._test_proper_values(model, {'ip_network': val}, expecting_stripped_string=True)
@@ -372,6 +374,7 @@ def test_ip_network(self, model, val):
         OrgConfigUpdateRequestIPNetwork,
         RegistrationRequestIPNetwork,
         CriteriaIPNetwork,
+        EntityIPNetwork,
     )
     def test_illegal_ip_network(self, model, val):
         expected_msg_pattern = r'\bnot a valid CIDR IPv4 network specification\b'
@@ -385,6 +388,7 @@ def test_illegal_ip_network(self, model, val):
         OrgConfigUpdateRequestIPNetwork,
         RegistrationRequestIPNetwork,
         CriteriaIPNetwork,
+        EntityIPNetwork,
     )
     def test_wrong_type_ip_network(self, model):
         val = ('127.0.0.1', '28')
@@ -872,7 +876,7 @@ def test_setting_ldap_not_safe_chars(self, model, tested_arg):
     )
     def test_string_based_fields_empty_or_whitespace_only_to_null(
             self, model, tested_arg, val):
-        # (note that same of these fields are NOT NULLABLE so "in the
+        # (note that some of these fields are NOT NULLABLE so "in the
         # real life" their NULL values will **not** be accepted on the
         # database level)
         obj = model(**{tested_arg: val})
@@ -933,6 +937,7 @@ def test_string_based_fields_empty_to_null(
         param(model=OrgConfigUpdateRequestIPNetwork, tested_arg='ip_network'),
         param(model=RegistrationRequestIPNetwork, tested_arg='ip_network'),
         param(model=CriteriaIPNetwork, tested_arg='ip_network'),
+        param(model=EntityIPNetwork, tested_arg='ip_network'),
         param(model=InsideFilterURL, tested_arg='url'),
         param(model=Source, tested_arg='source_id'),
         param(model=Source, tested_arg='anonymized_source_id'),

test_do_setup.py

@@ -1,6 +1,11 @@
 #!/usr/bin/env python
 
-# Copyright (c) 2013-2022 NASK. All rights reserved.
+# Copyright (c) 2013-2023 NASK. All rights reserved.
+
+"""
+Tests for the `do_setup.py` script.
+"""
+
 
 import argparse
 try:
@@ -25,7 +30,7 @@
 
 
 #
-# Some helpers (here we don't use any external libs, such as mock...)
+# Some helpers (note: here, for historical reasons, we don't use `unittest.mock`)
 
 class Case(namedtuple('Case', 'input, expected, py')):
     def __new__(cls, input, expected, py=None):
@@ -43,6 +48,7 @@ def copy_with(self, **kwargs):
 
 
 def using_template_and_cases(cls):
+
     """
     A class decorator that generates test methods and ads them to the class...
     """