Nwi 5 (#503)

* Remove RouteAutnumAuthentication and related logic.

* Fix one broken integration test.

* Fix some of the integration tests.

* Remove irrelevant integration tests.

* Out of region validations (#487)

* Added -NONAUTH source and validations for out of region objects.

* Do not allow objects other than autnum, route, route6 to be created with nonauth source.

* Added tests for out of region autnum creation.
Implemented (incorrect) source attribute warning/replace for in/out of region objects.

* Added tests for out of region autnum modification.

* Added support and tests for out of region routes.

* Use correct attribute to get route prefix.

* Added tests for modifying out of region routes.

* Added tests for creating out of region route6 objects.

* Added tests for modifying out of region route6 objects.

* Test cleanup.

* Fixed tests.

* Authoritative resource test data can now be specified per testcase.

* Fixed integration tests, removed tests related to pending authentication.

* Fixed integration tests.

* Process AttributeGenerators in a predictable order.

* Fixed tests.

* Moved out of region object detection for route(6) into AuthoritativeResource.

* Added specific method for checking if route(6) is out of region so GRS import behaviour remains the same.

* Added validation for not accepting reserved AS numbers as route origi… (#491)

* Added -NONAUTH source and validations for out of region objects.

* Do not allow objects other than autnum, route, route6 to be created with nonauth source.

* Added tests for out of region autnum creation.
Implemented (incorrect) source attribute warning/replace for in/out of region objects.

* Added tests for out of region autnum modification.

* Added support and tests for out of region routes.

* Use correct attribute to get route prefix.

* Added tests for modifying out of region routes.

* Added tests for creating out of region route6 objects.

* Added tests for modifying out of region route6 objects.

* Test cleanup.

* Fixed tests.

* Authoritative resource test data can now be specified per testcase.

* Added validation for not accepting reserved AS numbers as route origin and warning about AS numbers not present in the database.

* Use correct ObjectType for ipv6.

* Only check autnum existence in database when it is in region.

* Db 1581 query nonauth (#489)

* Added -NONAUTH source and validations for out of region objects.

* Do not allow objects other than autnum, route, route6 to be created with nonauth source.

* Added tests for out of region autnum creation.
Implemented (incorrect) source attribute warning/replace for in/out of region objects.

* Added tests for out of region autnum modification.

* Added support and tests for out of region routes.

* Use correct attribute to get route prefix.

* Added tests for modifying out of region routes.

* query support for RIPE-NONAUTH

* Added tests for creating out of region route6 objects.

* Added tests for modifying out of region route6 objects.

* to prevent null point exception

* Test cleanup.

* Fixed tests.

* Added groovy test and small fix

* Authoritative resource test data can now be specified per testcase.

* fixed e2e test and refactored NonAuthSourceDecorator

* Removed not needed field

* Changes according to PR comments

* Renamed ripeAuthAndNonauthSources field

* Fixed test.

* DB-1581 resources flag added into decorator

* DB-1581 resources flag

* Added end-to-end tests for querying out of region objects.

* Filter objects in search based on source.

* Only perform source filtering for nonauth objects on the main (non grs) source.
Don't filter related objects even when requesting a specific source that does not match
the source of the related objects.

* Cleanup unused code.

* Db 1581 lookup (#495)

* Added -NONAUTH source and validations for out of region objects.

* Do not allow objects other than autnum, route, route6 to be created with nonauth source.

* Added tests for out of region autnum creation.
Implemented (incorrect) source attribute warning/replace for in/out of region objects.

* Added tests for out of region autnum modification.

* Added support and tests for out of region routes.

* Use correct attribute to get route prefix.

* Added tests for modifying out of region routes.

* query support for RIPE-NONAUTH

* Added tests for creating out of region route6 objects.

* Added tests for modifying out of region route6 objects.

* to prevent null point exception

* Test cleanup.

* Fixed tests.

* Added groovy test and small fix

* Authoritative resource test data can now be specified per testcase.

* fixed e2e test and refactored NonAuthSourceDecorator

* Removed not needed field

* Changes according to PR comments

* Renamed ripeAuthAndNonauthSources field

* Fixed test.

* DB-1581 resources flag added into decorator

* DB-1581 resources flag

* Added end-to-end tests for querying out of region objects.

* redirect (see other) for nonauth requests made to default source

* Change from 303 to 301

* refactor and fix tests

* missing changes

* Fix tests

* Add redirect to delete - failing tests

* Fix tests

* Fixed merge error.

* Support redirect for autnum objects when it is out of region.

* Add redirect to PUT

* Add in region redirect

* Fixed whois json objects to return correct lookup links.

* Fixed whois json objects to return correct lookup links.

* Remove mnt-routes from autnum as it's not longer needed. (#494)

* Remove mnt-routes from autnum as it's not longer needed.

* Filter mnt-routes from aut-num updates, add a warning but don't fail the update.

* Skip validation of AUT-NUM mnt-routes (will be filtered out).

* Added tests for filtering out mnt-routes from aut-num updates.

* Updated template test.

* Removed deprecated mnt-routes from aut-num objects in tests.

* Nwi 5 autnum notification (#493)

* Notify the origin aut-num holder on route or route6 creation.

* Removed comment

* Added tests

* Various rest fixes.

* Retain request parameters when redirecting rest requests that specify the wrong source.
Fixed some tests.

* Overriding source attribute value is no longer possible, value is now always determined by whois.

* DB-1746 Added integration test for fultextsearch and nonauth objects (#500)

* Nrtm filtering sources (#502)

* [maven-release-plugin] prepare release whois-1.92

* [maven-release-plugin] prepare for next development iteration

* Test that comment separator(s) are not returned by the Whois REST API.

* Added nonauth source to NRTM.

* DB-1776 Filter objects based on source (don't show RIPE-NONAUTH updates for RIPE and vice versa)

* nrtm - I think we need value from property file

* nrtm test fix 🧐

* Fixed tests.

* Also filter on source for nrtm for objects other than autnum, route(6).

whois-api/src/main/java/net/ripe/db/whois/api/rest/WhoisRestService.java

@@ -21,6 +21,7 @@
 import net.ripe.db.whois.update.log.LoggerContext;
 import net.ripe.db.whois.update.sso.SsoTranslator;
 import org.apache.commons.collections.CollectionUtils;
+import org.apache.commons.lang.StringUtils;
 import org.springframework.beans.factory.annotation.Autowired;
 import org.springframework.stereotype.Component;
 
@@ -104,11 +105,11 @@ public Response delete(
             final UpdateContext updateContext = updatePerformer.initContext(origin, crowdTokenKey);
 
             if(requiresNonauthRedirect(source, objectType, key)) {
-                return redirect(request.getServletPath(), sourceContext.getNonauthSource().getName().toString(), objectType, key);
+                return redirect(request.getServletPath(), sourceContext.getNonauthSource().getName().toString(), objectType, key, request.getQueryString());
             }
 
             if(requiresRipeRedirect(source, objectType, key)) {
-                return redirect(request.getServletPath(), sourceContext.getWhoisMasterSource().getName().toString(), objectType, key);
+                return redirect(request.getServletPath(), sourceContext.getWhoisMasterSource().getName().toString(), objectType, key, request.getQueryString());
             }
 
             auditlogRequest(request);
@@ -163,11 +164,11 @@ public Response update(
             final UpdateContext updateContext = updatePerformer.initContext(origin, crowdTokenKey);
 
             if(requiresNonauthRedirect(source, objectType, key)) {
-                return redirect(request.getServletPath(), sourceContext.getNonauthSource().getName().toString(), objectType, key);
+                return redirect(request.getServletPath(), sourceContext.getNonauthSource().getName().toString(), objectType, key, request.getQueryString());
             }
 
             if(requiresRipeRedirect(source, objectType, key)) {
-                return redirect(request.getServletPath(), sourceContext.getWhoisMasterSource().getName().toString(), objectType, key);
+                return redirect(request.getServletPath(), sourceContext.getWhoisMasterSource().getName().toString(), objectType, key, request.getQueryString());
             }
 
             auditlogRequest(request);
@@ -285,11 +286,11 @@ public Response lookup(
         }
 
         if(requiresNonauthRedirect(source, objectType, key)) {
-            return redirect(request.getServletPath(), sourceContext.getNonauthSource().getName().toString(), objectType, key);
+            return redirect(request.getServletPath(), sourceContext.getNonauthSource().getName().toString(), objectType, key, request.getQueryString());
         }
 
         if(requiresRipeRedirect(source, objectType, key)) {
-            return redirect(request.getServletPath(), sourceContext.getWhoisMasterSource().getName().toString(), objectType, key);
+            return redirect(request.getServletPath(), sourceContext.getWhoisMasterSource().getName().toString(), objectType, key, request.getQueryString());
         }
 
         final QueryBuilder queryBuilder = new QueryBuilder().
@@ -352,11 +353,11 @@ private boolean requiresRipeRedirect(final String source, final String objectTyp
         return false;
     }
 
-    private Response redirect(final String context, final String source, final String objectType, final String pkey) {
-        final URI uri = URI.create(String.format("%s/%s/%s/%s",
-                context,
-                source,
-                objectType, pkey));
+    private Response redirect(final String context, final String source, final String objectType, final String pkey, String queryString) {
+        final URI uri = StringUtils.isBlank(queryString)?
+            URI.create(String.format("%s/%s/%s/%s", context, source, objectType, pkey)) :
+            URI.create(String.format("%s/%s/%s/%s", context, source, objectType, pkey) + "?" + queryString);
+
         return Response.status(Response.Status.MOVED_PERMANENTLY).location(uri).build();
 
     }

whois-api/src/test/java/net/ripe/db/whois/api/fulltextsearch/FullTextSearchTestIntegration.java

@@ -755,6 +755,165 @@ public void search_quote_escaping_json() {
         assertThat(searchResponse.getLsts().get(1).getLsts().get(0).getArrs().get(2).getStr().getValue(), is("\"<b>DEV</b> mntner\""));
     }
 
+    @Test
+    public void basic_search_nonauth_aut_num_object() throws Exception {
+        databaseHelper.addObject(
+                "aut-num: AS101\n" +
+                        "as-name: End-User-1\n" +
+                        "source: RIPE-NONAUTH\n");
+        databaseHelper.addObject(
+                "aut-num: AS102\n" +
+                        "as-name: End-User-2\n" +
+                        "source: RIPE-NONAUTH\n");
+        fullTextIndex.rebuild();
+
+        final QueryResponse queryResponse = query("q=AS101");
+
+        assertThat(queryResponse.getStatus(), is(0));
+        assertThat(queryResponse.getResults().getNumFound(), is(1L));
+        assertThat(queryResponse.getResults(), hasSize(1));
+    }
+
+    @Test
+    public void basic_search_nonauth_route_object() throws Exception {
+        databaseHelper.addObject(
+                "aut-num: AS101\n" +
+                        "as-name: End-User-1\n" +
+                        "source: RIPE-NONAUTH\n");
+        databaseHelper.addObject(
+                "aut-num: AS102\n" +
+                        "as-name: End-User-2\n" +
+                        "source: RIPE-NONAUTH\n");
+        databaseHelper.addObject(
+                "route: 211.43.192.0/19\n" +
+                        "origin: AS101\n" +
+                        "source: RIPE-NONAUTH\n");
+        fullTextIndex.rebuild();
+
+        final QueryResponse queryResponse = query("q=211.43.192.0/19");
+
+        assertThat(queryResponse.getStatus(), is(0));
+        assertThat(queryResponse.getResults().getNumFound(), is(1L));
+        assertThat(queryResponse.getResults(), hasSize(1));
+    }
+
+    @Test
+    public void basic_search_found_all_objects_with_nonauth_value_in_attribute() throws Exception {
+        databaseHelper.addObject(
+                "aut-num: AS101\n" +
+                        "as-name: End-User-1\n" +
+                        "descr: RIPE-NONAUTH\n" +
+                        "source: RIPE-NONAUTH\n");
+        databaseHelper.addObject(
+                "aut-num: AS102\n" +
+                        "as-name: End-User-2\n" +
+                        "descr: RIPE-NONAUTH\n" +
+                        "source: RIPE-NONAUTH\n");
+        databaseHelper.addObject(
+                "route: 211.43.192.0/19\n" +
+                        "origin: AS101\n" +
+                        "descr: RIPE-NONAUTH\n" +
+                        "source: RIPE-NONAUTH\n");
+        fullTextIndex.rebuild();
+
+        final QueryResponse queryResponse = query("q=RIPE-NONAUTH");
+
+        assertThat(queryResponse.getStatus(), is(0));
+        assertThat(queryResponse.getResults().getNumFound(), is(3L));
+        assertThat(queryResponse.getResults(), hasSize(3));
+    }
+
+    @Test
+    public void basic_search_ignore_source_attribute() throws Exception {
+        databaseHelper.addObject(
+                "aut-num: AS101\n" +
+                        "as-name: End-User-1\n" +
+                        "source: RIPE-NONAUTH\n");
+        databaseHelper.addObject(
+                "aut-num: AS102\n" +
+                        "as-name: End-User-2\n" +
+                        "source: RIPE-NONAUTH\n");
+        databaseHelper.addObject(
+                "route: 211.43.192.0/19\n" +
+                        "origin: AS101\n" +
+                        "source: RIPE-NONAUTH\n");
+        fullTextIndex.rebuild();
+
+        final QueryResponse queryResponse = query("q=RIPE-NONAUTH");
+
+        assertThat(queryResponse.getStatus(), is(0));
+        assertThat(queryResponse.getResults().getNumFound(), is(0L));
+        assertThat(queryResponse.getResults(), hasSize(0));
+    }
+
+    @Test
+    public void advanced_search_nonauth_aut_num_objects() throws Exception {
+        databaseHelper.addObject(
+                "aut-num: AS101\n" +
+                        "as-name: End-User-1\n" +
+                        "descr: RIPE-NONAUTH\n" +
+                        "source: RIPE\n");
+        databaseHelper.addObject(
+                "aut-num: AS102\n" +
+                        "as-name: End-User-2\n" +
+                        "descr: RIPE-NONAUTH\n" +
+                        "source: RIPE-NONAUTH\n");
+        fullTextIndex.rebuild();
+
+        final QueryResponse queryResponse = query("q=(descr:(RIPE%5C-NONAUTH))+AND+(object-type:aut-num)");
+
+        assertThat(queryResponse.getStatus(), is(0));
+        assertThat(queryResponse.getResults().getNumFound(), is(2L));
+        assertThat(queryResponse.getResults(), hasSize(2));
+    }
+
+    @Test
+    public void advanced_search_aut_num_route_with_same_descr() throws Exception {
+        databaseHelper.addObject(
+                "aut-num: AS101\n" +
+                        "as-name: End-User-1\n" +
+                        "descr: RIPE-NONAUTH\n" +
+                        "source: RIPE\n");
+        databaseHelper.addObject(
+                "aut-num: AS102\n" +
+                        "as-name: End-User-2\n" +
+                        "descr: RIPE-NONAUTH\n" +
+                        "source: RIPE-NONAUTH\n");
+        databaseHelper.addObject(
+                "route: 211.43.192.0/19\n" +
+                        "origin: AS101\n" +
+                        "descr: RIPE-NONAUTH\n" +
+                        "source: RIPE-NONAUTH\n");
+        fullTextIndex.rebuild();
+
+        final QueryResponse queryResponse = query("q=(descr:(RIPE%5C-NONAUTH))+AND+(object-type:aut-num+OR+object-type:route)");
+
+        assertThat(queryResponse.getStatus(), is(0));
+        assertThat(queryResponse.getResults().getNumFound(), is(3L));
+        assertThat(queryResponse.getResults(), hasSize(3));
+    }
+
+    @Test
+    public void advanced_search_nonauth_ignore_source() throws Exception {
+        databaseHelper.addObject(
+                "aut-num: AS101\n" +
+                        "as-name: End-User-1\n" +
+                        "descr: RIPE-NONAUTH\n" +
+                        "source: RIPE-NONAUTH\n");
+        databaseHelper.addObject(
+                "aut-num: AS102\n" +
+                        "as-name: End-User-2\n" +
+                        "descr: RIPE-NONAUTH\n" +
+                        "source: RIPE-NONAUTH\n");
+        fullTextIndex.rebuild();
+
+        final QueryResponse queryResponse = query("q=(source:(RIPE%5C-NONAUTH))+AND+(object-type:aut-num)");
+
+        assertThat(queryResponse.getStatus(), is(0));
+        assertThat(queryResponse.getResults().getNumFound(), is(0L));
+        assertThat(queryResponse.getResults(), hasSize(0));
+    }
+
     // helper methods
 
     private QueryResponse query(final String queryString) {

whois-api/src/test/java/net/ripe/db/whois/api/rest/RestClientTestIntegration.java

@@ -12,6 +12,7 @@
 import net.ripe.db.whois.common.rpsl.RpslObject;
 import net.ripe.db.whois.common.rpsl.RpslObjectBuilder;
 import net.ripe.db.whois.query.QueryFlag;
+import org.apache.commons.httpclient.HttpStatus;
 import org.joda.time.LocalDateTime;
 import org.junit.Before;
 import org.junit.Test;
@@ -24,6 +25,9 @@
 import java.util.Collection;
 import java.util.Iterator;
 
+import static net.ripe.db.whois.common.domain.CIString.ciString;
+import static net.ripe.db.whois.common.rpsl.AttributeType.DESCR;
+import static net.ripe.db.whois.common.rpsl.AttributeType.SOURCE;
 import static org.hamcrest.CoreMatchers.hasItem;
 import static org.hamcrest.CoreMatchers.hasItems;
 import static org.hamcrest.CoreMatchers.not;
@@ -157,8 +161,8 @@ public void lookup_not_found() throws Exception {
     public void lookup_with_wrong_password() throws Exception {
         final RpslObject object = restClient.request().lookup(ObjectType.MNTNER, OWNER_MNT.getKey().toString());
 
-        assertThat(object.getValueForAttribute(AttributeType.SOURCE).toString(), is("TEST"));
-        assertThat(object.findAttribute(AttributeType.SOURCE).getCleanComment(), is("Filtered"));
+        assertThat(object.getValueForAttribute(SOURCE).toString(), is("TEST"));
+        assertThat(object.findAttribute(SOURCE).getCleanComment(), is("Filtered"));
         assertThat(object.getValueForAttribute(AttributeType.AUTH).toString(), is("MD5-PW"));
         assertThat(object.findAttribute(AttributeType.AUTH).getCleanComment(), is("Filtered"));
     }
@@ -171,17 +175,17 @@ public void lookup_with_correct_password() throws Exception {
 
         assertThat(object.findAttribute(AttributeType.AUTH).getValue(), is("MD5-PW $1$d9fKeTr2$Si7YudNf4rUGmR71n/cqk/ # test"));
         assertThat(object.findAttribute(AttributeType.AUTH).getCleanComment(), is("test"));
-        assertThat(object.findAttribute(AttributeType.SOURCE).getCleanComment(), not(is("Filtered")));
-        assertThat(object.findAttribute(AttributeType.SOURCE).getCleanComment(), is(nullValue()));
+        assertThat(object.findAttribute(SOURCE).getCleanComment(), not(is("Filtered")));
+        assertThat(object.findAttribute(SOURCE).getCleanComment(), is(nullValue()));
     }
 
     @Test
     public void lookup_person_is_unfiltered() throws Exception {
         final RpslObject object = restClient.request()
                 .lookup(ObjectType.PERSON, TEST_PERSON.getKey().toString());
 
-        assertThat(object.findAttribute(AttributeType.SOURCE).getCleanComment(), not(is("Filtered")));
-        assertThat(object.findAttribute(AttributeType.SOURCE).getCleanComment(), is(nullValue()));
+        assertThat(object.findAttribute(SOURCE).getCleanComment(), not(is("Filtered")));
+        assertThat(object.findAttribute(SOURCE).getCleanComment(), is(nullValue()));
     }
 
     @Test
@@ -662,6 +666,87 @@ public void create_with_attribute_comment() {
         assertThat(created.toString(), containsString(forComparison.toString()));
     }
 
+    @Test
+    public void lookup_from_incorrect_source() {
+
+        databaseHelper.addObject("" +
+                "aut-num:        AS100\n" +
+                "as-name:        End-User-2\n" +
+                "descr:          description\n" +
+                "admin-c:        TP1-TEST\n" +
+                "tech-c:         TP1-TEST\n" +
+                "mnt-by:         OWNER-MNT\n" +
+                "source:         TEST-NONAUTH\n");
+
+        // restClient assumes TEST source, should be redirected to NONAUTH source and get correct object
+        final RpslObject autnum = restClient.request().lookup(ObjectType.AUT_NUM, "AS100");
+
+        assertThat(autnum.getValueForAttribute(SOURCE), is(ciString("TEST-NONAUTH")));
+    }
+
+    @Test
+    public void update_incorrect_source() {
+
+        databaseHelper.addObject("" +
+                "aut-num:        AS100\n" +
+                "as-name:        End-User-2\n" +
+                "descr:          description\n" +
+                "admin-c:        TP1-TEST\n" +
+                "tech-c:         TP1-TEST\n" +
+                "mnt-by:         OWNER-MNT\n" +
+                "source:         TEST-NONAUTH\n");
+
+        // restClient assumes TEST source, should be redirected to NONAUTH source and get correct object
+        final RpslObject autnum = restClient.request()
+                .addParam("password", "test")
+                .update(RpslObject.parse(
+                    "aut-num:        AS100\n" +
+                    "as-name:        End-User-2\n" +
+                    "descr:          description2\n" +
+                    "admin-c:        TP1-TEST\n" +
+                    "tech-c:         TP1-TEST\n" +
+                    "mnt-by:         OWNER-MNT\n" +
+                    "source:         TEST-NONAUTH\n"
+                )
+        );
+
+        assertThat(autnum.getValueForAttribute(DESCR), is(ciString("description2")));
+    }
+
+    @Test
+    public void delete_incorrect_source() {
+
+        databaseHelper.addObject("" +
+                "aut-num:        AS100\n" +
+                "as-name:        End-User-2\n" +
+                "descr:          description\n" +
+                "admin-c:        TP1-TEST\n" +
+                "tech-c:         TP1-TEST\n" +
+                "mnt-by:         OWNER-MNT\n" +
+                "source:         TEST-NONAUTH\n");
+
+        // restClient assumes TEST source, should be redirected to NONAUTH source and get correct object
+        final RpslObject autnum = restClient.request()
+                .addParam("password", "test")
+                .delete(RpslObject.parse(
+                        "aut-num:        AS100\n" +
+                        "as-name:        End-User-2\n" +
+                        "descr:          description2\n" +
+                        "admin-c:        TP1-TEST\n" +
+                        "tech-c:         TP1-TEST\n" +
+                        "mnt-by:         OWNER-MNT\n" +
+                        "source:         TEST-NONAUTH\n"
+                    )
+                );
+
+        try {
+            restClient.request().lookup(ObjectType.AUT_NUM, "AS100");
+            fail("Autnum was not deleted");
+        } catch (RestClientException rce) {
+            assertThat(rce.getStatus(), is(HttpStatus.SC_NOT_FOUND));
+        }
+    }
+
     @Test
     public void streaming_search_shows_correct_message_in_empty_result() {
         try {