[SOAR-15940] InsightIDR - advanced_query_on_log_set statistical result bug fix #2108

rbowden-r7 · 2023-11-10T11:13:16Z

Proposed Changes

Description

Describe the proposed changes:

to take the id that comes back from calling the /query/logs endpoint, then this ID can be used to the /query/ endpoint to get back the correct values for the statistics, as the will not return the results we need

PR Requirements

Developers, verify you have completed the following items by checking them off:

Testing

Unit Tests

Review our documentation on generating and writing plugin unit tests

Unit tests written for any new or updated code

new unit test have been added and all of the old tests are passing

Ran 82 tests in 0.038s

OK

In-Product Tests

If you are an InsightConnect customer or have access to an InsightConnect instance, the following in-product tests should be done:

Screenshot of job output with the plugin changes
Screenshot of the changed connection, actions, or triggers input within the InsightConnect workflow builder

testing via postman using a local docker container and showing the counts line up as expected

cd

Style

Review the style guide

For dependencies, pin OS package and Python package versions
For security, set least privileged account with USER nobody in the Dockerfile when possible
For size, use the slim SDK images when possible: rapid7/insightconnect-python-3-38-slim-plugin:{sdk-version-num} and rapid7/insightconnect-python-3-38-plugin:{sdk-version-num}
For error handling, use of PluginException and ConnectionTestException
For logging, use self.logger
For docs, use changelog style
For docs, validate markdown with insight-plugin validate which calls icon_validate to lint help.md

Functional Checklist

Work fully completed
Functional
- Any new actions/triggers include JSON test files in the tests/ directory created with insight-plugin samples
- Tests should all pass unless it's a negative test. Negative tests have a naming convention of tests/$action_bad.json
- Unsuccessful tests should fail by raising an exception causing the plugin to die and an object should be returned on successful test
- Add functioning test results to PR, sanitize any output if necessary
  - Single action/trigger insight-plugin run -T tests/example.json --debug --jq
  - All actions/triggers shortcut insight-plugin run -T all --debug --jq (use PR format at end)
- Add functioning run results to PR, sanitize any output if necessary
  - Single action/trigger insight-plugin run -R tests/example.json --debug --jq
  - All actions/triggers shortcut insight-plugin run --debug --jq (use PR format at end)

Assessment

You must validate your work to reviewers:

Run insight-plugin validate and make sure everything passes
Run the assessment tool: insight-plugin run -A. For single action validation: insight-plugin run tests/{file}.json -A
Copy (insight-plugin ... | pbcopy) and paste the output in a new post on this PR
Add required screenshots from the In-Product Tests section

igorski-r7 · 2023-11-10T11:22:51Z

plugins/rapid7_insightidr/unit_test/test_advanced_query_on_log_set_unit.py

Could you please add schema validation to those tests? 😄

…log_set was always coming back as 0.0

…e schema validation

cmcnally-r7 · 2023-11-10T13:20:02Z

plugins/rapid7_insightidr/help.md

@@ -2024,7 +2024,8 @@ Example output:

 # Version History

-* 6.0.1 - Action: `Advanced Query On Log Set` - Up the maximium events returned from 50 to 500
+* 6.0.2 - Action: `Advanced Query On Log set` - Fixed error where statistical queries would always return 0.0
+* 6.0.1 - Action: `Advanced Query On Log` - Up the maximium events returned from 50 to 500


Just some grammar and spelling check 😄

Suggested change

* 6.0.1 - Action: `Advanced Query On Log` - Up the maximium events returned from 50 to 500

* 6.0.1 - Action: `Advanced Query On Log` - Increase the maximum results returned from 50 to 500

this has been updated

cmcnally-r7 · 2023-11-10T13:22:38Z

plugins/rapid7_insightidr/komand_rapid7_insightidr/actions/advanced_query_on_log_set/action.py

+            stats_response = self.connection.session.get(stats_endpoint, params=params)
+            try:
+                stats_response.raise_for_status()
+            except Exception:


Exception is very broad here. Maybe narrow it down.

Suggested change

except Exception:

except HTTPError as error:

A more specific exception has been added here

cmcnally-r7 · 2023-11-10T13:23:07Z

plugins/rapid7_insightidr/komand_rapid7_insightidr/actions/advanced_query_on_log_set/action.py

+                raise PluginException(
+                    cause="Failed to get log sets from InsightIDR\n",
+                    assistance=f"Could not get statistical info from: {stats_endpoint}\n",
+                    data=stats_response.text,


Suggested change

data=stats_response.text,

data=f"{stats_response.text}, {error},

The error message will also be raised here

…t bug fix (#2108) * SOAR-15940-Fixing issue where statistical calls to advanced_query_on_log_set was always coming back as 0.0 * SOAR-15940-updated unit tests for advanced_query_on_log_set to include schema validation * SOAR-15940-updated unit tests for advanced_query_on_log_set to include schema validation * SOAR-15940-updating count to better reflect the total items matched * SOAR-15940-updating count to better reflect the total items matched * SOAR-15940-updating unit test to reflect change to count * SOAR-15940-adding more specific error handling

* [SOAR-15940] InsightIDR - advanced_query_on_log_set statistical result bug fix (#2108) * SOAR-15940-Fixing issue where statistical calls to advanced_query_on_log_set was always coming back as 0.0 * SOAR-15940-updated unit tests for advanced_query_on_log_set to include schema validation * SOAR-15940-updated unit tests for advanced_query_on_log_set to include schema validation * SOAR-15940-updating count to better reflect the total items matched * SOAR-15940-updating count to better reflect the total items matched * SOAR-15940-updating unit test to reflect change to count * SOAR-15940-adding more specific error handling * [PLGN-614] InsightIDR - Adding parity between advanced_query_on_log_set and advanced_query_on_log (#2110) * PLGN-614-Adding parity between advanced_query_on_log_set and advanced_query_on_log * PLGN-614-Updating the help md to use cpatial letter * PLGN-614-Updating the help md to match style format * [PLGN-651] insightidr - Fixing schema error with required fields and adding in unit tests for checking schemas (#2116) * PLGN-615-Fixing schema error with required fields and adding in unit test coverage for schema * PLGN-615-Updating help doc with new chnages * Bump aiohttp from 3.8.5 to 3.8.6 in /plugins/rapid7_insightidr (#2118) Bumps [aiohttp](https://github.com/aio-libs/aiohttp) from 3.8.5 to 3.8.6. - [Release notes](https://github.com/aio-libs/aiohttp/releases) - [Changelog](https://github.com/aio-libs/aiohttp/blob/master/CHANGES.rst) - [Commits](aio-libs/aiohttp@v3.8.5...v3.8.6) --- updated-dependencies: - dependency-name: aiohttp dependency-type: direct:production ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> --------- Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

rbowden-r7 requested review from cmcnally-r7, ablakley-r7, jerichardson-r7, igorski-r7, llaszuk-r7, dlaverty-r7, abmccann-r7 and joneill-r7 November 10, 2023 11:13

rbowden-r7 self-assigned this Nov 10, 2023

igorski-r7 approved these changes Nov 10, 2023

View reviewed changes

igorski-r7 force-pushed the develop branch from b840de1 to 44ed580 Compare November 10, 2023 12:03

rbowden-r7 added 4 commits November 10, 2023 12:41

SOAR-15940-Fixing issue where statistical calls to advanced_query_on_…

f4df4ca

…log_set was always coming back as 0.0

SOAR-15940-updated unit tests for advanced_query_on_log_set to includ…

ead7f73

…e schema validation

SOAR-15940-updated unit tests for advanced_query_on_log_set to includ…

1f3de26

…e schema validation

SOAR-15940-updating count to better reflect the total items matched

6af5c5b

rbowden-r7 force-pushed the SOAR-15940 branch from 21e01e7 to 6af5c5b Compare November 10, 2023 12:43

rbowden-r7 added 2 commits November 10, 2023 12:46

SOAR-15940-updating count to better reflect the total items matched

66a027c

SOAR-15940-updating unit test to reflect change to count

1d781b4

cmcnally-r7 requested changes Nov 10, 2023

View reviewed changes

SOAR-15940-adding more specific error handling

17343a6

rbowden-r7 requested review from igorski-r7 and cmcnally-r7 November 10, 2023 14:15

cmcnally-r7 approved these changes Nov 10, 2023

View reviewed changes

igorski-r7 approved these changes Nov 10, 2023

View reviewed changes

cmcnally-r7 changed the title ~~SOAR-15940-Fixing issue where statistical calls to advanced_query_on_log_set was always coming back as 0.0~~ [SOAR-15940] InsightIDR - advanced_query_on_log_set statistical result bug fix Nov 10, 2023

cmcnally-r7 merged commit 34642e7 into develop Nov 10, 2023
6 checks passed

cmcnally-r7 deleted the SOAR-15940 branch November 10, 2023 15:11

rbowden-r7 mentioned this pull request Nov 10, 2023

[PLGN-614] InsightIDR - Adding parity between advanced_query_on_log_set and advanced_query_on_log #2110

Merged

17 tasks

rbowden-r7 mentioned this pull request Nov 13, 2023

Rapid7_Insightidr 7.0.0 release #2112

Merged

17 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[SOAR-15940] InsightIDR - advanced_query_on_log_set statistical result bug fix #2108

[SOAR-15940] InsightIDR - advanced_query_on_log_set statistical result bug fix #2108

rbowden-r7 commented Nov 10, 2023

igorski-r7 Nov 10, 2023

cmcnally-r7 Nov 10, 2023

rbowden-r7 Nov 10, 2023

cmcnally-r7 Nov 10, 2023

rbowden-r7 Nov 10, 2023

cmcnally-r7 Nov 10, 2023

rbowden-r7 Nov 10, 2023

	* 6.0.1 - Action: `Advanced Query On Log` - Up the maximium events returned from 50 to 500
	* 6.0.1 - Action: `Advanced Query On Log` - Increase the maximum results returned from 50 to 500

	data=stats_response.text,
	data=f"{stats_response.text}, {error},

[SOAR-15940] InsightIDR - advanced_query_on_log_set statistical result bug fix #2108

[SOAR-15940] InsightIDR - advanced_query_on_log_set statistical result bug fix #2108

Conversation

rbowden-r7 commented Nov 10, 2023

Proposed Changes

Description

PR Requirements

Testing

Unit Tests

In-Product Tests

Style

Functional Checklist

Assessment

igorski-r7 Nov 10, 2023

Choose a reason for hiding this comment

cmcnally-r7 Nov 10, 2023

Choose a reason for hiding this comment

rbowden-r7 Nov 10, 2023

Choose a reason for hiding this comment

cmcnally-r7 Nov 10, 2023

Choose a reason for hiding this comment

rbowden-r7 Nov 10, 2023

Choose a reason for hiding this comment

cmcnally-r7 Nov 10, 2023

Choose a reason for hiding this comment

rbowden-r7 Nov 10, 2023

Choose a reason for hiding this comment