Feature/regex matching #105

notrepo05 · 2021-06-15T02:14:36Z

We would like to be able to match using regex. I'm opening this as a draft since we plan to continue working / redesigning before trying to get our changes merged in :).

Example use case:

nrohn-a01 nr $ BBL_IAAS=aws ./leftovers -f "pull-1408.*security" --aws-secret-access-key=*** --aws-access-key-id=***  --aws-region=*** --dry-run --filter-as-regex
[EC2 Security Group: nat_security_group (Name:pull-1408-nat-security-group, Environment:pull-1408, Application:Cloud Foundry)]
[EC2 Security Group: ops_manager_security_group (Environment:pull-1408, Application:Cloud Foundry, Name:pull-1408-ops-manager-security-group)]
[EC2 Security Group: web_lb_security_group (Name:pull-1408-lb-security-group, Environment:pull-1408, Application:Cloud Foundry)]
[EC2 Security Group: rds_security_group (Application:Cloud Foundry, Name:pull-1408-rds-security-group, Environment:pull-1408)]
[EC2 Security Group: tcp_lb_security_group (Environment:pull-1408, Application:Cloud Foundry, Name:pull-1408-tcp-lb-security-group)]
[EC2 Security Group: ssh_lb_security_group (Name:pull-1408-ssh-lb-security-group, Environment:pull-1408, Application:Cloud Foundry)]
...

This PR also reorders some AWS resource deletion by their type, and adds waits to a couple resources so we don't fail improperly (especially important for RDS, which takes ~10 minutes to delete)

Thanks!

cc @ciriarte

Authored-by: Nick Rohn <nrohn@vmware.com>

notrepo05 · 2021-06-15T02:34:42Z

TODOs

Remove debugging lines
Add test coverage for AWS reordering and waits

Authored-by: Nick Rohn <nrohn@vmware.com>

we need to wait for rds_instances to actually be deleted before continuing with rds subnet deletion, because you cannot delete a subnet that contains an instance Co-authored-by: Nick Rohn <nrohn@vmware.com> Co-authored-by: Jhonathan Aristizabal <jhonathana@vmware.com>

We believe that security groups need to be deleted after the db instance is. And that vpc should be deleted last Co-authored-by: Nick Rohn <nrohn@vmware.com> Co-authored-by: Jhonathan Aristizabal <jhonathana@vmware.com>

feature: wait for rds_instances to be deleted

Update help section with new command

- remove vendor Co-authored-by: Jhonathan Aristizabal <jhonathana@vmware.com> Co-authored-by: Nick Rohn <nrohn@vmware.com>

- revert openstack dependency Authored-by: Nick Rohn <nrohn@vmware.com>

…rs into feature/regex-matching

Authored-by: Nick Rohn <nrohn@vmware.com>

notrepo05 · 2021-06-22T16:02:21Z

Hi @genevieve, if we wanted to get this merged upstream I assume we'll need to reintroduce vendoring and add more test coverage? Would you also want us to refactor the regex filter flag? We can also maintain our own fork if that's preferable. Thank you!!

genevieve · 2021-06-22T16:33:33Z

Hey Nick! Yes, if you wouldnt mind reintroducing vendoring and adding more test coverage, we can merge these changes!

notrepo05 · 2021-06-23T20:06:04Z

Thanks! I'll try to get it ready to be merged in a couple weeks.

rowanjacobs

Looking good so far!

Has this been tested with invalid regex input yet? I didn't see any code catching the panic from MustCompile, but there were also 2205 file changes so I could have missed something.

rowanjacobs · 2021-07-06T20:26:05Z

vsphere/folders.go

@@ -27,7 +27,7 @@ func NewFolders(client client, logger logger) Folders {

 // List not only lists top-level folders, it also lists child and grandchild
 // folders, and all VMs within those folders.
-func (v Folders) List(filter string, rType string) ([]common.Deletable, error) {
+func (v Folders) List(filter string, rType string, regex bool) ([]common.Deletable, error) {


FWIW, I am no longer aware of any users who are using leftovers to delete folders on a multitenant vSphere cluster (since the Toolsmiths team no longer provides multitenant vSpheres to MAPBU). So I think you can safely implement deleting vSphere folders by regex, if that's something you want to do.

Thanks for pointing this out. I'll need to find a good way to deal with the potential panic in common/resource_filter.go before we merge.

rowanjacobs · 2021-07-06T20:27:33Z

cmd/leftovers/leftovers.go

@@ -32,6 +33,7 @@ func GetLeftovers(logger *app.Logger, o app.Options) (leftovers, error) {
 	case app.Azure:
 		l, err = azure.NewLeftovers(logger, o.AzureClientID, o.AzureClientSecret, o.AzureSubscriptionID, o.AzureTenantID)
 	case app.GCP:
+		fmt.Println("in gcp")


Reminder to self and Gen: make sure this test line is gone before merging the PR

notrepo05 · 2021-07-08T23:01:14Z

Thank you for the review @rowanjacobs. I still need to get around to getting this ready for merge. I'm trying to find time

notrepo05 added 4 commits June 14, 2021 18:37

feature: add flag to interpet filter as regex

a19bd8a

Authored-by: Nick Rohn <nrohn@vmware.com>

test: interpreting the filter as regex via a flag

0de4eeb

Authored-by: Nick Rohn <nrohn@vmware.com>

refactor: regex_match test to use ginkgo

2f09529

Authored-by: Nick Rohn <nrohn@vmware.com>

refactor: rename RegexMatch to ResourceMatches

7b106cb

Authored-by: Nick Rohn <nrohn@vmware.com>

notrepo05 and others added 23 commits June 14, 2021 19:46

refactor: remove redundant declaration

19962d8

Authored-by: Nick Rohn <nrohn@vmware.com>

refactor: kill a line of white space

58b1f09

Authored-by: Nick Rohn <nrohn@vmware.com>

fix: enable filter by regex for vsphere

5924211

Authored-by: Nick Rohn <nrohn@vmware.com>

fix: order of resource deletion

9832ef5

We believe that security groups need to be deleted after the db instance is. And that vpc should be deleted last Co-authored-by: Nick Rohn <nrohn@vmware.com> Co-authored-by: Jhonathan Aristizabal <jhonathana@vmware.com>

Merge pull request #1 from notrepo05/feature/dependent-rds-cleanup

ea813ea

feature: wait for rds_instances to be deleted

Update README.md

3742ab1

Update help section with new command

Update README.md

cd2d592

Update README.md

4df8225

refactor: rename describe to resource matches

1e7e4ab

feature: update our vendored dependencies

69ce9f9

- remove vendor Co-authored-by: Jhonathan Aristizabal <jhonathana@vmware.com> Co-authored-by: Nick Rohn <nrohn@vmware.com>

feat: use extended regex for resource filtering

723c575

- revert openstack dependency Authored-by: Nick Rohn <nrohn@vmware.com>

Merge branch 'feature/regex-matching' of github.com:notrepo05/leftove…

8efb52b

…rs into feature/regex-matching

Set version for beta release tag

2efedf4

Authored-by: Nick Rohn <nrohn@vmware.com>

bump version

633db7f

Authored-by: Nick Rohn <nrohn@vmware.com>

misc: revert version back to dev

1fe5290

Authored-by: Nick Rohn <nrohn@vmware.com>

Update readme to describe filtering

08d7c0b

Fix typo in readme

b16c7b6

Update README.md

341f42e

Update README.md

163a318

feat: add github actions unit tests on push

97bf0b7

Authored-by: Nick Rohn <nrohn@vmware.com>

fix: github action syntax

b93d2ab

Authored-by: Nick Rohn <nrohn@vmware.com>

fix: github actions syntax

36e523c

Authored-by: Nick Rohn <nrohn@vmware.com>

rowanjacobs reviewed Jul 6, 2021

View reviewed changes

nhsieh force-pushed the feature/regex-matching branch from cc050b9 to 36e523c Compare October 1, 2021 18:13

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Feature/regex matching #105

Feature/regex matching #105

notrepo05 commented Jun 15, 2021 •

edited

Loading

notrepo05 commented Jun 15, 2021 •

edited

Loading

notrepo05 commented Jun 22, 2021

genevieve commented Jun 22, 2021

notrepo05 commented Jun 23, 2021

rowanjacobs left a comment

rowanjacobs Jul 6, 2021

notrepo05 Jul 8, 2021

rowanjacobs Jul 6, 2021

notrepo05 commented Jul 8, 2021

Feature/regex matching #105

Are you sure you want to change the base?

Feature/regex matching #105

Conversation

notrepo05 commented Jun 15, 2021 • edited Loading

notrepo05 commented Jun 15, 2021 • edited Loading

notrepo05 commented Jun 22, 2021

genevieve commented Jun 22, 2021

notrepo05 commented Jun 23, 2021

rowanjacobs left a comment

Choose a reason for hiding this comment

rowanjacobs Jul 6, 2021

Choose a reason for hiding this comment

notrepo05 Jul 8, 2021

Choose a reason for hiding this comment

rowanjacobs Jul 6, 2021

Choose a reason for hiding this comment

notrepo05 commented Jul 8, 2021

notrepo05 commented Jun 15, 2021 •

edited

Loading

notrepo05 commented Jun 15, 2021 •

edited

Loading