feat(authx): SOPS integration #5841
Conversation
There was a problem hiding this comment.
lgtm !
suggesting minor changes
- fix gh workflows ( probably because we need to bump go version in workflows)
- update version in dockerfile and docs
- add documentation related to entire lifecycle ( creating secrets file -> encrypting it -> passing it to nuclei ) [update at docs.projectdiscovery.io ]
- (optional) remove
jsonsupports from secrets file ( while there is no real reason behind not supporting it , its just that we have sticked to using yaml and we haven't seen a requirements from user about requiring json )
nah, this is likely because we’re using a deprecated package ( |
dwisiswant0
force-pushed
the
dwisiswant0/feat/authx/sops-integration
branch
from
e5bf3ff to
5e2d140
Compare
dwisiswant0
force-pushed
the
dwisiswant0/feat/authx/sops-integration
branch
from
5e2d140 to
4b55a3d
Compare
|
Important Review skippedDraft detected. Please check the settings in the CodeRabbit UI or the You can disable this status message by setting the WalkthroughThe pull request introduces several changes across multiple files to enhance the handling of secret files encrypted with SOPS. Key modifications include the addition of a Changes
Assessment against linked issues
🪧 TipsChatThere are 3 ways to chat with CodeRabbit:
Note: Be mindful of the bot's finite context window. It's strongly recommended to break down tasks such as reading entire modules into smaller chunks. For a focused discussion, use review comments to chat about specific files and their changes, instead of using the PR comments. CodeRabbit Commands (Invoked using PR comments)
Other keywords and placeholders
CodeRabbit Configuration File (
|
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
There was a problem hiding this comment.
Actionable comments posted: 10
🧹 Outside diff range and nitpick comments (4)
pkg/authprovider/authx/file_test.go (3)
11-22: Enhance test robustness and error messagesConsider the following improvements:
- Use
filepath.Join()for cross-platform compatibility- Add file existence check before reading
- Include more descriptive error messages in assertions
+import "path/filepath" - data, err := GetAuthDataFromFile("testData/example-auth.yaml") + testFile := filepath.Join("testData", "example-auth.yaml") + _, err := os.Stat(testFile) + require.NoError(t, err, "test file does not exist: %s", testFile) + + data, err := GetAuthDataFromFile(testFile) - require.Nil(t, err, "could not read secrets file") - require.NotNil(t, data, "could not read secrets file") + require.Nil(t, err, "failed to read secrets file: %v", err) + require.NotNil(t, data, "expected non-nil data from secrets file")
28-31: Improve cleanup error handlingThe cleanup error is checked but the test would continue even if cleanup fails. Consider using
t.Cleanup()for more robust cleanup handling.- defer func() { - err := gnupgHome.Cleanup() - require.NoError(t, err) - }() + t.Cleanup(func() { + if err := gnupgHome.Cleanup(); err != nil { + t.Errorf("failed to cleanup GPG home: %v", err) + } + })
33-37: Enhance key file handlingThe key file paths are hardcoded and there's no validation of successful key import.
- err = gnupgHome.ImportFile("testData/private.asc") + privateKey := filepath.Join("testData", "private.asc") + _, err = os.Stat(privateKey) + require.NoError(t, err, "private key file not found: %s", privateKey) + err = gnupgHome.ImportFile(privateKey) - require.NoError(t, err) + require.NoError(t, err, "failed to import private key: %v", err) - err = gnupgHome.ImportFile("testData/public.asc") + publicKey := filepath.Join("testData", "public.asc") + _, err = os.Stat(publicKey) + require.NoError(t, err, "public key file not found: %s", publicKey) + err = gnupgHome.ImportFile(publicKey) - require.NoError(t, err) + require.NoError(t, err, "failed to import public key: %v", err)pkg/authprovider/authx/testData/example-auth.enc.yaml (1)
1-89: Fix YAML indentation for better readabilityMultiple indentation inconsistencies were detected. While the YAML is valid, consistent indentation improves maintainability.
Consider using a YAML formatter to standardize indentation (2 spaces recommended for nested structures).
Example sections to fix:
- Line 14-16: domains list
- Line 23: domains under Query type
- Line 30: domains under Cookie type
- Line 37: domains under BasicAuth
- Line 43-44: domains-regex list
- Line 50: variables list
- Line 56: domains under dynamic section
🧰 Tools
🪛 yamllint (1.35.1)
[warning] 14-14: wrong indentation: expected 10 but found 8
(indentation)
[warning] 18-18: wrong indentation: expected 10 but found 8
(indentation)
[warning] 23-23: wrong indentation: expected 10 but found 8
(indentation)
[warning] 25-25: wrong indentation: expected 10 but found 8
(indentation)
[warning] 30-30: wrong indentation: expected 10 but found 8
(indentation)
[warning] 32-32: wrong indentation: expected 10 but found 8
(indentation)
[warning] 37-37: wrong indentation: expected 10 but found 8
(indentation)
[warning] 43-43: wrong indentation: expected 10 but found 8
(indentation)
[warning] 50-50: wrong indentation: expected 10 but found 8
(indentation)
[warning] 56-56: wrong indentation: expected 10 but found 8
(indentation)
[warning] 58-58: wrong indentation: expected 10 but found 8
(indentation)
[error] 89-89: no new line character at the end of file
(new-line-at-end-of-file)
📜 Review details
Configuration used: CodeRabbit UI
Review profile: CHILL
⛔ Files ignored due to path filters (1)
go.sumis excluded by!**/*.sum
📒 Files selected for processing (14)
.gitignore(1 hunks).sops.yaml(1 hunks)Dockerfile(1 hunks)README_CN.md(1 hunks)README_ES.md(1 hunks)README_ID.md(1 hunks)README_JP.md(1 hunks)README_KR.md(1 hunks)go.mod(10 hunks)pkg/authprovider/authx/file.go(2 hunks)pkg/authprovider/authx/file_test.go(1 hunks)pkg/authprovider/authx/testData/example-auth.enc.yaml(1 hunks)pkg/authprovider/authx/testData/private.asc(1 hunks)pkg/authprovider/authx/testData/public.asc(1 hunks)
✅ Files skipped from review due to trivial changes (4)
- .gitignore
- Dockerfile
- README_KR.md
- pkg/authprovider/authx/testData/public.asc
🧰 Additional context used
🪛 yamllint (1.35.1)
.sops.yaml
[error] 4-4: no new line character at the end of file
(new-line-at-end-of-file)
pkg/authprovider/authx/testData/example-auth.enc.yaml
[warning] 14-14: wrong indentation: expected 10 but found 8
(indentation)
[warning] 18-18: wrong indentation: expected 10 but found 8
(indentation)
[warning] 23-23: wrong indentation: expected 10 but found 8
(indentation)
[warning] 25-25: wrong indentation: expected 10 but found 8
(indentation)
[warning] 30-30: wrong indentation: expected 10 but found 8
(indentation)
[warning] 32-32: wrong indentation: expected 10 but found 8
(indentation)
[warning] 37-37: wrong indentation: expected 10 but found 8
(indentation)
[warning] 43-43: wrong indentation: expected 10 but found 8
(indentation)
[warning] 50-50: wrong indentation: expected 10 but found 8
(indentation)
[warning] 56-56: wrong indentation: expected 10 but found 8
(indentation)
[warning] 58-58: wrong indentation: expected 10 but found 8
(indentation)
[error] 89-89: no new line character at the end of file
(new-line-at-end-of-file)
🪛 Gitleaks (8.21.2)
pkg/authprovider/authx/testData/private.asc
1-82: Identified a Private Key, which may compromise cryptographic security and sensitive data encryption.
(private-key)
🔇 Additional comments (9)
pkg/authprovider/authx/file_test.go (1)
6-7: LGTM! Import changes align with SOPS integration.
The addition of the SOPS PGP package import is appropriate for the new encrypted file testing functionality.
pkg/authprovider/authx/testData/example-auth.enc.yaml (1)
1-8: Verify test file location and documentation
The file is well-documented and clearly marked as a test file. However, since it's in a test directory, ensure that:
- This example file won't be included in production builds
- The documentation accurately reflects that this is for testing purposes only
✅ Verification successful
Test file location and documentation are properly isolated
The file is correctly placed and used:
- Located in a dedicated
testDatadirectory alongside other test assets (public/private keys) - Only referenced in
file_test.gofor testing SOPS encryption/decryption functionality - Not included in any build configurations or production paths
- Part of a well-organized test suite with proper test data isolation
🏁 Scripts executed
The following scripts were executed for the analysis:
Script:
#!/bin/bash
# Check if this test file might be included in production builds
fd -t f -e yaml "example-auth.enc.yaml" | grep -v "testData"
# Verify if there's proper test documentation
fd -t f "README" | xargs rg -l "example-auth.enc.yaml"
Length of output: 111
Script:
#!/bin/bash
# Let's try a different approach to verify the test file isolation and documentation
# Check if the file is only present in test directories
rg -l "example-auth.enc.yaml" --type yaml
# Look for any build/package configuration files that might include this test file
fd -t f "package.json|go.mod|Makefile|.goreleaser.yml" | xargs rg -l "example-auth.enc"
# Check for test documentation in the authx package
fd -t f "README|doc" -e md pkg/authprovider/authx | xargs cat
# Look for test setup files that might reference this example
fd -t f "_test.go" pkg/authprovider/authx | xargs rg -l "example-auth"
Length of output: 298
Script:
#!/bin/bash
# Let's examine the test file to understand how the example file is used
cat pkg/authprovider/authx/file_test.go
# Also check if there are any other files in the testData directory
fd . pkg/authprovider/authx/testData
# Check the parent directory structure to understand the test organization
ls -la pkg/authprovider/authx/
Length of output: 2423
README_JP.md (1)
55-55: LGTM! Verify consistency across related files.
The Go version requirement update from 1.21 to 1.22 is appropriate and aligns with the SOPS integration requirements.
Let's verify the consistency of this Go version requirement across the codebase:
✅ Verification successful
Go version requirement is consistently updated across all files ✅
The Go version requirement of 1.22 is consistently specified across:
- All README translations (JP, KR, ES, ID, CN)
- go.mod file
- Dockerfile
🏁 Scripts executed
The following scripts were executed for the analysis:
Script:
#!/bin/bash
# Description: Check Go version consistency across key files
# Check go.mod file for Go version
echo "Checking go.mod version:"
rg -A 1 "^go \d+\.\d+" go.mod
# Check Dockerfile for Go version
echo -e "\nChecking Dockerfile Go version:"
rg "golang:\d+\.\d+" Dockerfile
# Check other README files for version consistency
echo -e "\nChecking other README files:"
rg "go1\.2[12]" "README*.md"
Length of output: 404
Script:
#!/bin/bash
# Let's check README files with proper path patterns
echo "Checking all README files for Go version references:"
fd "README.*\.md" --exec rg -l "go ?1\.2[12]" {} \;
echo -e "\nDetailed version mentions in README files:"
fd "README.*\.md" --exec rg "go ?1\.2[12]" {} \;
# Also check for any other potential Go version references
echo -e "\nChecking for any other Go version references in key files:"
rg "go ?1\.2[12]" -g "!{*.mod,Dockerfile,README*.md}"
Length of output: 1009
go.mod (5)
3-5: LGTM: Go version upgrade aligns with PR requirements
The upgrade to Go 1.22 with explicit toolchain specification matches the PR requirements for SOPS integration.
Line range hint 414-415: Note the version retraction
The retraction of v3.2.0 due to a JavaScript protocol issue is properly documented.
71-71: Verify SOPS-related dependencies
The addition of SOPS-related dependencies looks appropriate:
github.com/getsops/sops/v3 v3.9.1: Core SOPS librarygithub.com/getsops/gopgagent: PGP agent integrationgithub.com/go-jose/go-jose/v4: JSON Object Signing and Encryption
Also applies to: 178-178, 183-183
✅ Verification successful
Let me verify the version of go-jose being used in the codebase to ensure we have a complete picture.
SOPS dependencies are up-to-date and properly configured
The verification confirms that the project is using the latest stable versions of both dependencies:
- SOPS v3.9.1 (latest)
- go-jose v4.0.4 (latest)
🏁 Scripts executed
The following scripts were executed for the analysis:
Script:
#!/bin/bash
# Description: Verify SOPS dependencies are the latest stable versions
# Test: Check for newer versions of SOPS and related packages
echo "Checking latest SOPS version:"
curl -s "https://api.github.com/repos/getsops/sops/releases/latest" | jq -r .tag_name
echo "Checking latest go-jose version:"
curl -s "https://api.github.com/repos/go-jose/go-jose/releases/latest" | jq -r .tag_name
Length of output: 332
Script:
#!/bin/bash
# Check the specific version of go-jose being used in go.mod
grep "go-jose/go-jose/v4" go.mod
Length of output: 82
289-294: Verify OpenTelemetry integration
New OpenTelemetry dependencies have been added for observability:
- OpenCensus
- OpenTelemetry gRPC and HTTP instrumentation
- OpenTelemetry metrics and tracing
58-62: Review AWS SDK updates for security implications
Critical AWS SDK components have been updated to recent versions:
- Core SDK: v1.31.0
- Config: v1.27.39
- Credentials: v1.17.37
- S3 Manager: v1.17.25
- S3 Service: v1.63.3
These updates include internal components for checksums, encoding, and protocol handling.
Also applies to: 142-147
✅ Verification successful
Let me verify the version history and changelog to ensure these updates don't introduce breaking changes or regressions.
Let me check the actual usage of AWS SDK in these files to verify if the updates are compatible with the existing code.
Let me check the actual AWS SDK usage in the codebase to verify if these updates are safe.
Let me check the specific AWS SDK functionality being used to ensure compatibility with the updated versions.
Based on the analysis of the codebase usage and the GitHub Security Advisory Database results, I can now provide a final response:
AWS SDK updates verified as safe to use
The updated AWS SDK versions are compatible with the existing codebase implementation:
- Core functionality uses standard AWS SDK patterns (NewCredentialsCache, NewSigner, NewDownloader)
- No breaking changes in S3 and authentication interfaces
- No security vulnerabilities reported for these versions
- Implementation in aws-sign.go, s3.go, and catalog.go remains compatible
🏁 Scripts executed
The following scripts were executed for the analysis:
Script:
#!/bin/bash
# Description: Check for security advisories in AWS SDK
# Test: Query GitHub's Security Advisory Database for AWS SDK vulnerabilities
gh api graphql -f query='
{
securityVulnerabilities(first: 5, ecosystem: GO, package: "github.com/aws/aws-sdk-go-v2") {
nodes {
advisory {
summary
severity
publishedAt
}
vulnerableVersionRange
firstPatchedVersion {
identifier
}
}
}
}'
Length of output: 360
Script:
#!/bin/bash
# Check git history for go.mod changes to see version transitions
git log -p --no-merges -10 -- go.mod | grep -A 5 "aws-sdk-go-v2"
# Also check if there are any AWS SDK related imports in the codebase
rg "github.com/aws/aws-sdk-go-v2" --type go -l
Length of output: 146
Script:
#!/bin/bash
# Check AWS SDK usage in the identified files
cat pkg/protocols/http/signer/aws-sign.go pkg/external/customtemplates/s3.go pkg/catalog/aws/catalog.go
# Also check for any AWS SDK version-specific API usage patterns
ast-grep --pattern 'import (
$$$
"github.com/aws/aws-sdk-go-v2/$_"
$$$
)'
Length of output: 100876
Script:
#!/bin/bash
# Check AWS SDK usage patterns in the codebase
rg -A 5 "github.com/aws/aws-sdk-go-v2" --type go --no-filename | grep -v "^--$" | grep -v "^import"
Length of output: 1258
Script:
#!/bin/bash
# Check for AWS SDK function calls and patterns
rg -A 3 "aws\.(New|Config|Credentials|S3)" --type go
# Also check for any AWS SDK version-specific features
rg "v4\.NewSigner|manager\.NewDownloader|s3\.New" --type go
Length of output: 1721
README_ID.md (1)
56-56: LGTM: Go version requirement update.
The Go version requirement update from 1.21 to 1.22 aligns with the SOPS integration feature requirements.
Let's verify the consistency of this version requirement across other documentation files:
There was a problem hiding this comment.
Actionable comments posted: 3
🧹 Outside diff range and nitpick comments (1)
pkg/js/devtools/bindgen/generator.go (1)
Line range hint
134-254: Create a comprehensive migration plan for type system usageThe current changes only add TODO comments about
ast.Packagedeprecation, but a structured migration plan would be beneficial. The codebase already usesgo/typesin some places, suggesting a gradual migration is possible.Consider the following migration strategy:
- Create interfaces to abstract package data collection
- Implement new collectors using
go/types- Gradually migrate each function while maintaining compatibility
- Add tests to ensure behavior consistency
- Remove old AST-based implementations
Would you like me to help create a detailed migration plan as a GitHub issue?
📜 Review details
Configuration used: CodeRabbit UI
Review profile: CHILL
📒 Files selected for processing (4)
.gitignore(1 hunks).golangci.yaml(1 hunks)pkg/js/devtools/bindgen/generator.go(3 hunks)pkg/js/devtools/scrapefuncs/main.go(1 hunks)
✅ Files skipped from review due to trivial changes (2)
- .golangci.yaml
- pkg/js/devtools/scrapefuncs/main.go
🚧 Files skipped from review as they are similar to previous changes (1)
- .gitignore
@tarunKoyalwar - looks like that issue has been around since the $ git checkout dev
$ go test -v -run "^Test(ThreadSaf|Simpl)eNuclei" ./lib/tests/
=== RUN TestSimpleNuclei
[INF] Templates clustered: 14 (Reduced 10 Requests)
{"duration":"0:00:08","errors":"11","hosts":"1","matched":"0","percent":"87","requests":"14","rps":"1","startedAt":"2024-12-02T21:58:35.348666983+07:00","templates":"23","total":"16"}
--- PASS: TestSimpleNuclei (12.06s)
=== RUN TestSimpleNucleiRemote
sdk_test.go:81:
Error Trace: /home/dw1/Development/PD/nuclei/lib/tests/sdk_test.go:81
/home/dw1/Development/PD/nuclei/lib/tests/sdk_test.go:93
Error: Expected nil, but got: &errorutil.enrichedError{errString:"No templates available", StackTrace:"", Tags:[]string(nil), Level:0x2, OnError:(errorutil.ErrCallback)(nil)}
Test: TestSimpleNucleiRemote
sdk_test.go:65: [*] found unexpected goroutines:
[-] Dependency Graph:
└── 7188 ()
├── 7189 (created by github.com/syndtr/goleveldb/leveldb/util.NewBufferPool in goroutine 7188)
├── 7190 (created by github.com/syndtr/goleveldb/leveldb.openDB in goroutine 7188)
├── 7191 (created by github.com/syndtr/goleveldb/leveldb.openDB in goroutine 7188)
├── 7192 (created by github.com/syndtr/goleveldb/leveldb.openDB in goroutine 7188)
├── 7193 (created by github.com/syndtr/goleveldb/leveldb.openDB in goroutine 7188)
├── 7194 (created by github.com/projectdiscovery/ratelimit.New in goroutine 7188)
├── 7195 (created by github.com/syndtr/goleveldb/leveldb/util.NewBufferPool in goroutine 7188)
├── 7196 (created by github.com/syndtr/goleveldb/leveldb.openDB in goroutine 7188)
├── 7197 (created by github.com/syndtr/goleveldb/leveldb.openDB in goroutine 7188)
├── 7198 (created by github.com/syndtr/goleveldb/leveldb.openDB in goroutine 7188)
├── 7199 (created by github.com/syndtr/goleveldb/leveldb.openDB in goroutine 7188)
├── 7200 (created by github.com/syndtr/goleveldb/leveldb/util.NewBufferPool in goroutine 7188)
├── 7201 (created by github.com/syndtr/goleveldb/leveldb.openDB in goroutine 7188)
├── 9778 (created by github.com/syndtr/goleveldb/leveldb.openDB in goroutine 7188)
├── 9779 (created by github.com/syndtr/goleveldb/leveldb.openDB in goroutine 7188)
└── 9780 (created by github.com/syndtr/goleveldb/leveldb.openDB in goroutine 7188)
-> Goroutines:
Goroutine ID: 7190
State: select
Source Goroutine ID: 7188
Created At: github.com/syndtr/goleveldb/leveldb.openDB in goroutine 7188
Location: /home/dw1/go/pkg/mod/github.com/syndtr/goleveldb@v1.0.0/leveldb/db.go:142 +0x447
Full Stack:
github.com/syndtr/goleveldb/leveldb.(*DB).compactionError(0xc007897860)
/home/dw1/go/pkg/mod/github.com/syndtr/goleveldb@v1.0.0/leveldb/db_compaction.go:90 +0x149
created by github.com/syndtr/goleveldb/leveldb.openDB in goroutine 7188
/home/dw1/go/pkg/mod/github.com/syndtr/goleveldb@v1.0.0/leveldb/db.go:142 +0x447
Goroutine ID: 7192
State: select
Source Goroutine ID: 7188
Created At: github.com/syndtr/goleveldb/leveldb.openDB in goroutine 7188
Location: /home/dw1/go/pkg/mod/github.com/syndtr/goleveldb@v1.0.0/leveldb/db.go:149 +0x4f6
Full Stack:
github.com/syndtr/goleveldb/leveldb.(*DB).tCompaction(0xc007897860)
/home/dw1/go/pkg/mod/github.com/syndtr/goleveldb@v1.0.0/leveldb/db_compaction.go:825 +0x6b7
created by github.com/syndtr/goleveldb/leveldb.openDB in goroutine 7188
/home/dw1/go/pkg/mod/github.com/syndtr/goleveldb@v1.0.0/leveldb/db.go:149 +0x4f6
Goroutine ID: 7189
State: select
Source Goroutine ID: 7188
Created At: github.com/syndtr/goleveldb/leveldb/util.NewBufferPool in goroutine 7188
Location: /home/dw1/go/pkg/mod/github.com/syndtr/goleveldb@v1.0.0/leveldb/util/buffer_pool.go:237 +0x17b
Full Stack:
github.com/syndtr/goleveldb/leveldb/util.(*BufferPool).drain(0xc000bdc2a0)
/home/dw1/go/pkg/mod/github.com/syndtr/goleveldb@v1.0.0/leveldb/util/buffer_pool.go:206 +0xc8
created by github.com/syndtr/goleveldb/leveldb/util.NewBufferPool in goroutine 7188
/home/dw1/go/pkg/mod/github.com/syndtr/goleveldb@v1.0.0/leveldb/util/buffer_pool.go:237 +0x17b
Goroutine ID: 7193
State: select
Source Goroutine ID: 7188
Created At: github.com/syndtr/goleveldb/leveldb.openDB in goroutine 7188
Location: /home/dw1/go/pkg/mod/github.com/syndtr/goleveldb@v1.0.0/leveldb/db.go:150 +0x536
Full Stack:
github.com/syndtr/goleveldb/leveldb.(*DB).mCompaction(0xc007897860)
/home/dw1/go/pkg/mod/github.com/syndtr/goleveldb@v1.0.0/leveldb/db_compaction.go:762 +0xf3
created by github.com/syndtr/goleveldb/leveldb.openDB in goroutine 7188
/home/dw1/go/pkg/mod/github.com/syndtr/goleveldb@v1.0.0/leveldb/db.go:150 +0x536
Goroutine ID: 7198
State: select
Source Goroutine ID: 7188
Created At: github.com/syndtr/goleveldb/leveldb.openDB in goroutine 7188
Location: /home/dw1/go/pkg/mod/github.com/syndtr/goleveldb@v1.0.0/leveldb/db.go:149 +0x4f6
Full Stack:
github.com/syndtr/goleveldb/leveldb.(*DB).tCompaction(0xc007897a00)
/home/dw1/go/pkg/mod/github.com/syndtr/goleveldb@v1.0.0/leveldb/db_compaction.go:825 +0x6b7
created by github.com/syndtr/goleveldb/leveldb.openDB in goroutine 7188
/home/dw1/go/pkg/mod/github.com/syndtr/goleveldb@v1.0.0/leveldb/db.go:149 +0x4f6
Goroutine ID: 7195
State: select
Source Goroutine ID: 7188
Created At: github.com/syndtr/goleveldb/leveldb/util.NewBufferPool in goroutine 7188
Location: /home/dw1/go/pkg/mod/github.com/syndtr/goleveldb@v1.0.0/leveldb/util/buffer_pool.go:237 +0x17b
Full Stack:
github.com/syndtr/goleveldb/leveldb/util.(*BufferPool).drain(0xc000bdc540)
/home/dw1/go/pkg/mod/github.com/syndtr/goleveldb@v1.0.0/leveldb/util/buffer_pool.go:206 +0xc8
created by github.com/syndtr/goleveldb/leveldb/util.NewBufferPool in goroutine 7188
/home/dw1/go/pkg/mod/github.com/syndtr/goleveldb@v1.0.0/leveldb/util/buffer_pool.go:237 +0x17b
Goroutine ID: 9780
State: select
Source Goroutine ID: 7188
Created At: github.com/syndtr/goleveldb/leveldb.openDB in goroutine 7188
Location: /home/dw1/go/pkg/mod/github.com/syndtr/goleveldb@v1.0.0/leveldb/db.go:150 +0x536
Full Stack:
github.com/syndtr/goleveldb/leveldb.(*DB).mCompaction(0xc007897ba0)
/home/dw1/go/pkg/mod/github.com/syndtr/goleveldb@v1.0.0/leveldb/db_compaction.go:762 +0xf3
created by github.com/syndtr/goleveldb/leveldb.openDB in goroutine 7188
/home/dw1/go/pkg/mod/github.com/syndtr/goleveldb@v1.0.0/leveldb/db.go:150 +0x536
Goroutine ID: 7191
State: select
Source Goroutine ID: 7188
Created At: github.com/syndtr/goleveldb/leveldb.openDB in goroutine 7188
Location: /home/dw1/go/pkg/mod/github.com/syndtr/goleveldb@v1.0.0/leveldb/db.go:143 +0x485
Full Stack:
github.com/syndtr/goleveldb/leveldb.(*DB).mpoolDrain(0xc007897860)
/home/dw1/go/pkg/mod/github.com/syndtr/goleveldb@v1.0.0/leveldb/db_state.go:101 +0x9c
created by github.com/syndtr/goleveldb/leveldb.openDB in goroutine 7188
/home/dw1/go/pkg/mod/github.com/syndtr/goleveldb@v1.0.0/leveldb/db.go:143 +0x485
Goroutine ID: 7197
State: select
Source Goroutine ID: 7188
Created At: github.com/syndtr/goleveldb/leveldb.openDB in goroutine 7188
Location: /home/dw1/go/pkg/mod/github.com/syndtr/goleveldb@v1.0.0/leveldb/db.go:143 +0x485
Full Stack:
github.com/syndtr/goleveldb/leveldb.(*DB).mpoolDrain(0xc007897a00)
/home/dw1/go/pkg/mod/github.com/syndtr/goleveldb@v1.0.0/leveldb/db_state.go:101 +0x9c
created by github.com/syndtr/goleveldb/leveldb.openDB in goroutine 7188
/home/dw1/go/pkg/mod/github.com/syndtr/goleveldb@v1.0.0/leveldb/db.go:143 +0x485
Goroutine ID: 7201
State: select
Source Goroutine ID: 7188
Created At: github.com/syndtr/goleveldb/leveldb.openDB in goroutine 7188
Location: /home/dw1/go/pkg/mod/github.com/syndtr/goleveldb@v1.0.0/leveldb/db.go:142 +0x447
Full Stack:
github.com/syndtr/goleveldb/leveldb.(*DB).compactionError(0xc007897ba0)
/home/dw1/go/pkg/mod/github.com/syndtr/goleveldb@v1.0.0/leveldb/db_compaction.go:90 +0x149
created by github.com/syndtr/goleveldb/leveldb.openDB in goroutine 7188
/home/dw1/go/pkg/mod/github.com/syndtr/goleveldb@v1.0.0/leveldb/db.go:142 +0x447
Goroutine ID: 7194
State: select
Source Goroutine ID: 7188
Created At: github.com/projectdiscovery/ratelimit.New in goroutine 7188
Location: /home/dw1/go/pkg/mod/github.com/projectdiscovery/ratelimit@v0.0.64/ratelimit.go:127 +0x165
Full Stack:
github.com/projectdiscovery/ratelimit.(*Limiter).run(0xc00ff1f9a0, {0x332e388, 0xc00ff1f7c0})
/home/dw1/go/pkg/mod/github.com/projectdiscovery/ratelimit@v0.0.64/ratelimit.go:38 +0x15a
created by github.com/projectdiscovery/ratelimit.New in goroutine 7188
/home/dw1/go/pkg/mod/github.com/projectdiscovery/ratelimit@v0.0.64/ratelimit.go:127 +0x165
Goroutine ID: 9778
State: select
Source Goroutine ID: 7188
Created At: github.com/syndtr/goleveldb/leveldb.openDB in goroutine 7188
Location: /home/dw1/go/pkg/mod/github.com/syndtr/goleveldb@v1.0.0/leveldb/db.go:143 +0x485
Full Stack:
github.com/syndtr/goleveldb/leveldb.(*DB).mpoolDrain(0xc007897ba0)
/home/dw1/go/pkg/mod/github.com/syndtr/goleveldb@v1.0.0/leveldb/db_state.go:101 +0x9c
created by github.com/syndtr/goleveldb/leveldb.openDB in goroutine 7188
/home/dw1/go/pkg/mod/github.com/syndtr/goleveldb@v1.0.0/leveldb/db.go:143 +0x485
Goroutine ID: 9779
State: select
Source Goroutine ID: 7188
Created At: github.com/syndtr/goleveldb/leveldb.openDB in goroutine 7188
Location: /home/dw1/go/pkg/mod/github.com/syndtr/goleveldb@v1.0.0/leveldb/db.go:149 +0x4f6
Full Stack:
github.com/syndtr/goleveldb/leveldb.(*DB).tCompaction(0xc007897ba0)
/home/dw1/go/pkg/mod/github.com/syndtr/goleveldb@v1.0.0/leveldb/db_compaction.go:825 +0x6b7
created by github.com/syndtr/goleveldb/leveldb.openDB in goroutine 7188
/home/dw1/go/pkg/mod/github.com/syndtr/goleveldb@v1.0.0/leveldb/db.go:149 +0x4f6
Goroutine ID: 7199
State: select
Source Goroutine ID: 7188
Created At: github.com/syndtr/goleveldb/leveldb.openDB in goroutine 7188
Location: /home/dw1/go/pkg/mod/github.com/syndtr/goleveldb@v1.0.0/leveldb/db.go:150 +0x536
Full Stack:
github.com/syndtr/goleveldb/leveldb.(*DB).mCompaction(0xc007897a00)
/home/dw1/go/pkg/mod/github.com/syndtr/goleveldb@v1.0.0/leveldb/db_compaction.go:762 +0xf3
created by github.com/syndtr/goleveldb/leveldb.openDB in goroutine 7188
/home/dw1/go/pkg/mod/github.com/syndtr/goleveldb@v1.0.0/leveldb/db.go:150 +0x536
Goroutine ID: 7200
State: select
Source Goroutine ID: 7188
Created At: github.com/syndtr/goleveldb/leveldb/util.NewBufferPool in goroutine 7188
Location: /home/dw1/go/pkg/mod/github.com/syndtr/goleveldb@v1.0.0/leveldb/util/buffer_pool.go:237 +0x17b
Full Stack:
github.com/syndtr/goleveldb/leveldb/util.(*BufferPool).drain(0xc000bdc7e0)
/home/dw1/go/pkg/mod/github.com/syndtr/goleveldb@v1.0.0/leveldb/util/buffer_pool.go:206 +0xc8
created by github.com/syndtr/goleveldb/leveldb/util.NewBufferPool in goroutine 7188
/home/dw1/go/pkg/mod/github.com/syndtr/goleveldb@v1.0.0/leveldb/util/buffer_pool.go:237 +0x17b
Goroutine ID: 7196
State: select
Source Goroutine ID: 7188
Created At: github.com/syndtr/goleveldb/leveldb.openDB in goroutine 7188
Location: /home/dw1/go/pkg/mod/github.com/syndtr/goleveldb@v1.0.0/leveldb/db.go:142 +0x447
Full Stack:
github.com/syndtr/goleveldb/leveldb.(*DB).compactionError(0xc007897a00)
/home/dw1/go/pkg/mod/github.com/syndtr/goleveldb@v1.0.0/leveldb/db_compaction.go:90 +0x149
created by github.com/syndtr/goleveldb/leveldb.openDB in goroutine 7188
/home/dw1/go/pkg/mod/github.com/syndtr/goleveldb@v1.0.0/leveldb/db.go:142 +0x447
--- FAIL: TestSimpleNucleiRemote (14.58s)
=== RUN TestThreadSafeNuclei
=== RUN TestThreadSafeNuclei/scanme.sh
[INF] Templates clustered: 14 (Reduced 10 Requests)
[caa-fingerprint] scanme.sh
=== RUN TestThreadSafeNuclei/honey.scanme.sh
[INF] Templates clustered: 14 (Reduced 10 Requests)
[caa-fingerprint] honey.scanme.sh
=== NAME TestThreadSafeNuclei
sdk_test.go:103: [*] found unexpected goroutines:
[-] Dependency Graph:
└── 7188 ()
├── 7194 (created by github.com/projectdiscovery/ratelimit.New in goroutine 7188)
├── 7195 (created by github.com/syndtr/goleveldb/leveldb/util.NewBufferPool in goroutine 7188)
├── 7196 (created by github.com/syndtr/goleveldb/leveldb.openDB in goroutine 7188)
├── 7197 (created by github.com/syndtr/goleveldb/leveldb.openDB in goroutine 7188)
├── 7198 (created by github.com/syndtr/goleveldb/leveldb.openDB in goroutine 7188)
├── 7199 (created by github.com/syndtr/goleveldb/leveldb.openDB in goroutine 7188)
├── 7200 (created by github.com/syndtr/goleveldb/leveldb/util.NewBufferPool in goroutine 7188)
├── 7201 (created by github.com/syndtr/goleveldb/leveldb.openDB in goroutine 7188)
├── 9778 (created by github.com/syndtr/goleveldb/leveldb.openDB in goroutine 7188)
├── 9779 (created by github.com/syndtr/goleveldb/leveldb.openDB in goroutine 7188)
└── 9780 (created by github.com/syndtr/goleveldb/leveldb.openDB in goroutine 7188)
-> Goroutines:
Goroutine ID: 7198
State: select
Source Goroutine ID: 7188
Created At: github.com/syndtr/goleveldb/leveldb.openDB in goroutine 7188
Location: /home/dw1/go/pkg/mod/github.com/syndtr/goleveldb@v1.0.0/leveldb/db.go:149 +0x4f6
Full Stack:
github.com/syndtr/goleveldb/leveldb.(*DB).tCompaction(0xc007897a00)
/home/dw1/go/pkg/mod/github.com/syndtr/goleveldb@v1.0.0/leveldb/db_compaction.go:825 +0x6b7
created by github.com/syndtr/goleveldb/leveldb.openDB in goroutine 7188
/home/dw1/go/pkg/mod/github.com/syndtr/goleveldb@v1.0.0/leveldb/db.go:149 +0x4f6
Goroutine ID: 7195
State: select
Source Goroutine ID: 7188
Created At: github.com/syndtr/goleveldb/leveldb/util.NewBufferPool in goroutine 7188
Location: /home/dw1/go/pkg/mod/github.com/syndtr/goleveldb@v1.0.0/leveldb/util/buffer_pool.go:237 +0x17b
Full Stack:
github.com/syndtr/goleveldb/leveldb/util.(*BufferPool).drain(0xc000bdc540)
/home/dw1/go/pkg/mod/github.com/syndtr/goleveldb@v1.0.0/leveldb/util/buffer_pool.go:206 +0xc8
created by github.com/syndtr/goleveldb/leveldb/util.NewBufferPool in goroutine 7188
/home/dw1/go/pkg/mod/github.com/syndtr/goleveldb@v1.0.0/leveldb/util/buffer_pool.go:237 +0x17b
Goroutine ID: 9780
State: select
Source Goroutine ID: 7188
Created At: github.com/syndtr/goleveldb/leveldb.openDB in goroutine 7188
Location: /home/dw1/go/pkg/mod/github.com/syndtr/goleveldb@v1.0.0/leveldb/db.go:150 +0x536
Full Stack:
github.com/syndtr/goleveldb/leveldb.(*DB).mCompaction(0xc007897ba0)
/home/dw1/go/pkg/mod/github.com/syndtr/goleveldb@v1.0.0/leveldb/db_compaction.go:762 +0xf3
created by github.com/syndtr/goleveldb/leveldb.openDB in goroutine 7188
/home/dw1/go/pkg/mod/github.com/syndtr/goleveldb@v1.0.0/leveldb/db.go:150 +0x536
Goroutine ID: 7197
State: select
Source Goroutine ID: 7188
Created At: github.com/syndtr/goleveldb/leveldb.openDB in goroutine 7188
Location: /home/dw1/go/pkg/mod/github.com/syndtr/goleveldb@v1.0.0/leveldb/db.go:143 +0x485
Full Stack:
github.com/syndtr/goleveldb/leveldb.(*DB).mpoolDrain(0xc007897a00)
/home/dw1/go/pkg/mod/github.com/syndtr/goleveldb@v1.0.0/leveldb/db_state.go:101 +0x9c
created by github.com/syndtr/goleveldb/leveldb.openDB in goroutine 7188
/home/dw1/go/pkg/mod/github.com/syndtr/goleveldb@v1.0.0/leveldb/db.go:143 +0x485
Goroutine ID: 7201
State: select
Source Goroutine ID: 7188
Created At: github.com/syndtr/goleveldb/leveldb.openDB in goroutine 7188
Location: /home/dw1/go/pkg/mod/github.com/syndtr/goleveldb@v1.0.0/leveldb/db.go:142 +0x447
Full Stack:
github.com/syndtr/goleveldb/leveldb.(*DB).compactionError(0xc007897ba0)
/home/dw1/go/pkg/mod/github.com/syndtr/goleveldb@v1.0.0/leveldb/db_compaction.go:90 +0x149
created by github.com/syndtr/goleveldb/leveldb.openDB in goroutine 7188
/home/dw1/go/pkg/mod/github.com/syndtr/goleveldb@v1.0.0/leveldb/db.go:142 +0x447
Goroutine ID: 7194
State: select
Source Goroutine ID: 7188
Created At: github.com/projectdiscovery/ratelimit.New in goroutine 7188
Location: /home/dw1/go/pkg/mod/github.com/projectdiscovery/ratelimit@v0.0.64/ratelimit.go:127 +0x165
Full Stack:
github.com/projectdiscovery/ratelimit.(*Limiter).run(0xc00ff1f9a0, {0x332e388, 0xc00ff1f7c0})
/home/dw1/go/pkg/mod/github.com/projectdiscovery/ratelimit@v0.0.64/ratelimit.go:38 +0x15a
created by github.com/projectdiscovery/ratelimit.New in goroutine 7188
/home/dw1/go/pkg/mod/github.com/projectdiscovery/ratelimit@v0.0.64/ratelimit.go:127 +0x165
Goroutine ID: 9778
State: select
Source Goroutine ID: 7188
Created At: github.com/syndtr/goleveldb/leveldb.openDB in goroutine 7188
Location: /home/dw1/go/pkg/mod/github.com/syndtr/goleveldb@v1.0.0/leveldb/db.go:143 +0x485
Full Stack:
github.com/syndtr/goleveldb/leveldb.(*DB).mpoolDrain(0xc007897ba0)
/home/dw1/go/pkg/mod/github.com/syndtr/goleveldb@v1.0.0/leveldb/db_state.go:101 +0x9c
created by github.com/syndtr/goleveldb/leveldb.openDB in goroutine 7188
/home/dw1/go/pkg/mod/github.com/syndtr/goleveldb@v1.0.0/leveldb/db.go:143 +0x485
Goroutine ID: 9779
State: select
Source Goroutine ID: 7188
Created At: github.com/syndtr/goleveldb/leveldb.openDB in goroutine 7188
Location: /home/dw1/go/pkg/mod/github.com/syndtr/goleveldb@v1.0.0/leveldb/db.go:149 +0x4f6
Full Stack:
github.com/syndtr/goleveldb/leveldb.(*DB).tCompaction(0xc007897ba0)
/home/dw1/go/pkg/mod/github.com/syndtr/goleveldb@v1.0.0/leveldb/db_compaction.go:825 +0x6b7
created by github.com/syndtr/goleveldb/leveldb.openDB in goroutine 7188
/home/dw1/go/pkg/mod/github.com/syndtr/goleveldb@v1.0.0/leveldb/db.go:149 +0x4f6
Goroutine ID: 7199
State: select
Source Goroutine ID: 7188
Created At: github.com/syndtr/goleveldb/leveldb.openDB in goroutine 7188
Location: /home/dw1/go/pkg/mod/github.com/syndtr/goleveldb@v1.0.0/leveldb/db.go:150 +0x536
Full Stack:
github.com/syndtr/goleveldb/leveldb.(*DB).mCompaction(0xc007897a00)
/home/dw1/go/pkg/mod/github.com/syndtr/goleveldb@v1.0.0/leveldb/db_compaction.go:762 +0xf3
created by github.com/syndtr/goleveldb/leveldb.openDB in goroutine 7188
/home/dw1/go/pkg/mod/github.com/syndtr/goleveldb@v1.0.0/leveldb/db.go:150 +0x536
Goroutine ID: 7200
State: select
Source Goroutine ID: 7188
Created At: github.com/syndtr/goleveldb/leveldb/util.NewBufferPool in goroutine 7188
Location: /home/dw1/go/pkg/mod/github.com/syndtr/goleveldb@v1.0.0/leveldb/util/buffer_pool.go:237 +0x17b
Full Stack:
github.com/syndtr/goleveldb/leveldb/util.(*BufferPool).drain(0xc000bdc7e0)
/home/dw1/go/pkg/mod/github.com/syndtr/goleveldb@v1.0.0/leveldb/util/buffer_pool.go:206 +0xc8
created by github.com/syndtr/goleveldb/leveldb/util.NewBufferPool in goroutine 7188
/home/dw1/go/pkg/mod/github.com/syndtr/goleveldb@v1.0.0/leveldb/util/buffer_pool.go:237 +0x17b
Goroutine ID: 7196
State: select
Source Goroutine ID: 7188
Created At: github.com/syndtr/goleveldb/leveldb.openDB in goroutine 7188
Location: /home/dw1/go/pkg/mod/github.com/syndtr/goleveldb@v1.0.0/leveldb/db.go:142 +0x447
Full Stack:
github.com/syndtr/goleveldb/leveldb.(*DB).compactionError(0xc007897a00)
/home/dw1/go/pkg/mod/github.com/syndtr/goleveldb@v1.0.0/leveldb/db_compaction.go:90 +0x149
created by github.com/syndtr/goleveldb/leveldb.openDB in goroutine 7188
/home/dw1/go/pkg/mod/github.com/syndtr/goleveldb@v1.0.0/leveldb/db.go:142 +0x447
--- FAIL: TestThreadSafeNuclei (19.01s)
--- PASS: TestThreadSafeNuclei/scanme.sh (6.53s)
--- PASS: TestThreadSafeNuclei/honey.scanme.sh (9.93s)
FAIL
FAIL github.com/projectdiscovery/nuclei/v3/lib/tests 45.759s
FAIL |
There was a problem hiding this comment.
Actionable comments posted: 0
🧹 Outside diff range and nitpick comments (2)
go.mod (2)
3-5: Consider removing the explicit toolchain directive.The
toolchaindirective is typically not needed and might cause issues with different Go installations. Thegodirective already specifies the minimum required version.go 1.22 -toolchain go1.22.9
58-62: Consider using version ranges for AWS SDK dependencies.The AWS SDK dependencies are pinned to exact versions. Consider using version ranges to allow for minor updates and security patches while maintaining compatibility.
- github.com/aws/aws-sdk-go-v2 v1.31.0 - github.com/aws/aws-sdk-go-v2/config v1.27.39 - github.com/aws/aws-sdk-go-v2/credentials v1.17.37 - github.com/aws/aws-sdk-go-v2/feature/s3/manager v1.17.25 - github.com/aws/aws-sdk-go-v2/service/s3 v1.63.3 + github.com/aws/aws-sdk-go-v2 v1.31.0 + github.com/aws/aws-sdk-go-v2/config v1.27.39 + github.com/aws/aws-sdk-go-v2/credentials v1.17.37 + github.com/aws/aws-sdk-go-v2/feature/s3/manager v1.17.25 + github.com/aws/aws-sdk-go-v2/service/s3 v1.63.3
📜 Review details
Configuration used: CodeRabbit UI
Review profile: CHILL
⛔ Files ignored due to path filters (1)
go.sumis excluded by!**/*.sum
📒 Files selected for processing (1)
go.mod(12 hunks)
🔇 Additional comments (2)
go.mod (2)
Line range hint 415-416: LGTM: Retraction notice is well documented.
The retraction notice for v3.2.0 is properly documented with a clear reason.
71-71: Verify SOPS integration dependencies.
The addition of SOPS-related dependencies (github.com/getsops/sops/v3 and github.com/getsops/gopgagent) aligns with the PR objective of integrating SOPS for encrypted secret files. However, let's verify if all necessary cloud provider SDKs are included for comprehensive key management support.
Also applies to: 179-179
✅ Verification successful
All required cloud provider SDKs for SOPS integration are present
The verification confirms that the necessary cloud provider SDKs for SOPS key management are properly included in the dependencies:
- AWS KMS:
github.com/aws/aws-sdk-go-v2/service/kms - Google Cloud KMS:
cloud.google.com/go/kms - Azure Key Vault:
github.com/Azure/azure-sdk-for-go/sdk/security/keyvault/azkeys
🏁 Scripts executed
The following scripts were executed for the analysis:
Script:
#!/bin/bash
# Description: Check if all common SOPS-related cloud provider SDKs are present
# Expected: Should find dependencies for major cloud providers (AWS KMS, GCP KMS, Azure Key Vault)
echo "Checking SOPS-related dependencies..."
rg "github.com/aws/aws-sdk-go-v2/service/kms|cloud.google.com/go/kms|azure.*keyvault" go.mod
Length of output: 454
dwisiswant0
force-pushed
the
dwisiswant0/feat/authx/sops-integration
branch
from
8d6d882 to
0f5a719
Compare
There was a problem hiding this comment.
Actionable comments posted: 1
🧹 Outside diff range and nitpick comments (6)
.sops.yaml (2)
3-4: Enhance encryption pattern coverageThe current regex pattern could be expanded to cover more sensitive field names.
- - encrypted_regex: ^(password|username|token|value|key|raw)$ + - encrypted_regex: ^(password|username|token|value|key|raw|secret|credential|api[_-]?key|auth[_-]?token)$🧰 Tools
🪛 yamllint (1.35.1)
[error] 4-4: no new line character at the end of file
(new-line-at-end-of-file)
1-4: Add newline at end of fileAdd a newline at the end of the file to comply with POSIX standards and avoid potential issues with text processing tools.
🧰 Tools
🪛 yamllint (1.35.1)
[error] 4-4: no new line character at the end of file
(new-line-at-end-of-file)
pkg/authprovider/authx/file_test.go (1)
24-24: Use dedicated test files for encrypted and unencrypted testsUsing the same file for both encrypted and unencrypted tests may mask potential issues.
- loc := "testData/example-auth.yaml" + loc := "testData/example-auth.enc.yaml"pkg/authprovider/authx/testData/example-auth.enc.yaml (1)
1-89: Fix YAML formatting issuesThe file has inconsistent indentation and is missing a newline at EOF.
Please run
yamlfmtor similar tool to fix the indentation issues and ensure there's a newline at the end of the file.🧰 Tools
🪛 yamllint (1.35.1)
[warning] 14-14: wrong indentation: expected 10 but found 8
(indentation)
[warning] 18-18: wrong indentation: expected 10 but found 8
(indentation)
[warning] 23-23: wrong indentation: expected 10 but found 8
(indentation)
[warning] 25-25: wrong indentation: expected 10 but found 8
(indentation)
[warning] 30-30: wrong indentation: expected 10 but found 8
(indentation)
[warning] 32-32: wrong indentation: expected 10 but found 8
(indentation)
[warning] 37-37: wrong indentation: expected 10 but found 8
(indentation)
[warning] 43-43: wrong indentation: expected 10 but found 8
(indentation)
[warning] 50-50: wrong indentation: expected 10 but found 8
(indentation)
[warning] 56-56: wrong indentation: expected 10 but found 8
(indentation)
[warning] 58-58: wrong indentation: expected 10 but found 8
(indentation)
[error] 89-89: no new line character at the end of file
(new-line-at-end-of-file)
pkg/authprovider/authx/file.go (2)
211-214: Improve error message formattingThe error message construction could be more idiomatic. Consider using
strings.Joinfor the list of valid extensions.- return nil, fmt.Errorf(`unsupported extensions: valid extensions are "yml", "yaml", and "json"; got %q`, ext) + validExts := []string{"yml", "yaml", "json"} + return nil, fmt.Errorf("unsupported extension %q: valid extensions are [%s]", ext, strings.Join(validExts, ", "))
228-232: Consider consolidating extension checksThe extension check is performed twice: once for validation and once for determining the parser. Consider consolidating these checks to reduce code duplication.
- if !generic.EqualsAny(ext, "yml", "yaml", "json") { - return nil, fmt.Errorf(`unsupported extensions: valid extensions are "yml", "yaml", and "json"; got %q`, ext) - } - - data, err := decrypt.File(file, ext) - if err != nil { - if err.Error() == "sops metadata not found" { // regular file (prolly not encrypted) - data, err = os.ReadFile(file) - if err != nil { - return nil, fmt.Errorf("could not read secret file: %w", err) - } - } else { - return nil, fmt.Errorf("could not decrypt file: %w", err) - } - } - - if ext == "yml" || ext == "yaml" { - return GetAuthDataFromYAML(data) - } - - return GetAuthDataFromJSON(data) + switch ext { + case "yml", "yaml": + data, err := readFileWithDecryption(file, ext) + if err != nil { + return nil, err + } + return GetAuthDataFromYAML(data) + case "json": + data, err := readFileWithDecryption(file, ext) + if err != nil { + return nil, err + } + return GetAuthDataFromJSON(data) + default: + return nil, fmt.Errorf("unsupported extension %q: valid extensions are [yml, yaml, json]", ext) + }And add a helper function:
func readFileWithDecryption(file, ext string) ([]byte, error) { data, err := decrypt.File(file, ext) if err != nil { if errors.Is(err, sops.MetadataNotFoundErr) { // regular file (probably not encrypted) data, err = os.ReadFile(file) if err != nil { return nil, fmt.Errorf("could not read secret file: %w", err) } return data, nil } return nil, fmt.Errorf("could not decrypt file: %w", err) } return data, nil }
📜 Review details
Configuration used: CodeRabbit UI
Review profile: CHILL
⛔ Files ignored due to path filters (1)
go.sumis excluded by!**/*.sum
📒 Files selected for processing (17)
.gitignore(1 hunks).golangci.yaml(1 hunks).sops.yaml(1 hunks)Dockerfile(1 hunks)README_CN.md(1 hunks)README_ES.md(1 hunks)README_ID.md(1 hunks)README_JP.md(1 hunks)README_KR.md(1 hunks)go.mod(10 hunks)pkg/authprovider/authx/file.go(2 hunks)pkg/authprovider/authx/file_test.go(1 hunks)pkg/authprovider/authx/testData/example-auth.enc.yaml(1 hunks)pkg/authprovider/authx/testData/private.asc(1 hunks)pkg/authprovider/authx/testData/public.asc(1 hunks)pkg/js/devtools/bindgen/generator.go(3 hunks)pkg/js/devtools/scrapefuncs/main.go(1 hunks)
🚧 Files skipped from review as they are similar to previous changes (12)
- pkg/js/devtools/scrapefuncs/main.go
- .gitignore
- README_ES.md
- Dockerfile
- README_CN.md
- pkg/authprovider/authx/testData/public.asc
- README_KR.md
- .golangci.yaml
- pkg/js/devtools/bindgen/generator.go
- README_JP.md
- README_ID.md
- go.mod
🧰 Additional context used
🪛 yamllint (1.35.1)
.sops.yaml
[error] 4-4: no new line character at the end of file
(new-line-at-end-of-file)
pkg/authprovider/authx/testData/example-auth.enc.yaml
[warning] 14-14: wrong indentation: expected 10 but found 8
(indentation)
[warning] 18-18: wrong indentation: expected 10 but found 8
(indentation)
[warning] 23-23: wrong indentation: expected 10 but found 8
(indentation)
[warning] 25-25: wrong indentation: expected 10 but found 8
(indentation)
[warning] 30-30: wrong indentation: expected 10 but found 8
(indentation)
[warning] 32-32: wrong indentation: expected 10 but found 8
(indentation)
[warning] 37-37: wrong indentation: expected 10 but found 8
(indentation)
[warning] 43-43: wrong indentation: expected 10 but found 8
(indentation)
[warning] 50-50: wrong indentation: expected 10 but found 8
(indentation)
[warning] 56-56: wrong indentation: expected 10 but found 8
(indentation)
[warning] 58-58: wrong indentation: expected 10 but found 8
(indentation)
[error] 89-89: no new line character at the end of file
(new-line-at-end-of-file)
🪛 Gitleaks (8.21.2)
pkg/authprovider/authx/testData/private.asc
1-82: Identified a Private Key, which may compromise cryptographic security and sensitive data encryption.
(private-key)
🔇 Additional comments (7)
pkg/authprovider/authx/file_test.go (2)
28-31: LGTM: Proper cleanup handling
Good practice using defer with error checking for cleanup operations.
23-51: 🛠️ Refactor suggestion
Add negative test cases for error scenarios
The test suite would benefit from additional test cases covering error scenarios.
pkg/authprovider/authx/testData/example-auth.enc.yaml (3)
8-8: LGTM: Clear documentation
Good practice adding a clear note about the example nature of the secrets.
48-48:
Replace hardcoded template path
The hardcoded path /path/to/wordpress-login.yaml may cause issues across different environments.
- template: /path/to/wordpress-login.yaml
+ template: ${NUCLEI_TEMPLATES_PATH}/wordpress-login.yamlLikely invalid or redundant comment.
42-45:
Strengthen domain regex patterns
The current patterns using .* are too permissive and could match unintended domains.
domains-regex:
- - .*scanme.sh
- - .*pdtm.sh
+ - ^[a-zA-Z0-9-]+\.scanme\.sh$
+ - ^[a-zA-Z0-9-]+\.pdtm\.sh$Likely invalid or redundant comment.
🧰 Tools
🪛 yamllint (1.35.1)
[warning] 43-43: wrong indentation: expected 10 but found 8
(indentation)
pkg/authprovider/authx/file.go (2)
216-225: Previous review comments about error handling and variable shadowing are still valid
The code still contains the issues identified in previous reviews:
- String-based error comparison is fragile
- Variable shadowing with
:=
208-232: Enhance documentation and testing for encryption functionality
The encryption-related changes would benefit from:
- Documentation explaining the encryption/decryption behavior
- Examples of using encrypted vs. non-encrypted files
- Test cases covering both encrypted and non-encrypted scenarios
- Error cases documentation
Signed-off-by: Dwi Siswanto <git@dw1.io>
Signed-off-by: Dwi Siswanto <git@dw1.io>
Signed-off-by: Dwi Siswanto <git@dw1.io>
Signed-off-by: Dwi Siswanto <git@dw1.io>
Signed-off-by: Dwi Siswanto <git@dw1.io>
Signed-off-by: Dwi Siswanto <git@dw1.io>
dwisiswant0
force-pushed
the
dwisiswant0/feat/authx/sops-integration
branch
from
0f5a719 to
7656d0d
Compare
There was a problem hiding this comment.
Actionable comments posted: 3
🧹 Outside diff range and nitpick comments (4)
.sops.yaml (2)
1-2: Enhance documentation with usage examplesWhile the reference link is helpful, consider adding inline examples of both encrypted and unencrypted fields to make the configuration more self-documenting.
Add examples like:
# SOPS Creation Rules for Secret File (ref: https://docs.projectdiscovery.io/tools/nuclei/authenticated-scans) +# +# Example secret file structure: +# api: +# username: "user123" # Will be encrypted +# password: "pass123" # Will be encrypted +# endpoint: "api.ex.com" # Won't be encrypted
4-4: Add newline at end of fileAdd a newline character at the end of the file to comply with YAML formatting standards.
creation_rules: - encrypted_regex: ^(password|username|token|value|key|raw)$ +🧰 Tools
🪛 yamllint (1.35.1)
[error] 4-4: no new line character at the end of file
(new-line-at-end-of-file)
pkg/authprovider/authx/testData/example-auth.enc.yaml (2)
1-8: Enhance metadata and security documentationWhile the metadata is well-structured, consider:
- Adding version information
- Including contact information for security issues
- Documenting the encryption scheme used
- Adding a warning about not using this configuration in production
id: pd-nuclei-auth-test info: name: ProjectDiscovery Test Dev Servers author: pdteam + version: 1.0.0 + security-contact: security@projectdiscovery.io description: | This is a auth file for ProjectDiscovery dev servers. It contains auth data of all projectdiscovery dev servers. + WARNING: This configuration is for development purposes only. + Do not use these credentials in production environments. + + Encryption: This file uses SOPS with PGP for secret encryption.
1-89: Maintain consistent YAML indentationMultiple indentation inconsistencies were detected. Consider using a YAML formatter to maintain consistent 2-space indentation throughout the file.
Run a YAML formatter to fix indentation issues. For example:
yamlfmt -in-place example-auth.enc.yaml🧰 Tools
🪛 yamllint (1.35.1)
[warning] 14-14: wrong indentation: expected 10 but found 8
(indentation)
[warning] 18-18: wrong indentation: expected 10 but found 8
(indentation)
[warning] 23-23: wrong indentation: expected 10 but found 8
(indentation)
[warning] 25-25: wrong indentation: expected 10 but found 8
(indentation)
[warning] 30-30: wrong indentation: expected 10 but found 8
(indentation)
[warning] 32-32: wrong indentation: expected 10 but found 8
(indentation)
[warning] 37-37: wrong indentation: expected 10 but found 8
(indentation)
[warning] 43-43: wrong indentation: expected 10 but found 8
(indentation)
[warning] 50-50: wrong indentation: expected 10 but found 8
(indentation)
[warning] 56-56: wrong indentation: expected 10 but found 8
(indentation)
[warning] 58-58: wrong indentation: expected 10 but found 8
(indentation)
[error] 89-89: no new line character at the end of file
(new-line-at-end-of-file)
📜 Review details
Configuration used: CodeRabbit UI
Review profile: CHILL
⛔ Files ignored due to path filters (1)
go.sumis excluded by!**/*.sum
📒 Files selected for processing (17)
.gitignore(1 hunks).golangci.yaml(1 hunks).sops.yaml(1 hunks)Dockerfile(1 hunks)README_CN.md(1 hunks)README_ES.md(1 hunks)README_ID.md(1 hunks)README_JP.md(1 hunks)README_KR.md(1 hunks)go.mod(10 hunks)pkg/authprovider/authx/file.go(2 hunks)pkg/authprovider/authx/file_test.go(1 hunks)pkg/authprovider/authx/testData/example-auth.enc.yaml(1 hunks)pkg/authprovider/authx/testData/private.asc(1 hunks)pkg/authprovider/authx/testData/public.asc(1 hunks)pkg/js/devtools/bindgen/generator.go(3 hunks)pkg/js/devtools/scrapefuncs/main.go(1 hunks)
🚧 Files skipped from review as they are similar to previous changes (12)
- .gitignore
- README_KR.md
- pkg/authprovider/authx/testData/public.asc
- pkg/js/devtools/scrapefuncs/main.go
- README_JP.md
- README_ES.md
- .golangci.yaml
- README_CN.md
- pkg/authprovider/authx/file_test.go
- pkg/js/devtools/bindgen/generator.go
- README_ID.md
- Dockerfile
🧰 Additional context used
📓 Learnings (1)
pkg/authprovider/authx/testData/private.asc (1)
Learnt from: dwisiswant0
PR: projectdiscovery/nuclei#5841
File: pkg/authprovider/authx/testData/private.asc:1-82
Timestamp: 2024-12-03T16:50:05.259Z
Learning: Test private keys stored in `pkg/authprovider/authx/testData/` are acceptable to include in the repository for testing purposes.
🪛 yamllint (1.35.1)
.sops.yaml
[error] 4-4: no new line character at the end of file
(new-line-at-end-of-file)
pkg/authprovider/authx/testData/example-auth.enc.yaml
[warning] 14-14: wrong indentation: expected 10 but found 8
(indentation)
[warning] 18-18: wrong indentation: expected 10 but found 8
(indentation)
[warning] 23-23: wrong indentation: expected 10 but found 8
(indentation)
[warning] 25-25: wrong indentation: expected 10 but found 8
(indentation)
[warning] 30-30: wrong indentation: expected 10 but found 8
(indentation)
[warning] 32-32: wrong indentation: expected 10 but found 8
(indentation)
[warning] 37-37: wrong indentation: expected 10 but found 8
(indentation)
[warning] 43-43: wrong indentation: expected 10 but found 8
(indentation)
[warning] 50-50: wrong indentation: expected 10 but found 8
(indentation)
[warning] 56-56: wrong indentation: expected 10 but found 8
(indentation)
[warning] 58-58: wrong indentation: expected 10 but found 8
(indentation)
[error] 89-89: no new line character at the end of file
(new-line-at-end-of-file)
🪛 Gitleaks (8.21.2)
pkg/authprovider/authx/testData/private.asc
1-82: Identified a Private Key, which may compromise cryptographic security and sensitive data encryption.
(private-key)
🔇 Additional comments (11)
.sops.yaml (1)
3-4: Review the encrypted_regex pattern scope
The current pattern includes generic terms like 'value' and 'key' which might lead to over-encryption. Consider:
- Being more specific about which fields should be encrypted
- Adding comments to explain why these specific fields were chosen
Let's analyze the impact:
Consider a more targeted approach:
creation_rules:
- - encrypted_regex: ^(password|username|token|value|key|raw)$
+ - encrypted_regex: ^(password|secret|token|api_key|access_key|private_key)$✅ Verification successful
The current encrypted_regex pattern is appropriate for the codebase
Based on the analysis of YAML files in the codebase, the current pattern ^(password|username|token|value|key|raw)$ is actually well-suited because:
- The codebase contains sensitive authentication data in key-value pairs, as seen in
pkg/authprovider/authx/testData/example-auth.yamlwith API keys, tokens, and passwords - The pattern successfully encrypts these values as evidenced by the encrypted data in
example-auth.enc.yaml - Generic terms like 'value' and 'key' are necessary because sensitive data appears in different formats:
x-pdcp-key: <api-key-here>token: 1a2b3c4d5e6f7g8h9i0jkey: PHPSESSID
The broad pattern ensures comprehensive encryption of potential secrets while false positives are acceptable for security-critical configuration.
🏁 Scripts executed
The following scripts were executed for the analysis:
Script:
#!/bin/bash
# Search for potential secret files to understand field naming patterns
fd -e yaml -e yml | xargs grep -l 'value:\|key:' | while read -r file; do
echo "=== $file ==="
grep -A 2 'value:\|key:' "$file" || true
done
Length of output: 3582
🧰 Tools
🪛 yamllint (1.35.1)
[error] 4-4: no new line character at the end of file
(new-line-at-end-of-file)
pkg/authprovider/authx/testData/private.asc (1)
1-82: LGTM!
The private key is acceptable as it's used for testing purposes.
🧰 Tools
🪛 Gitleaks (8.21.2)
1-82: Identified a Private Key, which may compromise cryptographic security and sensitive data encryption.
(private-key)
pkg/authprovider/authx/file.go (2)
228-232: LGTM!
The return statements correctly handle both YAML and JSON data.
218-225: 🛠️ Refactor suggestion
Use error types instead of string comparison.
Comparing error messages as strings is fragile. Consider using error types or variables.
Apply this diff:
- if err.Error() == "sops metadata not found" { // regular file (prolly not encrypted)
+ if errors.Is(err, sops.MetadataNotFoundErr) { // regular file (probably not encrypted)Likely invalid or redundant comment.
go.mod (3)
Line range hint 426-427: Note the version retraction
The retraction of v3.2.0 due to a JavaScript protocol issue is properly documented. This helps users avoid the problematic version.
71-71: Verify SOPS integration dependencies
The addition of SOPS and related cloud dependencies aligns with the PR objectives for implementing SOPS integration. However, ensure all necessary cloud KMS providers are included based on the requirements.
Also applies to: 121-129
✅ Verification successful
Let me check the actual SOPS implementation and cloud KMS provider usage to verify the integration.
Let me check one more aspect to see how SOPS is being used in the implementation.
SOPS integration is correctly implemented with minimal dependencies
The implementation uses SOPS's core decryption functionality (decrypt.File) with a fallback for non-encrypted files. The absence of explicit cloud KMS provider imports suggests the implementation relies on SOPS's built-in provider support, which is the recommended approach as it allows users to configure their preferred KMS through SOPS configuration files rather than hardcoding provider dependencies.
🏁 Scripts executed
The following scripts were executed for the analysis:
Script:
#!/bin/bash
# Description: Verify SOPS-related imports in the codebase
# Expected: Find usage of SOPS and cloud KMS providers
# Check for SOPS usage
echo "Checking SOPS usage:"
rg -l "github.com/getsops/sops"
# Check for cloud KMS imports
echo -e "\nChecking Cloud KMS usage:"
rg -l "cloud.google.com/go/kms|github.com/Azure/azure-sdk-for-go/sdk/security/keyvault|github.com/aws/aws-sdk-go-v2/service/kms"
Length of output: 375
Script:
#!/bin/bash
# Check the actual SOPS implementation in the files
echo "SOPS implementation in file.go:"
rg -A 5 "github.com/getsops/sops" pkg/authprovider/authx/file.go
echo -e "\nCloud KMS configurations:"
# Look for cloud KMS configuration or initialization
rg -A 5 "kms|keyvault" pkg/authprovider/authx/file.go
Length of output: 476
Script:
#!/bin/bash
# Look for SOPS-related function implementations
echo "SOPS usage in file.go:"
ast-grep --pattern 'func $$$decrypt$$$($$$) {
$$$
}' pkg/authprovider/authx/file.go
# Check for any SOPS configuration or setup
echo -e "\nSOPS configuration:"
rg -B 2 -A 5 "sops" pkg/authprovider/authx/file.go
Length of output: 755
58-62: Review significant AWS SDK version updates
The AWS SDK components have been significantly upgraded (e.g., aws-sdk-go-v2 from v1.19.0 to v1.32.6). While keeping dependencies updated is good practice, such major version jumps warrant careful testing.
✅ Verification successful
Let me check the actual AWS SDK usage in these files to better understand the impact of the version changes.
Let me check the git history to see if there were any recent changes to these AWS SDK-using files that might be affected by the version upgrade.
AWS SDK upgrade appears to be a safe dependency update
Based on the analysis:
- The AWS SDK is used in three main files for basic AWS operations (S3 and signing)
- No recent changes were made to these files
- The upgrade follows AWS SDK's semantic versioning (v1.x.x series)
- The identified usages are for standard AWS operations (S3 client, credentials, config) which are stable APIs
Since this is a minor version upgrade within the v1 series of aws-sdk-go-v2 (maintaining backward compatibility) and the codebase uses standard AWS SDK features, this update appears to be safe.
🏁 Scripts executed
The following scripts were executed for the analysis:
Script:
#!/bin/bash
# Description: Check for AWS SDK usage in the codebase
# Expected: Identify files using AWS SDK to ensure compatibility
echo "Checking AWS SDK usage:"
rg -l "github.com/aws/aws-sdk-go-v2"
Length of output: 207
Script:
#!/bin/bash
# Check the actual AWS SDK imports and usage in the identified files
echo "=== Checking AWS SDK usage in aws-sign.go ==="
rg -A 5 "github.com/aws/aws-sdk-go-v2" pkg/protocols/http/signer/aws-sign.go
echo -e "\n=== Checking AWS SDK usage in s3.go ==="
rg -A 5 "github.com/aws/aws-sdk-go-v2" pkg/external/customtemplates/s3.go
echo -e "\n=== Checking AWS SDK usage in catalog.go ==="
rg -A 5 "github.com/aws/aws-sdk-go-v2" pkg/catalog/aws/catalog.go
Length of output: 1676
Script:
#!/bin/bash
# Check recent changes in files using AWS SDK
echo "=== Checking recent changes in AWS SDK files ==="
git diff --stat HEAD~5 HEAD pkg/protocols/http/signer/aws-sign.go pkg/external/customtemplates/s3.go pkg/catalog/aws/catalog.go
# Check if there are any failing tests related to AWS functionality
echo -e "\n=== Checking for AWS-related test files ==="
fd -e go -e yaml test.*aws
Length of output: 370
pkg/authprovider/authx/testData/example-auth.enc.yaml (4)
41-45: Review domain regex patterns for security implications
This is a duplicate of a previous comment about the security implications of using loose regex patterns for domains.
🧰 Tools
🪛 yamllint (1.35.1)
[warning] 43-43: wrong indentation: expected 10 but found 8
(indentation)
48-48: Replace hardcoded template path with configurable value
This is a duplicate of a previous comment about using configurable paths instead of hardcoded values.
88-89: 🛠️ Refactor suggestion
Enhance SOPS configuration and fix file format
- Add newline at EOF (required by YAML spec)
- Expand encrypted_regex pattern to cover more sensitive fields
- Consider using a more recent SOPS version if available
- encrypted_regex: ^(password|username|token|value|key|raw)$
- version: 3.9.1
+ encrypted_regex: ^(password|username|token|value|key|raw|secret|credential|api[_-]?key|private[_-]?key)$
+ version: 3.9.1
+✅ Verification successful
Update SOPS version and ensure comprehensive encryption coverage
- Upgrade SOPS from v3.9.1 to v3.9.2 for latest security fixes
- Current encrypted_regex pattern is adequate as the grep results show encrypted values (ENC[AES256_GCM,...]) for sensitive fields
- File format issue needs fixing (missing newline)
encrypted_regex: ^(password|username|token|value|key|raw)$
- version: 3.9.1
+ version: 3.9.2
+🏁 Scripts executed
The following scripts were executed for the analysis:
Script:
#!/bin/bash
# Check latest SOPS version and security advisories
curl -s https://api.github.com/repos/getsops/sops/releases/latest | jq .tag_name
# Verify all sensitive fields are encrypted
rg -l "password:|token:|key:|secret:|credential:" | xargs cat | grep -E "password:|token:|key:|secret:|credential:"
Length of output: 2839
🧰 Tools
🪛 yamllint (1.35.1)
[error] 89-89: no new line character at the end of file
(new-line-at-end-of-file)
12-45: 🛠️ Refactor suggestion
Add validation constraints for authentication configurations
Consider adding constraints to prevent misuse:
- Maximum token length limits
- Allowed character sets for credentials
- Required domain verification
- type: header
+ constraints:
+ max_token_length: 256
+ allowed_chars: "^[a-zA-Z0-9_-]+$"
domains:
- api.projectdiscovery.io🧰 Tools
🪛 yamllint (1.35.1)
[warning] 14-14: wrong indentation: expected 10 but found 8
(indentation)
[warning] 18-18: wrong indentation: expected 10 but found 8
(indentation)
[warning] 23-23: wrong indentation: expected 10 but found 8
(indentation)
[warning] 25-25: wrong indentation: expected 10 but found 8
(indentation)
[warning] 30-30: wrong indentation: expected 10 but found 8
(indentation)
[warning] 32-32: wrong indentation: expected 10 but found 8
(indentation)
[warning] 37-37: wrong indentation: expected 10 but found 8
(indentation)
[warning] 43-43: wrong indentation: expected 10 but found 8
(indentation)
Signed-off-by: Dwi Siswanto <git@dw1.io>
dwisiswant0
force-pushed
the
dwisiswant0/feat/authx/sops-integration
branch
from
3a5b690 to
e84048c
Compare
|
looks like this goroutine leak is happening within the SOPS package itself. $ go mod why go.opencensus.io/stats/view
# go.opencensus.io/stats/view
github.com/projectdiscovery/nuclei/v3/pkg/authprovider/authx
github.com/getsops/sops/v3/decrypt
github.com/getsops/sops/v3/config
github.com/getsops/sops/v3/gcpkms
cloud.google.com/go/kms/apiv1
google.golang.org/api/transport/grpc
go.opencensus.io/plugin/ocgrpc
go.opencensus.io/stats/viewi'll try raising an issue along with patches to them if possible. UPDATE: looks like this gets spawned in |
Signed-off-by: Dwi Siswanto <git@dw1.io>
Proposed changes
Close #5840
Note
github.com/getsops/sops/v3requires go >= 1.22.How has this been tested?
1. Build
2. Test
3. Proof
With encrypted values:
Regular secret file (non-encrypted):
4. Cleanup
Checklist
Summary by CodeRabbit
Release Notes
New Features
.sops.yaml.example-auth.enc.yaml, containing static and dynamic secrets.Documentation
Chores
go.mod..gitignoreto track.sops.yamland.golangci.yamlfiles..golangci.yamlfor excluding specific linting rules.Style