rpm: consult dnf database for repository information #869

hdonnay · 2023-03-13T20:42:36Z

These changes make the rhel machinery consult both the rpm and dnf databases when possible.

See-also: #809

codecov · 2024-01-12T17:19:14Z

Codecov Report

Attention: Patch coverage is 29.52381% with 74 lines in your changes missing coverage. Please review.

Project coverage is 55.23%. Comparing base (c2fc225) to head (57bb054).

Files	Patch %	Lines
rpm/dnf.go	25.71%	52 Missing ⚠️
rpm/packagescanner.go	14.28%	18 Missing ⚠️
rhel/coalescer.go	57.14%	1 Missing and 2 partials ⚠️
rpm/native_db.go	85.71%	1 Missing ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##             main     #869      +/-   ##
==========================================
- Coverage   55.39%   55.23%   -0.17%     
==========================================
  Files         282      283       +1     
  Lines       17836    17923      +87     
==========================================
+ Hits         9881     9899      +18     
- Misses       6923     6990      +67     
- Partials     1032     1034       +2

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

hdonnay · 2024-12-19T23:12:57Z

rpm/dnf.go

+	if fi, err := fs.Stat(sys, `root/buildinfo/content_manifests`); errors.Is(err, nil) && fi.IsDir() {
+		// This is a RHEL layer, skip.
+		return nil, nil
+	}


Update this to check the files for a marker value and continue if found.

Should look for from_dnf_hint key in the yaml files.
See also: containerbuildsystem/atomic-reactor#2140

Punting on this unless we determine it's needed.

cgwalters · 2025-01-06T13:42:12Z

I created rpm-software-management/rpm#3505 related to this

Signed-off-by: Hank Donnay <hdonnay@redhat.com>

Using these should give us a nice "todo" list when the time comes. Signed-off-by: Hank Donnay <hdonnay@redhat.com>

Signed-off-by: Hank Donnay <hdonnay@redhat.com>

This adds two helpers: one to discover repoids, and one to annotate a stream of Packages. Signed-off-by: Hank Donnay <hdonnay@redhat.com>

Signed-off-by: Hank Donnay <hdonnay@redhat.com>

RTann

made it through most of the files, but I'll need to come back to this another time. Would you mind making more commits and then squashing things at the end so it's easier to see the diffs after each round? Thanks

RTann · 2025-01-22T22:14:53Z

internal/dnf/dnf.go

+// outside of the Red Hat build system's builder's context. See [CLAIRDEV-45]
+// for more information.
+//
+// [CLAIRDEV-45]: https://issues.redhat.com/browse/CLAIRDEV-45


This ticket specifically calls out OSBS. What about Konflux? Should we update the ticket to mention Konflux, or is this not true for Konflux?

RTann · 2025-01-22T22:16:29Z

internal/dnf/dnf.go

+	"github.com/quay/claircore/internal/rpm"
+	"github.com/quay/zlog"
+	_ "modernc.org/sqlite" // register the sqlite driver


just sanity checking here

"github.com/quay/claircore/internal/rpm" is in the same module, so I expected to see this in the last block

RTann · 2025-01-22T22:17:45Z

internal/dnf/dnf.go

+	Wrap(context.Context, iter.Seq[claircore.Package]) (iter.Seq[claircore.Package], func() error)
+}
+
+// Identity is an [Annotator] who's [Annotator.Wrap] method does nothing.


RTann · 2025-01-22T22:18:25Z

internal/dnf/dnf.go

+var Identity Annotator
+
+func init() {
+	Identity = ident{}
+}


curious why not just var Identity = ident{}?

RTann · 2025-01-22T22:19:54Z

internal/dnf/dnf.go

+	return nil
+}
+
+// NewAnnotator holds book-keeping for producing multiple independent mapping


This is where the convention of doing something like NewAnnotator instead of newAnnotator for the godoc confuses me because now we have two of these

RTann · 2025-01-23T00:03:00Z

internal/dnf/sql/repoid_for_package.sql

looks good, but I don't know the tables well enough to verify this is definitely the right query

RTann · 2025-01-23T00:05:04Z

internal/dnf/dnf.go

+					continue
+				default:
+					final = fmt.Errorf("internal/dnf: database error: %w", err)
+					return


I'm still getting used to iterators in Go. It's ok to have a path where we don't yield?

RTann · 2025-01-23T01:19:57Z

internal/rpm/find.go

+		switch n {
+		case `Packages`:
+			db.kind = kindBDB
+			if ok, err := testpath(p, bdb.CheckMagic); !ok {


confirming we want to ignore true, err != nil?

RTann · 2025-01-23T01:40:15Z

internal/rpm/packages.go

+		blobs, dbErr := db.All(ctx)
+		seq, parseErr := parseBlob(ctx, blobs)
+		defer func() {
+			final = errors.Join(err, parseErr(), dbErr(), db.Close())


just confirming we want to close the DB here?

RTann · 2025-01-23T01:46:02Z

internal/rpm/info.go

+	//
+	// [yara]: https://github.com/VirusTotal/yara
+	pat := []string{
+		`^.*/[^/]+\.jar$`, // Jar files


I know this is copied over, but I'm wondering if we should extend this to the other file extensions we support like WAR, EAR, etc?

hdonnay force-pushed the hack/dnf branch from 4b2aa2d to 9284a22 Compare April 5, 2023 17:55

hdonnay force-pushed the hack/dnf branch 2 times, most recently from 5acce04 to 5a9d76e Compare April 13, 2023 15:03

hdonnay mentioned this pull request Jan 12, 2024

rpm et. al.: lints #1205

Merged

hdonnay force-pushed the hack/dnf branch from 5a9d76e to cb74edd Compare January 12, 2024 17:10

hdonnay force-pushed the hack/dnf branch from cb74edd to 9e040ff Compare January 12, 2024 19:32

hdonnay force-pushed the hack/dnf branch from 9e040ff to 57bb054 Compare August 27, 2024 16:41

hdonnay force-pushed the hack/dnf branch from 57bb054 to f86dd34 Compare December 19, 2024 22:59

hdonnay changed the title ~~wip: rpm: consult dnf database for repository information~~ rpm: consult dnf database for repository information Dec 19, 2024

hdonnay force-pushed the hack/dnf branch from f86dd34 to 18e9818 Compare December 19, 2024 23:04

hdonnay commented Dec 19, 2024

View reviewed changes

hdonnay force-pushed the hack/dnf branch from 18e9818 to 69ba836 Compare December 20, 2024 22:17

hdonnay force-pushed the hack/dnf branch 3 times, most recently from 10b07ff to e59f165 Compare January 13, 2025 15:57

hdonnay force-pushed the hack/dnf branch from e59f165 to 4862693 Compare January 22, 2025 19:23

hdonnay added 6 commits January 22, 2025 13:52

internal/rpm: new iterator-based rpm package

58c5c61

Signed-off-by: Hank Donnay <hdonnay@redhat.com>

internal/rpm/sqlite: better error return for Validate

787c3e0

Signed-off-by: Hank Donnay <hdonnay@redhat.com>

wart: add helpers for iter.Seq[T] implementations

78c26b0

Using these should give us a nice "todo" list when the time comes. Signed-off-by: Hank Donnay <hdonnay@redhat.com>

rpm: move to use new internal/rpm package

cc4677a

Signed-off-by: Hank Donnay <hdonnay@redhat.com>

internal/rpm: change "RepositoryHint" format

79f59a7

Signed-off-by: Hank Donnay <hdonnay@redhat.com>

internal/dnf: add dnf-handling package

1eb75f1

This adds two helpers: one to discover repoids, and one to annotate a stream of Packages. Signed-off-by: Hank Donnay <hdonnay@redhat.com>

hdonnay force-pushed the hack/dnf branch from 4862693 to 8c86d88 Compare January 22, 2025 19:54

hdonnay added 5 commits January 22, 2025 13:58

rpmtest: allow specifying repository hint

77b8b68

Signed-off-by: Hank Donnay <hdonnay@redhat.com>

rhel: add package scanner

ff75397

Signed-off-by: Hank Donnay <hdonnay@redhat.com>

rhel: fix testcase names

9f75bb4

Signed-off-by: Hank Donnay <hdonnay@redhat.com>

rhel: have repository scanner consult dnf

26642a6

Signed-off-by: Hank Donnay <hdonnay@redhat.com>

rhel: use repoids in coalescer

9ea862e

Signed-off-by: Hank Donnay <hdonnay@redhat.com>

containerapi: add deprecation notices

7a459ba

Signed-off-by: Hank Donnay <hdonnay@redhat.com>

hdonnay force-pushed the hack/dnf branch from 8c86d88 to 7a459ba Compare January 22, 2025 20:08

hdonnay marked this pull request as ready for review January 22, 2025 20:22

hdonnay requested a review from a team as a code owner January 22, 2025 20:22

hdonnay requested review from RTann and removed request for a team January 22, 2025 20:22

RTann requested a review from crozzy January 22, 2025 22:11

RTann reviewed Jan 23, 2025

View reviewed changes

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

rpm: consult dnf database for repository information #869

rpm: consult dnf database for repository information #869

hdonnay commented Mar 13, 2023 •

edited

Loading

codecov bot commented Jan 12, 2024 •

edited

Loading

hdonnay Dec 19, 2024

hdonnay Jan 22, 2025

cgwalters commented Jan 6, 2025

RTann left a comment

RTann Jan 22, 2025

RTann Jan 22, 2025

RTann Jan 22, 2025

RTann Jan 22, 2025

RTann Jan 22, 2025

RTann Jan 23, 2025

RTann Jan 23, 2025

RTann Jan 23, 2025

RTann Jan 23, 2025

RTann Jan 23, 2025

rpm: consult dnf database for repository information #869

Are you sure you want to change the base?

rpm: consult dnf database for repository information #869

Conversation

hdonnay commented Mar 13, 2023 • edited Loading

codecov bot commented Jan 12, 2024 • edited Loading

Codecov Report

Choose a reason for hiding this comment

Choose a reason for hiding this comment

cgwalters commented Jan 6, 2025

RTann left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

hdonnay commented Mar 13, 2023 •

edited

Loading

codecov bot commented Jan 12, 2024 •

edited

Loading