[sos] Add 'upload' component to upload existing reports and files #3746

jcastill · 2024-08-09T12:00:52Z

This commit marks the beginning of the addition of a new 'upload' component for sos, which can be used to upload already created sos reports, collects, or other files like logs or vmcores to a policy defined location.

The user needs to specify a file location, and can make use of any of the options that exist nowadays for the --upload option.

This first commit includes:

The initial framework for the 'upload' component.
The new man page for 'sos upload'.
The code in the component 'help' to show information about the component.
The code in sos/init.py to deal with the component.
And modifications to setup.py to build the man pages.

Related: RHEL-23032, SUPDEV-138, CLIOT-481

Please place an 'X' inside each '[]' to confirm you adhere to our Contributor Guidelines

Is the commit message split over multiple lines and hard-wrapped at 72 characters?
Is the subject and message clear and concise?
Does the subject start with [plugin_name] if submitting a plugin patch or a [section_name] if part of the core sosreport code?
Does the commit contain a Signed-off-by: First Lastname email@example.com?
Are any related Issues or existing PRs properly referenced via a Closes (Issue) or Resolved (PR) line?
Are all passwords or private data gathered by this PR obfuscated?

jcastill · 2024-08-09T12:01:41Z

Huge thanks to @pmoravec for all the help reviewing this, suggesting improvements, and finding bugs.
This is a first implementation of the 'upload' component, that can help users to upload already created sos or other files that support organizations can find useful.
In the future, the idea is to hook 'report' and 'collector' components to this one if the maintainers think is a good idea.

packit-as-a-service · 2024-08-09T12:03:53Z

Congratulations! One of the builds has completed. 🍾

You can install the built RPMs by following these steps:

sudo yum install -y dnf-plugins-core on RHEL 8
sudo dnf install -y dnf-plugins-core on Fedora
dnf copr enable packit/sosreport-sos-3746
And now you can install the packages.

Please note that the RPMs should be used only in a testing environment.

arif-ali

Great idea

Some initial comments

man/en/sos-upload.1

sos/upload/__init__.py

jcastill · 2024-08-09T12:37:07Z

@arif-ali about these ones:

sos/upload/init.py:103:0: C0325: Unnecessary parens after '=' keyword (superfluous-parens)
Fixed
sos/upload/init.py:141:0: C0325: Unnecessary parens after 'if' keyword (superfluous-parens)
Fixed
sos/upload/init.py:142:0: C0325: Unnecessary parens after 'if' keyword (superfluous-parens)
Fixed

************* Module sos.upload
sos/upload/init.py:47:8: R1725: Consider using Python 3 style super() without arguments (super-with-arguments)
Do I need to fix this one? Are we doing the same in the rest of the code, or is a nice-thing to have but not necessary?

sos/upload/init.py:42:46: W0613: Unused argument 'in_place' (unused-argument)
This is there to help with hooking report and others in the future, but can remove it now.

sos/upload/init.py:43:17: W0613: Unused argument 'hook_commons' (unused-argument)
Same as above as far as I know.

sos/upload/init.py:155:24: R1722: Consider using 'sys.exit' instead (consider-using-sys-exit)
I'll make this change now.

arif-ali · 2024-08-09T13:15:58Z

With R1725, I made the changes a few months back, and hence enabled the check, so let's do this here too.

With the unused variable. If your 100% sure you're going to be using them then potentially you could add the following before the line

#pylint: disable=unused-argument

jcastill · 2024-08-09T13:49:44Z

With R1725, I made the changes a few months back, and hence enabled the check, so let's do this here too.

Done, should be in the version I just pushed.

With the unused variable. If your 100% sure you're going to be using them then potentially you could add the following before the line
#pylint: disable=unused-argument

Nice one! But I ended up removing it. I'll re-add them in the future when I have ready the code for hooking report etc.

pmoravec · 2024-08-13T11:08:38Z

man/en/sos-upload.1

+
+.PP
+.SH DESCRIPTION
+upload is an sos subcommand to upload sos reports, logs, vmcores, or other files to a policy defined remote location, or an user defined one.


Nitpick: s/an user/a user/.

I think is 'an' because 'user' starts with a vowel, no?

The rule is "first sound of word", not first letter :) E.g. https://english.stackexchange.com/questions/105116/is-it-a-user-or-an-user (though I am not sure how authoritative that source is).

Nice, TIL. Fixed in the next push.

man/en/sos-upload.1

sos/help/__init__.py

pmoravec · 2024-08-13T11:25:46Z

sos/policies/distros/redhat.py

@@ -298,7 +298,7 @@ def get_upload_url(self):
            self.ui_log.info("No case id provided, uploading to SFTP")
            return RH_SFTP_HOST
        rh_case_api = "/support/v1/cases/%s/attachments"
-        return RH_API_HOST + rh_case_api % self.case_id
+        return RH_API_HOST + rh_case_api % self.commons['cmdlineopts'].case_id


Why this change? AFAIK self.case_id might be blank (and common's case_id set) only in scenario "case id not in cmdline, batch not in cmdline" - should not upload query for case_id, then? (or am I wrong here with my assumption)?

(that concern is valid for sure (while I can be wrong on its impact to this code change):

# python3 bin/sos upload /var/tmp/sosreport-pmoravec-rhel8-012345678-2024-08-13-gbiatgg.tar.xz sos upload (version 4.7.2) This utility is used to upload files to a policy-default location. The archive to be uploaded may contain data considered sensitive and its content should be reviewed by the originating organization before being passed to any third party. No configuration changes will be made to the system running this utility. Press ENTER to continue, or CTRL-C to quit Attempting to upload file /var/tmp/sosreport-pmoravec-rhel8-012345678-2024-08-13-gbiatgg.tar.xz to case No case id provided, uploading to SFTP No case id provided, uploading to SFTP Attempting upload to Red Hat Secure FTP ..

This is one of the things we talked about internally when I first started playing with 'upload'. If you remember, the issue was that without this change, we were getting 'None' on the case_id and it was failing to build the url, and so failed to upload. I have the feeling that I've done something wrong on the upload side and I'm not passing the case_id correctly.
My hope is that more experienced eyes, or at least fresher, can tell me where I'm failing.

sos/policies/distros/redhat.py

pmoravec · 2024-08-13T11:38:58Z

When I run:

python3 bin/sos upload /var/tmp/sosreport-pmoravec-rhel8-01234567-2024-08-13-gbiatgg.tar.xz

and "Press ENTER to continue", and then nothing, then I get a final error:

..
Attempting to upload file /var/tmp/sosreport-pmoravec-rhel8-01234567-2024-08-13-gbiatgg.tar.xz to case 
No case id provided, uploading to SFTP
No case id provided, uploading to SFTP
Attempting upload to Red Hat Secure FTP
Please visit the following URL to authenticate this device: https://sso.redhat.com/device?user_code=SOME-CODE
User anonUser used for anonymous upload. Please inform your support engineer so they may retrieve the data.
Upload attempt failed: 'RHELPolicy' object has no attribute 'upload_directory'

I think the upload did not succeed at the end..

pmoravec · 2024-08-13T11:40:31Z

sos/upload/__init__.py

+    defined location. These files can be either sos reports,
+    sos collections, or other kind of files like: vmcores,
+    application cores, logs, etc.
+


Extra line..?

I don't think this has been addressed, we could just remove the extra line

sos/upload/__init__.py

pmoravec · 2024-08-13T11:55:27Z

When pressing Ctrl+C on ress ENTER to continue, or CTRL-C to quit prompt, I get backtrace:

Press ENTER to continue, or CTRL-C to quit
^CTraceback (most recent call last):
  File "/root/sos-main/sos/upload/__init__.py", line 120, in intro
    input(prompt)
KeyboardInterrupt

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "bin/sos", line 22, in <module>
    sos.execute()
  File "/root/sos-main/sos/__init__.py", line 187, in execute
    self._component.execute()
  File "/root/sos-main/sos/upload/__init__.py", line 137, in execute
    self.intro()
  File "/root/sos-main/sos/upload/__init__.py", line 123, in intro
    self.exit("Exiting on user cancel", 130)
AttributeError: 'SoSUpload' object has no attribute 'exit'

jcastill · 2024-08-13T12:18:40Z

@pmoravec thank you for finding this, I thought we solved these issues:

When I run:

python3 bin/sos upload /var/tmp/sosreport-pmoravec-rhel8-01234567-2024-08-13-gbiatgg.tar.xz

and "Press ENTER to continue", and then nothing, then I get a final error:

..
Attempting to upload file /var/tmp/sosreport-pmoravec-rhel8-01234567-2024-08-13-gbiatgg.tar.xz to case 
No case id provided, uploading to SFTP
No case id provided, uploading to SFTP

I'll check the double messaging here, looks horrible.

Attempting upload to Red Hat Secure FTP
Please visit the following URL to authenticate this device: https://sso.redhat.com/device?user_code=SOME-CODE
User anonUser used for anonymous upload. Please inform your support engineer so they may retrieve the data.
Upload attempt failed: 'RHELPolicy' object has no attribute 'upload_directory'

I'll check this one as well, I remember we had a similar issue with a previous implementation.

I think the upload did not succeed at the end..

No, it should not succeed in that case.

jcastill · 2024-08-13T12:19:26Z

When pressing Ctrl+C on ress ENTER to continue, or CTRL-C to quit prompt, I get backtrace:

Press ENTER to continue, or CTRL-C to quit
^CTraceback (most recent call last):
  File "/root/sos-main/sos/upload/__init__.py", line 120, in intro
    input(prompt)
KeyboardInterrupt

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "bin/sos", line 22, in <module>
    sos.execute()
  File "/root/sos-main/sos/__init__.py", line 187, in execute
    self._component.execute()
  File "/root/sos-main/sos/upload/__init__.py", line 137, in execute
    self.intro()
  File "/root/sos-main/sos/upload/__init__.py", line 123, in intro
    self.exit("Exiting on user cancel", 130)
AttributeError: 'SoSUpload' object has no attribute 'exit'

I'll check this, should be easy to fix.

jcastill · 2024-08-13T12:35:59Z

When pressing Ctrl+C on ress ENTER to continue, or CTRL-C to quit prompt, I get backtrace:

Press ENTER to continue, or CTRL-C to quit
^CTraceback (most recent call last):
  File "/root/sos-main/sos/upload/__init__.py", line 120, in intro
    input(prompt)
KeyboardInterrupt

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "bin/sos", line 22, in <module>
    sos.execute()
  File "/root/sos-main/sos/__init__.py", line 187, in execute
    self._component.execute()
  File "/root/sos-main/sos/upload/__init__.py", line 137, in execute
    self.intro()
  File "/root/sos-main/sos/upload/__init__.py", line 123, in intro
    self.exit("Exiting on user cancel", 130)
AttributeError: 'SoSUpload' object has no attribute 'exit'

Fixed. I used exit() instead of _exit(), which is the one implemented in Soscomponent.

TurboTurtle

At a bare minimum, a new component should be implementing all the abstractions that it needs to operate solo, not acting as a wrapper to existing functionality.

This means the upload logic needs to be separated from its current location in Policy, and implemented as a discrete unit. Policy should then control the default setting, and users should be able to direct sos to chose an upload target/profile/whatever we want to call it as an override. E.G. if I have an sos report locally on my Fedora workstation that was taken from a RHEL box, and I am unable due to some network policy to directly upload from the RHEL box, then on my Fedora system I should be able to send that sos report to Red Hat.

Further, any current or future usage of the component's functionality should go through the actual component code. Much like we do with sos clean, when --clean is used for a report being generated. We hook into the component from within report, to ensure we use the exact code flow for cleaning the archive as we would by running a clean after-the-fact.

jcastill · 2024-08-13T15:32:12Z

At a bare minimum, a new component should be implementing all the abstractions that it needs to operate solo, not acting as a wrapper to existing functionality.

This means the upload logic needs to be separated from its current location in Policy, and implemented as a discrete unit. Policy should then control the default setting, and users should be able to direct sos to chose an upload target/profile/whatever we want to call it as an override. E.G. if I have an sos report locally on my Fedora workstation that was taken from a RHEL box, and I am unable due to some network policy to directly upload from the RHEL box, then on my Fedora system I should be able to send that sos report to Red Hat.

Further, any current or future usage of the component's functionality should go through the actual component code. Much like we do with sos clean, when --clean is used for a report being generated. We hook into the component from within report, to ensure we use the exact code flow for cleaning the archive as we would by running a clean after-the-fact.

I agree with everything above, but the idea behind this PR is to be a first implementation to get the upload component started, and then move things carefully from policy to upload. Could this approach be acceptable?

pmoravec · 2024-08-14T10:06:11Z

I support this initial implementation of the feature to let enhance sos capabilities for a low cost. The discussion about refactorization (what precisely should be moved to some new *Upload* classes) can be lengthy, while we can already offer this feature as is.

I was thinking to raise the same concern, but I realized I would see beneficial for the discussion about refactorization if we already has some implementation in hand. With the current code, it is hard for me to specify "cut this away from here and put it (there)", if we have no "(there)". With the SosUpload class, I have better description of the "(there)". So having this initial implementation merged, it will be much more easier to have tat conversation - at least for me, this can be subjective.

If somebody sees as a potential threat "we accept this initial implementation, but will never refactor the code as needed, and we dont want that technical debt here", then I can make a commitment: once there will be an agreement about the refactorisation and if nobody(*) will have time to implement it, I will work on such PR.

(*) nobody including Jose as the primary person to implement. I assume he will be the primary person to make his own feature to make it complete. On the other side, there can be various reasons he won't be able to do the refactorisation (time, other work on sos, willingness, whatever). And then anybody else (with me as the volunteer with above commitment) can contribute that way.

jcastill · 2024-08-14T10:27:15Z

If somebody sees as a potential threat "we accept this initial implementation, but will never refactor the code as needed, and we dont want that technical debt here", then I can make a commitment: once there will be an agreement about the refactorisation and if nobody(*) will have time to implement it, I will work on such PR.

(*) nobody including Jose as the primary person to implement. I assume he will be the primary person to make his own feature to make it complete. On the other side, there can be various reasons he won't be able to do the refactorisation (time, other work on sos, willingness, whatever). And then anybody else (with me as the volunteer with above commitment) can contribute that way.

On this note, I already started moving things around from policies/distros just after I sent this PR - this is not something I want to leave abandoned, or done in six months time or more, but as soon as possible. But also I want to make sure I cover all the possible cases, and the upload code in policies has been there for a long time, working perfectly, so want to be extra careful while refactoring.

TrevorBenson · 2024-10-31T17:05:30Z

man/en/sos-upload.1

+          [--case-id id]\fR
+          [--upload-url url]\fR
+          [--upload-user user]\fR
+          [--upload-pass pass]\fR
+          [--upload-directory dir]\fR
+          [--upload-method]\fR
+          [--upload-no-ssl-verify]\fR
+          [--upload-protocol protocol]\fR


@jcastill Could the --upload-protocol s3 flags be included in this work? Unfortunately, it contains unique flags that made S3 easier to implement at the time.

[--upload-s3-endpoint endpoint] [--upload-s3-region region] [--upload-s3-bucket bucket] [--upload-s3-access-key access_key] [--upload-s3-secret-key secret_key] [--upload-s3-object-prefix object_prefix]

The existing flags and how the provided values were used were not well aligned for S3, even though valid for FTP, HTTP, etc. protocols. I didn't want to cause any breakage for existing upload protocols while trying to make them work for all protocols, so S3 ended up with unique flags.

I planned to attempt a refactor at some point (sos v5?) where the original protocols and s3 overlap. For example, allowing synonymous flags:

--upload-user ~ --upload-s3-access-key

--upload-pass ~ --upload-s3-secret-key

--upload-directory ~ --upload-s3-object-prefix

--upload-url ~ --upload-s3-endpoint

However, I haven't been able to dedicate the time yet.

Could the --upload-protocol s3 flags be included in this work? Unfortunately, it contains unique flags that made S3 easier to implement at the time.

Yes, I'll make sure I include them in the next iteration of this PR.

I planned to attempt a refactor at some point (sos v5?) where the original protocols and s3 overlap. For example, allowing synonymous flags:

--upload-user ~ --upload-s3-access-key --upload-pass ~ --upload-s3-secret-key --upload-directory ~ --upload-s3-object-prefix --upload-url ~ --upload-s3-endpoint

However, I haven't been able to dedicate the time yet.

Let me know if I can help. My original idea was to have this PR as a starting point and then move stuff out of the generic policy and the OS-specific ones in a second PR, but that was rejected, so I'm working on the full change now. As soon as I finish with that, we can start working on the refactor of S3 it that's OK with you. In fact we need to do some work with S3 uploads for the RH customer portal, so we could do both things in parallel.

Let me know if I can help. My original idea was to have this PR as a starting point and then move stuff out of the generic policy and the OS-specific ones in a second PR, but that was rejected, so I'm working on the full change now. As soon as I finish with that, we can start working on the refactor of S3 it that's OK with you. In fact we need to do some work with S3 uploads for the RH customer portal, so we could do both things in parallel.

When you have a branch published for public view and somewhat functional let me know. I'll branch off of it and start migrating the s3 portions in then submit a PR targeting your branch for you to review.

As for the s3 refactoring, we can look into it and I'd be more than happy to try and make some time. I believe a few lend themselves easily, or at least I don't immediately recall any issues with using them, like URL, user, and password. One I do recall bringing up some questions is the --upload-directory. For example, should this be only the prefixes inside the bucket? Or should it split the directory like {bucket}/{prefix} on only the first slash? There may have been others, but I would have to review the LinuxPolicy.get_upload_xxxx() functions and internal self._vars again.

Without some "group think" I decided not to implement something I (or others) may have been unhappy with later but stuck with unless making breaking changes.

There may be less to refactor than I first thought as I haven't reviewed the code in almost a year. I guess ended up implementing some of it already. Hope I'm still happy with my choices after a year 😄

sos/sos/policies/distros/__init__.py

Lines 615 to 629 in 2aa4fcf

def get_upload_s3_bucket(self):

"""Helper function to determine if we should use the policy default

upload bucket or one provided by the user

:returns: The S3 bucket to use for upload

:rtype: ``str``

"""

if self.upload_url and self.upload_url.startswith('s3://'):

bucket_and_prefix = self.upload_url[5:].split('/', 1)

self.upload_s3_bucket = bucket_and_prefix[0]

if len(bucket_and_prefix) > 1:

self.upload_s3_object_prefix = bucket_and_prefix[1]

if not self.upload_s3_bucket:

self.prompt_for_upload_s3_bucket()

return self.upload_s3_bucket or self._upload_s3_bucket

jcastill · 2025-01-20T11:17:42Z

This could be the fact that /tmp is set in the config in /etc/sos/sos.conf, and therefore by default the sos is created in /tmp. On Ubuntu platforms /tmp is a tmpfs filesystem.

Ah that makes sense. OK, as I said, it seems to be a global option so I'm not sure if I should touch it in this PR or leave it to @pmoravec 's

arif-ali · 2025-01-20T21:01:29Z

This could be the fact that /tmp is set in the config in /etc/sos/sos.conf, and therefore by default the sos is created in /tmp. On Ubuntu platforms /tmp is a tmpfs filesystem.

Ah that makes sense. OK, as I said, it seems to be a global option so I'm not sure if I should touch it in this PR or leave it to @pmoravec 's

yup, sounds good

jcastill · 2025-01-21T11:46:11Z

I fixed @pmoravec's find about upload_directory. I think this is ready - I'm sure there are things to change, unnecessary code that I brought from policy, text strings in English that can be improved, but it's my first subsystem so I think it's normal, and I reviewed this so often that I have reached a kind of semantic satiation, and I need extra eyes.

pmoravec · 2025-01-22T15:11:03Z

man/en/sos-upload.1

+
+.PP
+.SH DESCRIPTION
+upload is an sos subcommand to upload sos reports, logs, vmcores, or other files to a policy defined remote location, or a user defined one.


Nitpick: put this to multiple lines likewise e.g. options? Or likewise https://github.com/sosreport/sos/blob/main/man/en/sos-report.1#L49 .

Fixed, I think

pmoravec · 2025-01-22T18:40:41Z

sos/upload/__init__.py

+        package = sos.upload.targets
+        supported_upload_targets = {}
+        upload_targets = self._load_modules(package, 'targets')


Why not simplier:

supported_upload_targets = {} upload_targets = self._load_modules(sos.upload.targets, 'targets')

package variable is used just once..

pmoravec · 2025-01-22T18:41:52Z

sos/upload/__init__.py

+        package = sos.upload.targets
+        supported_upload_targets = {}
+        upload_targets = self._load_modules(package, 'targets')
+        for upload_target in upload_targets:


.. and maybe you can even merge the previous line to this..? `upload_targets is also not used anywhere else.

And fixed as well

sos/upload/__init__.py

pmoravec

https://github.com/sosreport/sos/pull/3746/files#r1901976808 needs addressing.

I added a few points, mostly nitpicks or some minor stuff. Otherwise the code looks good to me and it passed all tests I invented :) 👍

pmoravec · 2025-01-27T12:01:30Z

Hi @TurboTurtle and @arif-ali , can you please review the current version? It seems good to me already.

Thanks in advance.

arif-ali · 2025-01-27T12:03:23Z

Hi @TurboTurtle and @arif-ali , can you please review the current version? It seems good to me already.

Thanks in advance.

I was away again last week, so will have a look through this week, my initial tests a week before looked good, so will go through the code this week

arif-ali

A few comments, but I am not overly precious about them and maybe could be tackled later.

man/en/sos-upload.1

sos/upload/targets/__init__.py

arif-ali · 2025-01-27T12:26:37Z

sos/upload/targets/__init__.py

+            'ftp': self.upload_ftp,
+            'sftp': self.upload_sftp,
+            'https': self.upload_https,
+            's3': self.upload_s3


could we not potentially have http protocol?

We don't support http protocol uploads atm... do you guys support it in Canonical?
I could add http now, or we could add it in the future if requested. I have already a bunch of things to work on in the next iteration of the upload subsystem so I could tackle http in the next cycle as a low priority PR if that's ok with you

We would need to support user-specified targets which could include plain http endpoints. We can't enforce https-only just because packaged vendor targets are only https.

I would split the http support to a new PR. Better to have iterative approach in implementation of new feature(s) than a monolithic one.

But it's not a new feature. Users today can specify a plain HTTP upload url. If we make this shift, we'd be introducing a feature regression.

Are you sure?

# sos report --help .. --upload-protocol {auto,https,ftp,sftp,s3} Manually specify the upload protocol ..

and trying to upload to http fails:

# sos report -o qpid --upload-url http://127.0.0.1 .. Your sos report has been generated and saved in: /var/tmp/sosreport-pmoravec-rhel9-2025-01-28-ljrbhrf.tar.xz Size 3.43KiB Owner root sha256 d6a4e254b4c92331a76a3f14f19157bdb1f8eec02bc9fa735a34e3a83b11b226 Please send this file to your support representative. Upload attempt failed: Unsupported or unrecognized protocol: http

Oh, well, that's fun. Ok, yeah, disregard. Plain http can come later.

TurboTurtle · 2025-01-27T17:22:46Z

sos/upload/targets/redhat.py

+        return self.RH_API_HOST + rh_case_api\
+            % self.commons['cmdlineopts'].case_id


f-strings, please.

TurboTurtle · 2025-01-27T17:24:41Z

sos/upload/targets/__init__.py

+            'ftp': self.upload_ftp,
+            'sftp': self.upload_sftp,
+            'https': self.upload_https,
+            's3': self.upload_s3


We would need to support user-specified targets which could include plain http endpoints. We can't enforce https-only just because packaged vendor targets are only https.

TurboTurtle · 2025-01-27T17:25:16Z

sos/upload/targets/redhat.py

+                    _("Optionally, please enter the case id that you are "
+                        "generating this report for [%s]: ") % caseid


convert to f-string, please. I know this is an in place lift & shift but we should make this kind of correction while we're doing the moving around.

TurboTurtle · 2025-01-27T17:26:53Z

sos/upload/targets/__init__.py

+    BOTO3_LOADED = False
+
+
+class Upload():


Let's make this a bit more obvious as e.g. UploadTarget or somesuch, just to make future discussions around the feature vs. the target abstraction crystal clear.

TurboTurtle · 2025-01-27T17:27:29Z

sos/upload/targets/__init__.py

+        'upload_s3_secret_key': None,
+        'upload_s3_object_prefix': None,
+        'upload_target': None,
+        'skip_plugins': [],


Is there a use case for pulling in this option for upload?

I think this is a "must" due to #3906 .

I'm not following how this option becomes a must-have due to that issue. This is in the upload target abstraction, not the "global" component level.

Ah you are right. It is really obsolete here in upload targets.

TurboTurtle · 2025-01-27T18:17:56Z

Also, do we have confirmation testing for all currently supported upload methods? HTTP, HTTPS, FTP, SFTP, and S3?

pmoravec · 2025-01-28T11:04:00Z

Also, do we have confirmation testing for all currently supported upload methods? HTTP, HTTPS, FTP, SFTP, and S3?

I successfully tested HTTPS, SFTP, both with and without a proxy.

sos/upload/targets/redhat.py

pmoravec · 2025-01-28T12:49:18Z

sos/upload/targets/redhat.py

+    REQUESTS_LOADED = False
+
+
+class RHELUpload(UploadTarget):


Maybe worth renaming this to RHELUploadTarget to align with UploadTarget class name logic? (and ditto UbuntuUpload)

makes sense, let me do that

Actually, I'm thinking that here we will be perhaps too much? We'll call this "target RHELUpload" or "RHELUpload target" but adding target to the name itself may be unnecessary? But I'm not a native speaker so even if it sounds weird to me, it may be absolutely right to change it

What about RHELTarget instead of RHELUploadTarget? Does that sound good?

Well the RHELUploadTarget seems ugly long to me, but imho the class serves target functionality more than upload functionality. So more likely RHELTarget than RHELUploadTarget..?

Whatever works for others, esp. native speakers :)

sos/upload/targets/redhat.py

This commit marks the beginning of the addition of a new 'upload' component for sos, which can be used to upload already created sos reports, collects, or other files like logs or vmcores to a policy defined location. The user needs to specify a file location, and can make use of any of the options that exist nowadays for the --upload option. This first commit includes: - The initial framework for the 'upload' component. - The new man page for 'sos upload'. - The code in the component 'help' to show information about the component. - The code in sos/__init__.py to deal with the component. - The code for uploads to Red Hat and Ubuntu systems. - The code to allow uploads specifying remote destination, called targets in this implementation. For example, you could generate a sos report in a CentOS system and specify a target defined as 'redhat' or 'RedHatUpload' to upload to the Red Hat Customer Portal. - And modifications to setup.py to build the man pages. Related: RHEL-23032, SUPDEV-138, CLIOT-481 Co-authored-by: Jose Castillo <jcastillo@redhat.com> Co-authored-by: Pavel Moravec <pmoravec@redhat.com> Co-authored-by: Trevor Benson <trevor.benson@gmail.com> Signed-off-by: Jose Castillo <jcastillo@redhat.com>

jcastill · 2025-01-28T13:53:27Z

On this last push:

I added all requested f-strings.
Changed the name of the Upload class to the suggested UploadTarget.
Changed the name of the distributions upload to RHELUploadTarget and UbuntuUploadTarget.
Added a string identifier for targets, and a function to return this string.

I've tested:

Upload from 'report'.
Upload from 'collect'.
Upload setting case-id in the command line (and failing, so it fails over to SFTP).
Upload directly to SFTP.
Upload from Fedora setting the RHEL target upload.
Upload from Centos Stream setting the RHEL target upload.
Upload from Ubuntu to (mock) Ubuntu target.
Upload from Ubuntu to RH portal.
Upload from RH system to (mock) Ubuntu target.

Also tested cases where:

The file doesn't exist.
The file is a directory.
The file has zero size.

All worked as expected.

Regarding adding 'http' protocol - I agree with @pmoravec here, I think it may be better to do it in a new PR. The reasoning is that this first implementation is just moving existing code around to its own subsystem, and adding new options in new and specific PRs will help a lot when debugging issues, and add clarity to the different commits.
As soon as this is merged I'll add a list of enhancements that we can start working on.

TurboTurtle · 2025-01-28T16:26:49Z

Regarding adding 'http' protocol - I agree with @pmoravec here, I think it may be better to do it in a new PR. The reasoning is that this first implementation is just moving existing code around to its own subsystem, and adding new options in new and specific PRs will help a lot when debugging issues, and add clarity to the different commits.
As soon as this is merged I'll add a list of enhancements that we can start working on.

I replied above, but putting this here to so it doesn't get lost in the weeds.

Users today can specify a plain HTTP upload URL. So, as currently written this is a feature regression by not continuing to support that. HTTP is not a 'new' feature for uploads.

TurboTurtle · 2025-01-28T21:10:23Z

Oh, looks like we actually don't directly support http anymore. Disregard then, plain http support can follow.

Do we have S3 testing confirmation?

arif-ali reviewed Aug 9, 2024

View reviewed changes

man/en/sos-upload.1 Outdated Show resolved Hide resolved

man/en/sos-upload.1 Outdated Show resolved Hide resolved

sos/upload/__init__.py Outdated Show resolved Hide resolved

sos/upload/__init__.py Outdated Show resolved Hide resolved

jcastill force-pushed the jcastillo-add-upload-subsystem branch from d5b6c64 to 0e6bc72 Compare August 9, 2024 12:42

jcastill force-pushed the jcastillo-add-upload-subsystem branch from 0e6bc72 to 9c60d66 Compare August 9, 2024 13:46

jcastill requested review from pmoravec and arif-ali August 13, 2024 10:51