RPM Distributions delayed again?

This is what I get when I run rpm --checksig:

$ rpm --checksig -v opensearch-1.3.2-linux-x64.rpm
opensearch-1.3.2-linux-x64.rpm:
Header V4 RSA/SHA512 Signature, key ID 9310d3fc: NOKEY
Header SHA1 digest: OK
Payload SHA256 digest: NOTFOUND
V4 RSA/SHA512 Signature, key ID 9310d3fc: NOKEY
MD5 digest: NOTFOUND

$ rpm --checksig -v opensearch-dashboards-1.3.2-linux-x64.rpm
opensearch-dashboards-1.3.2-linux-x64.rpm:
Header V4 RSA/SHA512 Signature, key ID 9310d3fc: NOKEY
Header SHA1 digest: OK
Payload SHA256 digest: NOTFOUND
V4 RSA/SHA512 Signature, key ID 9310d3fc: NOKEY
MD5 digest: NOTFOUND

It appears there is no SHA256 payload digest. I am uncertain which keys you are using, so I am unsure if 9310d3fc is correct for the other signatures. Do you know where the PGP signatures are published to be verified against?

The PGP keys are published here - How to verify signatures ¡ OpenSearch . Btw, we are using the master key to sign the Yum repo. You can view more info on the signing here - [Question] RPM signing requires the master secret key, not subkey ¡ Issue #2041 ¡ opensearch-project/opensearch-build ¡ GitHub

Thanks for that link - I tried googling for the signature, but couldn’t find it. You link did list the location of that key. I was able to install that key and it did verify that the parts of the signatures that were present were valid, but that it still lacked the Payload SHA256 signatures. See output below after the key import:

$ rpm --checksig -v opensearch-1.3.2-linux-x64.rpm
opensearch-1.3.2-linux-x64.rpm:
Header V4 RSA/SHA512 Signature, key ID 9310d3fc: OK
Header SHA1 digest: OK
Payload SHA256 digest: NOTFOUND
V4 RSA/SHA512 Signature, key ID 9310d3fc: OK
MD5 digest: NOTFOUND

$ rpm --checksig -v opensearch-dashboards-1.3.2-linux-x64.rpm
opensearch-dashboards-1.3.2-linux-x64.rpm:
Header V4 RSA/SHA512 Signature, key ID 9310d3fc: OK
Header SHA1 digest: OK
Payload SHA256 digest: NOTFOUND
V4 RSA/SHA512 Signature, key ID 9310d3fc: OK
MD5 digest: NOTFOUND

We are using the master key to sign the RPM. You can view more info on the RPM signing here - [Question] RPM signing requires the master secret key, not subkey ¡ Issue #2041 ¡ opensearch-project/opensearch-build ¡

Similar issue trying to install through YUM with the instructions on that page:

[USERNAME@localhost ~]$ sudo yum clean all
43 files removed
[USERNAME@localhost ~]$ 
[USERNAME@localhost ~]$ sudo yum install opensearch
Rocky Linux 8 - AppStream                                                    12 MB/s |  10 MB     00:00    
Rocky Linux 8 - BaseOS                                                      8.2 MB/s | 7.7 MB     00:00    
Rocky Linux 8 - Extras                                                       97 kB/s |  12 kB     00:00    
OpenSearch 1.x                                                               67 kB/s |  18 kB     00:00    
Dependencies resolved.
============================================================================================================
 Package                   Architecture          Version                Repository                     Size
============================================================================================================
Installing:
 opensearch                x86_64                1.3.2-1                opensearch-1.x                369 M

Transaction Summary
============================================================================================================
Install  1 Package

Total download size: 369 M
Installed size: 604 M
Is this ok [y/N]: y
Downloading Packages:
opensearch-1.3.2-linux-x64.rpm                                                           10 MB/s | 369 MB     00:35    
------------------------------------------------------------------------------------------------------------------------
Total                                                                                    10 MB/s | 369 MB     00:35     
Running transaction check
Transaction check succeeded.
Running transaction test
The downloaded packages were saved in cache until the next successful transaction.
You can remove cached packages by executing 'yum clean packages'.
Error: Transaction test error:
  package opensearch-1.3.2-1.x86_64 does not verify: no digest

I am also not sure who controls that yum installation file (the ones stored at
https://artifacts.opensearch.org/releases/bundle/opensearch/1.x/opensearch-1.x.repo and https://artifacts.opensearch.org/releases/bundle/opensearch-dashboards/1.x/opensearch-dashboards-1.x.repo ) but they should also include repo_gpgcheck=1 for security. It appears repo metadata is being signed, or I don’t think it would have worked when I added that line.

Hi @justme,

I follow exactly how the documentation shows and install with no issues:

$ sudo yum install opensearch
Loaded plugins: extras_suggestions, kernel-livepatch, langpacks, priorities, update-motd
Resolving Dependencies
--> Running transaction check
---> Package opensearch.x86_64 0:1.3.2-1 will be installed
--> Finished Dependency Resolution

Dependencies Resolved

============================================================================================================================================================================================================================================================================================================================
 Package                                                                       Arch                                                                      Version                                                                    Repository                                                                         Size
============================================================================================================================================================================================================================================================================================================================
Installing:
 opensearch                                                                    x86_64                                                                    1.3.2-1                                                                    opensearch-1.x                                                                    369 M

Transaction Summary
============================================================================================================================================================================================================================================================================================================================
Install  1 Package

Total download size: 369 M
Installed size: 604 M
Is this ok [y/d/N]: y
Downloading packages:
warning: /var/cache/yum/x86_64/2/opensearch-1.x/packages/opensearch-1.3.2-linux-x64.rpm: Header V4 RSA/SHA512 Signature, key ID 9310d3fc: NOKEY 95% [==============================================================================================================================-      ]  37 MB/s | 351 MB  00:00:00 ETA
Public key for opensearch-1.3.2-linux-x64.rpm is not installed
opensearch-1.3.2-linux-x64.rpm                                                                                                                                                                                                                                                                       | 369 MB  00:00:10
Retrieving key from https://artifacts.opensearch.org/publickeys/opensearch.pgp
Importing GPG key 0x9310D3FC:
 Userid     : "OpenSearch project <opensearch@amazon.com>"
 Fingerprint: c5b7 4989 65ef d1c2 924b a9d5 39d3 1987 9310 d3fc
 From       : https://artifacts.opensearch.org/publickeys/opensearch.pgp
Is this ok [y/N]: y
Running transaction check
Running transaction test
Transaction test succeeded
Running transaction
  Installing : opensearch-1.3.2-1.x86_64                                                                                                                                                                                                                                                                                1/1
### NOT starting on installation, please execute the following statements to configure opensearch service to start automatically using systemd
 sudo systemctl daemon-reload
 sudo systemctl enable opensearch.service
### You can start opensearch service by executing
 sudo systemctl start opensearch.service
### Create opensearch demo certificates in /etc/opensearch/
 See demo certs creation log in /var/log/opensearch/install_demo_configuration.log
  Verifying  : opensearch-1.3.2-1.x86_64                                                                                                                                                                                                                                                                                1/1

Installed:
  opensearch.x86_64 0:1.3.2-1

I also get a rockylinux 8 to test and also works well.

It seems like in your case the public key listed in the repo file did not get retrieved, which I have never seen before.

We do have gpgcheck=1 which should force the gpgkey retrieval and check.

As for repo_gpgcheck=1 it is a miss from my side, I would add to the repo files later.

Thanks.

Are you running in FIPS mode?

[USERNAME@localhost ~]$ fips-mode-setup --check
FIPS mode is enabled.

In short, FIPS mode dramatically increases the RPM validation before installation. Going back to my earlier command which you should be able to replicate:

$ rpm --checksig -v opensearch-1.3.2-linux-x64.rpm
opensearch-1.3.2-linux-x64.rpm:
Header V4 RSA/SHA512 Signature, key ID 9310d3fc: OK
Header SHA1 digest: OK
Payload SHA256 digest: NOTFOUND
V4 RSA/SHA512 Signature, key ID 9310d3fc: OK
MD5 digest: NOTFOUND

FIPS mode will not install a RPM that has a Payload SHA256 digest of NOTFOUND.

Hi @justme, no I am not running with FIPS mode up.
And my output on RockyLinux8 is different from yours, assuming due to FIPS mode here:

[root@84c245a01715 /]# rpm --import https://artifacts.opensearch.org/publickeys/opensearch.pgp
[root@84c245a01715 /]# rpm --checksig -v opensearch-1.3.2-linux-x64.rpm
opensearch-1.3.2-linux-x64.rpm:
    Header V4 RSA/SHA512 Signature, key ID 9310d3fc: OK
    Header SHA1 digest: OK
    V4 RSA/SHA512 Signature, key ID 9310d3fc: OK
    MD5 digest: OK

And here is the rpmmacro we are using to sign the package, let me know if there is anything missed for payload sha256 to be empty, but in my prior experience I have never used FIPS so not familiar with this behavior:

%__gpg_sign_cmd %{__gpg} \
    gpg --no-verbose --no-armor --batch --yes --pinentry-mode loopback \
    --passphrase-file /path/to/passphrase-file \
    %{?_gpg_digest_algo:--digest-algo %{_gpg_digest_algo}} \
    --no-secmem-warning \
    -u "%{_gpg_name}" -sbo %{__signature_filename} --digest-algo sha512 %{__plaintext_filename}

Thanks.

I am quite familiar with needing the packages being signed, but not so much with the how the signing works. That being said, I worked with a different maintainer on a package and he noted that he needed to add %_gpg_digest_algo sha256 to sign it as such. See this semi-similar issue td-agent-bit won't install on RHEL 8 / FIPS ¡ Issue #3617 ¡ fluent/fluent-bit ¡ GitHub where the package also was lacking a Payload SHA256 digest preventing installation, but where it was ultimately resolved.

Thanks @justme, let me take a look and try to get a machine with FIPS on.

In the meantime we have updated the repo files to include repo_gpgcheck=1, please verify, same link.

Thanks.

I also do find out that using echo "%_pkgverify_level none" >/etc/rpm/macros.verify will allow you to bypass FIPS, probably a workaround you can consider being we permanently fix it on our pkg.

I can’t tell you how much that I dream about that, but local security policy prohibits that.

sudo fips-mode-setup --enable should get you what you need on a redhat-variant machine like rocky. You’ll likely need to reboot.

I can see the addition of repo_gpgcheck=1. Thanks.

Thanks @justme,

I have a centos8 and enable FIPS to see the exact behavior you are seeing.
https://github.com/opensearch-project/opensearch-build/issues/2099

I record my findings in above issue and will update it accordingly.

I will try to get a test rpm out and link to you for verification once I resolve it.

Thanks.

@justme

I think I got it here: https://github.com/opensearch-project/opensearch-build/issues/2099#issuecomment-1120109846

Could you help testing these 2 RPMs on your FIPS machine?

I can install them with no issues now.

If this runs on your machine I will send a PR later to fix it permanently.

Sorry for inconvinience.

Thanks.

The artifacts you linked to verify with --checksig now:

[USERNAME@localhost ~]$ rpm --checksig -v opensearch-1.3.2-linux-x64-fips-enabled.rpm 
opensearch-1.3.2-linux-x64-fips-enabled.rpm:
    Header V4 RSA/SHA512 Signature, key ID 9310d3fc: OK
    Header SHA256 digest: OK
    Header SHA1 digest: OK
    Payload SHA256 digest: OK
    V4 RSA/SHA512 Signature, key ID 9310d3fc: OK

[USERNAME@localhost ~]$ rpm --checksig -v opensearch-dashboards-1.3.2-linux-x64-fips-enabled.rpm 
opensearch-dashboards-1.3.2-linux-x64-fips-enabled.rpm:
    Header V4 RSA/SHA512 Signature, key ID 9310d3fc: OK
    Header SHA256 digest: OK
    Header SHA1 digest: OK
    Payload SHA256 digest: OK
    V4 RSA/SHA512 Signature, key ID 9310d3fc: OK

rpm -ivh succeeds as well. Thanks for the updates!

2 Likes

Many thanks, now everything works fine on our side too! :slight_smile:

2 Likes