Opensearch RPM build failing on Mac

Open Source Elasticsearch and Kibana
1

Hi Team,
As per RPM Distributions delayed again? - #16 by justme
RPM is not yet ready but i need the RPM urgently for my use case , hence i tried building the RPM as mentioned here
[Design] OpenSearch/Dashboards DEB/RPM Distribution Design Document · Issue #1452 · opensearch-project/opensearch-build · GitHub

and looks like its not working as expected.
Getting below error from the build command:
./build.sh manifests/2.0.0/opensearch-2.0.0.yml --distribution rpm --platform linux


See https://docs.gradle.org/7.3.3/userguide/command_line_interface.html#sec:command_line_warnings

BUILD SUCCESSFUL in 1m 6s
23 actionable tasks: 23 executed
+++ pwd
++ find /private/var/folders/1p/3z8hzlj16k51d7xnj6d2x4lc0000gq/T/tmpfopnklyg/k-NN -path '*build/distributions/*.zip'
+ zipPath=/private/var/folders/1p/3z8hzlj16k51d7xnj6d2x4lc0000gq/T/tmpfopnklyg/k-NN/build/distributions/opensearch-knn-2.0.0.0-rc1.zip
++ dirname /private/var/folders/1p/3z8hzlj16k51d7xnj6d2x4lc0000gq/T/tmpfopnklyg/k-NN/build/distributions/opensearch-knn-2.0.0.0-rc1.zip
+ distributions=/private/var/folders/1p/3z8hzlj16k51d7xnj6d2x4lc0000gq/T/tmpfopnklyg/k-NN/build/distributions
+ mkdir /private/var/folders/1p/3z8hzlj16k51d7xnj6d2x4lc0000gq/T/tmpfopnklyg/k-NN/build/distributions/lib
+ cp ./jni/release/libopensearchknn_common.jnilib ./jni/release/libopensearchknn_faiss.jnilib ./jni/release/libopensearchknn_nmslib.jnilib /private/var/folders/1p/3z8hzlj16k51d7xnj6d2x4lc0000gq/T/tmpfopnklyg/k-NN/build/distributions/lib
++ ldconfig -p
/private/var/folders/1p/3z8hzlj16k51d7xnj6d2x4lc0000gq/T/tmpfopnklyg/k-NN/scripts/build.sh: line 111: ldconfig: command not found
++ grep libgomp
++ cut -d ' ' -f 4
+ ompPath=
+ cp /private/var/folders/1p/3z8hzlj16k51d7xnj6d2x4lc0000gq/T/tmpfopnklyg/k-NN/build/distributions/lib
usage: cp [-R [-H | -L | -P]] [-fi | -n] [-apvXc] source_file target_file
       cp [-R [-H | -L | -P]] [-fi | -n] [-apvXc] source_file ... target_directory
2022-04-29 01:51:57 ERROR    Error building k-NN, retry with: ./build.sh manifests/2.0.0/opensearch-2.0.0.yml --component k-NN
Traceback (most recent call last):
  File "/Users/dkn/git-repo/Dinesh_Repo/opensearch-build/./src/run_build.py", line 80, in <module>
    sys.exit(main())
  File "/Users/dkn/git-repo/Dinesh_Repo/opensearch-build/./src/run_build.py", line 68, in main
    builder.build(build_recorder)
  File "/Users/dkn/git-repo/Dinesh_Repo/opensearch-build/src/build_workflow/builder_from_source.py", line 55, in build
    self.git_repo.execute(build_command)
  File "/Users/dkn/git-repo/Dinesh_Repo/opensearch-build/src/git/git_repository.py", line 83, in execute
    subprocess.check_call(command, cwd=cwd, shell=True)
  File "/usr/local/Cellar/python@3.9/3.9.12/Frameworks/Python.framework/Versions/3.9/lib/python3.9/subprocess.py", line 373, in check_call
    raise CalledProcessError(retcode, cmd)
subprocess.CalledProcessError: Command 'bash /private/var/folders/1p/3z8hzlj16k51d7xnj6d2x4lc0000gq/T/tmpfopnklyg/k-NN/scripts/build.sh -v 2.0.0 -q rc1 -p linux -a x64 -s false -o builds' returned non-zero exit status 64.

Has anyone build the RPM successfully in a Mac OS?

2

its failing because of :

/private/var/folders/1p/3z8hzlj16k51d7xnj6d2x4lc0000gq/T/tmpfopnklyg/k-NN/scripts/build.sh: line 111: ldconfig: command not found

and this seems to be not supported as per : sudo: ldconfig: command not found for C++ installation on Mac OS X · Issue #2570 · protocolbuffers/protobuf · GitHub
Any help will be appreciated.

1 Like
3

Hi @dinesh,

k-NN plugin requires to be built on LINUX.
If you remove knn section from the manifest file then run:

./build.sh manifests/2.0.0/opensearch-2.0.0.yml --platform linux --distribution rpm
./assemble.sh rpm/builds/opensearch/manifest.yml

It should build.
Dashboards is similar.

Thanks.

3 Likes
4

Thanks @zhujiaxi
i removed k-NN from manifest and retried .
it failed again with below error :

FAILURE: Build failed with an exception.

* Where:
Build file '/private/var/folders/1p/3z8hzlj16k51d7xnj6d2x4lc0000gq/T/tmpwt0lpkla/notifications-core/notifications/build.gradle' line: 105

* What went wrong:
A problem occurred evaluating root project 'opensearch-notifications'.
> A problem occurred configuring project ':notifications'.
   > Could not resolve all files for configuration ':notifications:classpath'.
      > Could not resolve com.netflix.nebula:gradle-ospackage-plugin:8.5.4.
        Required by:
            project :notifications > nebula.ospackage:nebula.ospackage.gradle.plugin:8.5.4
         > Could not resolve com.netflix.nebula:gradle-ospackage-plugin:8.5.4.
            > Could not get resource 'https://plugins.gradle.org/m2/com/netflix/nebula/gradle-ospackage-plugin/8.5.4/gradle-ospackage-plugin-8.5.4.module'.
               > Could not GET 'https://plugins.gradle.org/m2/com/netflix/nebula/gradle-ospackage-plugin/8.5.4/gradle-ospackage-plugin-8.5.4.module'.
                  > Read timed out
      > Could not resolve org.apache.commons:commons-lang3:3.4.
        Required by:
            project :notifications > com.dorongold.task-tree:com.dorongold.task-tree.gradle.plugin:1.5 > gradle.plugin.com.dorongold.plugins:task-tree:1.5
         > Skipped due to earlier error

so i removed both notifications from manifest and tried one more time. This time the build was successful.

BUILD SUCCESSFUL in 7m 17s
25 actionable tasks: 25 executed
+ mkdir -p builds/plugins
+ cp ./build/distributions/opensearch-performance-analyzer-2.0.0.0-rc1.zip builds/plugins
2022-04-29 10:37:16 INFO     Recording plugins artifact for performance-analyzer: plugins/opensearch-performance-analyzer-2.0.0.0-rc1.zip (from /var/folders/1p/3z8hzlj16k51d7xnj6d2x4lc0000gq/T/tmp104jb351/performance-analyzer/builds/plugins/opensearch-performance-analyzer-2.0.0.0-rc1.zip)
2022-04-29 10:37:16 INFO     Checked /var/folders/1p/3z8hzlj16k51d7xnj6d2x4lc0000gq/T/tmp104jb351/performance-analyzer/builds/plugins/opensearch-performance-analyzer-2.0.0.0-rc1.zip (2.0.0.0-rc1)
2022-04-29 10:37:17 INFO     Created build manifest /Users/dkn/git-repo/Dinesh_Repo/opensearch-build/rpm/builds/opensearch/manifest.yml
2022-04-29 10:37:44 INFO     Done.
dkn@m-c02g225xmd6n opensearch-build % ./assemble.sh rpm/builds/opensearch/manifest.yml

but the next command is not going through now.

dkn@m-c02g225xmd6n opensearch-build % ./assemble.sh rpm/builds/opensearch/manifest.yml

Installing dependencies in . ...
Installing dependencies from Pipfile.lock (9e9e8a)...
  🐍   ▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉ 0/0 — 00:00:00
To activate this project's virtualenv, run pipenv shell.
Alternatively, run a command inside the virtualenv with pipenv run.
Running ./src/run_assemble.py rpm/builds/opensearch/manifest.yml ...
2022-04-29 10:39:52 INFO     Bundling OpenSearch (x64) on linux into /Users/dkn/git-repo/Dinesh_Repo/opensearch-build/rpm/dist/opensearch ...
2022-04-29 10:39:52 INFO     Copied min bundle to /var/folders/1p/3z8hzlj16k51d7xnj6d2x4lc0000gq/T/tmpttf0_lc1/opensearch-min-2.0.0-rc1.x86_64.rpm.
2022-04-29 10:39:52 INFO     Start creating distribution rpm for OpenSearch.
2022-04-29 10:39:52 INFO     Extracting dist into /var/folders/1p/3z8hzlj16k51d7xnj6d2x4lc0000gq/T/tmpttf0_lc1.
2022-04-29 10:39:52 INFO     Convert rpm to cpio for extraction: /var/folders/1p/3z8hzlj16k51d7xnj6d2x4lc0000gq/T/tmpttf0_lc1/opensearch-min-2.0.0-rc1.x86_64.rpm to /var/folders/1p/3z8hzlj16k51d7xnj6d2x4lc0000gq/T/tmpttf0_lc1/opensearch-min-2.0.0-rc1.x86_64.cpio
Traceback (most recent call last):
  File "/Users/dkn/git-repo/Dinesh_Repo/opensearch-build/./src/run_assemble.py", line 58, in <module>
    sys.exit(main())
  File "/Users/dkn/git-repo/Dinesh_Repo/opensearch-build/./src/run_assemble.py", line 42, in main
    with Bundles.create(build_manifest, artifacts_dir, bundle_recorder, args.keep) as bundle:
  File "/Users/dkn/git-repo/Dinesh_Repo/opensearch-build/src/assemble_workflow/bundles.py", line 31, in create
    return klass(build_manifest, artifacts_dir, bundle_recorder, keep)  # type: ignore[no-any-return, operator]
  File "/Users/dkn/git-repo/Dinesh_Repo/opensearch-build/src/assemble_workflow/bundle.py", line 49, in __init__
    self.min_dist = self.__get_min_dist(build_manifest.components)
  File "/Users/dkn/git-repo/Dinesh_Repo/opensearch-build/src/assemble_workflow/bundle.py", line 140, in __get_min_dist
    min_dist.extract(self.tmp_dir.name)
  File "/Users/dkn/git-repo/Dinesh_Repo/opensearch-build/src/assemble_workflow/dist.py", line 75, in extract
    self.__extract__(dest)
  File "/Users/dkn/git-repo/Dinesh_Repo/opensearch-build/src/assemble_workflow/dist.py", line 115, in __extract__
    BundleRpm(self.filename, self.path, self.min_path).extract(dest)
  File "/Users/dkn/git-repo/Dinesh_Repo/opensearch-build/src/assemble_workflow/bundle_rpm.py", line 33, in extract
    subprocess.check_call(
  File "/usr/local/Cellar/python@3.9/3.9.12/Frameworks/Python.framework/Versions/3.9/lib/python3.9/subprocess.py", line 368, in check_call
    retcode = call(*popenargs, **kwargs)
  File "/usr/local/Cellar/python@3.9/3.9.12/Frameworks/Python.framework/Versions/3.9/lib/python3.9/subprocess.py", line 349, in call
    with Popen(*popenargs, **kwargs) as p:
  File "/usr/local/Cellar/python@3.9/3.9.12/Frameworks/Python.framework/Versions/3.9/lib/python3.9/subprocess.py", line 951, in __init__
    self._execute_child(args, executable, preexec_fn, close_fds,
  File "/usr/local/Cellar/python@3.9/3.9.12/Frameworks/Python.framework/Versions/3.9/lib/python3.9/subprocess.py", line 1821, in _execute_child
    raise child_exception_type(errno_num, err_msg, err_filename)
FileNotFoundError: [Errno 2] No such file or directory: 'rpm2cpio'

What does rpm2cpio does ?
How do i fix this ?

5

rpm2cpio is an application that converts rpms into a cpio archive. I think it’s specifically for working with RPMs on a Mac. You can install it with brew

I’ll keep looking, but I can’t quite find any other place to get it on your mac besides the brew formula. That should at least get you going.

Nate

6

thanks @nateynate for replying,
There seems to be lots of dependency , after rpm2cpio , it needed rpm install , and now its failing due to rpmbuild missing.
rpmbuild is not available via brew install and i dint find any other alternative for mac till now.

I guess i will wait for official release of RPM distribution as it might be available tomorrow , if not then i will continue with the manual build exploration.

7

Hi @dinesh - I think the brew formula you’re looking for is just ‘rpm’ - try this one:

This should install what you need to build an RPM on your mac. Let me know how it goes - I’m not on a mac over here so now you’ve got me very curious!

Nate

8

I already did that , rpmbuild is needed explicitly.

9

Hi @dinesh.

  1. We have RPM out for 2.0.0 already and will have for 1.3.2 soon now. See RPM Distributions delayed again? - #32 by bbarani.
  2. If you want to try with docker image then you can use images and args here:
    opensearch-build/opensearch-2.0.0-rc1.yml at main · opensearch-project/opensearch-build · GitHub

Building on linux is way easier than tweaking on mac, even I who develop on mac didnt test rpm building on mac.

rpmbuild is specifically needed as we use it to parse the spec file and create the final package. On my mac, I install rpm through brew and get rpmbuild automatically, so not sure why you dont have that available.

Thanks.

1 Like
10

@zhujiaxi I am already working on prod release 2.0.0 and wondering if you have any idea on how to run opensearch as a service ?

currently i am starting/stopping using systemctl command like

sudo systemctl start opensearch.service

sudo systemctl stop opensearch.service

i want it to be like :

sudo service opensearch stop/start/status/restart

Thanks in advance !

11

Hi @dinesh,

service commands are for sysvinit, which is the old process manager before systemd.
We have tested the product on systemd with systemctl and I do remember we still have sysvinit script, but we have not test them, those scripts are located in /etc/init.d/opensearch or similar.

If you are on RH/CentOS7 and AL2 I believe systemvinit has already been replaced with systemd, so you cannot run with service command. Afterall, systemctl is much better than service.

You can try to use older version of LINUX such as RH/CentOS6 if you really want the old way.

Thanks.

3 Likes
12

Thanks for the details @zhujiaxi