How to build OpenSearch including all plugins?

OpenDistro
1

I am trying to build OpenSearch including all plugins from the scratch but I’am struggling. I like to produce the release linux tar gz as well as the docker image including all plugins.

Is there a tutorial about this? Or how do you build the release on CI?

I came across issues with naming conventions, versioning conventions and missing git tags (see a-c below).

What i tried so far:

OS_VERSION="1.0.0"
OS_QUALIFIER="rc1"

#Save this as buildall.sh
#In the same folder like the script there shoudl be a git checkout of OpenSearch and opensearch-plugins repos

#Build OpenSearch and deploy atrifacts to local maven repo 
SCRIPT_DIR="$( cd "$( dirname "${BASH_SOURCE[0]}" )" &> /dev/null && pwd )"
cd "$SCRIPT_DIR/OpenSearch"
git fetch && git pull
git checkout "$OS_VERSION-$OS_QUALIFIER"
./gradlew clean :distribution:archives:linux-tar:assemble publishToMavenLocal -Dbuild.version_qualifier="$OS_QUALIFIER" -Dbuild.snapshot=false

#Try to iterste over the plugins and build an install them
cd "$SCRIPT_DIR/opensearch-plugins/plugins"

# a)
# Is it 1.0.0.0-rc1 (that is 4 digits)
# or is it 1.0.0-rc1 (that is 3 digits)
# Why have 4 digits here and not 3 like with OS itself

# b) 
# Only the job-scheduler plugin gets built and installed properly
# All other plugins are failing because of different naming conventions (paths, versioning)
# or because there is no rc1 in github (maybe the tags were not pushed properly)

# c) 
# Its hard to differentiate between dashboard and non-dashboard plugins because there seems to be no naming convention 

#meta exec "git fetch && git pull && git checkout v$OS_VERSION.0-$OS_QUALIFIER" #this fails because of missing tags
meta exec "./gradlew clean assemble -Dopensearch.version="$OS_VERSION-$OS_QUALIFIER" -Dbuild.snapshot=false \
          && $SCRIPT_DIR/OpenSearch/distribution/archives/linux-tar/build/install/opensearch-$OS_VERSION-$OS_QUALIFIER/bin/opensearch-plugin install -b file://\$(pwd)/build/distributions/opensearch-\$(basename \$(pwd))-$OS_VERSION.0-$OS_QUALIFIER.zip"

Here i gave up for now.

2 Likes
2

Let me find someone to help you with this - if you haven’t found these, they might help:

I know for the Alpha/Beta/RC releases there wasn’t a lot of automation (of course, will change for GA forward).

3

I see there are couple of other questions as well:

  1. Why is the plugin versioning different vs OpenSearch/Dashboards?
    Plugins versioning was inherited from ODFE which was kind of following the semver https://semver.org/ versioning.
    The reason behind that was to have plugin patch version added on to the ODFE semver so that individual plugins can be independent to update their patch versions.
    The same applies to OpenSearch and OpenSearch plugins.
    If you think there are better ways please open an issue in GitHub - opensearch-project/opensearch-plugins: For all things OpenSearch plugins. You want to install, or develop a plugin? You've come to the right place.

  2. Why is the naming convention for OpenSearch vs Dashboards plugins not distinct?
    The team is actively working on standardizing the plugin naming conventions and how the artifacts would look like.
    Here is the Github issue: Standardize plugin artifact names · Issue #47 · opensearch-project/opensearch-plugins · GitHub
    Feel free to drop comments if you prefer a different naming convention.

4

Would be great if there is a single shell script/command to build and deploy everything (including plugins) from the scratch.

I’d also suggest to add something like

java {
    withJavadocJar()
    withSourcesJar()
}

in server/build.gradle so that the source and javadoc artifacts will be pushed properly to maven.

5

Right now it’s a manual process to build the tarball. We basically check out each repo, run whatever build command is specified in the README, and then copy the artifacts into a temporary location.

To build the tarball, we unpack the OpenSearch tarball and run opensearch-plugin install file://path/to/zip for each plugin, then create a new tarball that rolls up the OpenSearch + installed plugins filesystem.

We’re working on automating this stuff, so you should see scripts start to appear in the opensearch-build repo over the next few weeks.

3 Likes
6

If you just want to build the Docker image with the plugins installed, you could do the following:

1.) Download the plugin *.zip files to /distribution/docker/src/docker/plugins

2.) Modify the /distribution/docker/build.gradle adding:

      into('plugins') {
        from project.projectDir.toPath().resolve("src/docker/plugins")
      }

3.) modify the Dockerfile /distribution/docker/src/docker/Dockerfile to install the plugins, e.g.

#NOTE: opensearch-job-scheduler must be installed first, other plugins depend on it!
COPY --chown=1000:0 plugins/*.zip /usr/share/opensearch/

USER opensearch

RUN /usr/share/opensearch/bin/opensearch-plugin install --batch file:///usr/share/opensearch/opensearch-job-scheduler-1.0.0.0-beta1.zip && \
    /usr/share/opensearch/bin/opensearch-plugin install --batch file:///usr/share/opensearch/opensearch-security-1.0.0.0-beta1.zip && \
    /usr/share/opensearch/bin/opensearch-plugin install --batch file:///usr/share/opensearch/opensearch-alerting-1.0.0.0-beta1.zip && \
    /usr/share/opensearch/bin/opensearch-plugin install --batch file:///usr/share/opensearch/opensearch-anomaly-detection-1.0.0.0-beta1.zip && \
    /usr/share/opensearch/bin/opensearch-plugin install --batch file:///usr/share/opensearch/opensearch-index-management-1.0.0.0-beta1.zip
7

Thanks but I also want to build all the plugins from source before installing them.

8

I have opened Document building end-to-end bundle · Issue #73 · opensearch-project/opensearch-build · GitHub to track this exact ask.

1 Like
9

Where can i find the plugins to download ?
Since i can’t download the plugin from the following page as “Access Denied” :
https://artifacts.opensearch.org/releases/plugins/repository-s3/1.0.0/repository-s3-1.0.0-rc1.zip
Please help!

10

Good catch. The team discovered this as well and they are working on getting those up.

1 Like