Download URLs for plugins not listed anywhere

there are currently two possibilities for installing opensearch:

  • use the normal aka “full” distribution with all standard plugins in it
  • use the “min” distribution which does not come with any plugins and install the plugins you want

and the second part of the second option is where things start to become difficult:
depending on the security setup your opensearch installation cannot access the open internet, instead you need to have a local mirror of things. so you can’t just run bin/opensearch-plugin install security. instead, you need to put the ZIP file of the plugin elsewhere and then reference that in the installation.

but: there’s no download link for the ZIP file to be found anywhere:

suggestions:

  • put the download link on most - if not all - of the places mentioned above
  • ensure that the final release ZIP is also attached to the github release for completeness sake

with Publish plugin distribution ZIPs to Maven, get rid of checked in snapshot builds · Issue #716 · opensearch-project/opensearch-build · GitHub this should also further improve as it can then be found on maven central & co. (plus, usually maven servers support directory listing if you’re not using some 3rd party website like https://mvnrepository.com/ to look at it)

EDIT: as a good first start i’d already be happy if somebody could give me the download URL for the latest security plugin or the general URL pattern for all plugins so that i can piece it together myself. thanks!

1 Like

Inside artifacts.zip is opensearch-security-1.2.3.0-SNAPSHOT.zip
I’m under the impression that opensearch-security-1.2.3.0-SNAPSHOT.zip is uncompressed into default path /usr/share/opensearch/plugins/opensearch-security, and that is the installation process.

(I’m just reviewing this myself as a community member, but it appears to be that simple. It seems to line up that way, comparing it to my running installation.)

I would still try to use the bin/opensearch-plugin tool to handle the install, though. Looks like it probably does accept a local path: Custom URL or file system | Elasticsearch Plugins and Integrations [8.4] | Elastic

thanks. as mentioned in my post: this is a SNAPSHOT release and thus does not correspond to the “real” release (it also is a SNAPSHOT release inside of the plugin-descriptor.properties file). this means that this is some snapshot of the plugin but you have no clue what it exactly contains, it is obviously not the artifact which went through all the tests.

i’d like to have the download URL for the plugin versions which are also used in the “full” distribution or when you run opensearch-plugin install [..] (i guess i could start digging in the code for plugin handling to see where it’s getting the plugins from? but that shouldn’t be the idea either, should it now)

@bbarani Let’s get an issue open for this in opensearch-build. All the plugins should be manageable through the same opensearch-plugin tool, and there are definitely some good requirements here for making zip URLs discoverable.

2 Likes

The answer is in manifest.yml in whatever build it is you’re looking to match plugins to.

So, I downloaded build 1.2.3 from here: https://artifacts.opensearch.org/releases/bundle/opensearch/1.2.3/opensearch-1.2.3-linux-x64.tar.gz

Then, in there, manifest.yml has all the URLs used by the bin/opensearch-plugins tool:

mhoydis@M1-16 opensearch-1.2.3 % cat manifest.yml 
build:
  architecture: x64
  id: '295'
  location: https://ci.opensearch.org/ci/dbc/distribution-build-opensearch/1.2.3/295/linux/x64/dist/opensearch/opensearch-1.2.3-linux-x64.tar.gz
  name: OpenSearch
  platform: linux
  version: 1.2.3
components:
- commit_id: 8a529d77c7432bc45b005ac1c4ba3b2741b57d4a
  location: https://ci.opensearch.org/ci/dbc/distribution-build-opensearch/1.2.3/295/linux/x64/builds/opensearch/dist/opensearch-min-1.2.3-linux-x64.tar.gz
  name: OpenSearch
  ref: '1.2'
  repository: https://github.com/opensearch-project/OpenSearch.git
- commit_id: 04f147f5e85ffd164849f8415b89436b51125edb
  location: https://ci.opensearch.org/ci/dbc/distribution-build-opensearch/1.2.3/295/linux/x64/builds/opensearch/plugins/opensearch-job-scheduler-1.2.3.0.zip
  name: job-scheduler
  ref: '1.2'
  repository: https://github.com/opensearch-project/job-scheduler.git
- commit_id: c20dec43456cf3382995257d966fb04ea5d52e9b
  location: https://ci.opensearch.org/ci/dbc/distribution-build-opensearch/1.2.3/295/linux/x64/builds/opensearch/plugins/opensearch-alerting-1.2.3.0.zip
  name: alerting
  ref: '1.2'
  repository: https://github.com/opensearch-project/alerting.git
- commit_id: 19a5c7467c0c46098abe6d2ff1c9e55c086b6c9d
  location: https://ci.opensearch.org/ci/dbc/distribution-build-opensearch/1.2.3/295/linux/x64/builds/opensearch/plugins/opensearch-asynchronous-search-1.2.3.0.zip
  name: asynchronous-search
  ref: '1.2'
  repository: https://github.com/opensearch-project/asynchronous-search.git
- commit_id: 4a5ca79280b67be20623daaac500a1186c2f7421
  location: https://ci.opensearch.org/ci/dbc/distribution-build-opensearch/1.2.3/295/linux/x64/builds/opensearch/plugins/opensearch-index-management-1.2.3.0.zip
  name: index-management
  ref: '1.2'
  repository: https://github.com/opensearch-project/index-management.git
- commit_id: 960c763862e904f479d1ab0044391ec61907797b
  location: https://ci.opensearch.org/ci/dbc/distribution-build-opensearch/1.2.3/295/linux/x64/builds/opensearch/plugins/opensearch-knn-1.2.3.0.zip
  name: k-NN
  ref: '1.2'
  repository: https://github.com/opensearch-project/k-NN.git
- commit_id: 456998475bc3987bcf365dfd2b9e7db9779ef5a7
  location: https://ci.opensearch.org/ci/dbc/distribution-build-opensearch/1.2.3/295/linux/x64/builds/opensearch/plugins/opensearch-security-1.2.3.0.zip
  name: security
  ref: '1.2'
  repository: https://github.com/opensearch-project/security.git
- commit_id: 7983848546a67bb38233e07a7e2286c0585f2928
  location: https://ci.opensearch.org/ci/dbc/distribution-build-opensearch/1.2.3/295/linux/x64/builds/opensearch/plugins/opensearch-performance-analyzer-1.2.3.0.zip
  name: performance-analyzer
  ref: '1.2'
  repository: https://github.com/opensearch-project/performance-analyzer.git
- commit_id: 4d45ebcf5faea8e3d180ed003659a2b98927911f
  location: https://ci.opensearch.org/ci/dbc/distribution-build-opensearch/1.2.3/295/linux/x64/builds/opensearch/plugins/opensearch-anomaly-detection-1.2.3.0.zip
  name: anomaly-detection
  ref: '1.2'
  repository: https://github.com/opensearch-project/anomaly-detection.git
- commit_id: 6323a5357de0c4a27572f6f192557e1528740aff
  location: https://ci.opensearch.org/ci/dbc/distribution-build-opensearch/1.2.3/295/linux/x64/builds/opensearch/plugins/opensearch-cross-cluster-replication-1.2.3.0.zip
  name: cross-cluster-replication
  ref: '1.2'
  repository: https://github.com/opensearch-project/cross-cluster-replication.git
- commit_id: 9bc9ddaf31e5704693969c8efd8a843395dc4be2
  location: https://ci.opensearch.org/ci/dbc/distribution-build-opensearch/1.2.3/295/linux/x64/builds/opensearch/plugins/opensearch-sql-1.2.3.0.zip
  name: sql
  ref: '1.2'
  repository: https://github.com/opensearch-project/sql.git
- commit_id: 0e815f43b3663a0b53abd78a82c2ac1c250cc870
  location: https://ci.opensearch.org/ci/dbc/distribution-build-opensearch/1.2.3/295/linux/x64/builds/opensearch/plugins/opensearch-reports-scheduler-1.2.3.0.zip
  name: dashboards-reports
  ref: '1.2'
  repository: https://github.com/opensearch-project/dashboards-reports.git
- commit_id: 02dc39920a2d81217ac1ab33d659929130598d77
  location: https://ci.opensearch.org/ci/dbc/distribution-build-opensearch/1.2.3/295/linux/x64/builds/opensearch/plugins/opensearch-observability-1.2.3.0.zip
  name: opensearch-observability
  ref: '1.2'
  repository: https://github.com/opensearch-project/observability
schema-version: '1.1'
4 Likes

As pointed by @ mhoydis You can download the plugin zips directly using the URL in the manifest file present inside the OpenSearch distribution artifact. We are working on adding support to install these plugins via ./opensearch-plugin command as well. We will keep you updated on the progress and you can track the status here

In the meantime you can still use the ./opensearch-plugin command to install the plugins by passing the URL of the plugin zips present in the distribution manifest file

Example:

./opensearch-plugin install https://ci.opensearch.org/ci/dbc/distribution-build-opensearch/1.2.3/295/linux/x64/builds/opensearch/plugins/opensearch-alerting-1.2.3.0.zip

i don’t know why, but the ./opensearch-plugin install command gives me this error

Hi @LailaBoullous

It should be ./opensearch-plugin, not /opensearch-plugin, as the latter assumes in the script is in root not in current directory.

You need to be in opensearch-<version>/bin folder running this.

Thanks.

3 Likes

I’m using Helm chart and i’ve tried gettin into the bin file associated to opensearch but i still get the same error

you’re in a windows powershell and not a (linux) bash, so using ./ won’t work. i think you can just directly write opensearch-plugin. but somebody familiar with PS might have to chime in here (and in that case the discussion should probably move to its own dedicated thread as it’s unrelated to the download links, it’s just about how to call the binary from powershell).

I found this thread while finding solution for my issue i.e. =

[opensearch-dashboards@gmsp-es-logging-dashboards-7d4f787c47-dnpzv bin]$ opensearch-dashboards-plugin install repository-s3
Found previous install attempt. Deleting…
Attempting to transfer from repository-s3
Attempting to transfer from https://ci.opensearch.org/ci/dbc/distribution-build-opensearch-dashboards/1.3.15/latest/linux/x64/tar/builds/opensearch-dashboards/plugins/repository-s3-1.3.15.zip
Plugin installation was unsuccessful due to error “No valid url specified.”

Question: Can I use following alternative? If yes how to use it via cmdline? Later on I need to embed in to the Dockerfile.

My existing Dockerfile =

FROM opensearchproject/opensearch-dashboards:1.3.15

RUN bin/opensearch-dashboards-plugin install --batch repository-s3
RUN bin/opensearch-plugin install --batch https://github.com/aiven/prometheus-exporter-plugin-for-opensearch/releases/download/1.3.15.0/prometheus-exporter-1.3.15.0.zip