How To Build k-NN Plugin?

Plugins k-NN
21

Thank you for testing the solution, it’s unfortunate that it doesn’t work for you.

I’ve double checked the ExtensiblePlugin interface, it has the default “empty” implementation so I think we can add it to the KNNPlugin class (that’s why it’s not required to provide your own implementation of the loadExtensions method as you’ve mentioned).
Our original concern was related to possible extension points because we are not planning to provide those for the kNN plugin at the moment.

1 Like
22

I’d like to provide some update on this topic.

We have made kNN plugin extensible, change is coming in 1.3 release. As of now this going to be a default implementation of the ExtensiblePlugin interface that is “empty” by means of not loading any extensions.

@asfoorial I’d like to understand more about the problem you are trying to solve. If you’re building your own plugin then why you need make it an extension of kNN?

Another question - how making plugin extensible solves your problem? Seems that you haven’t implement any extension points but just use kNN classes. You could achieve the same if you just define kNN as a build time dependency in gradle?

23

Thank you @martin.g for making the kNN plugin extensible. I am creating a plugin that utilizes the KNNQuery class. Everything worked fine at compile time, however when I install the plugin, the KNNQuery class becomes invisible to my plugin. The only way that made it visible for was by making the whole kNN plugin extensible.

24

Hi @asfoorial,
I think in this case the problem may be related to assembly of the final artifact for your plugin (I mean case when k-NN is not extensible). Could you please share the assembly script that you used (build.gradle or other similar)?
Also I wonder what changes for you after making plugin extensible? From what I understood you still need KNNQuery class in your project, and it visibility shouldn’t depend on the extensibility of the plugin.

25

apply plugin: ‘java’
apply plugin: ‘idea’
apply plugin: ‘opensearch.opensearchplugin’

opensearchplugin {
name ‘my-plugin’
description ‘My Plugin for OpenSearch’
classname ‘org.myplugin.search.opensearch.plugins.MyPlugin’
extendedPlugins = [‘opensearch-knn’]
licenseFile rootProject.file(‘LICENSE.txt’)
noticeFile rootProject.file(‘NOTICE.txt’)
}

// disabling some unnecessary validations for this plugin
testingConventions.enabled = false
loggerUsageCheck.enabled = false
validateNebulaPom.enabled = false

forbiddenApisMain.enabled = false
licenseHeaders.enabled = false
forbiddenPatterns.enabled = false

dependencyLicenses.enabled = false
thirdPartyAudit.enabled = false
jarHell.enabled = true
buildscript {
ext {
opensearch_version = System.getProperty(“opensearch.version”, “1.2.4”)
opensearch_group = “org.opensearch”
}
repositories {mavenLocal()
mavenCentral()
jcenter()

}

dependencies {
    classpath "org.opensearch.gradle:build-tools:1.2.4"
    //classpath 'org.opensearch:opensearch-knn:1.2.4.0'
}

}

dependencies {

compileOnly 'org.opensearch:opensearch-knn:1.2.4.0'
implementation 'com.github.haifengl:smile-nlp:2.6.0'
implementation 'com.fasterxml.jackson.core:jackson-databind:2.12.5'
implementation 'com.fasterxml.jackson.core:jackson-annotations:2.12.5'

}

repositories {
mavenCentral()
mavenLocal()
jcenter()
}

26

If I keep kNN in the dependencies without extendedPlugins = [‘opensearch-knn’] then I get jar hell errors when installing the plugin.

27

Thank you for the script provided. I’ve spotted few difference between your build,gradle and one that I’ve created for the sample plugin extension. They are: knn plugin version 1.3-snapshot instead of 1.2, usage of knn-plugin jar locally instead of getting it from the remote repo (compileOnly files), some validations are not toggled, specifically jarHell.enabled.

For testing purposes I’ve installed test extension plugin using standard opensearch script ( bin/opensearch-plugin install)

buildscript {
    repositories {
        mavenLocal()
        mavenCentral()
        jcenter()
    }

    dependencies {
        classpath "org.opensearch.gradle:build-tools:1.3.0-SNAPSHOT"
    }
}

plugins {
    id 'java'
}

group 'org.example'
version '1.0-SNAPSHOT'

repositories {
    mavenLocal()
    mavenCentral()
}

apply plugin: 'java'
apply plugin: 'idea'
apply plugin: 'opensearch.opensearchplugin'
apply plugin: 'opensearch.yaml-rest-test'

opensearchplugin {
    name 'opensearch-extension'
    description 'Custom OpenSearch plugin to test extension'
    classname 'org.knn.extension.ExampleKNNExtensionPlugin'
    extendedPlugins = ['opensearch-knn']
    licenseFile rootProject.file('LICENSE.txt')
    noticeFile rootProject.file('NOTICE.txt')
}

dependencies {
    compileOnly files('lib/opensearch-knn-1.3.0.0-SNAPSHOT.jar')
    implementation group: 'org.apache.logging.log4j', name: 'log4j-api', version: '2.17.1'
    implementation group: 'org.apache.logging.log4j', name: 'log4j-core', version: '2.17.1'
    testImplementation 'org.junit.jupiter:junit-jupiter-api:5.7.2'
    testRuntimeOnly 'org.junit.jupiter:junit-jupiter-engine:5.7.2'

    //jar {
    //    duplicatesStrategy = DuplicatesStrategy.EXCLUDE
    //    from (configurations.runtimeClasspath.findAll { it.name.endsWith('jar') }.collect { zipTree(it) } ) {
    //    }
    //}
}

dependencyLicenses.enabled = false
loggerUsageCheck.enabled = false
testingConventions.enabled = false
validateNebulaPom.enabled = false

test {
    useJUnitPlatform()
}
28

Hi @asfoorial,

Its been quite some time to this issue, but i have some ques around this and i hope you could help me.
I have two opensearch plugins (index-management and cross-cluster-replication), where I want to use classes from the ccr plugin in ism code.

I tried modifying ccr as an extensiblePlugin and using extendedPlugins = ['ccr" ] in ism build.gradle.
But when i try to install the plugin, i still get jarhell errors.
Have you seen this?
Any suggestions on this?