Greetings all,
I appreciate if someone can share with us the process to build the k-NN plugin.
When I run “./gradlew build” I get the below error:
Could not find blas (missing: BLAS_LIBRARIES)
I actually installed blas openblas and lapack but none of them helped.
The above was done on Rocky Linux 8 (which is supposed yo be equivalent to CentOS 8 and RHEL 8)
Thanks
I believe that blas stuff is very new. There is an open issue on opensearch-build about how the libraries are going to be integrated. @jmazane might know more.
@asfoorial are you trying to get to the new faiss stuff or just wanting to build a stable version of the library?
I am trying to see how I can extend it. Step zero is to build a stable version of the plugin.
Ah, ok - glad to hear it! Try building off a branch, for example: GitHub - opensearch-project/k-NN at 1.1 - @jmazane is a developer on the team, so he probably can give you more pointers.
The way OpenSearch repos work is that the main branch is always the active development branch, so it could be broken, etc. Stable versions are in version labeled branches.
I tried… but got the below error.
FAILURE: Build failed with an exception.
Could not resolve all artifacts for configuration ‘:classpath’.
Could not find org.opensearch.gradle:build-tools:1.1.0-SNAPSHOT.
Searched in the following locations:
- https://repo.maven.apache.org/maven2/org/opensearch/gradle/build-tools/1.1.0-SNAPSHOT/maven-metadata.xml
- https://repo.maven.apache.org/maven2/org/opensearch/gradle/build-tools/1.1.0-SNAPSHOT/build-tools-1.1.0-SNAPSHOT.pom
- https://plugins.gradle.org/m2/org/opensearch/gradle/build-tools/1.1.0-SNAPSHOT/maven-metadata.xml
- https://plugins.gradle.org/m2/org/opensearch/gradle/build-tools/1.1.0-SNAPSHOT/build-tools-1.1.0-SNAPSHOT.pom
- file:/home/hasan/.m2/repository/org/opensearch/gradle/build-tools/1.1.0-SNAPSHOT/maven-metadata.xml
- file:/home/hasan/.m2/repository/org/opensearch/gradle/build-tools/1.1.0-SNAPSHOT/build-tools-1.1.0-SNAPSHOT.pom
- https://jcenter.bintray.com/org/opensearch/gradle/build-tools/1.1.0-SNAPSHOT/maven-metadata.xml
- https://jcenter.bintray.com/org/opensearch/gradle/build-tools/1.1.0-SNAPSHOT/build-tools-1.1.0-SNAPSHOT.pom
Required by:
project :
Finally I got it working. Switching to 1.1 fixed one thing. But then I had to build OpenSearch and push it to a local maven repo. Also, it 9nly accepted compiling with JDK14.
Glad to hear you got it. Yeah, I’ve been bitten by the JDK thing before.
After all the struggle I found out that the kNN plugin is not extensible! Basically I wanted to create a search plugin that utilizes classes from the kNN plugin. So I added extended.plugins=opensearch-knn to the plugin descriptor. Later I got an error saying it is not extensible.
Is there a way to make it extensible?
Hi @asfoorial ,
It looks like to make the plugin extensible, the k-NN plugin would need to implement the ExtensiblePlugin interface.
There is no a reason (I know of) why we shouldn’t be able to implement this interface, but I need to look into what this would require a little bit closer. I will create an issue on the Github repo and move discussion over to there.
Side note: jni is changing in the 1.2 release. I am planning on updating developer guide with better instructions. Sorry for the inconvenience when building.
Jack
This is good news. I look forward for having kNN becoming extensible. I am sure it will pave the way for several extension projects in the future. Already we have seen projects leveraging kNN in past meetings and this step will enable them to work within the OpenSearch ecosystem.
Thats great to hear! I have not gotten a chance to look into it yet. I have been busy with 1.2 release, but will take a look after that.
Hi @asfoorial,
Could you please specify what was your plan for k-NN extension?
In order to make plugin extensible it has to have two components - be declared as extensible, and provide extension interface. This last extension interface is actually a mechanism to allow callbacks from extension be executed by the main plugin. For k-NN team it’s important to understand what events need to be exposed and in what is the context (information that is sent along with the event).
Hi @martin.g,
I wanted to create a custom query plugin that internally utilizes the classes provides by the kNN plugin. In fact I did that but during plugin installation I got the error saying it is not extensible. I could have replicated the kNN code and be done with it. But I prefer to utilize the kNN classes instead to maintain future kNN enhancements by your team
Thanks
Hi @asfoorial
Could you please give more concrete example of what you want to achieve? If plugin became extensible we as an owners can provide certain extension points, but we can’t open all plugin internals. For example, please check existing extension points for other plugins like Reindex - https://github.com/opensearch-project/OpenSearch/blob/33d8677796b399ca81156c8e043279eddbaf6cff/modules/reindex/src/main/java/org/opensearch/index/reindex/spi/RemoteReindexExtension.java#L19. In this example plugin owners provided two extension points - one is REST interceptor that can pre- or post-process REST requests to the plugin. Another extension point is an action listener for incoming requests to plugin and extension subscribes to such events.
To be more specific, I am creating a custom query plugin that overrides doToQuery of the AbstractQueryBuilder class. Inside this method, I am constructing an instance of org.opensearch.knn.index.KNNQuery.
My code compiles without any issue. However, When I install the plugin I get the error above.
I don’t want to create my own kNN but I want to create a custom query plugin that recieves an input text, preprocesses it, vectorizes it and finally passes it to the kNNQuery class to complete the kNN operation.
Note also that I have the kNN plugin as a dependency to my build.gradle.
dependencies {
classpath “org.opensearch:opensearch-knn:1.1.0.0”
}
Implementing the ExtensiblePlugin interface solves my issue. I just tested it. I hope that this change is applied to the main code in upcoming releases of this amazing k-NN plugin.
public class KNNPlugin extends Plugin implements MapperPlugin, SearchPlugin, ActionPlugin, EnginePlugin, ScriptPlugin, ExtensiblePlugin
{
…
}
Hi Hasan,
I see what you mean now. The problem with adding ExtensiblePlugin to the KNNPlugin is that it’s going to be an empty implementation of loadExtension method as the team doesn’t plan any extensions at the moment.
There is another way of achieving the same behavior without plugin class (KNNPlugin). It’s possible to extend KNNPlugin class and make that child class implementing ExtensiblePlugin. Code can be something like this:
public class MyPlugin extends KNNPlugin implements ExtensiblePlugin {
@Override
public void loadExtensions(ExtensionLoader loader) {
.......
}
}
Depending on use case you may not even need to fork plugin repo, but just add it as a compile/runtime dependency.
As far as I have seen, overriding loadExtensions is not required when implementing the ExtensiblePlugin interface. It is more of a type sharing thing that I am after. I will give your suggestion a try though and see if it works. Hopefully I won’t get Jar Hell and similar clashes.
Regards
Your earlier suggestions did not work. I got an error “plugin not extensible”. I tried implementing the ExtensiblePlugin inside my plugin and also tried extending the KNNPlugin but in both cases I got the same error.
The only way it worked is when making the kNN plugin implements ExtensiblePlugin. Note that overriding “loadExtensions” was not required.
Earlier you said that there are no plans to implement the ExtensiblePlugin inside the kNN plugin. Is there a technical reason for that? I don’t see how adding this word “ExtensiblePlugin” would create problems.
I highly appreciate your input to this.