Cohere connection failing

Plugins Machine Learning
,
1

Versions (relevant - OpenSearch/Dashboard/Server OS/Browser):

Opensearch -3.2.0

Describe the issue:

I am trying to connect to cohere using the connector :

{
“name”: “Cohere Chat Model”,
“description”: “The connector to Cohere’s public chat API”,
“version”: “1”,
“protocol”: “http”,
“allow_insecure”: true,
“credential”: {
“cohere_key”: “my_api_key”
},
“parameters”: {
“model”: “command-a-03-2025”
},
“actions”: [
{
“action_type”: “predict”,
“method”: “POST”,
“url”: “https://api.cohere.com/v2/chat”,
“headers”: {
“Authorization”: “Bearer ${credential.cohere_key}”,
“Request-Source”: “unspecified:opensearch”
},
“request_body”: “{ “message”: “${parameters.message}”, “model”: “${parameters.model}” }”
}
]
}

Relevant Logs or Screenshots:

When i do the predict it is failing with the following error:

image
image694×664 22.8 KB

in the logs i am seeing this:

Received error from remote service: Acquire operation took longer than the configured maximum time. This indicates that a request cannot get a connection from the pool within the specified maximum time. This can be due to high request rate.
Consider taking any of the following actions to mitigate the issue: increase max connections, increase acquire timeout, or slowing the request rate.
Increasing the max connections can increase client throughput (unless the network interface is already fully utilized), but can eventually start to hit operation system limitations on the number of file descriptors used by the process. If you already are fully utilizing your network interface or cannot further increase your connection count, increasing the acquire timeout gives extra time for requests to acquire a connection before timing out. If the connections doesn’t free up, the subsequent requests will still timeout.
If the above mechanisms are not able to fix the issue, try smoothing out your requests so that large traffic bursts cannot overload the client, being more efficient with the number of times you need to call AWS, or by increasing the number of hosts sending requests.
java.lang.Throwable: Acquire operation took longer than the configured maximum time. This indicates that a request cannot get a connection from the pool within the specified maximum time. This can be due to high request rate.
Consider taking any of the following actions to mitigate the issue: increase max connections, increase acquire timeout, or slowing the request rate.
Increasing the max connections can increase client throughput (unless the network interface is already fully utilized), but can eventually start to hit operation system limitations on the number of file descriptors used by the process. If you already are fully utilizing your network interface or cannot further increase your connection count, increasing the acquire timeout gives extra time for requests to acquire a connection before timing out. If the connections doesn’t free up, the subsequent requests will still timeout.
If the above mechanisms are not able to fix the issue, try smoothing out your requests so that large traffic bursts cannot overload the client, being more efficient with the number of times you need to call AWS, or by increasing the number of hosts sending requests.
at software.amazon.awssdk.http.nio.netty.internal.utils.NettyUtils.decorateException(NettyUtils.java:80) ~[?:?]
at software.amazon.awssdk.http.nio.netty.internal.NettyRequestExecutor.handleFailure(NettyRequestExecutor.java:307) ~[?:?]
at software.amazon.awssdk.http.nio.netty.internal.NettyRequestExecutor.makeRequestListener(NettyRequestExecutor.java:188) ~[?:?]
at io.netty.util.concurrent.DefaultPromise.notifyListener0(DefaultPromise.java:590) ~[?:?]
at io.netty.util.concurrent.DefaultPromise.notifyListenersNow(DefaultPromise.java:557) ~[?:?]
at io.netty.util.concurrent.DefaultPromise.access$200(DefaultPromise.java:35) ~[?:?]
at io.netty.util.concurrent.DefaultPromise$1.run(DefaultPromise.java:503) ~[?:?]
at io.netty.util.concurrent.AbstractEventExecutor.runTask(AbstractEventExecutor.java:173) ~[?:?]
at io.netty.util.concurrent.AbstractEventExecutor.safeExecute(AbstractEventExecutor.java:166) ~[?:?]
at io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:472) ~[?:?]
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:569) ~[?:?]
at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:998) ~[?:?]
at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) ~[?:?]
at java.base/java.lang.Thread.run(Thread.java:1583) [?:?]
Caused by: java.util.concurrent.TimeoutException: Acquire operation took longer than 10000 milliseconds.
at software.amazon.awssdk.http.nio.netty.internal.HealthCheckedChannelPool.timeoutAcquire(HealthCheckedChannelPool.java:77) ~[?:?]
at software.amazon.awssdk.http.nio.netty.internal.HealthCheckedChannelPool.lambda$acquire$0(HealthCheckedChannelPool.java:67) ~[?:?]
at io.netty.util.concurrent.PromiseTask.runTask(PromiseTask.java:98) ~[?:?]
at io.netty.util.concurrent.ScheduledFutureTask.run(ScheduledFutureTask.java:156) ~[?:?]

Please let me know how to resolve this

2

The same if i send through curl i am getting response

Is there any config i am missing

3

@tejashu I’ve just tested OpenSearch with Cohere and I had no issues. This is my working example.

POST /_plugins/_ml/connectors/_create
{
  "name": "Cohere model",
  "description": "Connector for Cohere",
  "protocol": "http",
  "version": 1,
  "parameters": {
    "model": "command-a-03-2025",
    "temperature": 0.7,
    "max_tokens": 500,
    "endpoint": "api.cohere.com"
  },
  "credential": {
    "api_key": "<api_key>"
  },
  "client_config" : {
    "read_timeout": 60000,
    "connection_timeout": 30000,
    "max_connection": 256,
    "max_retry_times": 3,
    "retry_backoff_policy": "exponential_full_jitter"
  },
  "actions": [
    {
      "action_type": "predict",
      "method": "POST",
      "url": "https://${parameters.endpoint}/v2/chat",
      "headers": {
        "Content-Type": "application/json",
        "Authorization": "Bearer ${credential.api_key}"
      },
      "request_body": "{ \"model\": \"${parameters.model}\",  \"messages\": ${parameters.messages}, \"temperature\": ${parameters.temperature}, \"max_tokens\": ${parameters.max_tokens}, \"stream\": false }"
    }
  ]
}

POST /_plugins/_ml/models/_register
{
  "name": "ollama",
  "function_name": "remote",
  "connector_id": "_M22WJoBLdsWuLU2pUmv"
}

POST /_plugins/_ml/models/Ac23WJoBLdsWuLU2Akrg/_predict
{
  "parameters": {
    "messages": [
      {
        "role": "system",
        "content": "You are a helpful assistant."
      },
      {
        "role": "user",
        "content": "Hello!What is the name of this model??"
      }
    ]
  }
}

Response:

{
  "inference_results": [
    {
      "output": [
        {
          "name": "response",
          "dataAsMap": {
            "id": "be1e2fde-20d0-4161-9d15-6dee82ec9efa",
            "message": {
              "role": "assistant",
              "content": [
                {
                  "type": "text",
                  "text": "Hello! I'm Command, a large language model built by Cohere. I'm designed to assist with a wide range of tasks and provide thorough, informative responses. How can I help you today?"
                }
              ]
            },
            "finish_reason": "COMPLETE",
            "usage": {
              "billed_units": {
                "input_tokens": 16,
                "output_tokens": 40
              },
              "tokens": {
                "input_tokens": 545,
                "output_tokens": 42
              },
              "cached_tokens": 496
            }
          }
        }
      ],
      "status_code": 200
    }
  ]
}
4

I am still seeing this error
How to debug
{
“error”: {
“root_cause”: [
{
“type”: “status_exception”,
“reason”: “Error communicating with remote model: Acquire operation took longer than the configured maximum time. This indicates that a request cannot get a connection from the pool within the specified maximum time. This can be due to high request rate.\nConsider taking any of the following actions to mitigate the issue: increase max connections, increase acquire timeout, or slowing the request rate.\nIncreasing the max connections can increase client throughput (unless the network interface is already fully utilized), but can eventually start to hit operation system limitations on the number of file descriptors used by the process. If you already are fully utilizing your network interface or cannot further increase your connection count, increasing the acquire timeout gives extra time for requests to acquire a connection before timing out. If the connections doesn’t free up, the subsequent requests will still timeout.\nIf the above mechanisms are not able to fix the issue, try smoothing out your requests so that large traffic bursts cannot overload the client, being more efficient with the number of times you need to call AWS, or by increasing the number of hosts sending requests.”
}
],
“type”: “status_exception”,
“reason”: “Error communicating with remote model: Acquire operation took longer than the configured maximum time. This indicates that a request cannot get a connection from the pool within the specified maximum time. This can be due to high request rate.\nConsider taking any of the following actions to mitigate the issue: increase max connections, increase acquire timeout, or slowing the request rate.\nIncreasing the max connections can increase client throughput (unless the network interface is already fully utilized), but can eventually start to hit operation system limitations on the number of file descriptors used by the process. If you already are fully utilizing your network interface or cannot further increase your connection count, increasing the acquire timeout gives extra time for requests to acquire a connection before timing out. If the connections doesn’t free up, the subsequent requests will still timeout.\nIf the above mechanisms are not able to fix the issue, try smoothing out your requests so that large traffic bursts cannot overload the client, being more efficient with the number of times you need to call AWS, or by increasing the number of hosts sending requests.”
},
“status”: 500
}

5

@tejashu This looks like a network issue. Can you reach the Cohere framework from the OpenSearch node/pod using curl or another tool like openssl?

curl https://api.cohere.com/v2/chat

openssl s_client -connect api.cohere.com:443