Versions (relevant - OpenSearch/Dashboard/Server OS/Browser):
OpenSearch - 2.18
Describe the issue:
Getting “Connection timed out” error when trying to deploy a model. Is there a way to increase the http timeout in OpenSearch?
Relevant Logs or Screenshots:
Caused by: java.net.ConnectException: Connection timed out
2024-11-09T19:01:31.158718036Z at java.base/sun.nio.ch.Net.connect0(Native Method) ~[?:?]
2024-11-09T19:01:31.158719937Z at java.base/sun.nio.ch.Net.connect(Net.java:589) ~[?:?]
2024-11-09T19:01:31.158721828Z at java.base/sun.nio.ch.Net.connect(Net.java:578) ~[?:?]
2024-11-09T19:01:31.158723859Z at java.base/sun.nio.ch.NioSocketImpl.connect(NioSocketImpl.java:583) ~[?:?]
2024-11-09T19:01:31.158725782Z at java.base/java.net.SocksSocketImpl.connect(SocksSocketImpl.java:327) ~[?:?]
2024-11-09T19:01:31.158727683Z at java.base/java.net.Socket.connect(Socket.java:751) ~[?:?]
2024-11-09T19:01:31.158729596Z at java.base/sun.security.ssl.SSLSocketImpl.connect(SSLSocketImpl.java:304) ~[?:?]
2024-11-09T19:01:31.158731615Z at java.base/sun.security.ssl.BaseSSLSocketImpl.connect(BaseSSLSocketImpl.java:181) ~[?:?]
2024-11-09T19:01:31.158733549Z at java.base/sun.net.NetworkClient.doConnect(NetworkClient.java:183) ~[?:?]
2024-11-09T19:01:31.158735440Z at java.base/sun.net.www.http.HttpClient.openServer(HttpClient.java:531) ~[?:?]
2024-11-09T19:01:31.158737363Z at java.base/sun.net.www.http.HttpClient.openServer(HttpClient.java:636) ~[?:?]
2024-11-09T19:01:31.158739286Z at java.base/sun.net.www.protocol.https.HttpsClient.<init>(HttpsClient.java:264) ~[?:?]
2024-11-09T19:01:31.158741209Z at java.base/sun.net.www.protocol.https.HttpsClient.New(HttpsClient.java:377) ~[?:?]
2024-11-09T19:01:31.158743122Z at java.base/sun.net.www.protocol.https.AbstractDelegateHttpsURLConnection.getNewHttpClient(AbstractDelegateHttpsURLConnection.java:193) ~[?:?]
2024-11-09T19:01:31.158745109Z at java.base/sun.net.www.protocol.http.HttpURLConnection.plainConnect0(HttpURLConnection.java:1237) ~[?:?]
2024-11-09T19:01:31.158747032Z at java.base/sun.net.www.protocol.http.HttpURLConnection.plainConnect(HttpURLConnection.java:1123) ~[?:?]
2024-11-09T19:01:31.158751577Z at java.base/sun.net.www.protocol.https.AbstractDelegateHttpsURLConnection.connect(AbstractDelegateHttpsURLConnection.java:179) ~[?:?]
2024-11-09T19:01:31.158753640Z at java.base/sun.net.www.protocol.http.HttpURLConnection.getInputStream0(HttpURLConnection.java:1675) ~[?:?]
2024-11-09T19:01:31.158755606Z at java.base/sun.net.www.protocol.http.HttpURLConnection.getInputStream(HttpURLConnection.java:1599) ~[?:?]
2024-11-09T19:01:31.158757572Z at java.base/sun.net.www.protocol.https.HttpsURLConnectionImpl.getInputStream(HttpsURLConnectionImpl.java:223) ~[?:?]
2024-11-09T19:01:31.158759699Z at ai.djl.util.Utils.openUrl(Utils.java:519) ~[?:?]
2024-11-09T19:01:31.158761654Z at ai.djl.util.Utils.openUrl(Utils.java:498) ~[?:?]
2024-11-09T19:01:31.158763545Z at ai.djl.util.Utils.openUrl(Utils.java:487) ~[?:?]
2024-11-09T19:01:31.158765458Z at ai.djl.pytorch.jni.LibUtils.downloadPyTorch(LibUtils.java:424) ~[?:?]
2024-11-09T19:01:31.158767370Z at ai.djl.pytorch.jni.LibUtils.findNativeLibrary(LibUtils.java:314) ~[?:?]
2024-11-09T19:01:31.158769272Z at ai.djl.pytorch.jni.LibUtils.getLibTorch(LibUtils.java:93) ~[?:?]
2024-11-09T19:01:31.158771173Z at ai.djl.pytorch.jni.LibUtils.loadLibrary(LibUtils.java:81) ~[?:?]
2024-11-09T19:01:31.158773086Z at ai.djl.pytorch.engine.PtEngine.newInstance(PtEngine.java:53) ~[?:?]
2024-11-09T19:01:31.158774998Z at ai.djl.pytorch.engine.PtEngineProvider.getEngine(PtEngineProvider.java:41) ~[?:?]
2024-11-09T19:01:31.158776921Z at ai.djl.engine.Engine.getEngine(Engine.java:190) ~[?:?]
2024-11-09T19:01:31.158778812Z at ai.djl.engine.Engine.getInstance(Engine.java:145) ~[?:?]
2024-11-09T19:01:31.158780735Z at ai.djl.onnxruntime.engine.OrtEngine.getAlternativeEngine(OrtEngine.java:75) ~[?:?]
2024-11-09T19:01:31.158782647Z at ai.djl.ndarray.BaseNDManager.<init>(BaseNDManager.java:64) ~[?:?]
2024-11-09T19:01:31.158784570Z at ai.djl.onnxruntime.engine.OrtNDManager.<init>(OrtNDManager.java:42) ~[?:?]
2024-11-09T19:01:31.158786504Z at ai.djl.onnxruntime.engine.OrtNDManager.<init>(OrtNDManager.java:35) ~[?:?]
2024-11-09T19:01:31.158788481Z at ai.djl.onnxruntime.engine.OrtNDManager$SystemManager.<init>(OrtNDManager.java:177) ~[?:?]
2024-11-09T19:01:31.158790426Z at ai.djl.onnxruntime.engine.OrtNDManager.<clinit>(OrtNDManager.java:37) ~[?:?]
2024-11-09T19:01:31.158792714Z at ai.djl.onnxruntime.engine.OrtEngine.newBaseManager(OrtEngine.java:134) ~[?:?]
2024-11-09T19:01:31.158794669Z at ai.djl.onnxruntime.engine.OrtEngine.newModel(OrtEngine.java:122) ~[?:?]
2024-11-09T19:01:31.158796582Z at ai.djl.Model.newInstance(Model.java:99) ~[?:?]
2024-11-09T19:01:31.158798462Z at ai.djl.repository.zoo.BaseModelLoader.createModel(BaseModelLoader.java:196) ~[?:?]
2024-11-09T19:01:31.158800363Z at ai.djl.repository.zoo.BaseModelLoader.loadModel(BaseModelLoader.java:159) ~[?:?]
2024-11-09T19:01:31.158804994Z at ai.djl.repository.zoo.Criteria.loadModel(Criteria.java:174) ~[?:?]
2024-11-09T19:01:31.158807003Z at org.opensearch.ml.engine.algorithms.DLModel.doLoadModel(DLModel.java:217) ~[?:?]
2024-11-09T19:01:31.158808947Z at org.opensearch.ml.engine.algorithms.DLModel.lambda$loadModel$1(DLModel.java:286) ~[?:?]
2024-11-09T19:01:31.158810860Z ... 14 more
@miroshuSan ,
to check whether the last trial of deploying a ML model in your nodes,
can you describe(i.e ls -la) their files/directories in /usr/share/opensearch/data/ml_cache?
In addition, does your cluster have a setting for plugins.ml_commons.model_auto_deploy.enable as “true”?
Or you can also try the below setting:
PUT _cluster/settings
{
"persistent": {
"plugins.ml_commons.ml_task_timeout_in_seconds": 600
}
}
The default valud is 600 and its value range is in [1, 86400]. (link)
Hey @yeonghyeonKo,
In /usr/share/opensearch/data/ml_cache i have models_cache folder with deploy, models and register folders but all of those are empty.
i didn’t explicitly set auto_deploy to true and i tried increasing the timeout to 86400 but that didnt help:
These are the settings im updating via PUT endpoint:
{
"persistent": {
"plugins.ml_commons.allow_registering_model_via_url": "true",
"plugins.ml_commons.allow_registering_model_via_local_file": "true",
"plugins.ml_commons.only_run_on_ml_node": "false",
"plugins.ml_commons.model_access_control_enabled": "true",
"plugins.ml_commons.native_memory_threshold": "99",
"plugins.ml_commons.ml_task_timeout_in_seconds": 86400
}
}
@miroshuSan ,
Is your computing environment a closed network with firewall conditions?
When MLModelManager tries to deploy ML models, it will check whether the model’s controller is enabled or not (git).
Usually your working directory in /usr/share/opensearch/data/ml_cache should be like this:
$ tree
.
├── pytorch
│ ├── 1.13.1-cpu-precxx11-linux-x86_64
│ │ ├── 0.28.0-libdjl_torch.so
│ │ ├── libc10.so
│ │ ├── libgomp-a34b3233.so.1
│ │ ├── libstdc++.so.6
│ │ ├── libtorch.so
│ │ └── libtorch_cpu.so
│ └── 1.13.1.txt
└── tokenizers
└── 0.19.1-0.28.0-linux-x86_64
└── libtokenizers.so
It not, doLoadModel method in DLModel in opensearch will download configuration files from djl.ai, but it fails if there is a barrier in your firewall from your host to https://publish.djl.ai.
@yeonghyeonKo ,
thanks for the explanation! Regarding the environment question:
no im working on my laptop and running opensearch in a docker container. Actually last week everything worked fine and it seems ever since Opensearch 2.18 update, im getting these Connection Timed out errors. Could it be that there are some newly introduced bugs in the newer versions of djl or ml-commons? it wonders me because the problem is fairly simple to reproduce. What i do is basically:
- Start OpenSearch container
- Send PUT request for settings as described above
- Create a model group
- Register a model. It seems it doesn’t matter if the model is local or default pretrained model provided by OpenSearch, as even when using “huggingface/sentence-transformers/paraphrase-multilingual-MiniLM-L12-v2” im getting the same error
- Try to deploy the model
Update: I just tried to delete the opensearch data volume, restart the container and tried to follow the above mentioned steps and now im getting Connection timed out error on the registration step already.
Below is the stacktrace:
java.security.PrivilegedActionException: null
2024-11-11T10:53:44.342078706Z at java.base/java.security.AccessController.doPrivileged(AccessController.java:575) ~[?:?]
2024-11-11T10:53:44.342084741Z at org.opensearch.ml.engine.ModelHelper.downloadPrebuiltModelMetaList(ModelHelper.java:226) ~[?:?]
2024-11-11T10:53:44.342089121Z at org.opensearch.ml.model.MLModelManager.registerPrebuiltModel(MLModelManager.java:833) ~[?:?]
2024-11-11T10:53:44.342093137Z at org.opensearch.ml.model.MLModelManager.uploadModel(MLModelManager.java:665) ~[?:?]
2024-11-11T10:53:44.342097423Z at org.opensearch.ml.model.MLModelManager.lambda$registerMLModel$17(MLModelManager.java:454) ~[?:?]
2024-11-11T10:53:44.342101033Z at org.opensearch.core.action.ActionListener$1.onResponse(ActionListener.java:82) [opensearch-core-2.17.0.jar:2.17.0]
2024-11-11T10:53:44.342104425Z at org.opensearch.action.support.TransportAction$1.onResponse(TransportAction.java:115) [opensearch-2.17.0.jar:2.17.0]
2024-11-11T10:53:44.342108181Z at org.opensearch.action.support.TransportAction$1.onResponse(TransportAction.java:109) [opensearch-2.17.0.jar:2.17.0]
2024-11-11T10:53:44.342111583Z at org.opensearch.core.action.ActionListener$1.onResponse(ActionListener.java:82) [opensearch-core-2.17.0.jar:2.17.0]
2024-11-11T10:53:44.342115120Z at org.opensearch.action.support.single.instance.TransportInstanceSingleOperationAction$AsyncSingleAction$1.handleResponse(TransportInstanceSingleOperationAction.java:235) [opensearch-2.17.0.jar:2.17.0]
2024-11-11T10:53:44.342118845Z at org.opensearch.action.support.single.instance.TransportInstanceSingleOperationAction$AsyncSingleAction$1.handleResponse(TransportInstanceSingleOperationAction.java:221) [opensearch-2.17.0.jar:2.17.0]
2024-11-11T10:53:44.342122611Z at org.opensearch.security.transport.SecurityInterceptor$RestoringTransportResponseHandler.handleResponse(SecurityInterceptor.java:404) [opensearch-security-2.17.0.0.jar:2.17.0.0]
2024-11-11T10:53:44.342126294Z at org.opensearch.transport.TransportService$ContextRestoreResponseHandler.handleResponse(TransportService.java:1505) [opensearch-2.17.0.jar:2.17.0]
2024-11-11T10:53:44.342129707Z at org.opensearch.transport.TransportService$DirectResponseChannel.processResponse(TransportService.java:1588) [opensearch-2.17.0.jar:2.17.0]
2024-11-11T10:53:44.342136792Z at org.opensearch.transport.TransportService$DirectResponseChannel.sendResponse(TransportService.java:1568) [opensearch-2.17.0.jar:2.17.0]
2024-11-11T10:53:44.342153033Z at org.opensearch.transport.TaskTransportChannel.sendResponse(TaskTransportChannel.java:72) [opensearch-2.17.0.jar:2.17.0]
2024-11-11T10:53:44.342156643Z at org.opensearch.core.action.ActionListener$1.onResponse(ActionListener.java:82) [opensearch-core-2.17.0.jar:2.17.0]
2024-11-11T10:53:44.342160274Z at org.opensearch.action.update.TransportUpdateAction.lambda$shardOperation$3(TransportUpdateAction.java:302) [opensearch-2.17.0.jar:2.17.0]
2024-11-11T10:53:44.342163697Z at org.opensearch.core.action.ActionListener$1.onResponse(ActionListener.java:82) [opensearch-core-2.17.0.jar:2.17.0]
2024-11-11T10:53:44.342167338Z at org.opensearch.action.bulk.TransportSingleItemBulkWriteAction.lambda$wrapBulkResponse$0(TransportSingleItemBulkWriteAction.java:84) [opensearch-2.17.0.jar:2.17.0]
2024-11-11T10:53:44.342170689Z at org.opensearch.core.action.ActionListener$1.onResponse(ActionListener.java:82) [opensearch-core-2.17.0.jar:2.17.0]
2024-11-11T10:53:44.342174413Z at org.opensearch.action.support.TransportAction$1.onResponse(TransportAction.java:115) [opensearch-2.17.0.jar:2.17.0]
2024-11-11T10:53:44.342177670Z at org.opensearch.action.support.TransportAction$1.onResponse(TransportAction.java:109) [opensearch-2.17.0.jar:2.17.0]
2024-11-11T10:53:44.342181269Z at org.opensearch.core.action.ActionListener$6.onResponse(ActionListener.java:301) [opensearch-core-2.17.0.jar:2.17.0]
2024-11-11T10:53:44.342184620Z at org.opensearch.action.bulk.TransportBulkAction$BulkOperation$1.finishHim(TransportBulkAction.java:706) [opensearch-2.17.0.jar:2.17.0]
2024-11-11T10:53:44.342190685Z at org.opensearch.action.bulk.TransportBulkAction$BulkOperation$1.onResponse(TransportBulkAction.java:679) [opensearch-2.17.0.jar:2.17.0]
2024-11-11T10:53:44.342194098Z at org.opensearch.action.bulk.TransportBulkAction$BulkOperation$1.onResponse(TransportBulkAction.java:665) [opensearch-2.17.0.jar:2.17.0]
2024-11-11T10:53:44.342198717Z at org.opensearch.core.action.ActionListener$6.onResponse(ActionListener.java:301) [opensearch-core-2.17.0.jar:2.17.0]
2024-11-11T10:53:44.342204034Z at org.opensearch.action.support.TransportAction$1.onResponse(TransportAction.java:115) [opensearch-2.17.0.jar:2.17.0]
2024-11-11T10:53:44.342208840Z at org.opensearch.action.support.TransportAction$1.onResponse(TransportAction.java:109) [opensearch-2.17.0.jar:2.17.0]
2024-11-11T10:53:44.342213293Z at org.opensearch.action.support.replication.TransportReplicationAction$ReroutePhase.finishOnSuccess(TransportReplicationAction.java:1242) [opensearch-2.17.0.jar:2.17.0]
2024-11-11T10:53:44.342218891Z at org.opensearch.action.support.replication.TransportReplicationAction$ReroutePhase$1.handleResponse(TransportReplicationAction.java:1150) [opensearch-2.17.0.jar:2.17.0]
2024-11-11T10:53:44.342224280Z at org.opensearch.action.support.replication.TransportReplicationAction$ReroutePhase$1.handleResponse(TransportReplicationAction.java:1136) [opensearch-2.17.0.jar:2.17.0]
2024-11-11T10:53:44.342230397Z at org.opensearch.transport.TransportService$9.handleResponse(TransportService.java:1723) [opensearch-2.17.0.jar:2.17.0]
2024-11-11T10:53:44.342235579Z at org.opensearch.security.transport.SecurityInterceptor$RestoringTransportResponseHandler.handleResponse(SecurityInterceptor.java:404) [opensearch-security-2.17.0.0.jar:2.17.0.0]
2024-11-11T10:53:44.342246264Z at org.opensearch.transport.TransportService$ContextRestoreResponseHandler.handleResponse(TransportService.java:1505) [opensearch-2.17.0.jar:2.17.0]
2024-11-11T10:53:44.342249739Z at org.opensearch.transport.TransportService$DirectResponseChannel.processResponse(TransportService.java:1588) [opensearch-2.17.0.jar:2.17.0]
2024-11-11T10:53:44.342253286Z at org.opensearch.transport.TransportService$DirectResponseChannel.sendResponse(TransportService.java:1568) [opensearch-2.17.0.jar:2.17.0]
2024-11-11T10:53:44.342256636Z at org.opensearch.transport.TaskTransportChannel.sendResponse(TaskTransportChannel.java:72) [opensearch-2.17.0.jar:2.17.0]
2024-11-11T10:53:44.342260153Z at org.opensearch.action.support.ChannelActionListener.onResponse(ChannelActionListener.java:62) [opensearch-2.17.0.jar:2.17.0]
2024-11-11T10:53:44.342263524Z at org.opensearch.action.support.ChannelActionListener.onResponse(ChannelActionListener.java:45) [opensearch-2.17.0.jar:2.17.0]
2024-11-11T10:53:44.342267103Z at org.opensearch.core.action.ActionListener$6.onResponse(ActionListener.java:301) [opensearch-core-2.17.0.jar:2.17.0]
2024-11-11T10:53:44.342270463Z at org.opensearch.action.support.replication.TransportReplicationAction$AsyncPrimaryAction.lambda$runWithPrimaryShardReference$2(TransportReplicationAction.java:627) [opensearch-2.17.0.jar:2.17.0]
2024-11-11T10:53:44.342274157Z at org.opensearch.core.action.ActionListener$1.onResponse(ActionListener.java:82) [opensearch-core-2.17.0.jar:2.17.0]
2024-11-11T10:53:44.342278319Z at org.opensearch.core.action.ActionListener$4.onResponse(ActionListener.java:182) [opensearch-core-2.17.0.jar:2.17.0]
2024-11-11T10:53:44.342282002Z at org.opensearch.action.support.replication.ReplicationOperation.finish(ReplicationOperation.java:439) [opensearch-2.17.0.jar:2.17.0]
2024-11-11T10:53:44.342285362Z at org.opensearch.action.support.replication.ReplicationOperation.decPendingAndFinishIfNeeded(ReplicationOperation.java:425) [opensearch-2.17.0.jar:2.17.0]
2024-11-11T10:53:44.342288931Z at org.opensearch.action.support.replication.ReplicationOperation$1.onResponse(ReplicationOperation.java:194) [opensearch-2.17.0.jar:2.17.0]
2024-11-11T10:53:44.342292250Z at org.opensearch.action.support.replication.ReplicationOperation$1.onResponse(ReplicationOperation.java:186) [opensearch-2.17.0.jar:2.17.0]
2024-11-11T10:53:44.342295860Z at org.opensearch.action.support.replication.TransportWriteAction$WritePrimaryResult$1.onSuccess(TransportWriteAction.java:372) [opensearch-2.17.0.jar:2.17.0]
2024-11-11T10:53:44.342299428Z at org.opensearch.action.support.replication.TransportWriteAction$AsyncAfterWriteAction.maybeFinish(TransportWriteAction.java:518) [opensearch-2.17.0.jar:2.17.0]
2024-11-11T10:53:44.342303174Z at org.opensearch.action.support.replication.TransportWriteAction$AsyncAfterWriteAction.lambda$run$1(TransportWriteAction.java:547) [opensearch-2.17.0.jar:2.17.0]
2024-11-11T10:53:44.342306534Z at org.opensearch.common.util.concurrent.AsyncIOProcessor.notifyList(AsyncIOProcessor.java:145) [opensearch-2.17.0.jar:2.17.0]
2024-11-11T10:53:44.342315003Z at org.opensearch.common.util.concurrent.AsyncIOProcessor.drainAndProcessAndRelease(AsyncIOProcessor.java:123) [opensearch-2.17.0.jar:2.17.0]
2024-11-11T10:53:44.342318489Z at org.opensearch.common.util.concurrent.AsyncIOProcessor.put(AsyncIOProcessor.java:97) [opensearch-2.17.0.jar:2.17.0]
2024-11-11T10:53:44.342322130Z at org.opensearch.index.shard.IndexShard.sync(IndexShard.java:4521) [opensearch-2.17.0.jar:2.17.0]
2024-11-11T10:53:44.342325439Z at org.opensearch.action.support.replication.TransportWriteAction$AsyncAfterWriteAction.run(TransportWriteAction.java:545) [opensearch-2.17.0.jar:2.17.0]
2024-11-11T10:53:44.342329018Z at org.opensearch.action.support.replication.TransportWriteAction$WritePrimaryResult.runPostReplicationActions(TransportWriteAction.java:379) [opensearch-2.17.0.jar:2.17.0]
2024-11-11T10:53:44.342332690Z at org.opensearch.action.support.replication.ReplicationOperation.handlePrimaryResult(ReplicationOperation.java:186) [opensearch-2.17.0.jar:2.17.0]
2024-11-11T10:53:44.342336332Z at org.opensearch.core.action.ActionListener$1.onResponse(ActionListener.java:82) [opensearch-core-2.17.0.jar:2.17.0]
2024-11-11T10:53:44.342339682Z at org.opensearch.core.action.ActionListener.completeWith(ActionListener.java:355) [opensearch-core-2.17.0.jar:2.17.0]
2024-11-11T10:53:44.342343188Z at org.opensearch.action.bulk.TransportShardBulkAction$2.finishRequest(TransportShardBulkAction.java:523) [opensearch-2.17.0.jar:2.17.0]
2024-11-11T10:53:44.342346528Z at org.opensearch.action.bulk.TransportShardBulkAction$2.doRun(TransportShardBulkAction.java:486) [opensearch-2.17.0.jar:2.17.0]
2024-11-11T10:53:44.342350086Z at org.opensearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:52) [opensearch-2.17.0.jar:2.17.0]
2024-11-11T10:53:44.342353415Z at org.opensearch.action.bulk.TransportShardBulkAction.performOnPrimary(TransportShardBulkAction.java:535) [opensearch-2.17.0.jar:2.17.0]
2024-11-11T10:53:44.342356994Z at org.opensearch.action.bulk.TransportShardBulkAction.dispatchedShardOperationOnPrimary(TransportShardBulkAction.java:416) [opensearch-2.17.0.jar:2.17.0]
2024-11-11T10:53:44.342360355Z at org.opensearch.action.bulk.TransportShardBulkAction.dispatchedShardOperationOnPrimary(TransportShardBulkAction.java:125) [opensearch-2.17.0.jar:2.17.0]
2024-11-11T10:53:44.342363986Z at org.opensearch.action.support.replication.TransportWriteAction$1.doRun(TransportWriteAction.java:275) [opensearch-2.17.0.jar:2.17.0]
2024-11-11T10:53:44.342367336Z at org.opensearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:1005) [opensearch-2.17.0.jar:2.17.0]
2024-11-11T10:53:44.342370863Z at org.opensearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:52) [opensearch-2.17.0.jar:2.17.0]
2024-11-11T10:53:44.342374161Z at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144) [?:?]
2024-11-11T10:53:44.342377594Z at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642) [?:?]
2024-11-11T10:53:44.342380851Z at java.base/java.lang.Thread.run(Thread.java:1583) [?:?]
2024-11-11T10:53:44.342389590Z Caused by: java.net.ConnectException: Connection timed out
2024-11-11T10:53:44.342392950Z at java.base/jdk.internal.reflect.DirectConstructorHandleAccessor.newInstance(DirectConstructorHandleAccessor.java:62) ~[?:?]
2024-11-11T10:53:44.342396977Z at java.base/java.lang.reflect.Constructor.newInstanceWithCaller(Constructor.java:502) ~[?:?]
2024-11-11T10:53:44.342400306Z at java.base/java.lang.reflect.Constructor.newInstance(Constructor.java:486) ~[?:?]
2024-11-11T10:53:44.342403604Z at java.base/sun.net.www.protocol.http.HttpURLConnection$10.run(HttpURLConnection.java:2055) ~[?:?]
2024-11-11T10:53:44.342407069Z at java.base/sun.net.www.protocol.http.HttpURLConnection$10.run(HttpURLConnection.java:2050) ~[?:?]
2024-11-11T10:53:44.342410544Z at java.base/java.security.AccessController.doPrivileged(AccessController.java:571) ~[?:?]
2024-11-11T10:53:44.342413821Z at java.base/sun.net.www.protocol.http.HttpURLConnection.getChainedException(HttpURLConnection.java:2049) ~[?:?]
2024-11-11T10:53:44.342417098Z at java.base/sun.net.www.protocol.http.HttpURLConnection.getInputStream0(HttpURLConnection.java:1619) ~[?:?]
2024-11-11T10:53:44.342420396Z at java.base/sun.net.www.protocol.http.HttpURLConnection.getInputStream(HttpURLConnection.java:1599) ~[?:?]
2024-11-11T10:53:44.342423996Z at java.base/sun.net.www.protocol.https.HttpsURLConnectionImpl.getInputStream(HttpsURLConnectionImpl.java:223) ~[?:?]
2024-11-11T10:53:44.342447218Z at ai.djl.training.util.DownloadUtils.download(DownloadUtils.java:78) ~[?:?]
2024-11-11T10:53:44.342474986Z at ai.djl.training.util.DownloadUtils.download(DownloadUtils.java:52) ~[?:?]
2024-11-11T10:53:44.342481531Z at org.opensearch.ml.engine.ModelHelper.lambda$downloadPrebuiltModelMetaList$2(ModelHelper.java:231) ~[?:?]
2024-11-11T10:53:44.342485339Z at java.base/java.security.AccessController.doPrivileged(AccessController.java:571) ~[?:?]
2024-11-11T10:53:44.342488980Z ... 72 more
2024-11-11T10:53:44.342492247Z Caused by: java.net.ConnectException: Connection timed out
2024-11-11T10:53:44.342495462Z at java.base/sun.nio.ch.Net.connect0(Native Method) ~[?:?]
2024-11-11T10:53:44.342498791Z at java.base/sun.nio.ch.Net.connect(Net.java:589) ~[?:?]
2024-11-11T10:53:44.342502287Z at java.base/sun.nio.ch.Net.connect(Net.java:578) ~[?:?]
2024-11-11T10:53:44.342505512Z at java.base/sun.nio.ch.NioSocketImpl.connect(NioSocketImpl.java:583) ~[?:?]
2024-11-11T10:53:44.342508966Z at java.base/java.net.SocksSocketImpl.connect(SocksSocketImpl.java:327) ~[?:?]
2024-11-11T10:53:44.342512181Z at java.base/java.net.Socket.connect(Socket.java:751) ~[?:?]
2024-11-11T10:53:44.342515385Z at java.base/sun.security.ssl.SSLSocketImpl.connect(SSLSocketImpl.java:304) ~[?:?]
2024-11-11T10:53:44.342518590Z at java.base/sun.security.ssl.BaseSSLSocketImpl.connect(BaseSSLSocketImpl.java:181) ~[?:?]
2024-11-11T10:53:44.342521846Z at java.base/sun.net.NetworkClient.doConnect(NetworkClient.java:183) ~[?:?]
2024-11-11T10:53:44.342534050Z at java.base/sun.net.www.http.HttpClient.openServer(HttpClient.java:531) ~[?:?]
2024-11-11T10:53:44.342537733Z at java.base/sun.net.www.http.HttpClient.openServer(HttpClient.java:636) ~[?:?]
2024-11-11T10:53:44.342541240Z at java.base/sun.net.www.protocol.https.HttpsClient.<init>(HttpsClient.java:264) ~[?:?]
2024-11-11T10:53:44.342544850Z at java.base/sun.net.www.protocol.https.HttpsClient.New(HttpsClient.java:377) ~[?:?]
2024-11-11T10:53:44.342812796Z at java.base/sun.net.www.protocol.https.AbstractDelegateHttpsURLConnection.getNewHttpClient(AbstractDelegateHttpsURLConnection.java:193) ~[?:?]
2024-11-11T10:53:44.342821858Z at java.base/sun.net.www.protocol.http.HttpURLConnection.plainConnect0(HttpURLConnection.java:1237) ~[?:?]
2024-11-11T10:53:44.342827414Z at java.base/sun.net.www.protocol.http.HttpURLConnection.plainConnect(HttpURLConnection.java:1123) ~[?:?]
2024-11-11T10:53:44.342832502Z at java.base/sun.net.www.protocol.https.AbstractDelegateHttpsURLConnection.connect(AbstractDelegateHttpsURLConnection.java:179) ~[?:?]
2024-11-11T10:53:44.342838193Z at java.base/sun.net.www.protocol.http.HttpURLConnection.getInputStream0(HttpURLConnection.java:1675) ~[?:?]
2024-11-11T10:53:44.342843561Z at java.base/sun.net.www.protocol.http.HttpURLConnection.getInputStream(HttpURLConnection.java:1599) ~[?:?]
2024-11-11T10:53:44.342848930Z at java.base/sun.net.www.protocol.http.HttpURLConnection.getHeaderField(HttpURLConnection.java:3235) ~[?:?]
2024-11-11T10:53:44.342854132Z at java.base/java.net.URLConnection.getHeaderFieldLong(URLConnection.java:649) ~[?:?]
2024-11-11T10:53:44.342858106Z at java.base/java.net.URLConnection.getContentLengthLong(URLConnection.java:511) ~[?:?]
2024-11-11T10:53:44.342861425Z at java.base/sun.net.www.protocol.https.HttpsURLConnectionImpl.getContentLengthLong(HttpsURLConnectionImpl.java:371) ~[?:?]
2024-11-11T10:53:44.342864744Z at ai.djl.training.util.DownloadUtils.download(DownloadUtils.java:73) ~[?:?]
2024-11-11T10:53:44.342867979Z at ai.djl.training.util.DownloadUtils.download(DownloadUtils.java:52) ~[?:?]
2024-11-11T10:53:44.342871236Z at org.opensearch.ml.engine.ModelHelper.lambda$downloadPrebuiltModelMetaList$2(ModelHelper.java:231) ~[?:?]
2024-11-11T10:53:44.342874513Z at java.base/java.security.AccessController.doPrivileged(AccessController.java:571) ~[?:?]
2024-11-11T10:53:44.342877780Z ... 72 more