[DEBUG]why a lot of logs about retention leases

General Feedback
#1

hi everyone!!
Recently, my opendistro is very unstable, all indexes had retention leases error.Would you know what’s happen and the detailed DEBUG ERROR is as follows:

[2021-03-25T00:00:05,117][DEBUG][o.e.i.s.ReplicationTracker] [aaa01] [.opendistro-ism-managed-index-history-2021.03.17-000166][0] no retention leases are expired from current retention leases [RetentionLeases{primaryTerm=6, version=64,
leases={peer_recovery/48sYCzT3TfGdPKMAtDQRjg=RetentionLease{id=‘peer_recovery/48sYCzT3TfGdPKMAtDQRjg’, retainingSequenceNumber=0, timestamp=1616590954938, source=‘peer recovery’}, peer_recovery/bYfsEWRwRPWg-AOLhFsUHg=RetentionLease{id=‘peer_recovery/bYfsEWRwRPWg-AOLhFsUHg’, retainingSequenceNumber=0, timestamp=1616590954938, source=‘peer recovery’}}}]
[2021-03-25T00:00:07,504][DEBUG][o.e.i.s.ReplicationTracker] [aaa01] [.opendistro-ism-managed-index-history-2021.03.11-000160][0] no retention leases are expired from current retention leases [RetentionLeases{primaryTerm=10, version=110,
leases={peer_recovery/bYfsEWRwRPWg-AOLhFsUHg=RetentionLease{id=‘peer_recovery/bYfsEWRwRPWg-AOLhFsUHg’, retainingSequenceNumber=0, timestamp=1616587627280, source=‘peer recovery’}}}]
[2021-03-25T00:00:10,311][DEBUG][o.e.i.s.ReplicationTracker] [aaa01] [.opendistro-alerting-alerts][0] no retention leases are expired from current retention leases [RetentionLeases{primaryTerm=19, version=636,
leases={peer_recovery/z3EzFRpYRFODJawOEsTeJw=RetentionLease{id=‘peer_recovery/z3EzFRpYRFODJawOEsTeJw’, retainingSequenceNumber=107, timestamp=1616598280071, source=‘peer recovery’}, peer_recovery/bYfsEWRwRPWg-AOLhFsUHg=RetentionLease{id=‘peer_recovery/bYfsEWRwRPWg-AOLhFsUHg’, retainingSequenceNumber=107, timestamp=1616598280071, source=‘peer recovery’}}}]
[2021-03-25T00:00:11,182][DEBUG][o.e.i.s.ReplicationTracker] [aaa01] [index01][1] no retention leases are expired from current retention leases [RetentionLeases{primaryTerm=21, version=5208,
leases={peer_recovery/48sYCzT3TfGdPKMAtDQRjg=RetentionLease{id=‘peer_recovery/48sYCzT3TfGdPKMAtDQRjg’, retainingSequenceNumber=41071641, timestamp=1616588140798, source=‘peer recovery’}, peer_recovery/bYfsEWRwRPWg-AOLhFsUHg=RetentionLease{id=‘peer_recovery/bYfsEWRwRPWg-AOLhFsUHg’, retainingSequenceNumber=41071641, timestamp=1616588140798, source=‘peer recovery’}}}]
[2021-03-25T00:00:11,287][DEBUG][o.e.i.s.ReplicationTracker] [aaa01] [index02][0] no retention leases are expired from current retention leases [RetentionLeases{primaryTerm=47, version=1360,
leases={peer_recovery/48sYCzT3TfGdPKMAtDQRjg=RetentionLease{id=‘peer_recovery/48sYCzT3TfGdPKMAtDQRjg’, retainingSequenceNumber=50891, timestamp=1616591291142, source=‘peer recovery’}, peer_recovery/bYfsEWRwRPWg-AOLhFsUHg=RetentionLease{id=‘peer_recovery/bYfsEWRwRPWg-AOLhFsUHg’, retainingSequenceNumber=50891, timestamp=1616591291142, source=‘peer recovery’}}}]
[2021-03-25T00:00:11,397][DEBUG][o.e.i.s.ReplicationTracker] [aaa01] [index03][0] no retention leases are expired from current retention leases [RetentionLeases{primaryTerm=6, version=1037,
leases={peer_recovery/z3EzFRpYRFODJawOEsTeJw=RetentionLease{id=‘peer_recovery/z3EzFRpYRFODJawOEsTeJw’, retainingSequenceNumber=860529, timestamp=1616587631082, source=‘peer recovery’}, peer_recovery/bYfsEWRwRPWg-AOLhFsUHg=RetentionLease{id=‘peer_recovery/bYfsEWRwRPWg-AOLhFsUHg’, retainingSequenceNumber=860529, timestamp=1616587631082, source=‘peer recovery’}}}]

Please help me this solution ! thanks so much.

#2

@Cindy What version are you using?

#3

Hi~ My version is elasticsearch 7.10.2, opendistroforelasticsearch-1.13.1.
Thanks a lot.

#4

I’m not an expert on this particular aspect of Elasticsearch (perhaps others can chime in), but it seems like this has replication and loss of communication with a shard. The ‘peer recovery’ part is the giveaway. Seems to be lots of information floating around about that. Hope it gives you a place to start looking.

#5

@searchymcsearchface Thanks to you！