high_disk_watermark_exceeded: high disk watermark [90%] exceeded / flood stage disk watermark [95%] exceeded(超过磁盘 high 水位线 [90%] / flood stage 水位线 [95%])
Elasticsearch含义
某个节点的磁盘使用超过了 high 水位线(默认 90%)或 flood stage(默认 95%)阈值。超过 high 水位线时,Elasticsearch 会尝试将分片从该节点移到其他节点;超过 flood stage 时,会将索引设为只读。
原因
磁盘正在被写满:持续增长的日志/指标索引、从未删除的旧索引、未清理的快照仓库、磁盘过小,或分片分布不均衡导致单个节点过满。
解决方法
检查磁盘使用:GET _cat/allocation?v。删除旧索引或生成快照后移除,并扩容磁盘。可以临时提高阈值(谨慎操作;在 8.x 中由于 max_headroom 校验,需要同时设置全部三项):PUT _cluster/settings {"persistent":{"cluster.routing.allocation.disk.watermark.low":"88%","cluster.routing.allocation.disk.watermark.high":"93%","cluster.routing.allocation.disk.watermark.flood_stage":"96%"}}。问题解决后,将相同设置设为 null 以恢复默认值(transient 设置已弃用;不要使用)。永久解决办法:用 ILM 进行生命周期管理、定期清理索引并做好容量规划。如果索引因 flood stage 变为只读,需要单独移除该块。
同一数据库的其他条目
- cluster_block_exceptionClusterBlockException[blocked by: [TOO_MANY_REQUESTS/12/disk usage exceeded flood-stage watermark, index has read-only-allow-delete block]] (before 7.x: FORBIDDEN/12/index read-only / allow delete (api))(磁盘使用超过 flood-stage 水位线,索引被设为 read-only-allow-delete)
- index_read_only_allow_deleteTOO_MANY_REQUESTS/12/disk usage exceeded flood-stage watermark, index has read-only-allow-delete block (legacy: FORBIDDEN/12/index read-only / allow delete (api))(磁盘使用超过 flood-stage 水位线,索引带有 read-only-allow-delete 块)
- circuit_breaking_exceptionCircuitBreakingException[[parent] Data too large, data for [<...>] would be [...], which is larger than the limit of [...]](数据过大,超过熔断器限制)
- mapper_parsing_exceptionmapper_parsing_exception: failed to parse field [...] of type [...](无法解析类型为 [...] 的字段 [...])
- strict_dynamic_mapping_exceptionstrict_dynamic_mapping_exception: mapping set to strict, dynamic introduction of [...] within [...] is not allowed(mapping 设为 strict,不允许动态添加字段)
- illegal_argument_exception_fielddataillegal_argument_exception: Fielddata is disabled on text fields by default. Set fielddata=true on [your_field] in order to load fielddata in memory(text 字段默认禁用 fielddata)