全部產品
Search
文件中心

:離線cluster配置

更新時間:Aug 30, 2024

cluster 配置中同時包含線上部分 (ha3) 配置和離線 (build_service) 兩部分配置;離線配置中包含了對builder 和merger 的各種配置。

customized_merge_config

customized_merge_config:使用者自訂配置。使用者可以對不同情境定製不同的merge策略。比如增量階段使用和預設策略不同的merge策略;可以配置多個,多個merge策略會同時生效,相同時間如果觸發了多個merge策略會按照配置中的順序先後觸發merge plan。 使用者除了指定merge_config,還可以配置下面兩個參數:

  • period: 索引的merge間隔時間,單位為s。或配置在某個固定時刻merge,比如在下午13pm觸發merge,格式是"period": "daytime=13:00"

  • need_wait_alter_field:如果當前有動態加欄位流程,是否要阻塞merge,預設是true。如果配成false,當前merge 策略將自動換成align_version merge策略生效。

  • merge_parallel_num: 每個partition參與merge的並發進程數量。

由於新增動態加欄位功能。禁止使用者自訂的merge策略命名為alter_field。

customized_merge_config預設值如下:

{
    "full":{
        "merge_config":{
            "keep_version_count":40,
            "merge_strategy":"optimize",
            "merge_strategy_param":"after-merge-max-segment-count=20",
            "merge_thread_count":4
        }
    },
    "large_segment_reclaim":{
        "merge_config":{
            "keep_version_count":40,
            "merge_strategy":"priority_queue",
            "merge_strategy_params":{
                "input_limits":"max-segment-size=20480",
                "output_limits":"max-merged-segment-size=13312;max-total-merged-size=15360",
                "strategy_conditions":"priority-feature=delete-doc-count#desc;conflict-segment-count=2;conflict-delete-percent=8"
            },
            "merge_thread_count":4
        },
        "period":"period=1800"
    },
    "segment_merge":{
        "merge_config":{
            "keep_version_count":40,
            "merge_strategy":"priority_queue",
            "merge_strategy_params":{
                "input_limits":"max-segment-size=12288",
                "output_limits":"max-merged-segment-size=13312;max-total-merged-size=15360",
                "strategy_conditions":"priority-feature=valid-doc-count#asc;conflict-segment-count=2;conflict-delete-percent=10"
            },
            "merge_thread_count":4
        },
        "period":"period=900"
    },
    "large_segment_merge":{
        "merge_config":{
            "keep_version_count":40,
            "merge_strategy":"priority_queue",
            "merge_strategy_params":{
                "input_limits":"max-segment-size=12288",
                "output_limits":"max-merged-segment-size=15360;max-total-merged-size=46080",
                "strategy_conditions":"priority-feature=valid-doc-count#desc;conflict-segment-count=2;conflict-delete-percent=10"
            },
            "merge_thread_count":4
        },
        "period":"period=3600"
    },
    "small_segment_merge":{
        "merge_config":{
            "keep_version_count":40,
            "merge_strategy":"priority_queue",
            "merge_strategy_params":{
                "input_limits":"max-segment-size=1536",
                "output_limits":"max-merged-segment-size=1536;max-total-merged-size=3072",
                "strategy_conditions":"priority-feature=valid-doc-count#asc;conflict-segment-count=2;conflict-delete-percent=20"
            },
            "merge_thread_count":4
        }
    }
}

segment_customize_metrics_updater

segment_customize_metrics_updater:用於產生各個segment的segment metric,可由使用者自訂產生。indexlib內建存在lifecycle updater,配置該updater後,每個segment會組建組態欄位的min,max值,並根據定義的lifecycle_param配置的打標方式,為每個sgement打標(merge階段,build階段按照預設tag打標)。打標可用於load策略和time_series_merge策略。預設值:[]。