Skip to main content
版本:1.9.0

安装部署

准备安装文件

安装文件在inlong-sort-standalone/sort-standalone-dist/target/目录下,文件名是apache-inlong-sort-standalone-${project.version}-bin.tar.gz。

启动inlong-sort-standalone应用

有了上述编译阶段产出的tar.gz包后,解压后就可以启动inlong-sort-standalone的应用了。
示例:

./bin/sort-start.sh

conf/common.properties配置

配置名是否必须默认值描述
clusterIdYNA用来唯一标识一个inlong-sort-standalone集群
sortSource.typeNorg.apache.inlong.sort.standalone.source.readapi.ReadApiSourceSource类名
sortChannel.typeNorg.apache.inlong.sort.standalone.channel.BufferQueueChannelChannel类型
sortSink.typeNorg.apache.inlong.sort.standalone.sink.hive.HiveSinkSink类名,不同的分发类型使用不同的Sink类
sortClusterConfig.typeNorg.apache.inlong.sort.standalone.config.loader.ClassResourceSortClusterConfigLoader分发集群配置加载类名,ClassResourceSortClusterConfigLoader从ClassPath的SortClusterConfig.conf源文件读取分发集群配置
sortClusterConfig.managerPathNNA分发集群配置加载类org.apache.inlong.sort.standalone.config.loader.ManagerSortClusterConfigLoader的参数,指定Inlong Manager的URL路径, 如http://${manager ip:port}/api/inlong/manager/openapi/sort/standalone/getClusterConfig
eventFormatHandlerNorg.apache.inlong.sort.standalone.sink.hive.DefaultEventFormatHandler分发Hive前的格式转换类名
maxThreadsN10sink的并行度
reloadIntervalN60000分发集群配置的更新加载周期,单位毫秒
processIntervalN100分发分组处理间隔,单位毫秒
metricDomainsNSort指标汇总域名
metricDomains.Sort.domainListenersNorg.apache.inlong.sort.standalone.metrics.prometheus.PrometheusMetricListener指标汇总监听器类名列表,空格分隔
prometheusHttpPortN8080org.apache.inlong.sort.standalone.metrics.prometheus.PrometheusMetricListener的参数,Prometheus的HttpServer端口
metricDomains.Sort.snapshotIntervalN60000订阅tube的重试超时时间,单位为ms

SortClusterConfig配置

  • 可以从ClassPath的SortClusterConfig.conf源文件读取,但不支持实时更新
  • 可以从Inlong Manager的HTTP接口获取配置
    | 配置名 | 是否必须 | 默认值 |描述 | | ------------ | ------------ | ------------ | ------------ | |clusterName | Y | NA | 用来唯一标识一个inlong-sort-standalone集群 | |sortTasks | Y | NA | 分发任务列表 |

SortTaskConfig配置

配置名是否必须默认值描述
nameYNA分发任务名
typeYNA分发任务类型,如HIVE("hive"), TUBE("tube"), KAFKA("kafka"), PULSAR("pulsar"), ElasticSearch("ElasticSearch"), UNKNOWN("n")
idParamsYNAInlong数据流参数列表
sinkParamsYNA分发任务的参数

Hive分发任务的idParams

配置名是否必须默认值描述
inlongGroupIdYNAinlongGroupId
inlongStreamIdYNAinlongStreamId
separatorYNA分隔符
partitionIntervalMsN3600000分区间隔时间,单位毫秒
idRootPathYNAInlong数据流的Hdfs根目录
partitionSubPathYNAInlong数据流的分区子目录
hiveTableNameYNAInlong数据流的Hive表名
partitionFieldNameNdtInlong数据流的分区字段名
partitionFieldPatternYNAInlong数据流的分区字段值格式,如{yyyyMMdd}、{yyyyMMddHH}、{yyyyMMddHHmm}
msgTimeFieldPatternYNA消息生成时间的字段值格式,Java时间格式
maxPartitionOpenDelayHourN8分区最大打开延迟时间,单位小时

Hive分发任务的sinkParams

配置名是否必须默认值描述
hdfsPathYNAHDFS的NameNode
maxFileOpenDelayMinuteN5单个HDFS文件最大写入时间,单位分钟
tokenOvertimeMinuteN60单个Inlong数据流的分区创建token最大占用时间,单位分钟
maxOutputFileSizeGbN2单个HDFS文件最大大小,单位GB
hiveJdbcUrlYNAHive的JDBC路径
hiveDatabaseYNAHive的数据库
hiveUsernameYNAHive的用户名
hivePasswordYNAHive的密码

Pulsar分发任务的idParams

配置名是否必须默认值描述
inlongGroupIdYNAinlongGroupId
inlongStreamIdYNAinlongStreamId
topicYNAPulsar的Topic

Pulsar分发任务的sinkParams

配置名是否必须默认值描述
serviceUrlYNAPulsar服务路径
authenticationYNAPulsar集群鉴权
enableBatchingNtrueenableBatching
batchingMaxBytesN5242880batchingMaxBytes
batchingMaxMessagesN3000batchingMaxMessages
batchingMaxPublishDelayN1batchingMaxPublishDelay
maxPendingMessagesN1000maxPendingMessages
maxPendingMessagesAcrossPartitionsN50000maxPendingMessagesAcrossPartitions
sendTimeoutN0sendTimeout
compressionTypeNNONEcompressionType
blockIfQueueFullNtrueblockIfQueueFull
roundRobinRouterBatchingPartitionSwitchFrequencyN10roundRobinRouterBatchingPartitionSwitchFrequency

Hive配置样例

{
"data":{
"clusterName":"hivev3-sz-sz1",
"sortTasks":[
{
"idParams":[
{
"inlongGroupId":"0fc00000046",
"inlongStreamId":"",
"separator":"|",
"partitionIntervalMs":3600000,
"idRootPath":"/user/hive/warehouse/t_inlong_v1_0fc00000046",
"partitionSubPath":"/{yyyyMMdd}/{yyyyMMddHH}",
"hiveTableName":"t_inlong_v1_0fc00000046",
"partitionFieldName":"dt",
"partitionFieldPattern":"yyyyMMddHH",
"msgTimeFieldPattern":"yyyy-MM-dd HH:mm:ss",
"maxPartitionOpenDelayHour":8
},
{
"inlongGroupId":"03600000045",
"inlongStreamId":"",
"separator":"|",
"partitionIntervalMs":3600000,
"idRootPath":"/user/hive/warehouse/t_inlong_v1_03600000045",
"partitionSubPath":"/{yyyyMMdd}/{yyyyMMddHH}",
"hiveTableName":"t_inlong_v1_03600000045",
"partitionFieldName":"dt",
"partitionFieldPattern":"yyyyMMddHH",
"msgTimeFieldPattern":"yyyy-MM-dd HH:mm:ss",
"maxPartitionOpenDelayHour":8
},
{
"inlongGroupId":"05100054990",
"inlongStreamId":"",
"separator":"|",
"partitionIntervalMs":3600000,
"idRootPath":"/user/hive/warehouse/t_inlong_v1_05100054990",
"partitionSubPath":"/{yyyyMMdd}/{yyyyMMddHH}",
"hiveTableName":"t_inlong_v1_05100054990",
"partitionFieldName":"dt",
"partitionFieldPattern":"yyyyMMddHH",
"msgTimeFieldPattern":"yyyy-MM-dd HH:mm:ss",
"maxPartitionOpenDelayHour":8
},
{
"inlongGroupId":"09c00014434",
"inlongStreamId":"",
"separator":"|",
"partitionIntervalMs":3600000,
"idRootPath":"/user/hive/warehouse/t_inlong_v1_09c00014434",
"partitionSubPath":"/{yyyyMMdd}/{yyyyMMddHH}",
"hiveTableName":"t_inlong_v1_09c00014434",
"partitionFieldName":"dt",
"partitionFieldPattern":"yyyyMMddHH",
"msgTimeFieldPattern":"yyyy-MM-dd HH:mm:ss",
"maxPartitionOpenDelayHour":8
},
{
"inlongGroupId":"0c900035509",
"inlongStreamId":"",
"separator":"|",
"partitionIntervalMs":3600000,
"idRootPath":"/user/hive/warehouse/t_inlong_v1_0c900035509",
"partitionSubPath":"/{yyyyMMdd}/{yyyyMMddHH}",
"hiveTableName":"t_inlong_v1_0c900035509",
"partitionFieldName":"dt",
"partitionFieldPattern":"yyyyMMddHH",
"msgTimeFieldPattern":"yyyy-MM-dd HH:mm:ss",
"maxPartitionOpenDelayHour":8
}
],
"name":"sid_hive_inlong6th_v3",
"sinkParams":{
"hdfsPath":"hdfs://127.0.0.1:9000",
"maxFileOpenDelayMinute":"5",
"tokenOvertimeMinute":"60",
"maxOutputFileSizeGb":"2",
"hiveJdbcUrl":"jdbc:hive2://127.0.0.2:10000",
"hiveDatabase":"default",
"hiveUsername":"hive",
"hivePassword":"hive"
},
"type":"HIVE"
}
]
},
"errCode":0,
"md5":"md5",
"result":true
}