为现有集群增加白屏监控 - 安装部署工具 OBD V3.4.0

obd 自 V1.6.0 开始支持 prometheus 和 grafana 组件的部署，若您想为之前部署的集群新增白屏监控，可参考本文进行操作。

本文分为三个场景进行介绍，您可根据根据自身集群情况选择合适的场景参考。

说明

本文所提供的配置示例仅供参考，详细配置及配置说明请切换到 /usr/obd/example 目录下查看对应组件的示例。

场景一：现有集群未部署 OBAgent

当您现有集群未部署 OBAgent 时，若要增加白屏监控，需新建一个包含 obagent、prometheus 和 grafana 组件的集群。

其中，OBAgent 通过单独的配置指定用于采集 OceanBase 数据库的监控信息；在配置文件中声明 Prometheus 依赖于 obagent 组件，Grafana 依赖于 prometheus 组件。

具体配置文件示例如下：

# user:
#   username: your username
#   password: your password if need
#   key_file: your ssh-key file path if need
#   port: your ssh port, default 22
#   timeout: ssh connection timeout (second), default 30
obagent:
  servers:
    # Please don't use hostname, only IP can be supported
    - 10.10.10.1
    - 10.10.10.2
    - 10.10.10.3
  global:
    # The working directory for obagent. obagent is started under this directory. This is a required field.
    home_path: /home/admin/obagent
    # The port of monitor agent. The default port number is 8088.
    monagent_http_port: 8088
    # The port of manager agent. The default port number is 8089.
    mgragent_http_port: 8089
    # Log path. The default value is log/monagent.log.
    log_path: log/monagent.log
    # The log level of manager agent.
    mgragent_log_level: info
    # The total size of manager agent.Log size is measured in Megabytes. The default value is 30M.
    mgragent_log_max_size: 30
    # Expiration time for manager agent logs. The default value is 30 days.
    mgragent_log_max_days: 30
    # The maximum number for manager agent log files. The default value is 15.
    mgragent_log_max_backups: 15
    # The log level of monitor agent.
    monagent_log_level: info
    # The total size of monitor agent.Log size is measured in Megabytes. The default value is 200M.
    monagent_log_max_size: 200
    # Expiration time for monitor agent logs. The default value is 30 days.
    monagent_log_max_days: 30
    # The maximum number for monitor agent log files. The default value is 15.
    monagent_log_max_backups: 15
    # Username for HTTP authentication. The default value is admin.
    http_basic_auth_user: admin
    # Password for HTTP authentication. The default value is a random password.
    # http_basic_auth_password: ******
    # Monitor password for OceanBase Database. The default value is empty. When a depends exists, obd gets this value from the oceanbase-ce of the depends. The value is the same as the ocp_agent_monitor_password in oceanbase-ce.
    monitor_password: ******
    # The SQL port for observer. The default value is 2881. When a depends exists, obd gets this value from the oceanbase-ce of the depends. The value is the same as the mysql_port in oceanbase-ce.
    sql_port: 2881
    # The RPC port for observer. The default value is 2882. When a depends exists, obd gets this value from the oceanbase-ce of the depends. The value is the same as the rpc_port in oceanbase-ce.
    rpc_port: 2882
    # Cluster name for OceanBase Database. When a depends exists, obd gets this value from the oceanbase-ce of the depends. The value is the same as the appname in oceanbase-ce.
    cluster_name: obcluster
    # Cluster ID for OceanBase Database. When a depends exists, obd gets this value from the oceanbase-ce of the depends. The value is the same as the cluster_id in oceanbase-ce.
    cluster_id: 1
    # The redo dir for Oceanbase Database. When a depends exists, obd gets this value from the oceanbase-ce of the depends. The value is the same as the redo_dir in oceanbase-ce.
    ob_log_path: /home/admin/observer/store
    # The data dir for Oceanbase Database. When a depends exists, obd gets this value from the oceanbase-ce of the depends. The value is the same as the data_dir in oceanbase-ce.
    ob_data_path: /home/admin/observer/store
    # The work directory for Oceanbase Database. When a depends exists, obd gets this value from the oceanbase-ce of the depends. The value is the same as the home_path in oceanbase-ce.
    ob_install_path: /home/admin/observer
    # The log path for Oceanbase Database. When a depends exists, obd gets this value from the oceanbase-ce of the depends. The value is the same as the {home_path}/log in oceanbase-ce.
    observer_log_path: /home/admin/observer/log
    # Monitor status for OceanBase Database.  Active is to enable. Inactive is to disable. The default value is active. When you deploy an cluster automatically, obd decides whether to enable this parameter based on depends.
    ob_monitor_status: active
  10.10.10.1:
    # Zone name for your observer. The default value is zone1. When a depends exists, obd gets this value from the oceanbase-ce of the depends. The value is the same as the zone name in oceanbase-ce.
    zone_name: zone1
  10.10.10.2:
    # Zone name for your observer. The default value is zone1. When a depends exists, obd gets this value from the oceanbase-ce of the depends. The value is the same as the zone name in oceanbase-ce.
    zone_name: zone2
  10.10.10.3:
    # Zone name for your observer. The default value is zone1. When a depends exists, obd gets this value from the oceanbase-ce of the depends. The value is the same as the zone name in oceanbase-ce.
    zone_name: zone3
prometheus:
  depends:
    - obagent
  servers:
    - 10.10.10.4
  global:
    home_path: /home/admin/prometheus
grafana:
  depends:
    - prometheus
  servers:
    - 10.10.10.4
  global:
    home_path: /home/admin/grafana
    login_password: *********

配置文件中各配置项含义可参见配置文件说明。修改配置文件后，执行如下命令部署并启动新集群：

obd cluster deploy <new deploy name> -c new_config.yaml
obd cluster start <new deploy name>

集群启动后，根据展示信息访问 Grafana 页面，即可查看现有集群的监控信息。

场景二：现有集群已部署 OBAgent

当您现有集群已部署 OBAgent 时，若要增加白屏监控，需新建一个包含 prometheus 和 grafana 组件的集群。

此时，由于无法声明 depends obagent，所以需要手动进行关联。查看原有集群中 OBAgent 安装目录下 conf/prometheus_config/prometheus.yaml 文件，将对应的配置复制到新集群配置中 prometheus 组件的 global -> conifg 下。具体配置示例如下：

# user:
#   username: your username
#   password: your password if need
#   key_file: your ssh-key file path if need
#   port: your ssh port, default 22
#   timeout: ssh connection timeout (second), default 30
prometheus:
  servers:
    - 10.10.10.4
  global:
    # The working directory for prometheus. prometheus is started under this directory. This is a required field.
    home_path: /home/admin/prometheus
    config: # Configuration of the Prometheus service. The format is consistent with the Prometheus config file. Corresponds to the `config.file` parameter.
      global:
        scrape_interval: 1s
        evaluation_interval: 10s

      rule_files:
        - "rules/*rules.yaml"

      scrape_configs:
        - job_name: prometheus
          metrics_path: /metrics
          scheme: http
          static_configs:
            - targets:
                - 'localhost:9090'
        - job_name: node
          basic_auth:
            username: ******
            password: ******
          metrics_path: /metrics/node/host
          scheme: http
          static_configs:
            - targets:
                - 10.10.10.1:8088
        - job_name: ob_basic
          basic_auth:
            username: ******
            password: ******
          metrics_path: /metrics/ob/basic
          scheme: http
          static_configs:
            - targets:
                - 10.10.10.1:8088
        - job_name: ob_extra
          basic_auth:
            username: ******
            password: ******
          metrics_path: /metrics/ob/extra
          scheme: http
          static_configs:
            - targets:
                - 10.10.10.1:8088
        - job_name: agent
          basic_auth:
            username: ******
            password: ******
          metrics_path: /metrics/stat
          scheme: http
          static_configs:
            - targets:
                - 10.10.10.1:8088
grafana:
  servers:
    - 10.10.10.4
  depends:
    - prometheus
  global:
    home_path: /home/admin/grafana
    login_password: ********* # Grafana login password. The default value is 'oceanbase'.

配置文件中各配置项含义可参见配置文件说明。上文配置文件示例中 basic_auth 配置项的用户名和密码需和 OBAgent 配置文件中 http_basic_auth_xxx 配置项相对应。

修改配置文件后，执行如下命令部署新集群：

obd cluster deploy <new deploy name> -c new_config.yaml

部署完成后，将 OBAgent 安装目录下 conf/prometheus_config/rules 目录复制到 Prometheus 的安装目录下。

执行如下命令启动新集群：

obd cluster start <new deploy name>

集群启动后，根据展示信息访问 Grafana 页面，即可查看现有集群的监控信息。

注意

scrape_configs 中的 Prometheus 监控项中 'localhost:9090' 需要根据当前 Prometheus 的监听地址进行修改，如果当前 Prometheus 开启了认证，也需要对应的配置 basic_auth。这里提到的监听地址为部署 Prometheus 的地址，即 Prometheus 配置中的 address 和 port 配置项。
如果原有集群的 obagent 节点有所变化，需通过 obd cluster edit-config 同步 OBAgent 安装目录下 conf/prometheus_config/prometheus.yaml 的内容。

场景三：监控多个集群并动态同步 OBAgent 变化

如若希望 Prometheus 可以采集多个集群的监控信息，或者动态同步 OBAgent 的变化，可在场景二的基础上做少许改动。

将 Prometheus 配置中的 static_configs 替换为 file_sd_config 来获取和同步 OBAgent 的节点信息。如下示例中表示收集 Prometheus 安装目录（home_path）下 targets 目录下的所有 yaml 文件。

说明

Prometheus 安装目录下的 targets 目录需在原有集群配置文件中 obagent 组件下进行相关配置才会被创建。具体配置参见修改被监控集群配置。

# user:
#   username: your username
#   password: your password if need
#   key_file: your ssh-key file path if need
#   port: your ssh port, default 22
#   timeout: ssh connection timeout (second), default 30
prometheus:
  servers:
    - 10.10.10.4
  global:
    # The working directory for prometheus. prometheus is started under this directory. This is a required field.
    home_path: /home/admin/prometheus
    config: # Configuration of the Prometheus service. The format is consistent with the Prometheus config file. Corresponds to the `config.file` parameter.
      global:
        scrape_interval: 1s
        evaluation_interval: 10s

      rule_files:
        - "rules/*rules.yaml"

      scrape_configs:
        - job_name: prometheus
          metrics_path: /metrics
          scheme: http
          static_configs:
            - targets:
                - 'localhost:9090'
        - job_name: node
          basic_auth:
            username: ******
            password: ******
          metrics_path: /metrics/node/host
          scheme: http
          file_sd_configs:
            - files:
              - targets/*.yaml
        - job_name: ob_basic
          basic_auth:
            username: ******
            password: ******
          metrics_path: /metrics/ob/basic
          scheme: http
          file_sd_configs:
            - files:
              - targets/*.yaml
        - job_name: ob_extra
          basic_auth:
            username: ******
            password: ******
          metrics_path: /metrics/ob/extra
          scheme: http
          file_sd_configs:
            - files:
              - targets/*.yaml
        - job_name: agent
          basic_auth:
            username: ******
            password: ******
          metrics_path: /metrics/stat
          scheme: http
          file_sd_configs:
            - files:
              - targets/*.yaml
grafana:
  servers:
    - 10.10.10.4
  depends:
    - prometheus
  global:
    home_path: /home/admin/grafana
    login_password: ********* # Grafana login password. The default value is 'oceanbase'.

修改配置文件后，执行如下命令部署新集群：

obd cluster deploy <new deploy name> -c new_config.yaml

部署完成后，将 OBAgent 安装目录下的 conf/prometheus_config/rules 目录复制到 Prometheus 的安装目录下。

执行如下命令启动新集群：

obd cluster start <new deploy name>

部署好新集群后，根据展示信息访问 Grafana 页面，此时无法查看被监控集群的监控信息，还需修改被监控集群的 obagent 配置。

修改被监控集群配置

若要在 Prometheus 安装目录下创建 targets 目录，需执行 obd cluster edit-config <deploy name> 命令修改配置文件，在配置文件中增加配置项 target_sync_configs 指向 Prometheus 安装目录下的 targets 目录（默认会使用当前集群的 user 配置，如果有差别可以按照示例进行设置）。

obagent:
  servers:
    # Please don't use hostname, only IP can be supported
    - 10.10.10.1
    - 10.10.10.2
    - 10.10.10.3
  global:
    ....
    target_sync_configs:
      - host: 10.10.10.4
        target_dir: /home/admin/prometheus/targets
    #    username: your username
    #    password: your password if need
    #    key_file: your ssh-key file path if need
    #    port: your ssh port, default 22
    #    timeout: ssh connection timeout (second), default 30
    ...

修改配置文件后需根据提示重启集群，重启后访问 Grafana 页面，即可查看现有集群的监控信息。

注意

scrape_configs 中的 Prometheus 监控项中 'localhost:9090' 需要根据当前 Prometheus 的监听地址进行修改，如果当前 Prometheus 开启了认证，也需要对应的配置 basic_auth。这里提到的监听地址为部署 Prometheus 的地址，即 Prometheus 配置中的 address 和 port 配置项。
被 Prometheus 采集的 OBAgent 的 http 用户名密码需要保持一致，如果不一致，请拆分采集项。