通过 OBD 部署 Config Server

部署 Config Server 时，您可以选择单独部署，也可以和 OceanBase 数据库以及 ODP 一起部署，本文以合并部署为例介绍如何通过 OBD 部署 Config Server，并简单介绍部署 Config Server 后如何使用。

组件介绍

• ODP

  OceanBase Database Proxy，OceanBase 数据库代理，是 OceanBase 数据库专用的代理服务器，简称为 ODP（又称为 OBProxy）。详细信息请参见 OceanBase 数据库代理。

• Config Server

  OceanBase Configserver，Config Server 可提供 OceanBase 的元数据注册，存储和查询服务。详细信息请参见 ob-configserver。

部署 Config Server

本节介绍如何和 OceanBase 数据库以及 ODP 一起部署 Config Server。一起部署的情况下，启动成功后该 OceanBase 数据库会被注册到 Config Server 中。

步骤一：修改配置文件

若您要同时部署 OceanBase 数据库、ODP 和 Config Server，只需在部署 OceanBase 数据库和 ODP 的配置文件下新增 ob-configserver 组件信息，并在 OceanBase 数据库和 ODP 的组件下配置依赖 ob-configserver。

创建一个配置文件 config.yaml，该配置文件中各个组件的配置及注意事项如下所示。

1. 用户配置

   ## Only need to configure when remote login is required
   user:
     username: admin
   #   password: your password if need
     key_file: /home/admin/.ssh/id_rsa
   #   port: your ssh port, default 22
   #   timeout: ssh connection timeout (second), default 30
   

   username 为登录到目标机器的用户名，确保您的用户名有 home_path 的写权限。password 和 key_file 均用于验证用户，通常情况下只需要填写一个。

   注意

   在配置秘钥路径后，如果您的秘钥不需要口令，请注释或者删除 password，以免 password 被视为秘钥口令用于登录，导致校验失败。

2. OceanBase 数据库

   oceanbase-ce:
     depends:
       - ob-configserver
     servers:
       - name: server1
         # Please don't use hostname, only IP can be supported
         ip: 10.10.10.2
       - name: server2
         ip: 10.10.10.3
       - name: server3
         ip: 10.10.10.4
     global:
       # The working directory for OceanBase Database. OceanBase Database is started under this directory. This is a required field.
       home_path: /home/admin/observer
       # The directory for data storage. The default value is $home_path/store.
       # data_dir: /data
       # The directory for clog, ilog, and slog. The default value is the same as the data_dir value.
       # redo_dir: /redo
       # Please set devname as the network adaptor's name whose ip is  in the setting of severs.
       # if set severs as "127.0.0.1", please set devname as "lo"
       # if current ip is 192.168.1.10, and the ip's network adaptor's name is "eth0", please use "eth0"
       devname: eth0
       cluster_id: 1
       # please set memory limit to a suitable value which is matching resource.
       memory_limit: 6G # The maximum running memory for an observer
       system_memory: 1G # The reserved system memory. system_memory is reserved for general tenants. The default value is 30G.
       datafile_size: 20G # Size of the data file.
       log_disk_size: 15G # The size of disk space used by the clog files.
       cpu_count: 16
       mysql_port: 2881 # External port for OceanBase Database. The default value is 2881. DO NOT change this value after the cluster is started.
       rpc_port: 2882 # Internal port for OceanBase Database. The default value is 2882. DO NOT change this value after the cluster is started.
       production_mode: false
       enable_syslog_wf: false # Print system logs whose levels are higher than WARNING to a separate log file. The default value is true.
       max_syslog_file_count: 4 # The maximum number of reserved log files before enabling auto recycling. The default value is 0.
       # Cluster name for OceanBase Database. The default value is obcluster. When you deploy OceanBase Database and obproxy, this value must be the same as the cluster_name for obproxy.
       appname: obcluster
       root_password: ****** # root user password
       proxyro_password: ****** # proxyro user password, consistent with obproxy's observer_sys_password, can be empty
     server1:
       zone: zone1
     server2:
       zone: zone2
     server3:
       zone: zone3
   

   配置文件中各配置项含义可参见 配置文件说明。需要注意的配置项如下：

   • depends：您需配置 oceanbase-ce 组件依赖于 ob-configserver 组件，否则无法将该 OceanBase 数据库注册到 Config Server 中。

   • appname：用于配置集群的名字，不配置的情况下 OceanBase 数据库无法注册到 Config Server 中。

3. ODP 组件

   obproxy-ce:
     # Set dependent components for the component.
     # When the associated configurations are not done, OBD will automatically get the these configurations from the dependent components.
     depends:
       - oceanbase-ce
       - ob-configserver
     servers:
       - 10.10.10.5
     global:
       listen_port: 2883 # External port. The default value is 2883.
       prometheus_listen_port: 2884 # The Prometheus port. The default value is 2884.
       home_path: /home/admin/obproxy
       # oceanbase root server list
       # format: ip:mysql_port;ip:mysql_port. When a depends exists, OBD gets this value from the oceanbase-ce of the depends.
       # rs_list: 10.10.10.1:2881;10.10.10.2:2881;10.10.10.3:2881
       enable_cluster_checkout: false
       # observer cluster name, consistent with oceanbase-ce's appname. When a depends exists, OBD gets this value from the oceanbase-ce of the depends.
       # cluster_name: obcluster
       skip_proxy_sys_private_check: true
       enable_strict_kernel_release: false
       # obproxy_sys_password: # obproxy sys user password, can be empty. When a depends exists, OBD gets this value from the oceanbase-ce of the depends.
       # observer_sys_password: # proxyro user password, consistent with oceanbase-ce's proxyro_password, can be empty. When a depends exists, OBD gets this value from the oceanbase-ce of the depends.
   

   配置文件中各配置项含义可参见 配置文件说明。您需注意请勿配置 rs_list 配置项，否则 ODP 无法获取 Config Server 中注册的集群信息。

4. Config Server 组件

   ob-configserver:
     servers:
       - 10.10.10.1
     global:
       listen_port: 8080 # The port of ob-configserver web
       server_ip: 0.0.0.0 # Listen to the ob-configserver server IP。When you want to listen to the specified IP address,use it.
       home_path: /home/admin/ob-configserver  # The working directory for prometheus. ob-configserver is started under this directory. This is a required field.
       ## log config
       # log_level: info # Log printing level of ob-configserver。The default value is `info`
       # log_maxsize: 30 # The total size of manager ob-configserver.Log size is measured in Megabytes.The default value is 30
       # log_maxage：7 # The days of manager expired ob-configserver.Log retention days. The default value is 7
       # log_maxbackups: 10  #The number of manager expired ob-configserver.Log. The default value is 10
       # log_localtime: true #  Switch of ob-configserver.Log naming with localtime. The default value is true
       # log_compress: true # Compress ob-configserver.Log switch. The default value is true
   
       ## vip config, configserver will generate url with vip_address and port and return it to the client
       ## do not use some random value that can't be connected
       # vip_address: "10.10.10.1"
       # vip_port：8080
       ## storage config
       # storage:
         ## database type, support sqlite3 or mysql
         # sqlite3:
         # database_type: sqlite3
         # connection_url: "/home/admin/ob-configserver/.data.db?cache=shared&_fk=1"
   
         # mysql:
         # database_type: mysql
         # connection_url: "$user:$password@tcp($IP:$PORT)/$metadb?parseTime=true"
   

   配置文件介绍：

   • server_ip：ob-configserver 服务访问白名单，默认为 0.0.0.0，表示 ob-configserver 所在服务器的所有 IP 都可以访问。

   • vip_address 和 vip_port：vip_address 和 vip_port 为负载均衡访问地址配置项。如果有使用负载均衡，请在配置好负载均衡后，填写负载均衡地址 IP 与端口。

   • database_type：数据库类型，提供 sqlite3 与 mysql，默认为 sqlite3 类型，推荐使用 mysql 类型。

   • connection_url：数据库连接 URL。选择 sqlite3 类型时默认为 $home_path/.data.db?cache=shared&_fk=1；选择 mysql 类型时，connection_url 可以是 OceanBase 数据库，也可以是原生 MySQL。指定的数据库用户需要有 URL 里对应库 DDL 和 DML 的权限，需注意的是，使用 OceanBase 数据库时这里的库名请勿使用 oceanbase 库，否则会提示没有权限。

   说明

   您可通过在 servers 下配置多台机器来部署多个 Config Server，此时 OceanBase 数据库和 ODP 默认关联第一台机器上启动的 Config Server。

步骤二：启动 Config Server

1. 部署 Config Server

   [admin@obtest ~]$ obd cluster deploy test -c config.yaml
   

2. 启动 Config Server

   [admin@obtest ~]$ obd cluster start test
   

   输出如下，可以看到 Config Server 访问地址的打印信息。

   +-----------------------------------------------------------+
   |                      ob-configserver                      |
   +---------+------+---------------+----------+--------+------+
   | server  | port | vip_address   | vip_port | status | pid  |
   +---------+------+---------------+----------+--------+------+
   | 0.0.0.0 | 8080 | 10.10.10.1    | 8080     | active | 6270 |
   +---------+------+---------------+----------+--------+------+
   curl -s 'http://10.10.10.1:8080/services?Action=GetObProxyConfig' |jq .
   Wait for observer init ok
   +-------------------------------------------------+
   |                     observer                    |
   +---------------+---------+------+-------+--------+
   | ip            | version | port | zone  | status |
   +---------------+---------+------+-------+--------+
   | 10.10.10.2    | 4.2.0.0 | 2881 | zone1 | ACTIVE |
   +---------------+---------+------+-------+--------+
   | 10.10.10.3    | 4.2.0.0 | 2881 | zone2 | ACTIVE |
   +---------------+---------+------+-------+--------+
   | 10.10.10.4    | 4.2.0.0 | 2881 | zone3 | ACTIVE |
   +---------------+---------+------+-------+--------+
   obclient -h10.10.10.2 -P2881 -uroot -p'********' -Doceanbase -A
   
   +-------------------------------------------------+
   |                     obproxy                     |
   +---------------+------+-----------------+--------+
   | ip            | port | prometheus_port | status |
   +---------------+------+-----------------+--------+
   | 10.10.10.5    | 2883 | 2884            | active |
   +---------------+------+-----------------+--------+
   obclient -h10.10.10.5 -P2883 -uroot -p'********' -Doceanbase -A
   

3. 复制执行 ob-configserver 下的访问信息

   说明

   访问信息中的 jq 命令的作用是格式化（美化）shell 命令行下 json 的输出，可通过 sudo yum install jq 安装。

   {
     "Code": 200,
     "Message": "successful",
     "Success": true,
     "Data": {
       "ObProxyBinUrl": "http://10.10.10.1:8080/client?Action=GetObProxy",
       "ObProxyDatabaseInfo": {
         "DataBase": "***",
         "MetaDataBase": "http://10.10.10.1:8080/services?Action=ObRootServiceInfo&User_ID=alibaba&UID=admin&ObRegion=obdv1",
         "Password": "***",
         "User": "***"
       },
       "ObRootServiceInfoUrlList": [
         {
           "ObRegion": "obcluster",
           "ObRootServiceInfoUrl": "http://10.10.10.1:8080/services?Action=ObRootServiceInfo&ObCluster=obcluster"
         }
       ],
       "Version": "4660b4b1f237893ba1da50a302e4c3e6"
     },
     "Trace": "07a7cac129713d00",
     "Server": "10.10.10.1",
     "Cost": 0
   }
   

   输出中 ObRootServiceInfoUrlList 中包含一起部署的 OceanBase 集群（obcluster）的集群信息，表示该集群已成功注册到 Config Server 中。您可再次请求 ObRootServiceInfoUrl 验证 Config Server 中是否可以解析出 OceanBase 集群的信息，命令如下。

   curl -s 'http://10.10.10.1:8080/services?Action=ObRootServiceInfo&ObCluster=obcluster' |jq .
   

   输出如下，成功解析出集群中 OBServer 节点的 IP 和端口。

   {
     "Code": 200,
     "Message": "successful",
     "Success": true,
     "Data": {
       "ObClusterId": 1,
       "ObRegionId": 1,
       "ObCluster": "obcluster",
       "ObRegion": "obcluster",
       "ReadonlyRsList": [],
       "RsList": [
         {
           "address": "10.10.10.4:2882",
           "role": "LEADER",
           "sql_port": 2881
         },
         {
           "address": "10.10.10.3:2882",
           "role": "FOLLOWER",
           "sql_port": 2881
         },
         {
           "address": "10.10.10.2:2882",
           "role": "FOLLOWER",
           "sql_port": 2881
         }
       ],
       "Type": "PRIMARY",
       "timestamp": 1694084002271443
     },
     "Trace": "2b2ee036276b068e",
     "Server": "10.10.10.1",
     "Cost": 3
   }
   

使用 Config Server

部署 Config Server 后，您可通过修改 OceanBase 集群配置文件或执行 SQL 命令将 OceanBase 集群注册到 Config Server 中，通过指定 Config Server 地址启动的 ODP 可以访问该 Config Server 中注册的所有 OceanBase 集群。

本节介绍如何将 OceanBase 数据库注册到 Config Server 中，以及如何通过指定 Config Server 地址启动 ODP。

注册 OceanBase 集群

本节介绍如何将 OceanBase 集群注册到 Config Server 中，有修改 OceanBase 集群配置文件和执行 SQL 命令两种方式。

启动 ODP

您可使用 OBD 新部署一个 ODP，并配置 obproxy_config_server_url 启动，配置文件示例如下：

##Only need to configure when remote login is required
user:
  username: admin
#   password: your password if need
  key_file: /home/admin/.ssh/id_rsa
#   port: your ssh port, default 22
#   timeout: ssh connection timeout (second), default 30
obproxy-ce:
  servers:
    - 10.10.10.2
  global:
    listen_port: 2883 # External port. The default value is 2883.
    prometheus_listen_port: 2884 # The Prometheus port. The default value is 2884.
    home_path: /home/admin/obproxy
    obproxy_config_server_url: http://10.10.10.1:8080/services?Action=GetObProxyConfig
    enable_cluster_checkout: false
    # observer cluster name, consistent with oceanbase-ce's appname. When a depends exists, OBD gets this value from the oceanbase-ce of the depends.
    skip_proxy_sys_private_check: true
    enable_strict_kernel_release: false
    # obproxy_sys_password: # obproxy sys user password, can be empty. When a depends exists, OBD gets this value from the oceanbase-ce of the depends.
    observer_sys_password: ******** # proxyro user password, consistent with oceanbase-ce's proxyro_password, can be empty. When a depends exists, OBD gets this value from the oceanbase-ce of the depends.
    obproxy_sys_password: ********

ODP 的详细参数介绍可参见 ODP 参数。

当 ODP 通过 obproxy_config_server_url 配置项启动后，可以使用 ODP 连接注册到 Config Server 中的集群。你可以通过如下命令查看 Config Server 中注册的 OceanBase 集群信息。

curl -s 'http://10.10.10.1:8080/services?Action=GetObProxyConfig' |jq .

输出如下，从输出中可以看到该 Config Server 中已注册 obcluster 和 test1 两个集群，此时通过 ODP 可成功连接这两个集群。

{
  "Code": 200,
  "Message": "successful",
  "Success": true,
  "Data": {
    "ObProxyBinUrl": "http://10.10.10.1:8080/client?Action=GetObProxy",
    "ObProxyDatabaseInfo": {
      "DataBase": "***",
      "MetaDataBase": "http://10.10.10.1:8080/services?Action=ObRootServiceInfo&User_ID=alibaba&UID=admin&ObRegion=obdv1",
      "Password": "***",
      "User": "***"
    },
    "ObRootServiceInfoUrlList": [
      {
        "ObRegion": "obcluster",
        "ObRootServiceInfoUrl": "http://10.10.10.1:8080/services?Action=ObRootServiceInfo&ObCluster=obcluster"
      }
      {
        "ObRegion": "test1",
        "ObRootServiceInfoUrl": "http://10.10.10.1:8080/services?Action=ObRootServiceInfo&ObCluster=test1"
      }
    ],
    "Version": "4840ade8a158753aa5b9ea69ba014fc1"
  },
  "Trace": "d4b4b27fae24e7b4",
  "Server": "10.10.10.1",
  "Cost": 1
}

删除集群注册信息

您可通过如下命令删去 Config Server 中注册的集群信息，此处以删除 test1 集群信息为例。

curl -X DELETE 'http://10.10.10.1:8080/services?Action=ObRootServiceInfo&ObCluster=test1&ObClusterId=1&version=2' |jq .

输出如下：

{
  "Code": 200,
  "Message": "successful",
  "Success": true,
  "Data": "success",
  "Trace": "caba1209b3a00c56",
  "Server": "10.10.10.1",
  "Cost": 4
}

再次查看 Config Server 中注册的集群信息。

curl -s 'http://10.10.10.1:8080/services?Action=GetObProxyConfig' |jq .

输出如下，可以看到 test1 的集群信息已不存在。

{
  "Code": 200,
  "Message": "successful",
  "Success": true,
  "Data": {
    "ObProxyBinUrl": "http://10.10.10.1:8080/client?Action=GetObProxy",
    "ObProxyDatabaseInfo": {
      "DataBase": "***",
      "MetaDataBase": "http://10.10.10.1:8080/services?Action=ObRootServiceInfo&User_ID=alibaba&UID=admin&ObRegion=obdv1",
      "Password": "***",
      "User": "***"
    },
    "ObRootServiceInfoUrlList": [
      {
        "ObRegion": "obcluster",
        "ObRootServiceInfoUrl": "http://10.10.10.1:8080/services?Action=ObRootServiceInfo&ObCluster=obcluster"
      }
    ],
    "Version": "4840ade8a158753aa5b9ea69ba014fc1"
  },
  "Trace": "d4b4b27fae24e7b4",
  "Server": "10.10.10.1",
  "Cost": 1
}

常见问题

启动 Config Server 时，报错 prepared statement not supported

问题现象

当使用 OceanBase 数据库做 metadb 时，启动 Config Server 报错 prepared statement not supported，查看日志如下所示。

2022-11-16T09:52:14.47799+08:00 ERROR [12258,] caller=cmd/main.go:36:func1: start configserver failed: start config server: create configserver schema: sql/schema: reading schema information Error 1235: while parameter _ob_enable_prepared_statement is disabled, prepared statement not supported
[10.10.10.1:2882] [2022-11-16 09:52:14.459102] [YB42AC1EC731-0005EB5F1D83A0A7] fields: args:="[]"

可能原因

在启动 Config Server 时，作为元数据库的 OceanBase 数据库未开启 prepared statement 功能。

解决方法

您可通过如下两种方法解决。

• 方法一：在配置文件里为 Config Server 的 connection_url 配置项添加上 interpolateParams=true，示例如下。

  connection_url: "user:password@tcp(IP:PORT)/test?parseTime=true&interpolateParams=true"
  

• 方法二：连接 OceanBase 集群开启 _ob_enable_prepared_statement 参数。

  • 查询是否开启 _ob_enable_prepared_statement 参数。

    obclient> select name,value,svr_ip,svr_port from oceanbase.__all_virtual_sys_parameter_stat where name='_ob_enable_prepared_statement';
    +-------------------------------+-------+---------------+----------+
    | name                          | value | svr_ip        | svr_port |
    +-------------------------------+-------+---------------+----------+
    | _ob_enable_prepared_statement | False | 10.10.10.1    |     2882 |
    +-------------------------------+-------+---------------+----------+
    1 row in set
    

  • 开启 _ob_enable_prepared_statement 参数。

    obclient> alter system set _ob_enable_prepared_statement='True';
    

通过 ODP 连接 Config Server 中注册的集群时报错

问题现象

通过 ODP 连接 Config Server 中注册的集群时报错，示例如下。

$ obclient -h10.10.10.2 -P2883 -uroot@sys#cluster -p -Doceanbase -A
ERROR 2013 (HY000): Lost connection to MySQL server at 'reading authorization packet', system error: 11

可能原因

ODP 使用配置的查询 OceanBase 数据库的用户和密码无法访问 OceanBase 集群查询数据。

解决方法

您可通过如下两种方法解决。

• 方法一：执行 obd cluster edit-config 命令查看配置文件，您需查看 OceanBase 集群配置文件中的 proxyro_password 和 ODP 配置文件中的 observer_sys_password 两个配置项设置是否保持一致。若配置文件中对应配置项不一致，将两者修改统一后根据输出执行对应命令重启，重启后即可通过 ODP 连接该集群。

• 方法二：查询 OceanBase 数据库中是否存在 proxyro 用户

  • 使用 root 用户登录 OceanBase 数据库的 sys 租户

    obclient -h10.10.10.2 -P2881 -uroot@sys -p -Doceanbase -A
    

  • 查看集群中是否有 proxyro 用户

    select user,password from mysql.user;
    

  • 根据输出结果有如下两种处理方法

    • 结果一：集群中无 proxyro 用户，执行如下命令创建 proxyro 用户，并将密码配置为和 ODP 配置文件中 observer_sys_password 配置项一致

      create user 'proxyro' identified by '*****';
      

    • 结果二：集群中存在 proxyro 用户，执行如下命令修改 proxyro 用户密码，使其与 ODP 配置文件中 observer_sys_password 配置项一致

      alter user 'proxyro' identified by '*****';