SYS 租户下建索引表导致 1 号表无主

问题现象

SYS 租户下有自建表；集群重启，启动过程中，集群无法登录，1 号表无主，CLOG 频繁卸任，SYS 租户 MemStore内存空间不足。

问题诊断

• 检查业务是否在 sys 租户下自行建表。

• 使用 OBStack 工具查看是否存在如下堆栈信息:

  Threads (359280-MinorScan)
  #0    0x00007f30c002d4fd in __nanosleep_nocancel from /usr/lib64/libc-2.17.so
  #1    0x00007f30c005e394 in usleep from /usr/lib64/libc-2.17.so
  #2    0x0000000009367345 in oceanbase::observer::ObInnerSQLConnection::process_retry(oceanbase::observer::ObInnerSQLResult&, int&, long, bool&, long, bool) from /home/admin/oceanbase/bin/observer
  #3    0x000000000936ca43 in oceanbase::observer::ObInnerSQLConnection::query(oceanbase::common::sqlclient::ObIExecutor&, oceanbase::observer::ObInnerSQLResult&, oceanbase::observer::ObVirtualTableIteratorFactory*, bool) from /home/admin/oceanbase/bin/observer
  #4    0x000000000936d98d in oceanbase::observer::ObInnerSQLConnection::execute_read(unsigned long, char const*, oceanbase::common::ObISQLClient::ReadResult&, bool, bool) from /home/admin/oceanbase/bin/observer^M^
  M
  #5    0x000000000caa6908 in oceanbase::common::ObCommonSqlProxy::read(oceanbase::common::sqlclient::ObISQLConnection*, oceanbase::common::ObISQLClient::ReadResult&, unsigned long, char const*) from /home/admin/oc
    eanbase/bin/observer
  #6    0x000000000caa6c34 in oceanbase::common::ObCommonSqlProxy::read(oceanbase::common::ObISQLClient::ReadResult&, unsigned long, char const*) from /home/admin/oceanbase/bin/observer
  #7    0x00000000096b274f in oceanbase::common::ObSQLClientRetryWeak::read(oceanbase::common::ObISQLClient::ReadResult&, unsigned long, char const*) from /home/admin/oceanbase/bin/observer
  #8    0x0000000003a4aa1b in oceanbase::share::schema::ObSchemaServiceSQLImpl::construct_schema_version_history(oceanbase::share::schema::ObRefreshSchemaStatus const&, oceanbase::common::ObISQLClient&, long, ocean
  base::share::schema::VersionHisKey const&, oceanbase::share::schema::VersionHisVal&) from /home/admin/oceanbase/bin/observer
  #9    0x000000000494f2c3 in oceanbase::share::schema::ObServerSchemaService::construct_schema_version_history(oceanbase::share::schema::ObRefreshSchemaStatus const&, long, oceanbase::share::schema::VersionHisKey
  const&, oceanbase::share::schema::VersionHisVal&) from /home/admin/oceanbase/bin/observer
  #10   0x0000000003ed1cac in oceanbase::share::schema::ObMultiVersionSchemaService::get_schema_version_history(oceanbase::share::schema::ObRefreshSchemaStatus const&, unsigned long, long, oceanbase::share::schema:
  :VersionHisKey const&, oceanbase::share::schema::VersionHisVal&, bool&) from /home/admin/oceanbase/bin/observer
  #11   0x0000000003ee16f6 in oceanbase::share::schema::ObMultiVersionSchemaService::get_schema(oceanbase::share::schema::ObSchemaMgr const*, oceanbase::share::schema::ObRefreshSchemaStatus const&, oceanbase::share
  ::schema::ObSchemaType, unsigned long, long, oceanbase::common::ObKVCacheHandle&, oceanbase::share::schema::ObSchema const*&) from /home/admin/oceanbase/bin/observer
  #12   0x0000000003d61a51 in int oceanbase::share::schema::ObSchemaGetterGuard::get_schema_v2<oceanbase::share::schema::ObTableSchema>(oceanbase::share::schema::ObSchemaType, unsigned long, oceanbase::share::schem
  a::ObTableSchema const*&, long) from /home/admin/oceanbase/bin/observer
  #13   0x0000000003c89140 in oceanbase::share::schema::ObSchemaGetterGuard::get_table_schema_inner(unsigned long, oceanbase::share::schema::ObTableSchema const*&) from /home/admin/oceanbase/bin/observer
  #14   0x0000000003c89294 in oceanbase::share::schema::ObSchemaGetterGuard::get_table_schema(unsigned long, oceanbase::share::schema::ObTableSchema const*&) from /home/admin/oceanbase/bin/observer
  #15   0x0000000003eda812 in oceanbase::share::schema::ObMultiVersionSchemaService::retry_get_schema_guard(long, unsigned long, oceanbase::share::schema::ObSchemaGetterGuard&, long&) from /home/admin/oceanbase/bin
  /observer
  #16   0x00000000088b2b2a in oceanbase::storage::ObPartitionStore::get_index_status(long, oceanbase::common::ObIArray<oceanbase::share::schema::ObIndexTableStat>&, oceanbase::common::ObIArray<unsigned long>&) from
  /home/admin/oceanbase/bin/observer
  #17   0x00000000088b325b in oceanbase::storage::ObPartitionStore::check_all_merged(oceanbase::memtable::ObMemtable&, long, bool&, bool&) from /home/admin/oceanbase/bin/observer
  #18   0x000000000895b01e in oceanbase::storage::ObPGStorage::check_can_release_pg_memtable_(oceanbase::common::ObSEArray<oceanbase::common::ObPartitionKey, 4l, oceanbase::common::ModulePageAllocator, false> const&, oceanbase::storage::ObTablesHandle&, oceanbase::storage::ObTablesHandle&, oceanbase::storage::ObTablesHandle&) from /home/admin/oceanbase/bin/observer
  #19   0x000000000897a253 in oceanbase::storage::ObPGStorage::check_release_memtable(oceanbase::storage::ObIPartitionReport&, long) from /home/admin/oceanbase/bin/observer
  #20   0x00000000085a1bc9 in oceanbase::storage::ObPartitionGroup::check_release_memtable(oceanbase::storage::ObIPartitionReport&) from /home/admin/oceanbase/bin/observer
  #21   0x0000000007fdce2e in oceanbase::storage::ObPartitionScheduler::check_release_memtable() from /home/admin/oceanbase/bin/observer
  #22   0x0000000007fe6450 in oceanbase::storage::ObPartitionScheduler::MinorMergeScanTask::runTimerTask() from /home/admin/oceanbase/bin/observer
  #23   0x000000000cb7cc83 in oceanbase::common::ObTimer::run1() from /home/admin/oceanbase/bin/observer
  #24   0x00000000037df1f9 in oceanbase::lib::ThreadPool::run0() from /home/admin/oceanbase/bin/observer
  #25   0x000000000c9c3e87 in oceanbase::lib::CoSetSched::Worker::run() from /home/admin/oceanbase/bin/observer
  #26   0x000000000c9c0d70 in oceanbase::lib::CoRoutine::__start(boost::context::detail::transfer_t) from /home/admin/oceanbase/bin/observer
  #27   0x000000000c9bfc0f in make_fcontext from /home/admin/oceanbase/bin/observer
  

问题原因

MemStore 内存空间不足时，需要释放 MemTable，MemTable 释放时需要检查索引表是否完成转储需要检查索引状态，检查索引状态需要依赖 Schema 获取索引表的列表，刷新 Schema 触发了 InnerSql，InnerSql 访问 SYS 租户但是因为无主导致访问失败，所以释放 MemTable 卡住。 释放 MemTable 卡住时，因为 MemStore 是内存爆的状态，CLOG 将会 Reconfirm 超时，然后强制 Election 卸任，最终导致 1 号表无主并造成循环依赖。

问题的风险及影响

可能导致集群重启时，无法重启成功，集群无法登录。

影响的版本

OceanBase 数据库 V4.0 之前所有版本。

解决方法

• 适当调大 memstore_limit_percentage (默认值 50)，看是否能恢复。

  例如:

  obclient> ALTER SYSTEM SET memstore_limit_percentage = 60;
  

• 启动时调大 SYS 租户的内存规格。

规避方式

不建议在 SYS 租户下建的用户表中写入大量数据。