OceanBase Database Community Edition V3.1.3 is released to provide more features

Since June 1, 2021, when it became open-source, OceanBase Database Community Edition has evolved from V3.1.0 to V3.1.1 and V3.1.2. V3.1.0 was an early version that offered only the database service and OceanBase Deployer (OBD). V3.1.1 and V3.1.2 were released before 2022. More tools tailored for the Community Edition were supported, such as OceanBase Cloud Platform (OCP) for O&M management, OceanBase Developer Center (ODC) for database development, OceanBase Migration Service (OMS) for data synchronization, and OBDUMPER/OBLOADER for data export and import. Thanks to the joint efforts of community developers and large amounts of feedback, OceanBase Database Community Edition has gained complete database capabilities required for enterprise applications and has been used by hundreds of companies to support their critical businesses.

OceanBase Database Community Edition V3.1.3 was officially released on March 30, 2022.

This version was released to:

Provide major features of OceanBase Database Enterprise Edition. The new Community Edition supports multiple models and JSON data types and is compatible with HBase API.

Respond to feedback and expectations from community members. The new Community Edition is easier to deploy, more user-friendly, more stable, and has better cloud adaptability. It is also compatible with Kubernetes.

Features and benefits of the new version:

• Supports OBKV to provide NoSQL capabilities for HBase and Table models. The new Community Edition delivers higher read/write performance than HBase and can reuse the underlying strong consistency and geo-disaster recovery features of OceanBase Database without the performance jitters of HBase. Various companies, including Ant Group, have already replaced HBase with OceanBase Database in their critical businesses that are latency-sensitive.

• Supports JSON data types of MySQL V5.7 to store semi-structured data. JSON data types strengthen the weak points of relational models. provide better storage for semi-structured data compared with TEXT and LOB data types. JSON data types also support efficient indexing and validation.

• Allows users to restore objects from the recycle bin. This reduces the risk of accidental deletion of objects. This feature uses Flashback Table to restore an accidentally deleted table from the recycle bin.

• OceanBase Change Data Capture (CDC) supports large transactions. A new persistence mode is introduced to cache data locally before it can be synchronized. This solves the issue that the connection may fail because of out-of-memory (OOM) errors when CDC handles large transactions.

• Kubernetes Operator is supported. Users can run OceanBase Database in a container in a Kubernetes cluster that is deployed on a public or private cloud.

• Easier deployment provides a better experience for developers. The hardware specifications that are required to run a Docker image are reduced from 2C10G to 2C8G (2 CPU cores and 8-GB memory). In this way, developers can use OceanBase Database on PCs. ARM architecture is also supported.

Take note of these changes in this version:

By default, a local index is created for a partitioned table. In earlier versions, a global index is created by default. We made this change because we found that users often adjust the indexing performance of partitioned tables. In addition, a local index is better for most business scenarios. To create a global index, add the global keyword in the statement. For example, the statement can be: create table example(c1 int primary key, c2 int, index(c2) global) partition by key(c1) partitions 32. This change does not affect existing indexes but only works on indexes to be created in the future. Database administrators must keep this in mind.

The non-strict mode is adopted for table creation. In earlier versions, you can create a table only when all replicas of the table are created. In this new version, you can create a table if the majority of the replicas, such as 2 out of 3 or 3 out of 5, are created. To enable strict mode, set the system variable ob_create_table_strict_mode to True.

OBKV capabilities

OceanBase Database Community Edition supports OBKV to provide NoSQL capabilities for HBase and Table models.

Why multiple models are supported

Many NoSQL storage systems, such as Bigtable, HBase, Cassandra, Zeppelin, ScyllaDB, Pegasus, and Riak KV, use storage engines based on the LSM-Tree structure to bring out the full potential of modern hardware. OceanBase Database implements a whole set of scalable distributed LSM-Tree storage engines that support multiple models. NoSQL add-ons to these engines benefit users from the following proven architectural advantages:

1. Enterprise-level data security and high availability in a NoSQL environment

2. Global deployment, such as the Five IDCs Across Three Regions deployment mode of OceanBase Database

3. Infinite scalability with separated read/write resources and weak-consistency read mode

4. High elasticity that enables instant online resource scaling to handle large amounts of data during the Double 11 festival

How does OceanBase Database support multiple models

OceanBase Database supports relational SQL APIs of MySQL syntax. The storage engines of OceanBase Database provide NoSQL capabilities for table models and HBase models by using TableAPI. Compatibility services are also introduced. In this way, multiple data models can be processed in an OceanBase database.

OBClient: OceanBase Database stores tables and table partitions on different servers. To read or write table data, you must locate the table or partition to which the data belongs first. OceanBase Database does not support routing through TableAPI but only executes a request locally. OBClient will calculate the partition number based on the table name and row key first and then use the partition number to obtain the IP address of the OceanBase service node. After that, OBClient sends the request to that node for processing. OBClient not only reads and writes data but also serves as a router. Note that TableAPI currently does not support cross-partition transactions.

Transport layer: TableAPI is implemented based on the Remote Procedure Call (RPC) protocol for OceanBase Database. The protocol follows the standard request-response model. Requests are separated from each other, and the OBServer concurrently processes multiple requests. Multiple TCP connections can be created between an OBClient and an OBServer. After a request is sent through the RPC-based transport layer to the OBServer, the packet is parsed to obtain the message type and data body.

OBServer: An OBServer processes TableAPI requests from an OBClient by using the producer-consumer model. After the OBServer receives a request from the OBClient, an independent thread puts the request into a message queue. Then, a worker thread pulls the request from the queue and processes it. The OBServer arranges requests of various types, such as MySQL and RPC requests, in a priority queue. This prevents requests that have a higher priority from being starved because of timeout.

In addition, TableAPI uses the same transaction framework and storage engines as those of OceanBase Database. The LSM-Tree-structured storage engines support subpartitioned tables, so TableAPI can also use automatic partition load balancing to prevent partition and server hotspots, which is fully transparent to users.

Benefits of multi-model architecture

OceanBase Database uses partitioned tables. The primary partition, or leader, can be distributed to different service nodes, so the data write operations on different partitions are distributed to different nodes. This multi-point write mode can improve the overall system write throughput. The data storage system adopts the LSM-Tree structure. Business data is written directly to the memory, which delivers write performance that can be compared with an in-memory database. OceanBase Database also supports block cache and row cache, Bloom filter, and query optimizer, which help maximize the data access performance. Compared with SQL APIs, TableAPI bypasses the SQL syntax parsing and provides direct access from OBClients to OBServers.

Figure 1: Performance differences between OceanBase Database and HBase

JSON data types

MySQL 5.7.8 and later versions support JSON data types defined in RFC 7159 to efficiently access the data in JSON files and provide the following three advantages to users:

1. Data validation. Compared with TEXT or LOB data types, JSON data types support syntax validation before text data is stored in a database. Errors are reported if any invalid data is identified. This ensures the accuracy of the inserted data.

2. Easy to use. Users do not need to convert TEXT or LOB types to JSON types. Various functions and path expressions are also provided to help users access JSON data.

3. High performance. JSON data types support indexing and reduce JOIN queries. This improves performance and relieves the service load on the database.

OceanBase Database Community Edition V3.1.3 supports JSON data types. The implementation of support for JSON data types involves the following aspects:

• JSON Type: It converts data types, compares JSON data types, and handles character set issues.

• JSON Binary: It stores binary values that are parsed and encoded from JSON types and supports query and traversal based on JSON BINARY, as well as binary search. This reduces resources that are consumed when you expand JSON data during queries.

• JSON Tree: It is a supplementary structure to support memory and is built only when you modify the JSON data.

• JSON Path: It provides the path to access data in JSON files and JSON subobjects.

• JSON Function: It provides function expressions that are used to manipulate JSON data, such as functions for creating, querying, and modifying JSON data.

OceanBase Database Community Edition V3.1.3 is compatible with all JSON functions in MySQL 5.7 and some JSON functions in MySQL 8.0. OceanBase Database Community Edition V3.1.3 also supports DDL operations, index creation, SQL query, and data type conversion. Features of V3.1.3 are described as follows:

• JSON data types include four basic types (string, number, Boolean, and null) and two structured types (object and array).

• You can create columns containing data of JSON data types when you create a table or add such columns.

• You can index the generated column of a column that contains data of JSON data types.

• You can reference JSON objects by using the -> and ->> operators.

• You can use JSON text in SQL statements such as SELECT, INSERT, UPDATE, and DELETE.

• All JSON functions in MySQL 5.7 and some JSON functions in MySQL 8.0, such as JSON_OVERLAPS() and JSON_VALUE(), are supported.

Object restoration from recycle bin

A recycle bin is used to store deleted objects. In Oracle 10g, a recycle bin object is introduced to store information about the database, tables, and other objects that are dropped or deleted by users.

The recycle bin is equivalent to a data dictionary table that stores information about objects deleted by users. Objects in the recycle bin still occupy physical space unless they are manually purged or periodically cleared by the system. In a database, if a table is deleted, then objects that are associated with and subordinated to the table, such as indexes, constraints, and other objects, are also moved to the recycle bin. Likewise, when the table is restored from the recycle bin, objects that are associated with and subordinated to the table are also restored.

Earlier versions of OceanBase Database Community Edition support the recycle bin feature. By default, the recycle bin is enabled but you cannot restore deleted objects. OceanBase Database Community Edition V3.1.3 provides the Flashback Tenant/Database/Table/Index commands for users to restore a deleted object to its original location by modifying the metadata about the object. Users can also restore the object to a new location by using the Rename To command to rename the object.

By running these commands, users can restore accidentally deleted objects. This reduces the risk of accidental deletion of objects.

Large transactions in data synchronization

CDC pulls changed data from OceanBase Database and sends the data to downstream consumers. CDC supports data synchronization at different levels (cluster, tenant, database, and table) and allows users to customize synchronization rules by creating blacklists and whitelists. Currently, data synchronization can be completed by CDC together with Canal, Flink, and OMS.

A bank once experienced a disconnection incident when it was testing CDC data synchronization because of an OOM error triggered by a large transaction. The error occurred because the changed data of the same transaction was stored in memory before it was sent to downstream consumers. So, a considerable part of memory resources would be occupied by a large or extra-large transaction even with throttling strategies in place, causing system exceptions and eventually impacting the stability of data synchronization.

To solve this issue, a persistence mode is introduced in OceanBase Database Community Edition V3.1.3 to cache data in local storage before the synchronization starts. The cached transaction data is sent downstream by rows and then cleared after being consumed. You can choose between the memory mode and the persistence mode based on your business requirements by setting the working_mode parameter. To improve the data synchronization efficiency, we recommend that you use the memory mode when no large transactions are involved. However, we recommend the persistence mode to ensure synchronization stability when large DML operations transactions are synchronized.

OB Operator

Kubernetes container orchestration supported

OB Operator is provided to run OceanBase Database in a container in a Kubernetes cluster that is deployed on a public or private cloud. Although a basic functional framework is in place, OB Operator is still in incubation. Further development is required to improve OB Operator.

OB Operator consists of three modules: StatefulApp Controller, OBServer Controller, and Operator Orchestrator.

StatefulApp Controller maintains Pod, PVC, and PV. OBServer Controller maintains OBServers. Operator Orchestrator manages OB Operator across multiple Kubernetes clusters. Operator Orchestrator is under development. We encourage developers in the community to help develop this module.

Figure 2. Hierarchical architecture of OB Operator

You can deploy OB Operator using YAML or Kustomize and use the following services:

• Deploy an OceanBase cluster on the Kubernetes platform.

• Create, delete, and scale clusters, and manage a cluster over its entire lifecycle.

• Create and delete zones.

• Create, delete, restart, and fail over OBServers.

• Run the kubectl commands to view the status of clusters, zones, and OBServers.

• Call HTTP APIs to manage Kubernetes resources.

More features, such as tenant management, OBProxy management, and multi-Kubernetes clusters, will be provided in the future to help users deploy and manage OceanBase Database.

Final Words

With the introduction of multiple model support, JSON data types, and HBase compatibility, OceanBase Database Community Edition V3.1.3 has set another major milestone in our journey to pursue higher stability and ease of use.

We would like to thank developers and users who have contributed to this new version in the community. Please check the Release Notes to learn more about OceanBase Database. If you have any suggestions for OceanBase Database Community Edition, visit open.oceanbase.com and share your ideas with us.