How OceanBase Wins a Leading Merchant Service Provider with Performance beyond Expectations

Founded in 2011, Wuhan Lichu Business Service (Lichu for short) is one of the first early birds flapping their high-tech wings in the development and application of integrated payment technologies in China. It is also a leading digital service provider for Chinese merchants. In addition to its three major product systems, namely, Saobei, an integrated payment platform, FU+, a lightweight SaaS solution for merchants, and Youyu, a uni-marketing platform, Lichu also owns a bunch of qualification certificates and more than 30 software copyrights. The powerful system development and business operation strength enables the company to establish itself as an innovative pacesetter in China's merchant service industry.

In this post, Lin Zhe, Lichu CTO, Vice President, and a senior system architect in the payment industry, shares the story between Lichu and OceanBase flavored with a brief history of distributed databases.

Why distributed databases

In the age of Internet, data can be vital for a business. As a leader in the integrated payment field, Lichu worked with more than 1,000 partners to serve over 1.1 million brick-and-mortar stores in 600 plus cities across China. With such a business scale, Lichu needed to handle more than 23 million transactions on a daily basis, which amounted to a transaction amount of RMB 350 billion in 2021. Facing such an immense volume of payment transactions, data not only determines the service quality of Lichu, but is also the source of vitality, even the future of the company. Therefore, for the health of data, the root of all our services, the stability of databases overwhelms it all. Stable operation day and night all year round is the basic requirement, as service continuity is the bottom line in terms of service quality. The application of highly available databases to avoid single point of failures (SPOFs) and perform fast standby switchover for disaster recovery without data loss has become an irreversible trend in the world of database technologies.

The development of database technology

The history of database technologies has witnessed the emergence of standalone databases, sharded databases, and distributed databases. Looking at our business, we also started early the upgrades to microservices and the cellular architecture. Such upgrades are essentially the evolution to distributed services, where applications and components are coordinated over a distributed consistency protocol to achieve high availability and high throughput of services. Although the development of distributed database technologies started late and still lags behind the technical architecture of application services, the development trend assures me that distributed databases will definitely be the most sought-after option someday.

Standalone databases

Ever since its introduction in 1970 by Dr. Edgar F. Codd, an IBM researcher, the relational model has been pushed forward for decades in an all-round manner and become the go-to model for database development. Many big names have been established in the standalone database market: Oracle, DB2, MySQL, and PostgreSQL to name a few.

Easy to use, standalone databases provide guaranteed atomicity, consistency, isolation, and durability (ACID) with low single-node maintenance costs. However, as the business volume swings up, standalone databases show their downsides under the great pressure of highly concurrent reads and writes and highly efficient processing of mighty data floods, while the specifications of a single node are eventually bottlenecked. In a word, the business today is growing faster than the speed of transistor integration defined by Mr. Moore.

Let me use a black-and-white picture to remember the product family in the prehistoric computer age

Database and table sharding

The weakness of standalone databases overstressed by high-concurrency massive data processing tasks gave rise to the sharding architecture. Still flourishing today, the sharding architecture easily handles highly concurrent data access brought by rapid business growth, and manages massive data read/write well by means of sharding rules. Typical middleware include TDDL, Sharding-JDBC, and MyCat.

After studying some mainstream cloud databases, we found that most of the "distributed" databases on cloud are secondary products with an architecture developed from a MySQL or PG kernel, where multiple database instances are managed with distributed components such as ZooKeeper and etcd to make it look like a distributed system. Essentially, however, it is similar to having an application connected to multiple databases, only that the metadata is managed by a separate component. This architecture does not guarantee ACID performance, and the instances must be split for data migration to support rapid business growth, incurring substantial O&M costs.

Distributed databases

The concept of distributed database was introduced in the beginning of 21st century. Compared with traditional and non-relational (NoSQL) databases, the new NewSQL databases inherited the features and advantages of their predecessors. In addition to guaranteed ACID performance and distributed scalability, NewSQL databases also provided ample storage, high availability, and high efficiency.

In the early 21st century, modern distributed databases began to take shape. Oracle Real Application Clusters (RAC), a typical database of the Shared Storage architecture featuring distributed computing nodes, could be seen as an attempt at building a distributed database. After all, high-end storage hardware was not something friendly to every budget, plus the storage layer was not horizontally scalable whatsoever.

In 2012, Google released Spanner, a closed source database. We could only take a glimpse of it from the published paper. At the same time, Shared Nothing, Paxos, ACID, and Multiversion concurrency control (MVCC) had basically formed the general architecture of distributed databases.

Among the many distributed database options, OceanBase Database came to our attention as it was ranked first in the TPC Benchmark C two years in a row, and outperformed Oracle by more than 20 times, which was really impressive. We also learned that the OceanBase team developed the distributed database fully on their own, instead of simply wrapping up a kernel of another product in all sorts of things. This was a major reason for us to give it a shot on OceanBase Database.

Verification of OceanBase

Seeing the development trend of databases, Lichu, as an innovative player in the Internet industry, always keeps pace with the development of technology. We understand that only by constantly updating the architecture of our platforms, can we keep bettering our service and stay organized while dealing with our growing business. So, based on our best knowledge in distributed databases, we chose OceanBase Database and tested it based on our business. It's kind of our first move towards a real distributed database product.

High compatibility

Migrating an application to another database is not only a matter of changing the database driver. While the driver takes care of the communication protocol between the application and the database, the SQL compatibility of the database directly affects the costs of our business migration. Among the many options, OceanBase Database is the only one that supports both Oracle and MySQL in one cluster.

In our compatibility assessment, OceanBase provided OceanBase Migration Assessment (OMA), a tool that fully scans a database before data migration and offers you a window to prepare targeted solutions. OMA can also scan the SQL statements in an application, and provide an assessment report that describes incompatible items in detail and gives suggestions for improvement. So, with the help of OMS, we assessed the SQL compatibility of our test architecture.

The migration assessment was performed in command-line mode and the report was quickly produced. According to the report, we also found two non-compatible problematic statements and corrected them before things got worse.

OMA also supports local code scan, which prevents our core code from being leaked. Anyway, the report indicated a 100% compatibility, and the assessment results were proven to be accurate in our application access test, provided, of course, that the performance_schema storage engine contained all the SQL statements involved in our business.

Kudos to OceanBase Database for the easy-to-use and accurate OMA.

High compression ratio

Now we have to mention OceanBase Migration Service (OMS), a user-friendly data migration tool that enabled us to migrate our data with a few settings. So, we tested its performance by migrating a couple of data copies.

After the migration, we compared the differences in database storage. Yes, we are a company that builds everything on data and the inflating data storage costs are driving us like crazy. The point is, we were surprised by the superb compression ratio of OceanBase Database because a high compression ratio means a great reduction in storage costs. We made the storage comparison of a 300 GB wide table in different database types, and OceanBase Database can achieve a space saving of higher than 70% for a whole database.

Table storage comparison

The surprisingly high compression ratio of OceanBase Database is made possible not only with traditional compression algorithms such as zlib and lz4. Advanced encoding and compression methods such as differential algorithms also play a role in pushing the results further.

Hybrid Transactional/Analytical Processing (HTAP)

OceanBase Database is more than online transaction processing (OLTP). Its HTAP capability is also quite eye-catching because we need to handle a huge and rapidly growing volume of payment transactions. We are also looking forward to the capability of real-time statistical analysis that, for example, enables our partners to check the summary statistics of their latest transactions. Timeliness is an important part of their experience here.

For that purpose, we compared our old database with the OceanBase solution in terms of transaction and analytical processing to make sure that it is capable of handling real-time statistical queries while maintaining stable payment services. We used a wide table with 490 million plus data rows to test the statistical and analytical performance of multiple databases in different scenarios. OceanBase Database stood out in query performance, and topped in transaction processing performance with high concurrency, as in the TPCC. Its HTAP engine is also capable of real-time data analysis.

OceanBase goes beyond expectations once again in the real-world comparison of SQL execution performance.

The first migration to OceanBase Database

OceanBase Database not only met our expectations for a distributed database in all tests under different circumstances, but also surprised us with its performance and high storage compression ratio.

However, for security's sake, we picked Laihehuo, an innovative app, for a migration trial. This app had been released for some time, but enjoyed fast business growth, posing a great challenge to the database. If it kept running on a traditional MySQL database, we would not have enough time to react against a traffic surge someday even by sharding the database or tables.

Again, OMS played a great role in data migration. In addition to full data migration, it also supports schema migration, full verification, incremental migration, and reverse synchronization. The full verification feature is extremely helpful for our payment service. It saves us from developing a dedicated program, and ensures data accuracy. The incremental migration feature enables real-time synchronization between the destination and the source, and provides a time window for us to fully verify our business data, thus ensuring system stability after the cutover.

After the full migration of our Laihehuo business, we compared the storage compression ratio and, as expected, one of its logical databases was compressed to 26.4% its original size in the OceanBase architecture.

A logical database of Laihehuo was compressed from 11.06 GB to 2.92 GB.

During the migration and cutover, we only modified the configuration file and switched the data source without a need to modify the application code. The compatibility of OceanBase Database with MySQL protocols exempted us from changing the driver, lowering the barrier for database migration. The OMA assessment report also helped a lot in the application validation.

After migration, we were able to manage the business in the OceanBase Cloud Platform (OCP) console, which enabled us to monitor the CPU utilization, TPS, QPS, and other performance metrics, and set alerts to ensure compliance to the service-level agreement (SLA). In addition to real-time diagnostics, a great tool for our database administrators, the console also provided Slow SQL and TopSQL modules for us to bind indexes to slow SQL queries without touching the business code, greatly improving the database O&M efficiency.

Final words

This is how we successfully migrated our first business line with the support of tools in the OceanBase ecosystem. As everything is now running stably under our ongoing monitoring, Lichu will migrate other business lines to OceanBase Database one step at a time. We are ready to embrace the era of distributed databases and cannot wait to enjoy the technical benefits that come along with it!

Postscript

Zhou Guiqing, OceanBase solution architect

Two years ago, we failed to win trust of Lichu. However, OceanBase has made breakthroughs over time in building an ecosystem of products that can solve real problems of our customers in their database assessment, migration, and O&M. Lichu's approval on the performance of both the database and other products in the ecosystem has attested our practice in keeping our word that we will streamline services for customers with every effort.

To fit the highly concurrent data floods that challenged Lichu, we optimized some core database schemas based on the proven solutions of Ant Group in the migration assessment, making the new database scalable and more productive without modifying their business code.

Lichu has got off to a good start by hosting a business line on OceanBase Database. As for other business lines of the company, especially the enormous volume of payment data, we will work closely in the expectation that Lichu can experience data migration with a few clicks, integrated O&M management, and HTAP performance in OceanBase Database.