What Is TcaplusDB?

TcaplusDB is a distributed NoSQL database produced by Tencent, which is completely self-developed. The characteristics of TcaplusDB include combination of cache and persistence, PB-level storage, millisecond delay, lossless horizontal scal, complex data structure and so on. It covers games, the Internet, government affairs, finance, manufacturing, the Internet of Things and other fields.

1. High performance. Hot and code data exchanged between memory and disk through LRU, data write to SSD disks, data distribute in multi-machine and so on, can keep high permormance，and one machine can support 100000 qps with a delay of less than 10 milliseconds

2. High availability. Dual-system hot backup disaster recovery mechanism ensuring rapid recovery in case of system failure. Fast recovery from hardware or operating system failures, with an availability rate of 99.999%.

3. Low cost. Switching ability of in-process data in memory and disk, active data in memory, and inactive data in disk. A cost savings of about 70% over full memory storage and about 40% over Redis + MySQL.

4. Dynamic expansion. Unlimited storage space, dynamic expansion and contraction of capacity according to the actual needs of the game, without affecting the game's operation, and easy coping with the rapid change of app scale.

5. Easy to use. Supporting API interface calls, WEB rendering of common operations (such as table addition, table modification, table deletion, and data deletion), and automation of operation and maintenance operating systems such as capacity expansion, capacity reduction, and backup.

1. Concept Mapping between TcaplusDB and Other Databases (MySQL, MongoDB)

2. Main Concepts of TcaplusDB

The data model of TcaplusDB includes App, game zone (Zone, table group), table (Table), index (Index) and other main concepts. The relationship between these concepts can be shown in the following figure.

An App includes multiple game zones. A game zone can create multiple tables, define multiple local indexes and create distributed indexes for each table.

2.1. App

The App is TcaplusDB's definition of a database user, usually a project team or company, such as the "Arena Of Valor" app. An App is a logical concept. Creating an App does not involve the allocation of resources.

An App can create multiple game zones (zones, table groups) across clusters, but a game zone can only use the resources of one cluster.

For details about how to create an App, see Create an App.

2.2. Game Zone (Zone, table group)

Similar to the concept of a traditional Database, a game Zone can also be understood as a cluster of tables. Multiple tables can be created under a game zone. Under the same game zone, the table cannot have the same name.

The app can plan the game zone according to its actual situation. It can create a "test" game zone and a "formal" game zone according to different purposes, or create a "WeChat zone" and a "QQ zone" according to different user platforms.

For details about how to create a game zone, see Create a Game Zone.

2.3. Table

Similar to the concept of tables in a traditional database, tables in TcaplusDB can be understood as sets of data with the same Schema.

Each TcaplusDB table has at least one data shard (which takes up one Shard resource). As the amount of data increases or decreases, the number of data shards in the table can be increased or decreased. At present, the system DBA is responsible for the expansion of data shards. After the user creates a table, if it is estimated that the table data and access may grow rapidly in a short time, the system DBA needs to be contacted in advance for expansion.

A TcaplusDB table consists of primary key fields and non-primary key fields. A maximum of eight primary key fields can be specified. A maximum of 256 regular fields (non-primary key fields) can be specified.

The tables in TcaplusDB can be divided into PB Tables(Protobuf) and TDR Tables according to the way the tables are defined and accessed.

PB tables use PB files to define the table structures and are accessed through the PB SDK. TDR tables use TDR files to define the table structures and are accessed through the TDR SDK. TDR is a serialization and deserialization protocol similar to PB. There is no essential difference between PB tables and TDR tables. Users can decide whether to use PB tables or TDR tables according to their familiarity with PB tables or TDR tables. It should be noted that the functions supported by the current TcaplusDB PB table and TDR table are somewhat different. When selecting which table type to use, users should also consider whether the function can meet the app requirements.

The main functional differences between TDR and PB are shown in the following table:

Tables in TcaplusDB can be devided into Generic Tables and List Tables according to table record types

Each Key in the Generic table corresponds to one record. Data keys must be unique. In the List table, multiple records with the same Key are stored in the data table in the form of a list.

For details on how to add a table, see Add a Table.

2.4. Primary Key

The table definition in TcaplusDB needs to set the primary key, which is the unique identifier of the data record and is used to uniquely locate one record.

2.5. Splittable Key

The table definition of TcaplusDB needs to set a splittable key attribute. The splittable key must be a subset of the primary key field. The splittable key can be a field or a combination of multiple fields.

The fields contained in the splittable key participate in the hash calculation, and the hash value determines which node in the cluster the record is stored to. Therefore, the values of splittable key fields for multiple records in a table should be varied to ensure a balanced distribution of data.

For example, if the primary key of a table is "uid, role_id, zone_id", the values of uid and role_id are varied enough, while the values of zone_id are only a few or at most a few hundred different. Using zone_id as a splittable key is risky. If the number of records corresponding to a specific zone_id is too large, the storage node of the Tcaplus is overloaded, or the data volume exceeds the storage capacity of the storage node. If the splittable key of a table is gender, the data will be distributed to two storage nodes at most, and the app distribution ability will be limited to the performance of two storage nodes at most.

The splittable key determines the physical distribution of data (the system hashes the data to different nodes according to this field). It is recommended to select a field with high dispersion to facilitate load balancing. If this parameter is not specified, all fields of the primary key are used by default.

2.6. Record

TcaplusDB records consist of a line of strings. Numbers in each field support nested types, which can be nested for up to 32 levels. The maximum size of a single record is 10MB. Regular object files can be serialized into binary files for storage.

2.7. Index

TcaplusDB supports two forms of indexes: Local Index and Global Index.

• Local index: an index based on the TcaplusDB primary key field is created with the table when it is being created.

• Global index: an index based on the first level fields (including primary key fields and non-primary key fields) of the TcaplusDB table.

Local and global indexes facilitate data query by using indexes. Advantages:

• The query based on the local index can satisfy the user to query the index by partial primary key fields.

• The query based on the global index can satisfy the user to query various forms through any first-level field, such as range, fuzzy, aggregate, pagination, etc.

3. Typical Application Scenarios of TcaplusDB

See Typical Application Scenarios of TcaplusDB

4. TcaplusDB Environment Preparation

At present, TcaplusDB provides three deployment modes: cloud service, local cluster deployment and local Docker container deployment. See TcaplusDB Environment Preparation for details.