Generic Table and List Table
TcaplusDB is divided into Generic tables and List tables according to different record types. A record of a Key (1 - 8 fields) in Generic table corresponds to a Record record, and a record of a Key (1 - 7 fields) in List table corresponds to multiple Record records. List tables are divided into regular list tables and sortlist tables according to whether sorting is required. For regular List tables, Tcaplus is inserted unordered inside. When users fetch all the values under a key, the values are unordered. SortList sorts and inserts values based on a specified Value field. Users can fetch values in the order from largest to smallest or from smallest to largest. These three schemas exist in both PB and TDR tables, and users can select the appropriate schema according to their needs.
1. Table and Related Rules
1.1. Table Rules
Table names shall not exceed 32 bytes (including '\0' ending).The input can only contain letters, numbers, and underscores, with the first character being a letter or an underscore.
Field names shall not exceed 32 bytes (including '\0' ending).
Each Key field shall not exceed 1024 bytes.
Each Value field shall not exceed 10MB.
Each record after Pack shall not exceed 10MB.
The total buffer length of all key fields in a Pb table after the sequence shall not exceed 1022 bytes.
1.2. Modification Rules
It is not allowed to make any modifications to the primarykey, split key, or index;
Value fields can be added but not deleted, and when adding fields, pay attention to updating the version number;
The default value of the Value field can be modified. The maximum length of the value field can only be increased.
It is not allowed to delete the Value field and change its name or type.
The maximum number of elements in a List type table can only be modified to a larger number but not a smaller one, and the maximum number of elements cannot exceed 10000;
1.3. Generic Table
Indexes can be created (see local index and global index).
The Generic table supports a maximum of 8 Key fields, 256 Value fields for api versions greater than or equal to 3.46, and 128 Value fields for versions lower than 3.46.
1.4. List Table
Indexes cannot be created.
The List table supports a maximum of 7 Key fields and a maximum of 255 Value fields. Each field has one reserved for the system.
The List table supports convenient head and tail operations, suitable for email, message boards, battle records and other scenarios.
The List table needs to specify the maximum number of elements under a single Key record (currently the maximum is 10,000). If the maximum number of elements exceeds that, it can specify that old elements be deleted from the head or tail. FIFO is used by default. The SortList table has the same maximum number of elements as a regular List table. When inserting more than the maximum number of elements N, the ascending order (from smallest to largest) always keeps the smallest N elements, and the reverse order, on the contrary, always keeps the largest N elements.
The SortList table supports specifying a maximum of 4 sorting fields at the same time.
All values of a single Key can be fetched at a time, and the internal values are unordered.
2. Generic Table
2.1. TDR Generic Examples
<struct name="PLAYERONLINECNT" version="1" primarykey="TimeStamp,GameSvrID" splittablekey="TimeStamp">
<entry name="TimeStamp" type="uint32" desc="in minute" />
<entry name="GameSvrID" type="string" size="64" />
<entry name="GameAppID" type="string" size="64" desc="gameapp id" />
<entry name="OnlineCntIOS" type="uint32" defaultvalue="0" desc="online count of ios" />
<entry name="OnlineCntAndroid" type="uint32" defaultvalue="0" desc="online count of android" />
<entry name="BinaryLen" type="smalluint" defaultvalue="1" desc="data length; ignore source checking when the length is 0"/>
<entry name="binary" type="tinyint" desc="binary" count= "1000" refer="BinaryLen" />
<entry name="binary2" type="tinyint" desc="binary2" count= "1000" refer="BinaryLen" />
<entry name="strstr" type="string" size="64" desc="string"/>
<index name="index_id" column="TimeStamp"/>
</struct>
2.2. PB-Gneric Examples
message pb_generic_index_shardingkey {
option(tcaplusservice.tcaplus_primary_key) = "openid,tconndid,timekey,svrid";
option(tcaplusservice.tcaplus_index) = "index_openid(openid)";
option(tcaplusservice.tcaplus_index) = "index_openid_tconndid(openid,tconndid)";
option(tcaplusservice.tcaplus_index) = "index_full_key(openid,tconndid,timekey,svrid)";
option(tcaplusservice.tcaplus_sharding_key) = "openid";
//Four keys
required uint32 openid = 1; //QQ Uin
required string timekey = 2[(tcaplusservice.tcaplus_size) = 20, (tcaplusservice.tcaplus_desc) = "bingo"];
required int32 tconndid = 3;
required string svrid = 4;
//value
required string gamesvrid = 5[(tcaplusservice.tcaplus_size) = 11];
repeated property other_property= 6 ;//Other extended attributes
optional item_bag items = 7;
repeated int64 lockid = 8 [packed = true];
optional bytes val = 9;
optional pay_info pay = 10;
optional uint32 id_uint32 = 11;
optional int32 id_int32 = 12;
}
3. List Table
TcaplusDB uses a hierarchical storage mechanism for List tables, including:
Index record, FullKey + (idx1, idx2, idx3,..., idxn)
Data record, [FullKey + idx1, value1] [FullKey + idx2, value2] [...] [FullKey + idxn, valuen]
Due to the implementation mechanism of the list table, which includes system index records (each key has an associated index record), the actual number of business records will be less than the TotalSize, which is the total number of records (the sum of key counts and actual records). For example, if you query and iterate through the table and find 41 actual records, with 26 keys, then the TotalSize of the count table would be 67.
3.1. TDR List Examples
<struct name="following_action_list" version="1" primarykey="game,myuin">
<entry name="game" type="uint64" defaultvalue="0" desc="game ID" />
<entry name="myuin" type="uint32" desc="QQ number"/>
<entry name="actiontype" type="uint8" defaultvalue="0" desc="1 - share pictures, 2 - like pictures, 3 - comment on pictures"/>
<entry name="uin2" type="uint32" desc="follower's QQ number"/>
<entry name="nick2" type="string" size="128" desc="follower's nickname"/>
<entry name="sex" type="uint8" defaultvalue="0" desc="follower's sex: male 0 female 1"/>
<entry name="pid" type="string" size="33" desc="picture ID operated by follower"/>
<entry name="time" type="uint32" desc="time"/>
<entry name="content" type="string" size="1024" desc="update content"/>
</struct>
3.2. PB List Examples
syntax = "proto3";
package myTcaplusTable;
import "tcaplusservice.optionv1.proto";
message tb_online_list {
option(tcaplusservice.tcaplus_primary_key) = "openid,tconndid,timekey";
option(tcaplusservice.tcaplus_customattr) = "TableType=LIST;ListNum=1900";
int32 openid = 1; //QQ Uin
int32 tconndid = 2;
string timekey = 3;
string gamesvrid = 4;
int32 logintime = 5 ;
repeated int64 lockid = 6;
pay_info pay = 7;
message pay_info {
uint64 total_money = 1;
uint64 pay_times = 2;
}
map<string, pay_info> projects = 8;
}
3.3. List Table Expiration Strategy
/**
@Brief: set the mode of deleting old elements at the time of inserting elements when LIST is full
@param [in] chListShiftFlag TCAPLUS_LIST_SHIFT_NONE: It is not allowed to delete elements. If LIST is full, the insertion fails; TCAPLUS_ LIST_ SHIFT_ HEAD: Remove the first element; TCAPLUS_ LIST_ SHIFT_ TAIL: Remove the last element
If the table is a sort List, the user will fail to call the interface in the case that expiration must be done, and the expiration rules are automatically determined based on the sorting order of the fields
@Retval 0: setting successful
@Retval non-0: setting failed. For specific errors, see \link ErrorCode \endlink
*/
int32_t SetListShiftFlag(IN const char chListShiftFlag = TCAPLUS_API_LIST_SHIFT_HEAD);
When the expiration strategy is set to TCAPLUS LIST SHIFT_HEAD (when the list is full, the elements in the head of the list are expired), it can be used together with tail insertion to achieve a similar FIFO queue.
When the expiration strategy is set to TCAPLUS LIST SHIFT_TAIL (when the list is full, the elements at the end of the list are expired), it can also be used together with head insertion to achieve a similar FIFO queue.
4. SortList Table
It is allowed to have up to four sort fields(first-level field), which are specified in the xml (tdr) at the time of creating a table. The specific format is as follows:
<struct name="following_action_list" version="1" primarykey="game,myuin" customattr="TableType=SORTLIST;SortRule=INSC;SortFieldNum=1">
<entry name="game" type="uint64" defaultvalue="0" desc="game ID" />
<entry name="myuin" type="uint32" desc="QQ number"/>
<entry name="actiontype" type="uint8" defaultvalue="0" desc="1 - share pictures, 2 - like pictures, 3 - comment on pictures"/>
<entry name="uin2" type="uint32" desc="follower's QQ number"/>
<entry name="nick2" type="string" size="128" desc="follower's nickname"/>
<entry name="sex" type="uint8" defaultvalue="0" desc="follower's sex: male 0 female 1"/>
<entry name="pid" type="string" size="33" desc="picture ID operated by follower"/>
<entry name="time" type="uint32" customattr="sort1" desc="time"/>
<entry name="content" type="string" size="1024" desc="update content"/>
</struct>
As shown above, customattr="TableType=SORTLIST; ListNum=1023; SortFieldNum=1" means the table type is SORTLIST, SortRule=INSC means that the table is sorted in ascending order, SortFieldNum=1 means there is one sort field, and customattr="sort1" means the first sort field.
The sorting order is set to smallest to largest by default.
It is not allowed to change from an unordered list to a SortList, or from a SortList to an unordered list, or change the order of the sorting fields, or add or delete the sorting fields (this is achieved by writing so to change the Key-Value).
The maximum number of bytes in the sort field is 8 B. The type of the sort field is byte, uint16, uint32, uint64, int16, int32, int64, float, double [string (containing \0 up to 8 B, not supported temporarily)].
Sorting refers to sorting Value fields, not Keys.
Instructions:
Sort
After the existing data structure is sorted, use ListAddAfter to insert data, and the effect of insertion is the best.
Allow multiple sorting fields to sort together (up to 4), sort by the first field first, and if the first field is the same, then sort by the second field, and so on.
Insert
For ListAddAfter, the process is: first check whether the list is full. If it is full and elements are not allowed to be expired, the insertion fails. If the list is full and the element is allowed to be expired, delete the element, obtain a BiggestIndex, insert the new element at the corresponding position, and move other elements.
When using ListAddAfter in the SortList, if the number of lists exceeds the maximum number of elements N, the ascending order (from smallest to largest) always keeps the smallest N elements, that is, when the inserted elements are larger than the current maximum, the insertion will fail (because they are expired immediately), while the reverse order, on the contrary, always keeps the largest N elements.
4.1. TDR-Sortlist Examples
<struct name="table_Sortlist_single" primarykey="key, name" customattr2="TableType=SORTLIST;ListNum=1023;SortFieldNum=1;SortRule=DESC" version="5" desc="used for the list table traversal test, requiring 4 shards, and the list number is 1023 at the maximum" >
<entry name="key" type="uint32" desc="when a single uint32 is a KEY, hashcode = key % 10000"/>
<entry name="name" type="int16" />
<entry name="level" type="uint32" />
<entry name="value1" type="string" size="102400" defaultvalue="" desc="maximum length: 100KB"/>
<entry name="value2" type="string" size="102400" defaultvalue="" desc="maximum length: 100KB"/>
<entry name="type_int8" type="int8" desc="type_int8" />
<entry name="type_uint8" type="uint8" desc="type_uint8"/>
<entry name="type_int16" type="int16" desc="type_int16" customattr2="sort1"/>
<entry name="type_uint16" type="uint16" desc="type_uint16"/>
<entry name="type_int32" type="int32" desc="type_int32"/>
<entry name="type_uint32" type="uint32" desc="type_uint32"/>
<entry name="type_int64" type="int64" desc="type_int64"/>
<entry name="type_uint64" type="uint64" desc="type_uint64"/>
<entry name="type_float" type="float" desc="type_float"/>
<entry name="type_double" type="double" desc="type_double"/>
<entry name="type_short" type="short" desc="type_short"/>
<entry name="type_string" type="string" desc="type_string" size="20"/>
<entry name="type_tinyint" type="tinyint" desc="type_tinyint"/>
<entry name="type_datetime" type="datetime" desc="type_datetime"/>
</struct>
4.2. PB-SortList Examples
message pb_sortedlist {
option(tcaplusservice.tcaplus_primary_key) = "openid,tconndid,timekey,svrid";
option(tcaplusservice.tcaplus_customattr) = "TableType=SORTLIST;ListNum=1900;SortField=id_int32";//Multi field sorting SortField=id_int32,id_uint32..
//Four keys
required uint32 openid = 1; //QQ Uin
required string timekey = 2[(tcaplusservice.tcaplus_size) = 20, (tcaplusservice.tcaplus_desc) = "bingo"];
required int32 tconndid = 3;
required string svrid = 4;
//value
required string gamesvrid = 5[(tcaplusservice.tcaplus_size) = 11];
repeated property other_property= 6 ;//Other extended attributes
optional item_bag items = 7;
repeated int64 lockid = 8 [packed = true];
optional bytes val = 9;
optional pay_info pay = 10;
optional uint32 id_uint32 = 11;
optional int32 id_int32 = 12;
}