Internet Engineering Task Force P. THOMY Internet-Draft Loco-labs Intended status: Informational 19 December 2023 Expires: 21 June 2024 NTV tabular format (NTV-TAB) draft-thomy-ntv-tab-00 Abstract This document describes a set of simple rules for unambiguously and concisely encoding semantic tabular and multidimensional data (NTV- TAB format). These rules are based on the NTV structure and its JSON representation (JSON-NTV format). Status of This Memo This Internet-Draft is submitted in full conformance with the provisions of BCP 78 and BCP 79. Internet-Drafts are working documents of the Internet Engineering Task Force (IETF). Note that other groups may also distribute working documents as Internet-Drafts. The list of current Internet- Drafts is at https://datatracker.ietf.org/drafts/current/. Internet-Drafts are draft documents valid for a maximum of six months and may be updated, replaced, or obsoleted by other documents at any time. It is inappropriate to use Internet-Drafts as reference material or to cite them other than as "work in progress." This Internet-Draft will expire on 21 June 2024. Copyright Notice Copyright (c) 2023 IETF Trust and the persons identified as the document authors. All rights reserved. This document is subject to BCP 78 and the IETF Trust's Legal Provisions Relating to IETF Documents (https://trustee.ietf.org/ license-info) in effect on the date of publication of this document. Please review these documents carefully, as they describe your rights and restrictions with respect to this document. Code Components extracted from this document must include Revised BSD License text as described in Section 4.e of the Trust Legal Provisions and are provided without warranty as described in the Revised BSD License. THOMY Expires 21 June 2024 [Page 1] Internet-Draft NTV tabular format (NTV-TAB) December 2023 Table of Contents 1. Introduction . . . . . . . . . . . . . . . . . . . . . . . . 2 1.1. Presentation . . . . . . . . . . . . . . . . . . . . . . 3 1.2. Key design features . . . . . . . . . . . . . . . . . . . 3 1.3. Conventions Used in This Document . . . . . . . . . . . . 4 2. Tabular data . . . . . . . . . . . . . . . . . . . . . . . . 4 2.1. Principles . . . . . . . . . . . . . . . . . . . . . . . 4 2.2. Tabular structure . . . . . . . . . . . . . . . . . . . . 5 2.3. Field structure . . . . . . . . . . . . . . . . . . . . . 8 2.4. Representation . . . . . . . . . . . . . . . . . . . . . 8 3. NTV-TAB format . . . . . . . . . . . . . . . . . . . . . . . 8 3.1. NTV structure . . . . . . . . . . . . . . . . . . . . . . 8 3.2. simple NTVfield formats . . . . . . . . . . . . . . . . . 9 3.3. default NTVfield formats . . . . . . . . . . . . . . . . 9 3.4. Optimized NTVfield formats . . . . . . . . . . . . . . . 12 3.5. Synthesis . . . . . . . . . . . . . . . . . . . . . . . . 13 4. Examples . . . . . . . . . . . . . . . . . . . . . . . . . . 15 4.1. Field examples . . . . . . . . . . . . . . . . . . . . . 15 4.2. Dataset examples . . . . . . . . . . . . . . . . . . . . 17 5. Properties . . . . . . . . . . . . . . . . . . . . . . . . . 19 5.1. JSON representation . . . . . . . . . . . . . . . . . . . 19 5.2. Dataset size . . . . . . . . . . . . . . . . . . . . . . 19 5.3. Nested NTV-TAB structure . . . . . . . . . . . . . . . . 20 6. Parsing a JSON-value . . . . . . . . . . . . . . . . . . . . 20 7. IANA Considerations . . . . . . . . . . . . . . . . . . . . . 21 8. Security Considerations . . . . . . . . . . . . . . . . . . . 22 9. References . . . . . . . . . . . . . . . . . . . . . . . . . 22 9.1. Normative References . . . . . . . . . . . . . . . . . . 22 9.2. Informative References . . . . . . . . . . . . . . . . . 22 Appendix A. Dataset sizing . . . . . . . . . . . . . . . . . . . 23 A.1. Methodology . . . . . . . . . . . . . . . . . . . . . . . 23 A.2. Formats . . . . . . . . . . . . . . . . . . . . . . . . . 25 Appendix B. Table schema compatibility . . . . . . . . . . . . . 27 B.1. Table schema . . . . . . . . . . . . . . . . . . . . . . 27 B.2. Compatibility . . . . . . . . . . . . . . . . . . . . . . 28 B.3. Example . . . . . . . . . . . . . . . . . . . . . . . . . 29 Acknowledgements . . . . . . . . . . . . . . . . . . . . . . . . 30 Contributors . . . . . . . . . . . . . . . . . . . . . . . . . . 30 Author's Address . . . . . . . . . . . . . . . . . . . . . . . . 30 1. Introduction THOMY Expires 21 June 2024 [Page 2] Internet-Draft NTV tabular format (NTV-TAB) December 2023 1.1. Presentation The main operational standard used to exchange textual tabular data is CSV format [RFC4180]. Unfortunately CSV format is obsolete (last revision in 2005) and current CSV tools do not comply with the standard. It is therefore important to define an alternative format that meets the expectations of tabular and multidimensional data exchanges. The NTV-TAB format proposed here is a response to this need. 1.2. Key design features The format's focus is on simplicity, lightness and web usage. The key features of this format are the following: * JSON as the base format JSON is simple and readable as simple text JSON supports rich structure including nesting and basic types JSON is web-native and very widely used and supported JSON format has binary representation (i.e. CBOR format) * optimized representations from the simplest to the most optimized, are available avoid data duplication, reduce the size of data, allows strict and unambiguous reversibility (lossless round- trip) * high semantic level of data (JSON-NTV as a grammar) Take into account most common data formats used in Internet standards wide variety of data typing meta-data (header or schema) can be integrate common format between tabular and multidimensional data THOMY Expires 21 June 2024 [Page 3] Internet-Draft NTV tabular format (NTV-TAB) December 2023 * simple, compact, extensible and self-describing 1.3. Conventions Used in This Document The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT", "SHOULD", "SHOULD NOT", "RECOMMENDED", "NOT RECOMMENDED", "MAY", and "OPTIONAL" in this document are to be interpreted as described in BCP 14 [RFC2119] [RFC8174] when, and only when, they appear in all capitals, as shown here. This document also uses the following terms: *JsonText, JsonValue, JsonObject, JsonMember, JsonElement, JsonArray, JsonNumber, JsonString, JsonFalse, JsonNull, JsonTrue :* These terms are defined in [JSON-NTV]. *NTV, NTVlist, NVlist, Vlist, TVlist, NTVsingle, NVsingle, TVsingle, Vsingle, NTVname, NTVtype, NTVvalue, JsonNTVtype, JsonNTVname, JsonPrimitive, JsonUnnamed, JsonNamed:* These terms are defined in [JSON-NTV]. *Row, Column, Table, Cell:* These terms are defined in [W3C_TAB]. *Dataset* A Dataset is equivalent to a Table *Field* A Field is equivalent to a Column 2. Tabular data 2.1. Principles _*Tabular data* is data that is structured into rows, each of which contains information about some things. Each row contains the same number of cells (although some of these cells may be empty), which provide values of properties of the thing described by the row. In tabular data, cells within the same column provide values for the same property of the things described by each row. This is what differentiates tabular data from other line-oriented formats._[W3C_TAB] Two main uses are identified for tabular data: THOMY Expires 21 June 2024 [Page 4] Internet-Draft NTV tabular format (NTV-TAB) December 2023 * a flow-oriented use for which each row is independent of the others. The dataset is then seen as a list of rows whose number can be variable. This is for example the case of a list of measurements from a sensor. * a structure-oriented use for which the rows are not independent and contribute to describing the same object. This is for example the case of a grade table for a class which integrates the students, courses, periods, etc. This document deals with this second use. 2.2. Tabular structure In structure-oriented use, columns and rows are not equivalent, the columns (or Fields) represent the 'semantics' of the data and the rows represent a specific combination of Field's values according to the structure defined by the tabular data (Dataset). The nature of the rows is often implicit. Two basic patterns are present in Datasets: * *Tree pattern*: A tree is represented in tabular form by a list of paths between each leaf and the node. The columns then represent the levels of the tree. * *Matrix pattern*: A matrix (or multidimensional data) is represented in tabular form by a column of the values of the matrix and additional columns represent the coordinates of each of the values. Table 1 and Table 2 present an example of such patterns +======+=========+=========+ | Root | level 1 | level 2 | +======+=========+=========+ | A | B | D | +------+---------+---------+ | A | B | E | +------+---------+---------+ | A | C | F | +------+---------+---------+ | A | C | G | +------+---------+---------+ Table 1: Tree pattern THOMY Expires 21 June 2024 [Page 5] Internet-Draft NTV tabular format (NTV-TAB) December 2023 +=======+=====+=====+ | Value | row | col | +=======+=====+=====+ | 1 | A | C | +-------+-----+-----+ | 2 | A | D | +-------+-----+-----+ | 3 | B | C | +-------+-----+-----+ | 4 | B | D | +-------+-----+-----+ Table 2: Matrix pattern Taking these structures into account leads to significant duplication of data. In the general case, Datasets mix these different structures. If we now observe the relationships between Fields [TAB-ANA], we can identify four main uses: * *association*: this consists of coupling each value of a Field to a single value of another Field ("coupled" relationship between two fields), * *classification*: This involves grouping the data by category in order - for example - to be able to make a statistical use of it, ("derived" relationship between two fields), * *crossing*: This consists of representing all the combinations between the two Fields, such as in matrix representations ("crossed" relationship between two fields), * *characterization*: It corresponds to the documentation of defined properties (no specific relationship). _Example: Price list of different foods based on packaging for the year 2022._Table 3 THOMY Expires 21 June 2024 [Page 6] Internet-Draft NTV tabular format (NTV-TAB) December 2023 +==+=======+=========+=========+======+=====+========+==============+ |Id|Product|Food |Packaging|Weight|Price| Period | Availability | +==+=======+=========+=========+======+=====+========+==============+ |11|apple |fruit |bag |1 kg |1 | 2nd | Yes | | | | | | | | half | | | | | | | | | 2022 | | +--+-------+---------+---------+------+-----+--------+--------------+ |12|apple |fruit |cardboard|10 kg |9 | 2nd | Yes | | | | | | | | half | | | | | | | | | 2022 | | +--+-------+---------+---------+------+-----+--------+--------------+ |13|orange |fruit |bag |1 kg |2 | 2nd | end of 2022 | | | | | | | | half | | | | | | | | | 2022 | | +--+-------+---------+---------+------+-----+--------+--------------+ |14|orange |fruit |cardboard|10 kg |18 | 2nd | end of 2022 | | | | | | | | half | | | | | | | | | 2022 | | +--+-------+---------+---------+------+-----+--------+--------------+ |15|pepper |vegetable|bag |1 kg |1.5 | 2nd | end of 2022 | | | | | | | | half | | | | | | | | | 2022 | | +--+-------+---------+---------+------+-----+--------+--------------+ |16|pepper |vegetable|cardboard|10 kg |13 | 2nd | end of 2022 | | | | | | | | half | | | | | | | | | 2022 | | +--+-------+---------+---------+------+-----+--------+--------------+ |17|banana |fruit |bag |1 kg |0.5 | 2nd | Yes | | | | | | | | half | | | | | | | | | 2022 | | +--+-------+---------+---------+------+-----+--------+--------------+ |18|banana |fruit |cardboard|10 kg |4 | 2nd | Yes | | | | | | | | half | | | | | | | | | 2022 | | +--+-------+---------+---------+------+-----+--------+--------------+ Table 3: Price list _We find here:_ * _association: between "Packaging" and "Weight",_ * _classification: between "Product" and "Food",_ * _crossing: between "Product" and "Weight",_ * _characterization: between "Product" and "Availability"_ THOMY Expires 21 June 2024 [Page 7] Internet-Draft NTV tabular format (NTV-TAB) December 2023 2.3. Field structure A Field is an ordered set of Cells. To represent this structure, several representations are possible depending on the nature of the data: * the simplest format is to represent a Field by the list of Cells with the same order for all Fields. This format is interesting when the data is little duplicated, * when the data is repetitive, a second option is to represent on the one hand the list of different data and on the other hand their position in the list (i.e. categorical data), * another special case also concerns repetitive data for which one value is highly predominant (sparse data). In this case, it is sufficient to provide only the position in the list of data except for the one that is predominant, * a last option consists in representing the Field according to its dependence with another Field (coupled, derived or crossed relationship). This leads to an optimized data volume. 2.4. Representation Three representations are available for a tabular object : row- oriented (list of Rows), cells-oriented (list of Cells), field- oriented (list of Fields). The field-oriented representation is retained because it takes into account the semantics carried by the Fields as well as the inter- Field analysis presented above. A Dataset is then seen as a set of Fields representing the properties of the entire Dataset. The order of Fields or Rows is not relevant. 3. NTV-TAB format 3.1. NTV structure A Dataset is represented by the following NTV entities: * NTVcell represents a Cell. NTVcell is a NTVsingle. THOMY Expires 21 June 2024 [Page 8] Internet-Draft NTV tabular format (NTV-TAB) December 2023 * NTVfield represents a Field. NTVfield is a NTV entity depending on the format chosen to represent the Field (simple format, default format, optimized format). A NTVfield contains a NTVlist of part of the NTVcells (Codec) and optionnaly coding data. * NTVdataset represents the Dataset. NTVdataset is a NVlist where the NTVname is the name of the Dataset and the NTVvalue is the list of NTVfields. The JSON format of a NTVdataset is his JSON-NTV format. 3.2. simple NTVfield formats This category is the usual representation of a Field with different values (Full format) or with several identical values (Unique format). *Full format* : The Full format is the format that does not use any coding. Codec and NTVfield are identical. The NTVfield is therefore a NTVlist where the NTVname is the name of the Field, the NTVtype is the default type of the NTVcells and the NTVvalue is the list of NTVcells. _Example JsonNTVvalue ( "price" Field)_ : _[ 1, 9, 2, 18, 1.5, 13, 0.5, 4 ]_ *Unique format* : The Unique format is used when all NTVcells are identical. The Codec is the NTVcell.Codec and NTVfield are identical (coding is implict). The NTVfield is therefore the NTVcell. _Example JsonNTVvalue ( "period" Field)_ : _"2nd half 2022"_ Note : The Unique format also makes it possible to represent tabular metadata 3.3. default NTVfield formats This category completes the simple formats with the other most common representations of a Field : THOMY Expires 21 June 2024 [Page 9] Internet-Draft NTV tabular format (NTV-TAB) December 2023 * Categorical Field (Complete format) * Periodic Field (Primary format) * Sparse Field (Sparse format) In those formats, Codec is explicit and is the TVlist of different Field NTVcells (Codec). The NTVfield is a NVlist where the NTVname is the name of the Field. *Complete format* : The "complete format" is equivalent to the format used to store categorical variables. The NTVfield is a NVlist composed with two NTV entities : * Codec: TVlist of different Field NTVcells (Codec), * Coding: Vlist of indexes of NTVcells in Component (Keys) The list of NTVcells is reconstituted by replacing the integers in the coding Vlist with the NTVcell at the coding index in the Codec (e.g. pandas categories and codes). _Example JsonNTVvalue ( "product" Field)_ : _[ [ "orange" , "pepper" , "apple" , "banana" ], [ 2, 2, 0, 0, 1, 1, 3, 3 ] ]_ *Sparse format* : A specific format (one dimensional sparse LIL format) is used for sparse data. It is defined by: * 'fill_value': it should be most common value * 'sp_value': it is a list storing only values distinct from the 'fill_value' * 'sp_index': list of index of 'sp-value' in the sparse data list The NTVfield is a NVlist composed with three NTV entities : * Codec: TVlist of different Field NTVcells (Codec), * Ref: Vlist of indexes of Codec value in 'sp_value' , THOMY Expires 21 June 2024 [Page 10] Internet-Draft NTV tabular format (NTV-TAB) December 2023 * Coding: Vlist of 'sp_index' The list of NTVcells is reconstituted by replacing in a list of 'fill_value', the values with index in the Coding Vlist by the corresponding value defined by the Ref index in the Codec TVlist. _Example JsonNTVvalue ( "food" Field)_ : _[ [ "vegetable" , "fruit"], [0, 0], [ 4, 5 ] ] 'fruit' is the 'fill_value' - 0 is the index of "vegetable"_ *Primary format* : This format is equivalent to the Complete format where the Keys Vlist is calculated with the "repetition coefficient". The NTVfield is a NVlist composed with two NTV entities : * Codec: TVlist of different Field NTVcells (Codec), * Coding: Vlist with a single integer (Repetition coefficient) The Keys Vlist is generated with the formula: keys[ikey] = ( ikey % ( coef * period ) ) // coef where: keys: is the Keys Vlist ikey: is the index of a key value coef: is the Repetition coefficient period: is the length of Codec _Example: coef = 2, period = 3, Keys length = 12_ _keys = [0, 0, 1, 1, 2, 2, 0, 0, 1, 1, 2, 2]_ The Repetition coefficient is the number of adjacent identical values in the Keys list. _Example "packaging"_ : _Codec: [ "bag" , "cardboard" ]_ _Coefficient: 1_ THOMY Expires 21 June 2024 [Page 11] Internet-Draft NTV tabular format (NTV-TAB) December 2023 _(implicit Keys : [ 0, 1, 0, 1, 0, 1, 0, 1 ] )_ _Example "product"_ : _Codec: [ "apple" , "orange" , "pepper" , "banana" ]_ _Coefficient: 2_ _(implicit Keys : [ 0, 0, 1, 1, 2, 2, 3, 3 ] )_ 3.4. Optimized NTVfield formats This category of formats reduces the size of Complete format with optimized Keys. The length of Keys is reduced with using of derived (Relative format) or coupled (Implicit format) relationships between two Fields. In those formats, Codec is explicit and is the TVlist of different Field NTVcells (Codec). The NTVfield is a NVlist where the NTVname is the name of the Field. *Implicit format* : This representation is associated with "coupled" Fields. These Fields have a one-to-one correspondence. The NTVfield is a NVlist composed with two NTV entities : * Codec: TVlist of different field NTVcells (Codec), * Ref: Vsingle entity index or name of the coupled Field. This format is equivalent to the Complete format where Keys is the Keys of the Field (with Complete format) defined by Ref. _Example JsonNTVvalue ( "weight" Field is associated with "packaging" Field )_ : _[ [ "1 kg" , "10 kg" ], "packaging"]_ _( implicit Keys : [ 0, 1, 0, 1, 0, 1, 0, 1 ] )_ *Relative format* : This representation is associated with "derived" Fields. These Fields have a one-to-many correspondence. THOMY Expires 21 June 2024 [Page 12] Internet-Draft NTV tabular format (NTV-TAB) December 2023 The values of a "derived" Field are inferred from the values of the parent Field. The Field is a NVlist composed with three NTV entities : * Codec: TVlist of different field NTVcells (Codec), * Ref: Vsingle entity index or name of the parent Field, * Coding : Vlist of relative indexes of NTVcells in Codec (Relative Keys). This format is equivalent to the Complete format where the Keys Vlist is obtained by replacing the values of the Keys Vlist of the parent Field with the corresponding values in the Relative Keys (the length of the Relative Keys is the length of the Codec of the parent Field). _Example JsonNTVvalue ( "food" Field - "product" Field is the parent Field of "food" Field)_ : _[ [ "fruit" , "vegetable" ], "product", [ 0, 1, 0, 0 ] ]_ _(the Vlist Keys is obtained by replacing the values 0, 1, 2, 3 of the Vlist Keys of the "product" Field by the values 0, 1, 0, 0 of the Relative Keys i.e.: [ 0, 0, 0, 0, 1 , 1, 0, 0] )_ 3.5. Synthesis The NTVfield structure corresponding to the format defined above are in Table 4: THOMY Expires 21 June 2024 [Page 13] Internet-Draft NTV tabular format (NTV-TAB) December 2023 +====================+========+===============+==================+ | Structure | Codec | Ref | Coding | +==========+=========+========+===============+==================+ | format | NTV | TVlist | Vsingle | Vlist | +==========+=========+========+===============+==================+ | Relative | NTVlist | x | index | Relative Keys | | | | | | | | | len = 3 | | or name | len < len(Field) | +----------+---------+--------+---------------+------------------+ | Complete | NTVlist | x | | Keys | | | | | | | | | len = 2 | | | len = len(Field) | +----------+---------+--------+---------------+------------------+ | Sparse | NTVlist | x | list of index | sp_index | | | | | | | | | len = 3 | | sp_value | 1. [RFC4180] Shafranovich, Y., "Common Format and MIME Type for Comma- Separated Values (CSV) Files", RFC 4180, DOI 10.17487/RFC4180, October 2005, . [RFC8174] Leiba, B., "Ambiguity of Uppercase vs Lowercase in RFC 2119 Key Words", BCP 14, RFC 8174, DOI 10.17487/RFC8174, May 2017, . [RFC8259] Bray, T., Ed., "The JavaScript Object Notation (JSON) Data Interchange Format", STD 90, RFC 8259, DOI 10.17487/RFC8259, December 2017, . 9.2. Informative References [TABLE] "FrictionLess", "Table Schema", 2021, . [JSON-NTV] Thomy, P., "JSON semantic format (JSON-NTV)", 2023, . [TAB-ANA] Thomy, P., "Tabular dataset analysis", 2022, . THOMY Expires 21 June 2024 [Page 22] Internet-Draft NTV tabular format (NTV-TAB) December 2023 [W3C_TAB] "W3C", "Recommendation : Model for Tabular Data and Metadata on the Web", 17 December 2015, . Appendix A. Dataset sizing This appendix presents an analysis of NTVdataset size optimization with the defined formats. A.1. Methodology The principle of defined formats is to replace duplicated data with encoding based on integers. We define the size of a Dataset representation (SZ) as the sum of the encoding size and the size of unencoded unduplicated values. The coding is modeled as being the product of the values remaining to be represented (nv - nc) with an average coding size (sc): SZ = nc * sv + (nv - nc) * sc where : nv : number of values sv : mean value size nc : number of different values sc : mean coding size example : Full format : {"product": ["orange","apple","apple","apple","orange","orange"]} Complete format : {"product": [ ["orange","apple"], [1, 0, 0, 0, 1, 1] ]} SZ = 9 + 8 + 7 + 6 * 1 = 30 (including double quotes), nv = 9 (including the Field name), sv = (9 + 8 * 3 + 7 * 3) / 7 = 7.71 nc = 3 (including the Field name) THOMY Expires 21 June 2024 [Page 23] Internet-Draft NTV tabular format (NTV-TAB) December 2023 sc = (30 - 3 * 7.71) / (9 - 3) = 1.15 (sc = (SZ - nc * sv) / (nv - nc)) In this example the JSON overhead (coma, space, curly bracket, square bracket) is not included. SZ is maximal when there is no coding (sc = sv) and minimal when the coding is perfect (sc = 0): SZmax = nv * sv SZmin = nc * sv We then define the following indicators: * unicity level UL = nc / nv UL = SZmin / SZmax 1 - UL = (SZmax - SZmin) / SZmax UL characterizes the nature of the data independently of the coding and represents the maximum achievable gain (1-UL). maximum UL = 1 (unduplicated data) minimum UL = 0 (full duplicated data = empty data) * object lightness OL = sc / sv OL = (SZ - SZmin) / (SZmax - SZmin) 1 - OL = (SZmax - SZ) / (SZmax - SZmin) OL characterizes coding efficiency maximum OL = 1 (no coding) minimum OL = 0 (perfect coding) The optimization of the size of the representation is then evaluated by comparing the size obtained without coding and that obtained with coding: G = (SZmax - SZ) / SZmax = (1 - UL) * (1 - OL) R = 1 - G = SZ / SZmax = UL + OL - UL * OL THOMY Expires 21 June 2024 [Page 24] Internet-Draft NTV tabular format (NTV-TAB) December 2023 The maximum G gain is 1 - UL, the minimum G gain is 0. If the data is empty, UL = OL = 0 and the gain is equal to 1. The indicators are deduced from the following four measurable values: * number of cells in the dataset (nv) * number of different cells in the dataset (nc) * size of the dataset with the format to study (SZ) * size of the dataset with Full format (SZmax) We then deduce sv = SZmax / nv as well as sc = (SZ - nc * sv) / (nv - nc) In the example above, the indicators are: UL = 3 / 9 = 0.33 OL = 1.15 / 7.71 = 0.15 G = 0.67 * 0.85 = 0.57 SZmax = 9 * 7.71 = 69.4 SZmin = 3 * 7.71 = 23.1 The Complete format is close to the minimum size (SZ = 30) and its size is less than half the size of the Full format (43 %). A.2. Formats The formats used to represent an NTVfield are in general form: * list of part of NTVcells * list of integers used to encode other NTVcells The size of this format can then be written (without taking into account the overhead linked to the format): SZ = nc * sv + k * nv * si where : nv : number of values THOMY Expires 21 June 2024 [Page 25] Internet-Draft NTV tabular format (NTV-TAB) December 2023 sv : mean value size nc : number of different Field values si : integer size k: specific coefficient of the coding used Comparison with the structure defined in the previous chapter allows us to deduce the parameters: UL = nc/nv sc = si * k * nv/(nv-nc) OL = si/sv * k * nv/(nv-nc) G = 1- nc/nv - si/sv * k R = nc/nv + si/sv * k The gain G is therefore equal to the maximum gain 1-UL reduced by the weight of the coding corresponding to the parameter k weighted by the average size of the values compared to an integer. Table 9 below specifies the values of k for the different formats: +==========+===============+===========================+ | Format | k coefficient | comments | +==========+===============+===========================+ | Full | 0 | R = 1 | +----------+---------------+---------------------------+ | Unique | 0 | R = 1/nv (nc = 1) | +----------+---------------+---------------------------+ | Complete | 1 | R = nc/nv + si/sv | +----------+---------------+---------------------------+ | Primary | 1 / nv | R = nc/nv + si/sv/nv | +----------+---------------+---------------------------+ | Coupled | 1 / nv | R = nc/nv + si/sv/nv | +----------+---------------+---------------------------+ | Sparse | 2 * ns / nv | R = nc/nv + 2*si/sv*ns/nv | +----------+---------------+---------------------------+ | Derived | nd / nv | R = nc/nv + si/sv*nd/nv | +----------+---------------+---------------------------+ Table 9: coding coefficient _ns: number of values distinct from the 'fill_value'_ THOMY Expires 21 June 2024 [Page 26] Internet-Draft NTV tabular format (NTV-TAB) December 2023 _nd: number of different values in the parent Field_ Appendix B. Table schema compatibility This appendix presents the compatibility between Tableschema [TABLE] and the NTV-TAB format. B.1. Table schema Table Schema is a simple language- and implementation-agnostic way to declare a schema for tabular data. A Table Schema is represented by a descriptor. The descriptor MUST be a JSON object with defined properties (JsonMember). Table Schema define following descriptors and properties: * Fields property Fields property MUST be an array where each entry in the array is a field descriptor (as defined below). * Field descriptor A field descriptor MUST be a JSON object that describes a single field. * Field properties The field descriptor object MAY contain any number of other properties. Some specific properties are defined below. Of these, only the name property is REQUIRED. Defined Properties: o name o title o description o example o type / format o constraints The constraints property on Table Schema Fields can be used by consumers to list constraints for validating field values. THOMY Expires 21 June 2024 [Page 27] Internet-Draft NTV tabular format (NTV-TAB) December 2023 * Table properties In additional to field descriptors, there are the following "table level" properties: o missingValues o primaryKey o foreignKeys B.2. Compatibility Three levels of compatibility are addressed : * Concepts The concepts are equivalent between Table Schema and NTV-TAB : o Table is equivalent to Dataset o Field in Table Schema is equivalent to the NTVfield o Name in Table Schema is equivalent to the NTVname of the NTVfield o Type / Format in Table Schema is equivalent to the NTVtype of the NTVfield * Type / Format NTVtype combines the concepts of type and format. The correspondence table in the NTV specification [JSON-NTV] gives the link between an NTVtype and the corresponding type/format. * Constraints Constraints are applicable to each value in a Table Field. Validating constraints for all values in a Table Field is equivalent to validating a constraint for all values in the Codec list. These compatibility levels are reached, which makes it possible to validate an NTVdataset with a schema defined according to the Table Schema format. The following principles should then be considered to validate an NTVdataset: THOMY Expires 21 June 2024 [Page 28] Internet-Draft NTV tabular format (NTV-TAB) December 2023 * The NTVfields names must be identical to the Schema names, * If the NTVtype of Codec data is not 'json', it must match the Type/Format defined in the Schema, * If the constrainst are valid with the Codec data, they are valid with the Field. B.3. Example Figure 2 is an example of Dataset with Full format ('tab_data1' without NTVtypes) and with other formats ('tab_data2'). tab_data1 = { "index": [100, 200, 300, 400, 500, 600], "dates": ["1964-01-01", "1985-02-05", "2022-01-21", "1964-01-01", "1985-02-05", "2022-01-21"], "value": [10, 10, 20, 20, 30, 30], "coord": [[1,2], [3,4], [5,6], [7,8], [3,4], [5,6]], "names": ["john", "eric", "judith", "mila", "hector", "maria"], "unique": ["true, "true", "true", "true", "true", "true"] } tab_data2 = { "index": [100, 200, 300, 400, 500, 600], "dates": {"::date":[["1964-01-01","1985-02-05","2022-01-21"],[1]}, "value": [[10, 20, 30], [2]], "coord::point": [[1,2], [3,4], [5,6], [7,8], [3,4], [5,6]], "names::string":["john", "eric", "judith", "mila", "hector", "maria"], "unique": True } Figure 2: Dataset example The schema in Figure 3 is valid with 'tab_data1' and 'tab_data2' formats tab_schema = { "fields": [ {"name":"index", "type":"integer", "constraint":{"minimum":50}}, {"name":"dates", "type":"date"}, {"name":"value", "type":"integer"}, {"name":"coord", "type":"geopoint", "format":"array"}, {"name":"names"}, {"name":"unique", "type":"boolean"} ] } THOMY Expires 21 June 2024 [Page 29] Internet-Draft NTV tabular format (NTV-TAB) December 2023 Figure 3: Schema example Acknowledgements TBD Contributors TBD Author's Address Philippe THOMY Loco-labs 476 chemin du gaf de Famian 84 500 BOLLENE France Email: philippe@loco-labs.io URI: https://github.com/loco-philippe/NTV/blob/main/README.md THOMY Expires 21 June 2024 [Page 30]