Data transformers are responsible for converting application types to and from IData types. They constitute the transformation layer between XIndex<TKey,TRecord> and XIndex table. All transformers look like:

 

    public interface IDataTransformer<T>
    {
        IData ToIData(T item);
        T FromIData(IData data);
        DataType DataType { get; }
    }

Each XIndex<TKey,TRecord> instance automatically generates two data transformers – one for the keys and one for the records. In that way every input TKey/TRecord instance is transformed to appropriate IData instance. Similarly, every IData instance from the XIndex is transformed back to TKey/TRecord. (For highly sophisticated types custom transformers can be provided.)

Suppose again that we have the following class:

 

    public class Tick
    {
        public string Symbol { get; set; }
        public DateTime Timestamp { get; set; }
        public double Bid { get; set; }
        public double Ask { get; set; }
        public long Volume { get; set; }
        public string Provider { get; set; }
    }

 

 

If we have XIndex<long, Tick>

 

IIndex<long, Tick> table = engine.OpenXIndex<long, Tick>("table");

 

Then the backend XIndex table will be opened with the following two types for the keys and for the records:

 

Data<long>
Data<string, DateTime, double, double, long, string>

 

We can open simultaneously the backend XIndex table:

 

    DataType keyType = DataType.Int64;

    DataType recordType = DataType.Slotes(
        DataType.String,
        DataType.DateTime,
        DataType.Double,
        DataType.Double,
        DataType.Int64,
        DataType.String);

    IIndex<IData, IData> table2 = engine.OpenXIndex(keyType, recordType, "table");

In this case table and table2 will refer to the same data.

If we decide to extend the Tick class with property of non-primitive type (Provider):

 

    public class Provider
    {
        public string Name { get; set; }
        public string Website { get; set; }
    }

    public class Tick
    {
        public string Symbol { get; set; }
        public DateTime Timestamp { get; set; }
        public double Bid { get; set; }
        public double Ask { get; set; }
        public long Volume { get; set; }
        public Provider Provider { get; set; }
    }

 

 

Then the backend XIndex will be opened with the types:

 

 Data<long>
 Data<string, DateTime, double, double, long, bool, string, string>

 

We can decide to exclude Symbol and Timestamp and use them as composite key:

public class Key
    {
        public string Symbol { get; set; }
        public DateTime Timestamp { get; set; }
    }
    
    public class Provider
    {
        public string Name { get; set; }
        public string Website { get; set; }
    }

    public class Tick
    {
        //public string Symbol { get; set; }
        //public DateTime Timestamp { get; set; }
        public double Bid { get; set; }
        public double Ask { get; set; }
        public long Volume { get; set; }
        public Provider Provider { get; set; }
    }

IIndex table = engine.OpenXIndex("table");

Then the backend XIndex will be with types:

Data<string, DateTime>

Data<double, double, long, bool, string, string>

Because data slots can be from primitive types only, an additional Boolean slot is added to indicate whether the Provider property of the current object is null. Thus we decompose user types.

Note: In the future releases slots will be extended to support IData recursively, so the additional slots will be eliminated.

If a XIndex uses composite keys with more than one slot in its Data, the engine compares sub-keys in BigEndian order – from small slot indexes to big one. (The engine automatically generates appropriate comparers for the relevant Data type.)

Transformed to IData user data is very suitable for compression. The engine again automatically generates compression classes for each XIndex type. These classes use parallel vertical compressions for each slot, depending on its primitive type. In that way the compressions are fast, while keeping high compression ratio. By default XIndex<TKey,TRecord> and XIndex keeps the data in compressed format.

Last edited Aug 17, 2013 at 10:56 AM by SvetoslavMateev, version 1

Comments

No comments yet.