Supporting your own format

In order to support your own serialization format, you need to implement a reader and a writer on top of whatever serialization libary you have chosen.

The reader and writer constitute a thin layer between the serialization library of your choice and reflect-cpp.

The requirements are laid down in the concepts IsReader and IsWriter and also documented below.

Using the reader and the writer you can define your parser like this:

template <class T>
using YourParser = rfl::parsing::Parser<YourReader, YourWriter, T>;

This can then be used to implement a read function and a write function:

using InputVarType = typename YourReader::InputVarType;
using OutputVarType = typename YourWriter::OutputVarType;

template <class T>
rfl::Result<T> read(const std::string& _str) {
    // This should be supported by whatever library you are
    // using for your format.
    const InputVarType root = str_to_input_var(_str);

    // You can pass variables to the constructor, if necessary
    const auto r = Reader(...); 

    return YourParser<T>::read(r, root);
}

template <class T>
std::string write(const T& _obj) {
    // You can pass variables to the constructor, if necessary
    auto w = Writer(...);

    OutputVarType var = Parser<T>::write(w, _obj);

    // This should be supported by whatever library you are
    // using for your format.
    return output_var_to_str(var);
}

In the following two sections, we will provide templates for your Reader and Writer. You should probably just copy + paste this into your own code and fill in the blanks.

As a reference, you can take a look at how this is done for JSON: https://github.com/getml/reflect-cpp/tree/main/include/rfl/json

Implementing your own writer

Because writers are somewhat simpler, we will start with them.

Any Writer needs to define the following types:

1) An OutputArrayType, which must be an array-like data structure. 2) An OutputObjectType, which must contain key-value pairs. 3) An OutputVarType, which must be able to represent either OutputArrayType, OutputObjectType or a basic type (bool, integral, floating point, std::string). We hesitate to call these "primitive types", because primitive types in C++ are defined as a slightly different group of types.

It also needs to support the following methods:

struct Writer {
    using OutputArrayType = ...;
    using OutputObjectType = ...;
    using OutputVarType = ...;

  /// Sets an empty array as the root element of the document.
  /// Some serialization formats require you to pass the expected size in
  /// advance. If you are not working with such a format, you can ignore the
  /// parameter `_size`. Returns the new array for further modification.
  OutputArrayType array_as_root(const size_t _size) const noexcept;

  /// Sets an empty object as the root element of the document.
  /// Some serialization formats require you to pass the expected size in
  /// advance. If you are not working with such a format, you can ignore the
  /// parameter `_size`.
  /// Returns the new object for further modification.
  OutputObjectType object_as_root(const size_t _size) const noexcept;

  /// Sets a null as the root element of the document. Returns OutputVarType
  /// containing the null value.
  OutputVarType null_as_root() const noexcept;

  /// Sets a basic value (bool, numeric, string) as the root element of the
  /// document. Returns an OutputVarType containing the new value.
  template <class T>
  OutputVarType value_as_root(const T& _var) const noexcept;

  /// Adds an empty array to an existing array. Returns the new
  /// array for further modification.
  OutputArrayType add_array_to_array(const size_t _size,
                                     OutputArrayType* _parent) const noexcept;

  /// Adds an empty array to an existing object. The key or name of the field is
  /// signified by `_name`. Returns the new array for further modification.
  OutputArrayType add_array_to_object(
      const std::string_view& _name, const size_t _size,
      OutputObjectType* _parent) const noexcept;

  /// Adds an empty object to an existing array. Returns the new
  /// object for further modification.
  OutputObjectType add_object_to_array(
      const size_t _size, OutputArrayType* _parent) const noexcept;

  /// Adds an empty object to an existing object. The key or name of the field
  /// is signified by `_name`. Returns the new object for further modification.
  OutputObjectType add_object_to_object(
      const std::string_view& _name, const size_t _size,
      OutputObjectType* _parent) const noexcept;

  /// Adds a basic value (bool, numeric, string) to an array. Returns an
  /// OutputVarType containing the new value.
  template <class T>
  OutputVarType add_value_to_array(const T& _var,
                                   OutputArrayType* _parent) const noexcept;

  /// Adds a basic value (bool, numeric, string) to an existing object. The key
  /// or name of the field is signified by `name`. Returns an
  /// OutputVarType containing the new value.
  template <class T>
  OutputVarType add_value_to_object(const std::string_view& _name, const T& _var,
                                    OutputObjectType* _parent) const noexcept;

  /// Adds a null value to an array. Returns an
  /// OutputVarType containing the null value.
  OutputVarType add_null_to_array(OutputArrayType* _parent) const noexcept;

  /// Adds a null value to an existing object. The key
  /// or name of the field is signified by `name`. Returns an
  /// OutputVarType containing the null value.
  OutputVarType add_null_to_object(const std::string_view& _name,
                                   OutputObjectType* _parent) const noexcept;

  /// Signifies to the writer that we do not want to add any further elements to
  /// this array. Some serialization formats require this. If you are working
  /// with a serialization format that doesn't, just leave the function empty.
  void end_array(OutputArrayType* _arr) const noexcept;

  /// Signifies to the writer that we do not want to add any further elements to
  /// this object. Some serialization formats require this. If you are working
  /// with a serialization format that doesn't, just leave the function empty.
  void end_object(OutputObjectType* _obj) const noexcept;
};

Implementing your own reader

Any Reader needs to define the following:

1) An InputArrayType, which must be an array-like data structure. 2) An InputObjectType, which must contain key-value pairs. 3) An InputVarType, which must be able to represent either InputArrayType, InputObjectType or a basic type (bool, integral, floating point, std::string). 4) A static constexpr bool has_custom_constructor<T>, that determines whether the class in question as a custom constructor, which might be called something like from_json_obj(...). If you do not want to support this functionality, just set it to false.

It also needs to support the following methods:

struct Reader {
    using InputArrayType = ...;
    using InputObjectType = ...;
    using InputVarType = ...;

    /// If you do not want to support custom constructors,
    /// just set this to false.
    template <class T>
    static constexpr bool has_custom_constructor = false;

    /// Retrieves a particular field from an array.
    /// Returns an rfl::Error if the index is out of bounds.
    /// If your format is schemaful, you do not need this.
    rfl::Result<InputVarType> get_field_from_array(
        const size_t _idx, const InputArrayType _arr) const noexcept {...}

    /// Retrieves a particular field from an object.
    /// Returns an rfl::Error if the field cannot be found.
    /// If your format is schemaful, you do not need this.
    rfl::Result<InputVarType> get_field_from_object(
        const std::string& _name, const InputObjectType& _obj) const noexcept {...}

    /// Determines whether a variable is empty (the NULL type).
    bool is_empty(const InputVarType& _var) const noexcept {...}

    /// Cast _var as a basic type (bool, integral,
    /// floating point, std::string).
    /// Returns an rfl::Error if it cannot be cast
    /// as that type
    template <class T>
    rfl::Result<T> to_basic_type(const InputVarType& _var) const noexcept {...}

    /// Casts _var as an InputArrayType.
    /// Returns an rfl::Error if `_var` cannot be cast as an array.
    rfl::Result<InputArrayType> to_array(const InputVarType& _var) const noexcept {...}

    /// Casts _var as an InputObjectType.
    /// Returns an rfl::Error if `_var` cannot be cast as an object.
    rfl::Result<InputObjectType> to_object(
        const InputVarType& _var) const noexcept {...}

    /// Iterates through an array and inserts the values into the array
    /// reader. See below for a more detailed explanation.
    template <class ArrayReader>
    std::optional<Error> read_array(const ArrayReader& _array_reader,
                                    const InputArrayType& _arr) const noexcept {...}

    /// Iterates through an object and inserts the key-value pairs into the object 
    /// reader. See below for a more detailed explanation.
    template <class ObjectReader>
    std::optional<Error> read_object(const ObjectReader& _object_reader,
                                     const InputObjectType& _obj) const noexcept {...}

    /// Constructs T using its custom constructor. This will only be triggered if
    /// T was determined to have a custom constructor by
    /// static constexpr bool has_custom_constructor, as defined above.
    /// Returns an rfl::Error, if the custom constructor throws an exception.
    template <class T>
    rfl::Result<T> use_custom_constructor(
        const InputVarType& _var) const noexcept {
        // If you do not want to support this functionality,
        // just return this.
        return rfl::Error("Not supported.");
    }
};

Of these methods, read_array and read_object probably require further explanation.

read_array

read_array expects an ArrayReader class which might come in several forms. But all of these forms have a method with the following signature:

std::optional<Error> read(const InputVarType& _var) const noexcept;

Within your implementation of read_array, you must iterate through the array passed to the function and then insert the resulting values into array_reader.read. If array_reader.read returns an error, then you must return that error immediately.

read_object

read_object expects an ObjectReader class which might come in several forms. But all of these forms have a method with the following signature:

void read(const std::string_view& _name,
          const InputVarType& _var) const noexcept;

Within your implementation of read_object, you must iterate through the object passed to the function and then insert the resulting key-value-pairs into object_reader.read.

Additional requirements for schemaful formats

Schemaful formats, like Apache Avro or Cap'n Proto, are somewhat more complicated than schemaless ones. There are additional factors to consider which do not apply schemaless formats:

1. Schemaful formats needs to differentiate between objects, for which the field names are known at compile time and maps, for which the field names are not known at compile time. In schemaless formats, there is no differentiation.

2. Schemaful formats needs an explicit union types. This also means that many of the problems we have with serializing std::variant which requires us to develop concepts like rfl::TaggedUnion simply do not apply to schemaful formats - the problem is already solved.

Additional requirements for schemaful writers

Any schemaful reader additionally needs to define the following:

1) An OutputMapType, which must contain key-value pairs. 2) An OutputUnionType, which represents an explicit union.

struct Writer {
  using OutputArrayType = ...;
  using OutputMapType = ...;
  using OutputObjectType = ...;
  using OutputUnionType = ...;
  using OutputVarType = ...;

  /// Sets an empty map as the root element of the document.
  /// Some serialization formats require you to pass the expected size in
  /// advance. If you are not working with such a format, you can ignore the
  /// parameter `size`. Returns the new array for further modification.
  OutputMapType map_as_root(const size_t _size) const noexcept;

  /// Sets an empty union as the root element of the document.
  OutputUnionType union_as_root() const noexcept;

  /// Adds an empty array to an existing map. Returns the new
  /// array for further modification.
  OutputArrayType add_array_to_map(const std::string_view& _name,
                                   const size_t _size,
                                   OutputMapType* _parent) const noexcept;

  /// Adds an empty array to an existing union.
  /// The index refers to the index of the element in the union.
  /// Returns the new array for further modification.
  OutputArrayType add_array_to_union(const size_t _index, const size_t _size,
                                     OutputUnionType* _parent) const noexcept;

  /// Adds an empty map to an existing array. Returns the new
  /// map for further modification.
  OutputMapType add_map_to_array(const size_t _size,
                                 OutputArrayType* _parent) const noexcept;

  /// Adds an empty map to an existing map. The key or name of the field
  /// is signified by `name`. Returns the new map for further modification.
  OutputMapType add_map_to_map(const std::string_view& _name,
                               const size_t _size,
                               OutputMapType* _parent) const noexcept;

  /// Adds an empty map to an existing object. The key or name of the field
  /// is signified by `name`. Returns the new map for further modification.
  OutputMapType add_map_to_object(const std::string_view& _name,
                                  const size_t _size,
                                  OutputObjectType* _parent) const noexcept;

  /// Adds an empty map to an existing union.
  /// The index refers to the index of the element in the union.
  /// Returns the new map for further modification.
  OutputMapType add_map_to_union(const size_t _index, const size_t _size,
                                 OutputUnionType* _parent) const noexcept;

  /// Adds an empty object to an existing map. The key or name of the field
  /// is signified by `name`. Returns the new object for further modification.
  OutputObjectType add_object_to_map(const std::string_view& _name,
                                     const size_t _size,
                                     OutputMapType* _parent) const noexcept;

  /// Adds an empty object to an existing union.
  /// The index refers to the index of the element in the union.
  /// Returns the new object for further modification.
  OutputObjectType add_object_to_union(const size_t _index, const size_t _size,
                                       OutputUnionType* _parent) const noexcept;

  /// Adds an empty union to an existing array. Returns the new
  /// union for further modification.
  OutputUnionType add_union_to_array(OutputArrayType* _parent) const noexcept;

  /// Adds an empty union to an existing map. The key or name of the field
  /// is signified by `name`. Returns the new union for further modification.
  OutputUnionType add_union_to_map(const std::string_view& _name,
                                   OutputMapType* _parent) const noexcept;

  /// Adds an empty union to an existing object. The key or name of the field
  /// is signified by `name`. Returns the new union for further modification.
  OutputUnionType add_union_to_object(const std::string_view& _name,
                                      OutputObjectType* _parent) const noexcept;

  /// Adds an empty union to an existing union.
  /// The index refers to the index of the element in the union.
  /// Returns the new union for further modification.
  OutputUnionType add_union_to_union(const size_t _index,
                                     OutputUnionType* _parent) const noexcept;

  /// Adds a null value to a map. Returns an
  /// OutputVarType containing the null value.
  OutputVarType add_null_to_map(const std::string_view& _name,
                                OutputMapType* _parent) const noexcept;

  /// Adds a null value to a union. Returns an
  /// OutputVarType containing the null value.
  OutputVarType add_null_to_union(const size_t _index,
                                  OutputUnionType* _parent) const noexcept;

  /// Adds a basic value (bool, numeric, string) to an existing map. The key
  /// or name of the field is signified by `name`. Returns an
  /// OutputVarType containing the new value.
  template <class T>
  OutputVarType add_value_to_map(const std::string_view& _name, const T& _var,
                                 OutputMapType* _parent) const noexcept; 

  /// Adds a basic value (bool, numeric, string) to an existing union. The key
  /// or name of the field is signified by `name`. Returns an
  /// OutputVarType containing the new value.
  template <class T>
  OutputVarType add_value_to_union(const size_t _index, const T& _var,
                                   OutputUnionType* _parent) const noexcept;

  /// Signifies to the writer that we do not want to add any further elements to
  /// this map. Some serialization formats require this. If you are working
  /// with a serialization format that doesn't, just leave the function empty.
  void end_map(OutputMapType* _obj) const noexcept; 

};

Additional requirements for schemaful readers

Any schemaful reader additionally needs to define the following:

1) An InputMapType, which must contain key-value pairs. 2) An InputUnionType, which represents an explicit union.

struct Reader {
  using InputArrayType = ...;
  using InputObjectType = ...;
  using InputMapType = ...;
  using InputUnionType = ...;
  using InputVarType = ...;

  /// A schemaful reader needs to differentiate between objects, for which
  /// the field names are known at compile time and maps, for which the
  /// field names are not known at compile time.
  rfl::Result<InputMapType> to_map(const InputVarType& _var) const noexcept;

  /// read_map works exactly the same as read_object in schemaless formats.
  template <class MapReader>
  std::optional<Error> read_map(const MapReader& _map_reader,
                                const InputMapType& _map) const noexcept;

  /// A schemaful reader needs an explicit union type.
  rfl::Result<InputUnionType> to_union(const InputVarType& _var) const noexcept;

  /// read_union needs to be able to take an InputUnionType and return the corresponding
  /// variant (like std::variant or rfl::Variant). 
  template <class VariantType, class UnionReaderType>
  rfl::Result<VariantType> read_union(
      const InputUnionType& _union) const noexcept;
};