Astra Engine

Appearance

Streams Module ​

The Streams module provides a flexible serialization and deserialization framework for data persistence. It abstracts data sources and formats, allowing Astra to save and load game data, configurations, and scenes.

Purpose ​

The Streams module provides:

  • Unified serialization and deserialization abstraction
  • JSON data format support
  • File and memory stream adapters
  • Context-based serialization API with operator overloading
  • Type-safe data reading and writing via template methods

Key Components ​

SerializationContext ​

The main interface for serializing and deserializing data. Unlike many frameworks, there is no separate DeserializationContext - SerializationContext handles both directions:

cpp
class SerializationContext {
public:
    virtual ~SerializationContext() = default;

    // Key access for hierarchical data
    virtual ContextProxy operator[](const SerializableKey &key) = 0;

    // Set values (write)
    virtual void set_value(const SerializableValue &value) = 0;
    virtual void set_value(Ref<SerializationContext> ctx) = 0;

    // Type introspection
    virtual bool is_string() = 0;
    virtual bool is_int() = 0;
    virtual bool is_float() = 0;
    virtual bool is_bool() = 0;
    virtual bool is_array() = 0;
    virtual bool is_object() = 0;

    // Get values (read)
    virtual std::string as_string() = 0;
    virtual int as_int() = 0;
    virtual float as_float() = 0;
    virtual bool as_bool() = 0;
    virtual std::vector<std::any> as_array() = 0;

    // Factory methods
    static Ref<SerializationContext> create(SerializationFormat format);
    static Ref<SerializationContext> create(SerializationFormat format, Scope<StreamBuffer> buffer);
};

ContextProxy ​

Returned by operator[] to enable chained key access:

cpp
class ContextProxy {
public:
    ContextProxy operator[](const SerializableKey &sub_key);
    void operator=(const SerializableValue &value);
    void operator=(Ref<SerializationContext> ctx);

    template <typename T> T as();
    SerializationTypeKind kind();
    size_t size();
};

Supports natural syntax like:

cpp
ctx["player"]["health"] = 100;
int health = ctx["player"]["health"].as<int>();

StreamBuffer ​

Arena-backed memory buffer for stream I/O:

cpp
struct StreamBuffer {
    StreamBuffer(size_t capacity);

    void write(char *src, size_t size);
    void reset();
    char *data();
    size_t size() const;
};

StreamReader / StreamWriter ​

Base classes for different I/O adapters:

StreamReader - Abstract base for reading into buffers:

cpp
class StreamReader {
public:
    virtual void read() = 0;
    Scope<StreamBuffer> get_buffer();
};

StreamWriter - Abstract base for writing from buffers:

cpp
class StreamWriter {
public:
    StreamWriter(Scope<StreamBuffer> buffer);
    virtual void write() = 0;
    virtual void flush() = 0;
};

Concrete Adapters ​

FileStreamReader - Reads from files:

cpp
FileStreamReader reader("/path/to/file.json");
reader.read();
auto buffer = reader.get_buffer();

FileStreamWriter - Writes to files:

cpp
FileStreamWriter writer("/path/to/file.json", buffer);
writer.write();
writer.flush();

MemoryStreamWriter - Writes to memory (Note: MemoryStreamReader is not yet implemented):

cpp
MemoryStreamWriter writer;
// Use writer's buffer

JsonSerializationContext ​

The only currently implemented serialization format:

cpp
class JsonSerializationContext : public SerializationContext {
public:
    JsonSerializationContext();
    JsonSerializationContext(Scope<StreamBuffer> buffer);

    // Implements all SerializationContext methods
    ContextProxy operator[](const SerializableKey &key) override;
    // ... other methods
};

Architecture ​

Three-Layer Design ​

The Streams module uses a three-layer architecture:

SerializationContext (Format abstraction - JSON, etc.)
    ↓ to_buffer() / from_buffer()
StreamBuffer (Arena-backed memory)
    ↓ read() / write()
StreamReader/StreamWriter (I/O abstraction - File, Memory, etc.)

Type-Safe API with Operator Overloading ​

The Streams module provides type-safe serialization via operator[] and template methods:

cpp
// Create context
auto ctx = SerializationContext::create(SerializationFormat::Json);

// Write data using operator[]
ctx["health"] = 100;                    // int
ctx["speed"] = 5.5f;                    // float
ctx["name"] = std::string("Player");    // string
ctx["active"] = true;                   // bool

// Read data using as<T>()
int health = ctx["health"].as<int>();
float speed = ctx["speed"].as<float>();
std::string name = ctx["name"].as<std::string>();
bool active = ctx["active"].as<bool>();

Hierarchical Data ​

Support for nested objects via chained operator[]:

cpp
// Writing nested data
ctx["player"]["name"] = std::string("Hero");
ctx["player"]["stats"]["level"] = 5;
ctx["player"]["stats"]["health"] = 100;

// Reading nested data
std::string name = ctx["player"]["name"].as<std::string>();
int level = ctx["player"]["stats"]["level"].as<int>();
int health = ctx["player"]["stats"]["health"].as<int>();

Key Features ​

JSON Serialization ​

Write and read JSON data using the operator-based API:

cpp
// Create a JSON context
auto ctx = SerializationContext::create(SerializationFormat::Json);

// Write data
ctx["config"]["windowWidth"] = 1920;
ctx["config"]["windowHeight"] = 1080;
ctx["config"]["fullscreen"] = false;

// Serialize to file
FileStreamWriter writer("config.json", ctx->to_buffer(arena));
writer.write();
writer.flush();

Reading JSON Files ​

Load and deserialize JSON data:

cpp
// Read file into buffer
FileStreamReader reader("config.json");
reader.read();

// Create context from buffer
auto ctx = SerializationContext::create(SerializationFormat::Json, reader.get_buffer());

// Read values
int width = ctx["config"]["windowWidth"].as<int>();
int height = ctx["config"]["windowHeight"].as<int>();
bool fullscreen = ctx["config"]["fullscreen"].as<bool>();

Object Serialization ​

Serialize custom types by inheriting from Serializer:

cpp
class PlayerSerializer : public Serializer {
    std::string name;
    int level;
    float health;

public:
    void serialize() override {
        m_ctx["player"]["name"] = name;
        m_ctx["player"]["level"] = level;
        m_ctx["player"]["health"] = health;
    }

    void deserialize() override {
        name = m_ctx["player"]["name"].as<std::string>();
        level = m_ctx["player"]["level"].as<int>();
        health = m_ctx["player"]["health"].as<float>();
    }
};

Array Serialization ​

Handle arrays using std::vector:

cpp
// Note: Array serialization requires storing as nested contexts
// The current implementation doesn't provide a simple vector write API
// Arrays are accessed via the as_array() method which returns std::vector<std::any>

// Reading arrays
if (ctx["scores"].is_array()) {
    auto scores = ctx["scores"].as_array();
    for (const auto& score : scores) {
        // Process each element
    }
}

Memory Streams ​

Work with in-memory data (Note: MemoryStreamReader is not yet fully implemented):

cpp
// Write to memory
MemoryStreamWriter writer;
// Use writer's internal buffer

// Note: Full round-trip memory serialization requires
// a complete MemoryStreamReader implementation

Integration with Other Modules ​

With Project Module ​

The Project module uses ProjectSerializer to load and save project descriptor files. The serialization format is determined by ProjectConfig::serialization.format:

cpp
// ProjectSerializer reads/writes project.astra files
class ProjectSerializer : public Serializer {
    void serialize() override {
        m_ctx["name"] = project_config.name;
        m_ctx["directory"] = project_config.directory;
        m_ctx["resources"]["directory"] = project_config.resources.directory;
        // ... more fields
    }

    void deserialize() override {
        project_config.name = m_ctx["name"].as<std::string>();
        project_config.directory = m_ctx["directory"].as<std::string>();
        // ... more fields
    }
};

With Engine Module ​

Engine components can use Streams for scene persistence by creating custom serializers:

cpp
class SceneSerializer : public Serializer {
public:
    void serialize() override {
        // Serialize scene entities
        m_ctx["scene"]["name"] = scene_name;
        // Serialize each entity...
    }

    void deserialize() override {
        scene_name = m_ctx["scene"]["name"].as<std::string>();
        // Deserialize entities...
    }
};

With Editor Module ​

The Editor module can use Streams to save editor preferences:

cpp
class EditorPreferencesSerializer : public Serializer {
    void serialize() override {
        m_ctx["preferences"]["theme"] = m_Theme;
        m_ctx["preferences"]["autoSave"] = m_AutoSave;
        // ... more preferences
    }

    void deserialize() override {
        m_Theme = m_ctx["preferences"]["theme"].as<std::string>();
        m_AutoSave = m_ctx["preferences"]["autoSave"].as<bool>();
    }
};

Supported Types ​

The Streams module natively supports the following types via SerializableValue:

  • int - Integer values
  • float - Floating point values
  • std::string - String values
  • bool - Boolean values
  • Arrays - Via std::vector<std::any> returned by as_array()
  • Custom Types - Via custom Serializer subclasses

Note: Complex types like glm::vec3, glm::quat, glm::mat4 are NOT natively supported. You must serialize them as separate components (e.g., vec3 as three floats) or implement custom serialization logic.

File Locations ​

/src/modules/streams/
├── serialization-context.hpp      # Abstract serialization interface
├── serialization-context.cpp
├── context-proxy.hpp              # Chained key access proxy
├── context-proxy.cpp
├── serializer.hpp                 # Base serializer class
├── serializer.cpp
├── stream-reader.hpp              # Abstract stream reader
├── stream-writer.hpp              # Abstract stream writer
├── stream-buffer.hpp              # Arena-backed memory buffer
├── adapters/
│   ├── file/
│   │   ├── file-stream-reader.hpp
│   │   ├── file-stream-reader.cpp
│   │   ├── file-stream-writer.hpp
│   │   └── file-stream-writer.cpp
│   ├── memory/
│   │   ├── memory-stream-reader.hpp   # Not yet implemented
│   │   ├── memory-stream-writer.hpp
│   │   └── memory-stream-writer.cpp
│   └── json/
│       ├── json-serialization-context.hpp
│       └── json-serialization-context.cpp

Note: No binary format support currently exists. Only JSON is implemented.

  • Project - Uses Streams for project files
  • Engine - Uses Streams for scene serialization
  • Editor - Uses Streams for editor preferences

Best Practices ​

  1. Type Checking: Use is_*() methods to verify types before calling as<T>()
  2. Error Handling: Wrap deserialization in try-catch blocks (uses ASTRA_ENSURE macros internally)
  3. Versioning: Include version numbers in serialized data for compatibility
  4. Validation: Validate data after deserialization
  5. Flush Streams: Always call flush() on StreamWriter after writing
  6. Arena Management: When using to_buffer(), ensure the arena outlives the buffer
  7. Key Access: Use operator[] for hierarchical access rather than flattening keys

Performance Considerations ​

  • File I/O is slower than memory operations
  • JSON parsing has overhead (binary format not yet implemented)
  • StreamBuffer uses ElasticArena for efficient memory allocation
  • JsonSerializationContext maintains a stack for navigation; deep nesting has overhead
  • FileStreamReader loads entire file into memory at once
  • Consider serialization frequency; avoid serializing every frame

Common Patterns ​

Versioned Serialization ​

Support multiple data versions:

cpp
class SaveDataSerializer : public Serializer {
    int version = 2;
    int newField;
    int oldField;

public:
    void serialize() override {
        m_ctx["save"]["version"] = version;
        m_ctx["save"]["newField"] = newField;
        m_ctx["save"]["oldField"] = oldField;
    }

    void deserialize() override {
        int version = m_ctx["save"]["version"].as<int>();

        if (version >= 2) {
            newField = m_ctx["save"]["newField"].as<int>();
        }

        oldField = m_ctx["save"]["oldField"].as<int>();
    }
};

Optional Fields ​

Handle missing fields gracefully using type checking:

cpp
void deserialize() override {
    // Required field
    m_RequiredValue = m_ctx["data"]["required"].as<int>();

    // Optional field with default
    if (m_ctx["data"]["optional"].is_int()) {
        m_OptionalValue = m_ctx["data"]["optional"].as<int>();
    } else {
        m_OptionalValue = defaultValue;
    }
}

Batch Serialization ​

Serialize multiple objects efficiently:

cpp
class EntityBatchSerializer : public Serializer {
    std::vector<Entity> entities;

public:
    void serialize() override {
        // Note: Current API doesn't provide direct array serialization
        // You may need to serialize as individual indexed keys
        for (size_t i = 0; i < entities.size(); i++) {
            m_ctx["entities"][std::to_string(i)]["id"] = entities[i].id;
            m_ctx["entities"][std::to_string(i)]["name"] = entities[i].name;
        }
    }
};

Custom Type Serialization ​

Make custom types serializable (remember: only int, float, string, bool are natively supported):

cpp
struct Transform {
    float pos_x, pos_y, pos_z;
    float rot_x, rot_y, rot_z, rot_w;
    float scale_x, scale_y, scale_z;

    void serialize(Ref<SerializationContext> ctx) const {
        // Must serialize components individually
        (*ctx)["transform"]["position"]["x"] = pos_x;
        (*ctx)["transform"]["position"]["y"] = pos_y;
        (*ctx)["transform"]["position"]["z"] = pos_z;

        (*ctx)["transform"]["rotation"]["x"] = rot_x;
        (*ctx)["transform"]["rotation"]["y"] = rot_y;
        (*ctx)["transform"]["rotation"]["z"] = rot_z;
        (*ctx)["transform"]["rotation"]["w"] = rot_w;

        (*ctx)["transform"]["scale"]["x"] = scale_x;
        (*ctx)["transform"]["scale"]["y"] = scale_y;
        (*ctx)["transform"]["scale"]["z"] = scale_z;
    }

    void deserialize(Ref<SerializationContext> ctx) {
        pos_x = (*ctx)["transform"]["position"]["x"].as<float>();
        pos_y = (*ctx)["transform"]["position"]["y"].as<float>();
        pos_z = (*ctx)["transform"]["position"]["z"].as<float>();

        rot_x = (*ctx)["transform"]["rotation"]["x"].as<float>();
        rot_y = (*ctx)["transform"]["rotation"]["y"].as<float>();
        rot_z = (*ctx)["transform"]["rotation"]["z"].as<float>();
        rot_w = (*ctx)["transform"]["rotation"]["w"].as<float>();

        scale_x = (*ctx)["transform"]["scale"]["x"].as<float>();
        scale_y = (*ctx)["transform"]["scale"]["y"].as<float>();
        scale_z = (*ctx)["transform"]["scale"]["z"].as<float>();
    }
};