How to setup this complex configuration project with Pkl?

[gian-didom-emovo]

Hello and thanks eveeryone for giving this one a read.

I'm in charge of defining a configuration framework for a complex project, in which multiple heterogeneous nodes co-exist and each one has its own set of configuration files in different formats (JSON, Yaml, etc).

These configurations often share multiple parameters, and have unique sets of parameters as well. It is important to keep coherence between the shared parameters. Moreover, there are constraints across parameters, even among different nodes.

I tought of using Pkl for its capabilities of rendering multiple formats and handling constraints. However, I am struggling to set up the architecture of this project, probably for my inexperience with Pkl.

I tought of three layers:

1. First layer (libraries)
A first layers is mapped at device level. It contains, for every device, the declarations of the needed parameters and the definition of the output formats. I would like not to contain any definition in order to force users to define in order to get an output. For instance, I could have:

module obc

import "obc_network.pkl" // contains obc.network class definitions
import "obc_algorithms.pkl"  // contains obc.algorithms class definitions

obcNetwork : OBCNetwork

obcAlgorithms : OBCAlgorithms

output {

files {

[network.json] {
...
}

[algorithms.json] {
...
}

Apart from this, I also include general domain libraries, such as definition of network-related classes (Host, Endpoint, etc.)
At this point, the folder structure looks like

lib
|-- device1
|-- device2
|-- ...
|-- deviceN 
|
!-- network
|-- simulation
|-- ...

2. Second layer (templates)
A second layer is a template layer which defines the mappings between the domain-related libraries and the device-specific libraries. As an instance, I could define the network field of the obc configuration starting from the network library definitions, as:

import ".../lib/network/network.pkl" as Network // Network definitions
import "../lib/obc/obc.pkl" // OBC specific definitions

/* ... */

hidden subsystemEndpoint : Network.Host
fixed subsystemHostname : String = subsystemEndpoint.hostname
fixed subsystemPort : Int(this>0) = subsystemEndpoint.port

The idea is that, when amending this template, configuration usres only need to setup the subsystemEndpoint property, not the hostname and port, ensuring the hostname and port will always match the ones defined in the network configuration.

3. Thirst layer (user-defined configuration)
The third and final layer is the one directly editable from users, and is an amend of the previous layer. I would act as follows:

amends ".../templates/obc/obc.pkl"
import ".../impl/network/endpoints.pkl" as Endpoints

subsystemEndpoint = Endpoints.subsystem

From what I understood, the properties to be rendered in the output (i.e., subsystemHostname and subsystemPort would be automatically set due to the late-binding properties of Pkl.

Issues
On paper, this looks to me pretty clean and elegant. However, there are some issues I'm struggling with:

1. Outputs
I'd like to have the outputs defined in the first level, as there are device-specific and, in the future, could be directly provided from the developers working on that device or service. However, if I define the output property in the top-level library and run pkl eval with the -m flag of the third layer file, I simple get no output file, or empty variables.

I tried to inherit the output configuration in the template layer, i.e.,

/// templates/obc/obc.pkl

/*... */

output = import(".../lib/obc/obc.pkl).output

But I get an error that tells me that the top-level properties have not been defined. It seems Pkl is trying to render directly the output of the top-level module. So, I tought of changing the output property from the lib-level file to another name (e.g., _output not to trigger rendering, and assign it in the template-level file as:

/// templates/obc/obc.pkl

/* ... */

output = import(".../lib/obc/obc.pkl")._output

But it seems that the type mismatch for the inner porperty are not trivially resolved, as I get a bunch of Expected value of type 'Mapping', but got type 'Dynamic'., 'Expected value of type FileOutput, but got type Dynamic.', and so on. Even if I use typed properties in the top-level file, it seems Pkl is matching the values of the output files entries to the ones in the lib folder, which only contains declarations.

It seems the only way to avoid this is to have the template as an amend of the lib file; however, this prevents me to have hidden properties, fixed properties, and also typed annotations and constraint, which I need to enforce down to the implementation level (third level).

I am pretty confused as of now, and therefore I make a step back. Is this a XY problem? Is there a better project configuration to achieve what I'm looking for? In summary, I have the following requirements:

1. Independent declarations of device-specific constraints, properties, and output file mapping
2. An application-specific set of constraints and mappings to transform the complex, fragmented device-level configurations in a simpler, domain-specific one, which shall be hidden from the final user
3. An exposed layer which allows users to obtain the output configuration files of 1. by setting domain-specific properties, explotiing the mappings defined in 2.

Is there a suggested architecture or language features I'm missing to obtain these?

[bioball]

Your overall design makes sense to me!

This code seems incorrect, because .../lib/obc/obc.pkl is your base template, not where your data is defined:

output = import(".../lib/obc/obc.pkl").output

You likely want to be extending obc.pkl instead.

By the way, I don't know if you need a second layer? You can just one one base layer, and then amend it:

module obc

import "obc_network.pkl" // contains obc.network class definitions
import "obc_algorithms.pkl"  // contains obc.algorithms class definitions

// define your input properties
subsystemEndpoint: Network.Host

// ...

// then define your derived "output" properties
fixed subsystemHostname : String = subsystemEndpoint.hostname
fixed subsystemPort : Int(this>0) = subsystemEndpoint.port

fixed obcNetwork: OBCNetwork = new {
  endpoint = subsystemEndpoint
}

fixed obcAlgorithms: OBCAlgorithms = new {
  endpoint = subsystemEndpoint
}

// ...

// the define the output of your module
output {
  files {
    ["network.json"] = obcNetwork.output

    ["algorithms.json"] = obcAlgorithms.output
  }
}

But it seems that the type mismatch for the inner porperty are not trivially resolved, as I get a bunch of Expected value of type 'Mapping', but got type 'Dynamic'., 'Expected value of type FileOutput, but got type Dynamic.', and so on. Even if I use typed properties in the top-level file, it seems Pkl is matching the values of the output files entries to the ones in the lib folder, which only contains declarations.

This looks like a simple bug in your code; you are assigning the output to be a Dynamic value. But, without looking at a reproducer, I don't know exactly where the source of your bug is.

[gian-didom-emovo]

It seemed the extends keywork was the key here (pun intended). Thanks!

I seemed to obtain nice results with the second layer extending the first one, and the last one amending the second one. In theory, I could make it with one base layer, but that would result in a bit of cross-dependencies between modules, which won't be handled nicely in a future scenario in which every developer writes down its configuration and outputs for their module.