Update variable replacement during deserialization to use replacement settings class and add AKV replacement logic. #2882

aaronburtle · 2025-09-18T01:39:47Z

Why make this change?

Adds AKV variable replacement and expands our design for doing variable replacements to be more extensible when new variable replacement logic is added.

Closes #2708
Closes #2748
Related to #2863

What is this change?

Change the way that variable replacement is handled to instead of simply using a bool to indicate that we want env variable replacement, we add a class which holds all of the replacement settings. This will hold whether or not we will do replacement for each kind of variable that we will handle replacement for during deserialization. We also include the replacement failure mode, and put the logic for handling the replacements into a strategy dictionary which pairs the replacement variable type with the strategy for doing that replacement.

Because Azure Key Vault secret replacement requires having the retry and connection settings in order to do the AKV replacement, we must do a first pass where we only do non-AKV replacement and get the required settings so that if AKV replacement is used we have the required settings to do that replacement.

We also have to keep in mind that the legacy of the Configuration Controller will ignore all variable replacement, so we construct the replacement settings for this code path to not use any variable replacement at all.

How was this tested?

We have updated the logic for the tests to use the new system, however manual testing using an actual AKV is still required.

Sample Request(s)

Example REST and/or GraphQL request to demonstrate modifications
Example of CLI usage to demonstrate modifications

src/Config/Converters/AKVRetryPolicyOptionsConverterFactory.cs

src/Config/Converters/EntityCacheOptionsConverterFactory.cs

src/Config/Converters/StringJsonConverterFactory.cs

src/Config/FileSystemRuntimeConfigLoader.cs

src/Config/RuntimeConfigLoader.cs

RubenCerna2079

Still need to get rid of envVar in some places and resolve missing logic inside of an if statement.

src/Config/Converters/AzureKeyVaultOptionsConverterFactory.cs

src/Config/Converters/StringJsonConverterFactory.cs

src/Config/Converters/Utf8JsonReaderExtensions.cs

src/Config/RuntimeConfigLoader.cs

src/Config/DeserializationVariableReplacementSettings.cs

src/Config/RuntimeConfigLoader.cs

src/Config/DeserializationVariableReplacementSettings.cs

src/Config/RuntimeConfigLoader.cs

souvikghosh04

added comments

src/Config/RuntimeConfigLoader.cs

src/Config/DeserializationVariableReplacementSettings.cs

Aniruddh25

The changes look good to me. This is a good refactor. However, unit testing still remains and so does the test in AKV environment.

src/Config/RuntimeConfigLoader.cs

src/Service.Tests/Configuration/ConfigurationTests.cs

Aniruddh25 · 2025-10-30T01:45:01Z

src/Service.Tests/UnitTests/RuntimeConfigLoaderJsonDeserializerTests.cs

                {
                    Assert.IsTrue(RuntimeConfigLoader.TryParseConfig(
-                        GetModifiedJsonString(repValues, @"""postgresql"""), out expectedConfig, replaceEnvVar: replaceEnvVar),
+                        GetModifiedJsonString(repValues, @"""postgresql"""), out expectedConfig, replacementSettings: new DeserializationVariableReplacementSettings(azureKeyVaultOptions: null, doReplaceEnvVar: replaceEnvVar, doReplaceAKVVar: true)),


We should add similar tests for AKV replacement checks

src/Config/Converters/AzureKeyVaultOptionsConverterFactory.cs

src/Service.Tests/UnitTests/MySqlQueryExecutorUnitTests.cs

src/Service.Tests/UnitTests/SqlQueryExecutorUnitTests.cs

src/Service.Tests/UnitTests/PostgreSqlQueryExecutorUnitTests.cs

Copilot

Pull Request Overview

This PR refactors the configuration deserialization system to introduce Azure Key Vault (AKV) variable replacement support alongside environment variable replacement. The key change is replacing individual boolean parameters (replaceEnvVar, replacementFailureMode) with a unified DeserializationVariableReplacementSettings object that encapsulates all variable replacement configuration, including the new AKV functionality.

Introduces DeserializationVariableReplacementSettings class to centralize variable replacement configuration
Adds Azure Key Vault secret resolution support with configurable retry policies
Updates all JSON converters and configuration loading methods to use the new settings object

Reviewed Changes

Copilot reviewed 33 out of 33 changed files in this pull request and generated 15 comments.

Show a summary per file

File	Description
`src/Config/DeserializationVariableReplacementSettings.cs`	New class that encapsulates variable replacement logic for both environment variables and Azure Key Vault secrets
`src/Config/RuntimeConfigLoader.cs`	Updated to use new settings object and extract AKV options with two-pass parsing
`src/Config/Converters/StringJsonConverterFactory.cs`	Refactored to use pluggable replacement strategies from settings object
`src/Config/Converters/*ConverterFactory.cs`	Updated multiple converter factories to accept `DeserializationVariableReplacementSettings` instead of boolean parameters
`src/Config/ObjectModel/AzureKeyVaultOptions.cs`	Added constructor and flags to track user-provided properties
`src/Directory.Packages.props`	Added Azure.Security.KeyVault.Secrets package dependency
`src/Service/Controllers/ConfigurationController.cs`	Updated Initialize call with new settings object
Test files	Updated test code to use new settings object instead of boolean parameters

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

src/Config/RuntimeConfigLoader.cs

src/Config/DeserializationVariableReplacementSettings.cs

src/Config/Converters/DatasourceHealthOptionsConvertorFactory.cs

src/Config/Converters/EntityCacheOptionsConverterFactory.cs

src/Config/Converters/StringJsonConverterFactory.cs

src/Config/Converters/RuntimeHealthOptionsConvertorFactory.cs

src/Config/RuntimeConfigLoader.cs

src/Config/DeserializationVariableReplacementSettings.cs

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

Aniruddh25 · 2025-10-31T22:33:23Z

src/Cli/Exporter.cs

-                        ExportGraphQL(options, runtimeConfig, fileSystem, loader, logger).Wait();
-                        isSuccess = true;
-                        break;
+                        if (runtimeConfig is not null)


Why this change? The fact that TryLoadConfig returned true means runtimeConfig is going to be not null. Why do we need to check again here?

Aniruddh25 · 2025-10-31T22:34:43Z

src/Cli.Tests/EndToEndTests.cs


-        SqlConnectionStringBuilder builder = new(runtimeConfig.DataSource.ConnectionString);
-        Assert.AreEqual(ProductInfo.GetDataApiBuilderUserAgent(), builder.ApplicationName);
+        if (runtimeConfig is not null)


Same here, we are already Assert for true above. If the assert fails, it will error out. so runtimeConfig can never be null here.

Aniruddh25

Need to carefully look at what the default value of replaceEnvVar property should be when replaceMentSettings is null or new(). And ensure no regressions

Aniruddh25 · 2025-10-31T22:37:38Z

src/Config/Converters/AzureKeyVaultOptionsConverterFactory.cs

+
+using System.Text.Json;
+using System.Text.Json.Serialization;
+using Azure.DataApiBuilder.Config.ObjectModel;


This seems to be an unnecessary using statement

Aniruddh25 · 2025-10-31T22:38:40Z

src/Config/Converters/AzureKeyVaultOptionsConverterFactory.cs

+        // value or not while deserializing.
+        private readonly DeserializationVariableReplacementSettings? _replacementSettings;
+
+        /// <param name="replaceEnvVar">Whether to replace environment variable with its


Suggested change

/// <param name="replaceEnvVar">Whether to replace environment variable with its

/// <param name="replacementSettings">Whether to replace environment variable with its

Aniruddh25 · 2025-10-31T22:40:07Z

src/Config/Converters/AzureKeyVaultOptionsConverterFactory.cs

+                        case "retry-policy":
+                            if (reader.TokenType is JsonTokenType.StartObject)
+                            {
+                                // Pass the replaceEnvVar setting to the retry policy converter


how is this setting passed to the retry policy converter?

Aniruddh25 · 2025-10-31T22:49:06Z

src/Config/DeserializationVariableReplacementSettings.cs

+        // .*          :  any char except newline any number of times
+        // (?=\))      :  look ahead for end char of )
+        // This pattern greedy matches all characters that are not a part of @env()
+        // ie: @env('hello@env('goodbye')world') match: 'hello@env('goodbye')world'


In the match here, the substring @env( is a substring but the pattern [^@env\(] expects any substring that is not @env(. so how did it match?

Aniruddh25 · 2025-10-31T23:29:28Z

src/Config/FileSystemRuntimeConfigLoader.cs

        {
            // Need to remove the dependencies in startup on the RuntimeConfigProvider
-            // before we can have an ILogger here.
+            // before we can have anILogger here.


nit:

Suggested change

// before we can have anILogger here.

// before we can have an ILogger here.

Aniruddh25 · 2025-10-31T23:32:17Z

src/Config/FileSystemRuntimeConfigLoader.cs


-            if (!string.IsNullOrEmpty(json) && TryParseConfig(json, out RuntimeConfig, connectionString: _connectionString, replaceEnvVar: replaceEnvVar))
+            // Use default replacement settings if none provided
+            replacementSettings ??= new DeserializationVariableReplacementSettings(azureKeyVaultOptions: null, doReplaceEnvVar: true, doReplaceAkvVar: true);


Default for this function was to keep replaceEnvVar to false. But, here I see it is true, why is this change needed and if needed, did we ensure all the callers are updated accordingly?

Aniruddh25 · 2025-10-31T23:34:58Z

src/Config/ObjectModel/RuntimeConfig.cs

-                        _dataSourceNameToDataSource = _dataSourceNameToDataSource.Concat(config._dataSourceNameToDataSource).ToDictionary(kvp => kvp.Key, kvp => kvp.Value);
-                        _entityNameToDataSourceName = _entityNameToDataSourceName.Concat(config._entityNameToDataSourceName).ToDictionary(kvp => kvp.Key, kvp => kvp.Value);
-                        allEntities = allEntities.Concat(config.Entities.AsEnumerable());
+                        if (config is not null)


why do we need to check for config is not null when TryLoadConfig is true?

Aniruddh25 · 2025-10-31T23:39:46Z

src/Service.Tests/Configuration/ConfigurationTests.cs

        {
            string configJson = TestHelper.AddPropertiesToJson(TestHelper.BASE_CONFIG, entityJson);
-            RuntimeConfigLoader.TryParseConfig(configJson, out RuntimeConfig deserializedConfig, logger: null, GetConnectionStringFromEnvironmentConfig(environment: TestCategory.MSSQL));
+            RuntimeConfigLoader.TryParseConfig(


for TryParseConfig, the default value of replaceEnvVar was false. But with the addition of the replacementSettings, the default value for this call ends up being true. This will break the previous assumptions. In case of refactors, the best idea is to maintain previous behavior to avoid regressions.

Aniruddh25 · 2025-10-31T23:43:55Z

/azp run

azure-pipelines · 2025-10-31T23:44:16Z

Azure Pipelines successfully started running 6 pipeline(s).

aaronburtle added 13 commits June 20, 2025 15:55

allow unused cache-control options without error

2fc665d

Merge branch 'main' of github.com:Azure/data-api-builder

50e54aa

Merge branch 'main' of github.com:Azure/data-api-builder

455ff18

Merge branch 'main' of github.com:Azure/data-api-builder

e0574bf

Merge branch 'main' of github.com:Azure/data-api-builder

033eeb0

Merge branch 'main' of github.com:Azure/data-api-builder

c7ad08f

...

049b23e

extend variable replacment and include AKV

cfdc2b5

change some logic for first pass var replacement

5eb9f0e

fix old bools being used

bbcc159

more bool replacements

f6f92bd

refactor out bools

0dd7ad6

Merge branch 'main' of github.com:Azure/data-api-builder

802eb53

RubenCerna2079 reviewed Oct 7, 2025

View reviewed changes

RubenCerna2079 requested changes Oct 7, 2025

View reviewed changes

souvikghosh04 requested changes Oct 8, 2025

View reviewed changes

src/Config/RuntimeConfigLoader.cs Show resolved Hide resolved

src/Config/RuntimeConfigLoader.cs Show resolved Hide resolved

src/Config/DeserializationVariableReplacementSettings.cs Outdated Show resolved Hide resolved

aaronburtle added 3 commits October 9, 2025 13:29

Merge branch 'main' of github.com:Azure/data-api-builder

4e71c25

addressing comments

ac4356e

Merge branch 'main' of github.com:Azure/data-api-builder

c3dabe3

Aniruddh25 requested changes Oct 30, 2025

View reviewed changes

aaronburtle added 4 commits October 30, 2025 09:35

Merge branch 'main' of github.com:Azure/data-api-builder

6e77297

update names

3254af5

cleanup with fix

b28cd6b

cleanup

5e79339

Aniruddh25 reviewed Oct 31, 2025

View reviewed changes

src/Service.Tests/UnitTests/MySqlQueryExecutorUnitTests.cs Outdated Show resolved Hide resolved

Aniruddh25 reviewed Oct 31, 2025

View reviewed changes

src/Service.Tests/UnitTests/SqlQueryExecutorUnitTests.cs Outdated Show resolved Hide resolved

Aniruddh25 reviewed Oct 31, 2025

View reviewed changes

src/Service.Tests/UnitTests/PostgreSqlQueryExecutorUnitTests.cs Outdated Show resolved Hide resolved

format, cleanup

fa6582b

aaronburtle marked this pull request as ready for review October 31, 2025 16:48

aaronburtle requested a review from anushakolan as a code owner October 31, 2025 16:48

Copilot AI review requested due to automatic review settings October 31, 2025 16:48

aaronburtle requested review from Alekhya-Polavarapu, neeraj-sharma2592, rusamant, sourabh1007 and vadeveka as code owners October 31, 2025 16:48

Copilot AI reviewed Oct 31, 2025

View reviewed changes

aaronburtle and others added 17 commits October 31, 2025 10:12

Update src/Config/RuntimeConfigLoader.cs

b761333

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

Update src/Config/DeserializationVariableReplacementSettings.cs

9411180

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

Update src/Config/Converters/DatasourceHealthOptionsConvertorFactory.cs

890e2c6

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

Update src/Config/Converters/DatasourceHealthOptionsConvertorFactory.cs

9af3a4c

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

Update src/Config/Converters/EntityCacheOptionsConverterFactory.cs

11318c6

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

Update src/Config/Converters/EntityCacheOptionsConverterFactory.cs

b9ef8f9

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

Update src/Config/Converters/StringJsonConverterFactory.cs

3b1d682

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

Update src/Config/Converters/McpRuntimeOptionsConverterFactory.cs

15abe47

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

Update src/Config/Converters/RuntimeHealthOptionsConvertorFactory.cs

da9d974

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

Update src/Config/Converters/RuntimeHealthOptionsConvertorFactory.cs

3bde74c

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

format

025cbc3

remove last vestigates of replaceEnvVar

1c6f020

format

9a14dc3

format

4f38f92

using ordering

4742669

Merge branch 'main' of github.com:Azure/data-api-builder

6b73688

Merge branch 'main' into dev/aaronburtle/AKVReplacement

97fdc69

Aniruddh25 reviewed Oct 31, 2025

View reviewed changes

Aniruddh25 requested changes Oct 31, 2025

View reviewed changes

	/// <param name="replaceEnvVar">Whether to replace environment variable with its
	/// <param name="replacementSettings">Whether to replace environment variable with its

	// before we can have anILogger here.
	// before we can have an ILogger here.

Update variable replacement during deserialization to use replacement settings class and add AKV replacement logic. #2882

Are you sure you want to change the base?

Update variable replacement during deserialization to use replacement settings class and add AKV replacement logic. #2882

Uh oh!

Conversation

aaronburtle commented Sep 18, 2025

Why make this change?

What is this change?

How was this tested?

Sample Request(s)

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

RubenCerna2079 left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

souvikghosh04 left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Aniruddh25 left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull Request Overview

Reviewed Changes

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Aniruddh25 left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!