Add SparkPlanInfo constructor compatible with Databricks' Spark fork #9888

charlesmyu · 2025-10-30T17:07:30Z

What Does This Do

The primary addition in this PR is support for the Databricks-specific SparkPlanInfo constructor. The available public methods and constructors on Databricks 17.3 LTS (based on Spark 4.0.0), for example, look like this:

def <init>(
	nodeName: String, 
	simpleString: String, 
	children: Seq[org.apache.spark.sql.execution.SparkPlanInfo], 
	metadata: Map[String,String], 
	metrics: Seq[org.apache.spark.sql.execution.metric.SQLMetricInfo], 
	estRowCount: Option[BigInt], 
	rddScopeId: String, 
	explainId: Option[Int]
): org.apache.spark.sql.execution.SparkPlanInfo;
def <init>(
	nodeName: String, 
	simpleString: String, 
	children: Seq[org.apache.spark.sql.execution.SparkPlanInfo], 
	metadata: Map[String,String], 
	metrics: Seq[org.apache.spark.sql.execution.metric.SQLMetricInfo], 
	estRowCount: Option[BigInt], 
	rddScopeId: String
): org.apache.spark.sql.execution.SparkPlanInfo;

val nodeName: <?>;
val simpleString: <?>;
val children: <?>;
val metadata: <?>;
val metrics: <?>;
val estRowCount: <?>;
val rddScopeId: <?>;
val explainId: <?>;

The last three methods (estRowCount, rddScopeId, explainId) are non-standard, and thus must be accounted for in these specific constructors.

Note that we attempt the Databricks constructor first since it has more info, and then the standard constructor. Both are unlikely to exist simultaneously, but this is just to be defensive. In addition, we try to take advantage of the MethodHandles class where possible for reflection, but can't use it to load any of the getters in SparkPlanInfoUtils as all the methods are inherited (MethodHandles uses getDeclaredMethod which excludes inherited methods).

Additional, smaller changes worth noting:

Moves the SparkPlanInfo constructor logic into a dedicated class, SparkPlanInfoUtils.
Makes a small change in AbstractSparkPlanSerializer to improve how we reflect the simpleString method, specifically preferring the .invoke method provided by MethodHandlers as a nice wrapper for exception handling logic.
- This shouldn't change any behaviour of how we get simpleString.
Swaps us from matching on parameter count to matching on explicit parameter types to avoid false matches, especially as we handle a more diverse set of constructors.
We also explicitly reflect all getters (i.e. nodeName(), simpleString(), children(), and metrics()) to avoid version incompatibilities causing muzzle to reject the entire Spark tracer.
- e.g. in Scala 2.13.0 , scala.Seq (used by children and metrics) was migrated from scala.collections.Seq to scala.collections.immutable.Seq.

Motivation

We would like to support Spark jobs for column-level metadata extraction, but they use a different signature for their SparkPlanInfo constructors. This PR fixes that.

Additional Notes

Contributor Checklist

Format the title according the contribution guidelines
Assign the type: and (comp: or inst:) labels in addition to any useful labels
Don't use close, fix or any linking keywords when referencing an issue.
Use solves instead, and assign the PR milestone to the issue
Update the CODEOWNERS file on source file addition, move, or deletion
Update the public documentation in case of new configuration flag or behavior

Jira ticket: DJM-974

datadog-datadog-prod-us1 · 2025-10-30T17:19:39Z

🎯 Code Coverage
• Patch Coverage: 100.00%
• Total Coverage: 73.62% (+13.92%)

View detailed report

_{This comment will be updated automatically if new data arrives.

🔗 Commit SHA: 261e9b1 | Docs | Datadog PR Page | Was this helpful? Give us feedback!}

pr-commenter · 2025-10-30T17:49:56Z

Benchmarks

Startup

Parameters

	Baseline	Candidate
baseline_or_candidate	baseline	candidate
git_branch	master	charles.yu/djm-974/spark-plan-databricks
git_commit_date	1761921316	1761923536
git_commit_sha	`feb33de`	`261e9b1`
release_version	1.55.0-SNAPSHOT~feb33def65	1.55.0-SNAPSHOT~261e9b1d4b

See matching parameters

	Baseline	Candidate
application	insecure-bank	insecure-bank
ci_job_date	1761925518	1761925518
ci_job_id	1209519244	1209519244
ci_pipeline_id	80902820	80902820
cpu_model	Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz	Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz
kernel_version	Linux runner-zfyrx7zua-project-304-concurrent-0-9h59hfks 6.8.0-1031-aws #33~22.04.1-Ubuntu SMP Thu Jun 26 14:22:30 UTC 2025 x86_64 x86_64 x86_64 GNU/Linux	Linux runner-zfyrx7zua-project-304-concurrent-0-9h59hfks 6.8.0-1031-aws #33~22.04.1-Ubuntu SMP Thu Jun 26 14:22:30 UTC 2025 x86_64 x86_64 x86_64 GNU/Linux
module	Agent	Agent
parent	None	None

Summary

Found 0 performance improvements and 0 performance regressions! Performance is the same for 54 metrics, 11 unstable metrics.

Startup time reports for insecure-bank

gantt
    title insecure-bank - global startup overhead: candidate=1.55.0-SNAPSHOT~261e9b1d4b, baseline=1.55.0-SNAPSHOT~feb33def65

    dateFormat X
    axisFormat %s
section tracing
Agent [baseline] (1.043 s) : 0, 1042605
Total [baseline] (8.664 s) : 0, 8663585
Agent [candidate] (1.031 s) : 0, 1031452
Total [candidate] (8.623 s) : 0, 8622998
section iast
Agent [baseline] (1.16 s) : 0, 1160365
Total [baseline] (9.37 s) : 0, 9370015
Agent [candidate] (1.162 s) : 0, 1161876
Total [candidate] (9.336 s) : 0, 9335996

baseline results

Module	Variant	Duration	Δ tracing
Agent	tracing	1.043 s	-
Agent	iast	1.16 s	117.76 ms (11.3%)
Total	tracing	8.664 s	-
Total	iast	9.37 s	706.43 ms (8.2%)

candidate results

Module	Variant	Duration	Δ tracing
Agent	tracing	1.031 s	-
Agent	iast	1.162 s	130.424 ms (12.6%)
Total	tracing	8.623 s	-
Total	iast	9.336 s	712.999 ms (8.3%)

gantt
    title insecure-bank - break down per module: candidate=1.55.0-SNAPSHOT~261e9b1d4b, baseline=1.55.0-SNAPSHOT~feb33def65

    dateFormat X
    axisFormat %s
section tracing
crashtracking [baseline] (1.48 ms) : 0, 1480
crashtracking [candidate] (1.464 ms) : 0, 1464
BytebuddyAgent [baseline] (711.797 ms) : 0, 711797
BytebuddyAgent [candidate] (703.603 ms) : 0, 703603
GlobalTracer [baseline] (246.765 ms) : 0, 246765
GlobalTracer [candidate] (244.886 ms) : 0, 244886
AppSec [baseline] (32.789 ms) : 0, 32789
AppSec [candidate] (32.44 ms) : 0, 32440
Debugger [baseline] (6.535 ms) : 0, 6535
Debugger [candidate] (6.4 ms) : 0, 6400
Remote Config [baseline] (686.341 µs) : 0, 686
Remote Config [candidate] (680.036 µs) : 0, 680
Telemetry [baseline] (14.662 ms) : 0, 14662
Telemetry [candidate] (14.314 ms) : 0, 14314
Flare Poller [baseline] (6.48 ms) : 0, 6480
Flare Poller [candidate] (6.548 ms) : 0, 6548
section iast
crashtracking [baseline] (1.465 ms) : 0, 1465
crashtracking [candidate] (1.481 ms) : 0, 1481
BytebuddyAgent [baseline] (823.378 ms) : 0, 823378
BytebuddyAgent [candidate] (824.411 ms) : 0, 824411
GlobalTracer [baseline] (233.304 ms) : 0, 233304
GlobalTracer [candidate] (234.011 ms) : 0, 234011
IAST [baseline] (31.665 ms) : 0, 31665
IAST [candidate] (32.511 ms) : 0, 32511
AppSec [baseline] (29.673 ms) : 0, 29673
AppSec [candidate] (28.817 ms) : 0, 28817
Debugger [baseline] (6.215 ms) : 0, 6215
Debugger [candidate] (6.125 ms) : 0, 6125
Remote Config [baseline] (608.652 µs) : 0, 609
Remote Config [candidate] (591.928 µs) : 0, 592
Telemetry [baseline] (8.61 ms) : 0, 8610
Telemetry [candidate] (8.506 ms) : 0, 8506
Flare Poller [baseline] (4.098 ms) : 0, 4098
Flare Poller [candidate] (4.169 ms) : 0, 4169

Startup time reports for petclinic

gantt
    title petclinic - global startup overhead: candidate=1.55.0-SNAPSHOT~261e9b1d4b, baseline=1.55.0-SNAPSHOT~feb33def65

    dateFormat X
    axisFormat %s
section tracing
Agent [baseline] (1.032 s) : 0, 1032416
Total [baseline] (10.878 s) : 0, 10878145
Agent [candidate] (1.03 s) : 0, 1029580
Total [candidate] (10.858 s) : 0, 10857564
section appsec
Agent [baseline] (1.202 s) : 0, 1201836
Total [baseline] (10.915 s) : 0, 10915298
Agent [candidate] (1.21 s) : 0, 1209821
Total [candidate] (10.928 s) : 0, 10927710
section iast
Agent [baseline] (1.168 s) : 0, 1167747
Total [baseline] (11.081 s) : 0, 11081102
Agent [candidate] (1.159 s) : 0, 1159196
Total [candidate] (11.14 s) : 0, 11139512
section profiling
Agent [baseline] (1.182 s) : 0, 1181666
Total [baseline] (10.967 s) : 0, 10966996
Agent [candidate] (1.175 s) : 0, 1175226
Total [candidate] (10.91 s) : 0, 10910035

baseline results

Module	Variant	Duration	Δ tracing
Agent	tracing	1.032 s	-
Agent	appsec	1.202 s	169.421 ms (16.4%)
Agent	iast	1.168 s	135.332 ms (13.1%)
Agent	profiling	1.182 s	149.25 ms (14.5%)
Total	tracing	10.878 s	-
Total	appsec	10.915 s	37.153 ms (0.3%)
Total	iast	11.081 s	202.957 ms (1.9%)
Total	profiling	10.967 s	88.851 ms (0.8%)

candidate results

Module	Variant	Duration	Δ tracing
Agent	tracing	1.03 s	-
Agent	appsec	1.21 s	180.242 ms (17.5%)
Agent	iast	1.159 s	129.616 ms (12.6%)
Agent	profiling	1.175 s	145.646 ms (14.1%)
Total	tracing	10.858 s	-
Total	appsec	10.928 s	70.145 ms (0.6%)
Total	iast	11.14 s	281.947 ms (2.6%)
Total	profiling	10.91 s	52.471 ms (0.5%)

gantt
    title petclinic - break down per module: candidate=1.55.0-SNAPSHOT~261e9b1d4b, baseline=1.55.0-SNAPSHOT~feb33def65

    dateFormat X
    axisFormat %s
section tracing
crashtracking [baseline] (1.464 ms) : 0, 1464
crashtracking [candidate] (1.468 ms) : 0, 1468
BytebuddyAgent [baseline] (703.759 ms) : 0, 703759
BytebuddyAgent [candidate] (701.947 ms) : 0, 701947
GlobalTracer [baseline] (245.151 ms) : 0, 245151
GlobalTracer [candidate] (244.947 ms) : 0, 244947
AppSec [baseline] (32.486 ms) : 0, 32486
AppSec [candidate] (32.182 ms) : 0, 32182
Debugger [baseline] (6.414 ms) : 0, 6414
Debugger [candidate] (6.354 ms) : 0, 6354
Remote Config [baseline] (671.156 µs) : 0, 671
Remote Config [candidate] (674.046 µs) : 0, 674
Telemetry [baseline] (12.869 ms) : 0, 12869
Telemetry [candidate] (13.57 ms) : 0, 13570
Flare Poller [baseline] (8.309 ms) : 0, 8309
Flare Poller [candidate] (7.29 ms) : 0, 7290
section appsec
crashtracking [baseline] (1.466 ms) : 0, 1466
crashtracking [candidate] (1.465 ms) : 0, 1465
BytebuddyAgent [baseline] (725.845 ms) : 0, 725845
BytebuddyAgent [candidate] (731.177 ms) : 0, 731177
GlobalTracer [baseline] (235.518 ms) : 0, 235518
GlobalTracer [candidate] (237.022 ms) : 0, 237022
IAST [baseline] (25.005 ms) : 0, 25005
IAST [candidate] (25.122 ms) : 0, 25122
AppSec [baseline] (174.076 ms) : 0, 174076
AppSec [candidate] (174.793 ms) : 0, 174793
Debugger [baseline] (5.934 ms) : 0, 5934
Debugger [candidate] (5.955 ms) : 0, 5955
Remote Config [baseline] (618.472 µs) : 0, 618
Remote Config [candidate] (628.904 µs) : 0, 629
Telemetry [baseline] (8.436 ms) : 0, 8436
Telemetry [candidate] (8.479 ms) : 0, 8479
Flare Poller [baseline] (3.871 ms) : 0, 3871
Flare Poller [candidate] (3.904 ms) : 0, 3904
section iast
crashtracking [baseline] (1.476 ms) : 0, 1476
crashtracking [candidate] (1.457 ms) : 0, 1457
BytebuddyAgent [baseline] (828.92 ms) : 0, 828920
BytebuddyAgent [candidate] (822.073 ms) : 0, 822073
GlobalTracer [baseline] (234.692 ms) : 0, 234692
GlobalTracer [candidate] (233.568 ms) : 0, 233568
IAST [baseline] (35.365 ms) : 0, 35365
IAST [candidate] (33.595 ms) : 0, 33595
AppSec [baseline] (26.478 ms) : 0, 26478
AppSec [candidate] (27.992 ms) : 0, 27992
Debugger [baseline] (6.172 ms) : 0, 6172
Debugger [candidate] (6.132 ms) : 0, 6132
Remote Config [baseline] (601.657 µs) : 0, 602
Remote Config [candidate] (593.225 µs) : 0, 593
Telemetry [baseline] (8.446 ms) : 0, 8446
Telemetry [candidate] (8.329 ms) : 0, 8329
Flare Poller [baseline] (4.134 ms) : 0, 4134
Flare Poller [candidate] (4.182 ms) : 0, 4182
section profiling
crashtracking [baseline] (1.462 ms) : 0, 1462
crashtracking [candidate] (1.452 ms) : 0, 1452
BytebuddyAgent [baseline] (731.976 ms) : 0, 731976
BytebuddyAgent [candidate] (728.167 ms) : 0, 728167
GlobalTracer [baseline] (221.583 ms) : 0, 221583
GlobalTracer [candidate] (220.54 ms) : 0, 220540
AppSec [baseline] (32.641 ms) : 0, 32641
AppSec [candidate] (32.228 ms) : 0, 32228
Debugger [baseline] (11.556 ms) : 0, 11556
Debugger [candidate] (11.346 ms) : 0, 11346
Remote Config [baseline] (697.699 µs) : 0, 698
Remote Config [candidate] (1.431 ms) : 0, 1431
Telemetry [baseline] (11.472 ms) : 0, 11472
Telemetry [candidate] (10.522 ms) : 0, 10522
Flare Poller [baseline] (4.184 ms) : 0, 4184
Flare Poller [candidate] (4.161 ms) : 0, 4161
ProfilingAgent [baseline] (110.236 ms) : 0, 110236
ProfilingAgent [candidate] (109.66 ms) : 0, 109660
Profiling [baseline] (110.871 ms) : 0, 110871
Profiling [candidate] (110.328 ms) : 0, 110328

Load

Parameters

	Baseline	Candidate
baseline_or_candidate	baseline	candidate
git_branch	master	charles.yu/djm-974/spark-plan-databricks
git_commit_date	1761921316	1761923536
git_commit_sha	`feb33de`	`261e9b1`
release_version	1.55.0-SNAPSHOT~feb33def65	1.55.0-SNAPSHOT~261e9b1d4b

See matching parameters

	Baseline	Candidate
application	insecure-bank	insecure-bank
ci_job_date	1761925925	1761925925
ci_job_id	1209519245	1209519245
ci_pipeline_id	80902820	80902820
cpu_model	Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz	Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz
kernel_version	Linux runner-zfyrx7zua-project-304-concurrent-0-uhphenhh 6.8.0-1031-aws #33~22.04.1-Ubuntu SMP Thu Jun 26 14:22:30 UTC 2025 x86_64 x86_64 x86_64 GNU/Linux	Linux runner-zfyrx7zua-project-304-concurrent-0-uhphenhh 6.8.0-1031-aws #33~22.04.1-Ubuntu SMP Thu Jun 26 14:22:30 UTC 2025 x86_64 x86_64 x86_64 GNU/Linux

Summary

Found 3 performance improvements and 3 performance regressions! Performance is the same for 6 metrics, 12 unstable metrics.

scenario	Δ mean http_req_duration	Δ mean throughput	candidate mean http_req_duration	candidate mean throughput	baseline mean http_req_duration	baseline mean throughput
scenario:load:insecure-bank:profiling:high_load	better [-289.595µs; -250.463µs] or [-13.207%; -11.423%]	unstable [-1.106op/s; +560.606op/s] or [-0.054%; +27.161%]	1.923ms	2343.750op/s	2.193ms	2064.000op/s
scenario:load:insecure-bank:iast_GLOBAL:high_load	worse [+85.481µs; +198.406µs] or [+2.397%; +5.563%]	unstable [-198.965op/s; +102.402op/s] or [-15.474%; +7.964%]	3.708ms	1237.500op/s	3.566ms	1285.781op/s
scenario:load:petclinic:tracing:high_load	worse [+1.231ms; +1.630ms] or [+7.017%; +9.288%]	unstable [-47.418op/s; +7.418op/s] or [-17.833%; +2.790%]	18.977ms	245.906op/s	17.547ms	265.906op/s
scenario:load:petclinic:no_agent:high_load	worse [+0.954ms; +1.355ms] or [+5.342%; +7.592%]	unstable [-43.633op/s; +12.071op/s] or [-16.696%; +4.619%]	19.007ms	245.562op/s	17.852ms	261.344op/s
scenario:load:petclinic:iast:high_load	better [-1.250ms; -0.840ms] or [-6.397%; -4.301%]	unstable [-12.586op/s; +39.399op/s] or [-5.268%; +16.491%]	18.494ms	252.312op/s	19.539ms	238.906op/s
scenario:load:petclinic:code_origins:high_load	better [-2.303ms; -1.908ms] or [-11.673%; -9.670%]	unstable [+1.881op/s; +54.369op/s] or [+0.795%; +22.977%]	17.624ms	264.750op/s	19.729ms	236.625op/s

Request duration reports for petclinic

gantt
    title petclinic - request duration [CI 0.99] : candidate=1.55.0-SNAPSHOT~261e9b1d4b, baseline=1.55.0-SNAPSHOT~feb33def65
    dateFormat X
    axisFormat %s
section baseline
no_agent (17.852 ms) : 17671, 18034
.   : milestone, 17852,
appsec (19.35 ms) : 19148, 19552
.   : milestone, 19350,
code_origins (19.729 ms) : 19534, 19923
.   : milestone, 19729,
iast (19.539 ms) : 19347, 19732
.   : milestone, 19539,
profiling (20.615 ms) : 20407, 20824
.   : milestone, 20615,
tracing (17.547 ms) : 17373, 17720
.   : milestone, 17547,
section candidate
no_agent (19.007 ms) : 18815, 19199
.   : milestone, 19007,
appsec (19.321 ms) : 19126, 19517
.   : milestone, 19321,
code_origins (17.624 ms) : 17451, 17796
.   : milestone, 17624,
iast (18.494 ms) : 18306, 18682
.   : milestone, 18494,
profiling (20.547 ms) : 20342, 20752
.   : milestone, 20547,
tracing (18.977 ms) : 18780, 19174
.   : milestone, 18977,

baseline results

Variant	Request duration [CI 0.99]	Δ no_agent
no_agent	17.852 ms [17.671 ms, 18.034 ms]	-
appsec	19.35 ms [19.148 ms, 19.552 ms]	1.497 ms (8.4%)
code_origins	19.729 ms [19.534 ms, 19.923 ms]	1.876 ms (10.5%)
iast	19.539 ms [19.347 ms, 19.732 ms]	1.687 ms (9.4%)
profiling	20.615 ms [20.407 ms, 20.824 ms]	2.763 ms (15.5%)
tracing	17.547 ms [17.373 ms, 17.72 ms]	-305.805 µs (-1.7%)

candidate results

Variant	Request duration [CI 0.99]	Δ no_agent
no_agent	19.007 ms [18.815 ms, 19.199 ms]	-
appsec	19.321 ms [19.126 ms, 19.517 ms]	314.609 µs (1.7%)
code_origins	17.624 ms [17.451 ms, 17.796 ms]	-1.383 ms (-7.3%)
iast	18.494 ms [18.306 ms, 18.682 ms]	-512.755 µs (-2.7%)
profiling	20.547 ms [20.342 ms, 20.752 ms]	1.54 ms (8.1%)
tracing	18.977 ms [18.78 ms, 19.174 ms]	-29.848 µs (-0.2%)

Request duration reports for insecure-bank

gantt
    title insecure-bank - request duration [CI 0.99] : candidate=1.55.0-SNAPSHOT~261e9b1d4b, baseline=1.55.0-SNAPSHOT~feb33def65
    dateFormat X
    axisFormat %s
section baseline
no_agent (1.21 ms) : 1198, 1222
.   : milestone, 1210,
iast (3.251 ms) : 3207, 3296
.   : milestone, 3251,
iast_FULL (5.608 ms) : 5553, 5664
.   : milestone, 5608,
iast_GLOBAL (3.566 ms) : 3517, 3615
.   : milestone, 3566,
profiling (2.193 ms) : 2173, 2213
.   : milestone, 2193,
tracing (1.781 ms) : 1766, 1796
.   : milestone, 1781,
section candidate
no_agent (1.207 ms) : 1195, 1218
.   : milestone, 1207,
iast (3.23 ms) : 3190, 3270
.   : milestone, 3230,
iast_FULL (5.513 ms) : 5459, 5567
.   : milestone, 5513,
iast_GLOBAL (3.708 ms) : 3653, 3764
.   : milestone, 3708,
profiling (1.923 ms) : 1906, 1939
.   : milestone, 1923,
tracing (1.772 ms) : 1758, 1787
.   : milestone, 1772,

baseline results

Variant	Request duration [CI 0.99]	Δ no_agent
no_agent	1.21 ms [1.198 ms, 1.222 ms]	-
iast	3.251 ms [3.207 ms, 3.296 ms]	2.041 ms (168.6%)
iast_FULL	5.608 ms [5.553 ms, 5.664 ms]	4.398 ms (363.4%)
iast_GLOBAL	3.566 ms [3.517 ms, 3.615 ms]	2.356 ms (194.7%)
profiling	2.193 ms [2.173 ms, 2.213 ms]	982.458 µs (81.2%)
tracing	1.781 ms [1.766 ms, 1.796 ms]	570.499 µs (47.1%)

candidate results

Variant	Request duration [CI 0.99]	Δ no_agent
no_agent	1.207 ms [1.195 ms, 1.218 ms]	-
iast	3.23 ms [3.19 ms, 3.27 ms]	2.023 ms (167.7%)
iast_FULL	5.513 ms [5.459 ms, 5.567 ms]	4.306 ms (356.9%)
iast_GLOBAL	3.708 ms [3.653 ms, 3.764 ms]	2.502 ms (207.3%)
profiling	1.923 ms [1.906 ms, 1.939 ms]	716.051 µs (59.3%)
tracing	1.772 ms [1.758 ms, 1.787 ms]	565.76 µs (46.9%)

Dacapo

Parameters

	Baseline	Candidate
baseline_or_candidate	baseline	candidate
git_branch	master	charles.yu/djm-974/spark-plan-databricks
git_commit_date	1761921316	1761923536
git_commit_sha	`feb33de`	`261e9b1`
release_version	1.55.0-SNAPSHOT~feb33def65	1.55.0-SNAPSHOT~261e9b1d4b

See matching parameters

	Baseline	Candidate
application	biojava	biojava
ci_job_date	1761925687	1761925687
ci_job_id	1209519246	1209519246
ci_pipeline_id	80902820	80902820
cpu_model	Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz	Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz
kernel_version	Linux runner-zfyrx7zua-project-304-concurrent-1-0xhktn4v 6.8.0-1031-aws #33~22.04.1-Ubuntu SMP Thu Jun 26 14:22:30 UTC 2025 x86_64 x86_64 x86_64 GNU/Linux	Linux runner-zfyrx7zua-project-304-concurrent-1-0xhktn4v 6.8.0-1031-aws #33~22.04.1-Ubuntu SMP Thu Jun 26 14:22:30 UTC 2025 x86_64 x86_64 x86_64 GNU/Linux

Summary

Found 0 performance improvements and 0 performance regressions! Performance is the same for 11 metrics, 1 unstable metrics.

Execution time for tomcat

gantt
    title tomcat - execution time [CI 0.99] : candidate=1.55.0-SNAPSHOT~261e9b1d4b, baseline=1.55.0-SNAPSHOT~feb33def65
    dateFormat X
    axisFormat %s
section baseline
no_agent (1.473 ms) : 1461, 1484
.   : milestone, 1473,
appsec (3.684 ms) : 3466, 3901
.   : milestone, 3684,
iast (2.214 ms) : 2151, 2278
.   : milestone, 2214,
iast_GLOBAL (2.247 ms) : 2183, 2311
.   : milestone, 2247,
profiling (2.069 ms) : 2017, 2122
.   : milestone, 2069,
tracing (2.029 ms) : 1979, 2079
.   : milestone, 2029,
section candidate
no_agent (1.472 ms) : 1460, 1483
.   : milestone, 1472,
appsec (3.702 ms) : 3484, 3919
.   : milestone, 3702,
iast (2.202 ms) : 2139, 2266
.   : milestone, 2202,
iast_GLOBAL (2.245 ms) : 2181, 2309
.   : milestone, 2245,
profiling (2.07 ms) : 2017, 2123
.   : milestone, 2070,
tracing (2.044 ms) : 1994, 2095
.   : milestone, 2044,

baseline results

Variant	Execution Time [CI 0.99]	Δ no_agent
no_agent	1.473 ms [1.461 ms, 1.484 ms]	-
appsec	3.684 ms [3.466 ms, 3.901 ms]	2.211 ms (150.1%)
iast	2.214 ms [2.151 ms, 2.278 ms]	741.755 µs (50.4%)
iast_GLOBAL	2.247 ms [2.183 ms, 2.311 ms]	774.256 µs (52.6%)
profiling	2.069 ms [2.017 ms, 2.122 ms]	596.728 µs (40.5%)
tracing	2.029 ms [1.979 ms, 2.079 ms]	556.346 µs (37.8%)

candidate results

Variant	Execution Time [CI 0.99]	Δ no_agent
no_agent	1.472 ms [1.46 ms, 1.483 ms]	-
appsec	3.702 ms [3.484 ms, 3.919 ms]	2.23 ms (151.6%)
iast	2.202 ms [2.139 ms, 2.266 ms]	730.535 µs (49.6%)
iast_GLOBAL	2.245 ms [2.181 ms, 2.309 ms]	773.21 µs (52.5%)
profiling	2.07 ms [2.017 ms, 2.123 ms]	598.583 µs (40.7%)
tracing	2.044 ms [1.994 ms, 2.095 ms]	572.894 µs (38.9%)

Execution time for biojava

gantt
    title biojava - execution time [CI 0.99] : candidate=1.55.0-SNAPSHOT~261e9b1d4b, baseline=1.55.0-SNAPSHOT~feb33def65
    dateFormat X
    axisFormat %s
section baseline
no_agent (15.399 s) : 15399000, 15399000
.   : milestone, 15399000,
appsec (14.552 s) : 14552000, 14552000
.   : milestone, 14552000,
iast (18.77 s) : 18770000, 18770000
.   : milestone, 18770000,
iast_GLOBAL (18.203 s) : 18203000, 18203000
.   : milestone, 18203000,
profiling (15.132 s) : 15132000, 15132000
.   : milestone, 15132000,
tracing (15.332 s) : 15332000, 15332000
.   : milestone, 15332000,
section candidate
no_agent (14.837 s) : 14837000, 14837000
.   : milestone, 14837000,
appsec (14.837 s) : 14837000, 14837000
.   : milestone, 14837000,
iast (18.485 s) : 18485000, 18485000
.   : milestone, 18485000,
iast_GLOBAL (17.982 s) : 17982000, 17982000
.   : milestone, 17982000,
profiling (15.406 s) : 15406000, 15406000
.   : milestone, 15406000,
tracing (15.404 s) : 15404000, 15404000
.   : milestone, 15404000,

baseline results

Variant	Execution Time [CI 0.99]	Δ no_agent
no_agent	15.399 s [15.399 s, 15.399 s]	-
appsec	14.552 s [14.552 s, 14.552 s]	-847.0 ms (-5.5%)
iast	18.77 s [18.77 s, 18.77 s]	3.371 s (21.9%)
iast_GLOBAL	18.203 s [18.203 s, 18.203 s]	2.804 s (18.2%)
profiling	15.132 s [15.132 s, 15.132 s]	-267.0 ms (-1.7%)
tracing	15.332 s [15.332 s, 15.332 s]	-67.0 ms (-0.4%)

candidate results

Variant	Execution Time [CI 0.99]	Δ no_agent
no_agent	14.837 s [14.837 s, 14.837 s]	-
appsec	14.837 s [14.837 s, 14.837 s]	0.0 µs (0.0%)
iast	18.485 s [18.485 s, 18.485 s]	3.648 s (24.6%)
iast_GLOBAL	17.982 s [17.982 s, 17.982 s]	3.145 s (21.2%)
profiling	15.406 s [15.406 s, 15.406 s]	569.0 ms (3.8%)
tracing	15.404 s [15.404 s, 15.404 s]	567.0 ms (3.8%)

pawel-big-lebowski · 2025-10-31T09:34:01Z

...trumentation/spark/src/main/java/datadog/trace/instrumentation/spark/SparkPlanInfoUtils.java

+            SparkPlanInfo.class,
+            String.class,
+            String.class,
+            scala.collection.immutable.Seq.class,


One of the most painful discrepancies between Scala 2.12 and 2.13 is in scala.Seq. And these bugs are hard to tackle.

I think it makes sense having two separate Spark212PlanInfoUtils2 and Spark213PlanInfoUtils classes, at least for the constructors and databricksConstructor part. This assures code is compiled with the same Scala version as it's run.

Yeah, this makes sense - I was worried initially it was Spark version incompatibilities but if it's Scala there isn't much reason to not have two separate files. I ended up putting them in completely separate files with no common abstraction since I couldn't find a clean way of separating them for now, but I imagine down the line when we need to add more constructors that may become more appealing. Let me know what you think!

261e9b1 (#9888)

…rk fork

charlesmyu force-pushed the charles.yu/djm-974/spark-plan-databricks branch 2 times, most recently from 06f8fee to f649787 Compare October 30, 2025 22:09

charlesmyu added the inst: apache spark Apache Spark instrumentation label Oct 30, 2025

charlesmyu force-pushed the charles.yu/djm-974/spark-plan-databricks branch from f649787 to 03b891a Compare October 30, 2025 22:33

Base automatically changed from charles.yu/djm-974/reflect-spi-constructor to adrien.boitreaud/fix-muzzle-mismatch October 31, 2025 08:53

pawel-big-lebowski reviewed Oct 31, 2025

View reviewed changes

charlesmyu force-pushed the charles.yu/djm-974/spark-plan-databricks branch from 03b891a to 078c8db Compare October 31, 2025 13:49

Base automatically changed from adrien.boitreaud/fix-muzzle-mismatch to master October 31, 2025 14:35

charlesmyu added 2 commits October 31, 2025 11:12

Use SparkPlanInfo constructor that is compatible with Databricks' Spa…

cbe8e4a

…rk fork

Move Spark Plan utils to distinct files for Scala 2.12 and 2.13

261e9b1

charlesmyu force-pushed the charles.yu/djm-974/spark-plan-databricks branch from 078c8db to 261e9b1 Compare October 31, 2025 15:13

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Add SparkPlanInfo constructor compatible with Databricks' Spark fork #9888

Add SparkPlanInfo constructor compatible with Databricks' Spark fork #9888

Uh oh!

charlesmyu commented Oct 30, 2025 •

edited

Loading

Uh oh!

datadog-datadog-prod-us1 bot commented Oct 30, 2025 •

edited

Loading

Uh oh!

pr-commenter bot commented Oct 30, 2025 •

edited

Loading

Uh oh!

pawel-big-lebowski Oct 31, 2025

Uh oh!

charlesmyu Oct 31, 2025 •

edited

Loading

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Add SparkPlanInfo constructor compatible with Databricks' Spark fork #9888

Are you sure you want to change the base?

Add SparkPlanInfo constructor compatible with Databricks' Spark fork #9888

Uh oh!

Conversation

charlesmyu commented Oct 30, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What Does This Do

Motivation

Additional Notes

Contributor Checklist

Uh oh!

datadog-datadog-prod-us1 bot commented Oct 30, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

pr-commenter bot commented Oct 30, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Benchmarks

Startup

Parameters

Summary

Load

Parameters

Summary

Dacapo

Parameters

Summary

Uh oh!

pawel-big-lebowski Oct 31, 2025

Choose a reason for hiding this comment

Uh oh!

charlesmyu Oct 31, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

charlesmyu commented Oct 30, 2025 •

edited

Loading

datadog-datadog-prod-us1 bot commented Oct 30, 2025 •

edited

Loading

pr-commenter bot commented Oct 30, 2025 •

edited

Loading

charlesmyu Oct 31, 2025 •

edited

Loading