Testing Environment for Spans in a High Throughput Project

[dachakra]

Problem Statement

Our project is sending over 3.5B spans each month. Before pushing any changes to production, I normally use Sentry to trigger a few span samples and check our performance baselines. Since switching to the span dataset, I’ve been seeing discrepancies between Discover and Explore. Making the span widgets unusable.

Because I’m working in a staging environment, I need to view all the samples I generate locally to confirm everything looks right before deployment. Right now I’m seeing very few (sometimes none), which is breaking my workflow.

Solution Brainstorm

Special sampling rules for testing/staging environemtns

Product Area

Performance

[linear]

ENG-5773

[getsantry]

Assigning to @getsentry/support for routing ⏲️

[getsantry]

Routing to @getsentry/product-owners-discover for triage ⏲️

[bcoe]

This is a reality of how extrapolation and querying works today.

The best bet to answer questions about a staging environment separately from production is to use a separate project, and then when querying filter to the specific project; this may still have issues if you have All Projects selected, so your best bet is to select one project at a time with the project selector.