diff --git a/develop-docs/sdk/telemetry/spans/batch-processor.mdx b/develop-docs/sdk/telemetry/spans/batch-processor.mdx index 2d28342770699..ab63effca567b 100644 --- a/develop-docs/sdk/telemetry/spans/batch-processor.mdx +++ b/develop-docs/sdk/telemetry/spans/batch-processor.mdx @@ -10,7 +10,7 @@ title: Batch Processor This document uses key words such as "MUST", "SHOULD", and "MAY" as defined in [RFC 2119](https://www.ietf.org/rfc/rfc2119.txt) to indicate requirement levels. -The BatchProcessor batches spans and logs into one envelope to reduce the number of HTTP requests. When an SDK implements span streaming or logs, it MUST use a BatchProcessor, which is similar to [OpenTelemetry's Batch Processor](https://github.com/open-telemetry/opentelemetry-collector/blob/main/processor/batchprocessor/README.md). The BatchProcessor holds logs and finished spans in memory and batches them together into envelopes. It uses a combination of time and size-based batching. When writing this, the BatchProcessor only handles spans and logs, but an SDK MAY use it for other telemetry data in the future. +The BatchProcessor batches spans and logs into one envelope to reduce the number of HTTP requests. When an SDK implements span streaming or logs, it MUST use a BatchProcessor, which is similar to [OpenTelemetry's Batch Processor](https://github.com/open-telemetry/opentelemetry-collector/blob/main/processor/batchprocessor/README.md). The BatchProcessor tracks logs and finished spans, allowing it to batch them together into envelopes. It uses a combination of time and size-based batching. When writing this, the BatchProcessor only handles spans and logs, but an SDK MAY use it for other telemetry data in the future. ## Specification @@ -22,17 +22,16 @@ The BatchProcessor MUST send all items after the SDK when containing spans or lo When the BatchProcessor sends all spans or logs, it MUST reset its timeout and remove all spans and logs. The SDK MUST apply filtering and sampling before adding spans or logs to the BatchProcessor. The SDK MUST apply rate limits to spans and logs after they leave the BatchProcessor to send as much data as possible by dropping data as late as possible. -The BatchProcessor MUST send all spans and logs in memory to avoid data loss in the following scenarios: +The BatchProcessor MUST send all spans and logs to avoid data loss in the following scenarios: 1. When the user calls `SentrySDK.flush()`, the BatchProcessor MUST send all data in memory. 2. When the user calls `SentrySDK.close()`, the BatchProcessor MUST send all data in memory. 3. When the application shuts down gracefully, the BatchProcessor SHOULD send all data in memory. This is mostly relevant for mobile SDKs already subscribed to these hooks, such as [applicationWillTerminate](https://developer.apple.com/documentation/uikit/uiapplicationdelegate/applicationwillterminate(_:)) on iOS. 4. When the application moves to the background, the BatchProcessor SHOULD send all data in memory and stop the timer. This is mostly relevant for mobile SDKs. -5. We're working on concept for crashes, and will update the specification when we have more details. +5. If applicable to your environment, SDKs MUST minimize data loss when sudden process terminations occur. Refer to the [Sudden Process Terminations](#sudden-process-terminations) section for more details. The detailed specification is written in the [Gherkin syntax](https://cucumber.io/docs/gherkin/reference/). The specification uses spans as an example, but the same applies to logs or any other future telemetry data. - ```Gherkin Scenario: No spans in BatchProcessor 1 span added Given no spans in the BatchProcessor @@ -95,3 +94,109 @@ Scenario: 1 span added application crashes And loses the spans in the BatchProcessor ``` + +## Sudden Process Terminations + +The BatchProcessor MUST minimize the loss of logs for sudden process terminations, such as crashes or watchdog terminations. + +Each SDK environment is unique. Therefore, SDKs have three options to choose from to minimize data loss. As their number increases, the options get more complex. The first option is the simplest, and the last option is the most complicated. SDKs SHOULD implement the least complex option that is suitable for their environment. + +### 1. Flush All Data + +When the SDK detects a sudden process termination, it MUST put all remaining items in the BatchProcessor into one envelope and flush it. If your SDK has an offline cache, it MAY flush the envelope to disk and skip sending it to Sentry, if it ensures to send the envelope the next time the SDK starts. The BatchProcessor MUST keep its existing logic described in the [specification](#specification) above. + +Suppose your SDK can't reliably detect sudden process terminations, or it can't reliably flush envelopes to Sentry or disk when a sudden process termination happens. In that case, it SHOULD implement the [FileStream Cache](#2-file-stream-cache) or the [DoubleRotatingBuffer](#3-doublerotatingbuffer). It's acceptable to start with this option as a best effort interim solution before adding one of the more complex options. + +### 2. FileStream Cache + +SDKs for which blocking the main thread is a nogo, such as Android and Apple, SDKs MUST NOT implement this option. They SHOULD implement the [DoubleRotatingBuffer](#3-doublerotatingbuffer). + +With this option, the BatchProcessor stores the data on the calling thread directly to disk. The SDK SHOULD store the BatchProcessor files in a folder that is a sibling of the `envelopes` or `replay` folder, named `batch-processor`. This folder is scoped per DSN, so SDKs ensure not mixing up data for different DSNs. In the `batch-processor` folder, the SDK MUST store two types of cache files: + +- **`cache`** - The file the processor is actively writing to +- **`flushing`** - The file being converted to an envelope and sent to Sentry + +When the timeout expires or the cache file hits the size limit, the BatchProcessor renames the `cache` file to `flushing`, creates a new `cache` file for incoming data, converts the data in the `flushing` file to an envelope, sends it to Sentry, and then deletes the `flushing` file. When the SDK starts again, it MUST check if there are any cache files in the cache directory (both `cache` and `flushing`) and if so, it MUST load the data from the files and send it to Sentry. + + +### 3. DoubleRotatingBuffer + +SDKs should only consider implementing this option when options [1](#1-flush-all-data) or [2](#2-file-stream-cache) are insufficient to prevent data loss within their ecosystem. We recommend this option only if SDKs are unable to reliably detect sudden process terminations or consistently store envelopes to disk during such terminations. + +The BatchProcessor uses two buffers to minimize data loss in the event of an abnormal process termination: +* **Crash-Safe List**: A list stored in a crash-safe space to prevent data loss during detectable abnormal process terminations. +* **Async IO Cache**: When a process terminates without the SDK being able to detect it, the crash-safe list loses all its elements. Therefore, the BatchProcessor uses a second buffer, the async IO cache, that stores elements to disk on a background thread to avoid blocking the calling thread, which ensures minimal data loss when such terminations occur. + +As the BatchProcessor MUST prevent data loss during flushing, it uses a double-buffering solution. The crash-safe list has two lists `crash-safe-list-1` and `crash-safe-list-2`, and the async IO cache has two files `async-io-cache-1` and `async-io-cache-2`. When `crash-safe-list-1` is full, the BatchProcessor stores any new incoming items in `crash-safe-list-2` until it successfully stores items from `crash-safe-list-1` to disk as an envelope. Then it can delete items in `crash-safe-list-1`. The same applies to the async IO cache. + +#### BatchProcessor Files + +The SDK SHOULD store the BatchProcessor files in a folder that is a sibling of the `envelopes` or `replay` folder, named `batch-processor`. This folder is scoped per DSN, so SDKs ensure not mixing up data for different DSNs. The `batch-processor` folder MAY contain the following files: + +- `async-io-cache-1` and `async-io-cache-2` - The async IO cache files. +- `detected-termination-x` - The file containing items from the crash-safe list from a previous detected abnormal termination. +- `envelope-x` - The envelope that the BatchProcessor is about to move to the envelopes cache folder, so the SDK can send it to Sentry, where `x` is the an increasing index of the file starting from 0. + + +#### Receiving Items + +The BatchProcessor has two lists `crash-safe-list-1` and `crash-safe-list-2` and two files `async-io-cache-1` and `async-io-cache-2`. When it receives items, it performs the following steps: + +1. Put the item into the crash-safe `crash-safe-list-1` on the calling thread. +2. On a background thread, store the item in the `async-io-cache-1`. + +#### Flushing + +When the `crash-safe-list1` exceeds the [above described](#specification) 1MiB in size or the timeout exceeds, the BatchProcessor performs the following flushing steps: + +1. Store new incoming items to the `crash-safe-list-2` and `async-io-cache-2`. +2. Put the items of `crash-safe-list-1` into an envelope named `envelope-x`. +3. Delete the items in `crash-safe-list-1` and `async-io-cache-1`. +4. Move the `envelope-x` to the envelopes cache folder, in which all the other envelopes are stored, so the SDK can send it to Sentry. + +The BatchProcessor stores the `envelope-x` not directly in the envelope cache folder because, if an abnormal process termination occurs before deleting the items `crash-safe-list-1` and `async-io-cache-1`, the SDKs might send duplicate items. + + +#### Abnormal Process Termination + +When SDKs detect an abnormal process termination, they MUST write the items in both `crash-safe-list-1` and `crash-safe-list-2` to the `detected-termination-x` file where `x` is the an increasing index of the file starting from 0. + +When the process terminates abnormally and the SDKs can't detect it, the SDKs lose items from the crash-safe lists, which we consider preferable to blocking the calling thread, which could be the main thread. However, the SDKs don't lose items from the async IO cache. + +So the SDK MAY lose items due to undetectable abnormal process terminations that occur immediately after receiving an item, but it won't lose items due to detectable abnormal process terminations. + +``` +// SDKs keep this log +Sentry.logger.info("We might crash now."); + +crash(); +``` + +``` +// SDKs might lose this log +Sentry.logger.info("We are going to allocate loads of memory now."); + +// Something allocating a lot of memory leading to a watchdog termination due to out of memory. +allocateAllMemory(); +``` + +#### SDK Initialization + +Whenever the SDKs initialize, they MUST check if there is any data in the batch processor folder that needs to be recovered. They MUST perform the following steps when initializing: + +1. Load all items from `async-io-cache-1`, `async-io-cache-2` and `detected-termination-x` into memory if they exist. When the application terminates normally, these files don't exist. +2. Deduplicate the items based on the IDs of the items and store the deduplicated items in the `envelope-x` file. +3. Create new `async-io-cache-1` and `async-io-cache-2` files, and delete the `detected-termination-x` file. +4. Now the BatchProcessor can start receiving new items. +5. Move the `envelope-x` to the envelopes cache folder. + +#### SDK Closes + +Whenever the users closes the SDK or the application terminates normally, the BatchProcessor MUST perform the steps described in the [Flushing](#flushing) section and the SDK MUST delete all items in the `async-io-cache-1` and `async-io-cache-2` files. + +#### Miscellaneous + +The BatchProcessor maintains its logic of batching multiple logs and spans together into a single envelope to avoid multiple HTTP requests. + +Hybrid SDKs pass every log and span down to the native SDKs, which will put every log and span in their BatchProcessor and its cache when logs and spans are ready for sending, meaning after they go through beforeLog, integrations, processors, etc. +