feat(low-code): add ignore status to AsyncJobStatus #825
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
👋 Greetings, Airbyte Team Member!Here are some helpful tips and reminders for your convenience. Testing This CDK VersionYou can test this version of the CDK using the following: # Run the CLI from this branch:
uvx 'git+https://github.com/airbytehq/airbyte-python-cdk.git@daryna/low-code/add-ignored-status-to-async-retriever#egg=airbyte-python-cdk[dev]' --help
# Update a connector to use the CDK from this branch ref:
cd airbyte-integrations/connectors/source-example
poe use-cdk-branch daryna/low-code/add-ignored-status-to-async-retrieverHelpful ResourcesPR Slash CommandsAirbyte Maintainers can execute the following slash commands on your PR:
|
![github-code-quality[bot]](https://avatars.githubusercontent.com/u/9919?s=60&v=4){kind=small}
Contributor
📝 WalkthroughWalkthroughA new IGNORE status is introduced to the async job orchestration system. The changes add an IGNORE enum member to AsyncJobStatus as a terminal state, update the schema and model to include an ignore field for status mapping configuration, extend the parser to map "ignore" to AsyncJobStatus.IGNORE and skip mapping when no API statuses are provided, and update the orchestrator to treat IGNORE as a terminal state and stop running partitions reporting it. Changes
Sequence DiagramsequenceDiagram
participant Parser as ModelToComponentFactory
participant StatusMap as AsyncJobStatusMap
participant Orchestrator as AsyncJobOrchestrator
participant Partition as AsyncPartition
Parser->>StatusMap: _create_async_job_status_mapping("ignore", None)
Note over StatusMap: api_statuses is None → skip mapping for "ignore"
Parser->>StatusMap: _get_async_job_status("ignore")
StatusMap->>Parser: AsyncJobStatus.IGNORE
Orchestrator->>Partition: Check partition status
Partition->>Orchestrator: returns AsyncJobStatus.IGNORE
Note over Partition: IGNORE treated as terminal
alt Running partition reports IGNORE
Orchestrator->>Orchestrator: Stop partition
Orchestrator->>Orchestrator: Log warning (ignored partition)
Note over Orchestrator: Skip reallocation / restart logic
end
Estimated code review effort🎯 3 (Moderate) | ⏱️ ~20 minutes
Pre-merge checks and finishing touches✅ Passed checks (2 passed)
✨ Finishing touches
🧪 Generate unit tests (beta)
📜 Recent review detailsConfiguration used: CodeRabbit UI Review profile: CHILL Plan: Pro 📒 Files selected for processing (1)
🚧 Files skipped from review as they are similar to previous changes (1)
⏰ Context from checks skipped due to timeout of 90000ms. You can increase the timeout in your CodeRabbit configuration to a maximum of 15 minutes (900000ms). (14)
Comment |
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
Contributor
There was a problem hiding this comment.
Actionable comments posted: 1
📜 Review details
Configuration used: CodeRabbit UI
Review profile: CHILL
Plan: Pro
📒 Files selected for processing (5)
airbyte_cdk/sources/declarative/async_job/job_orchestrator.py(3 hunks)airbyte_cdk/sources/declarative/async_job/status.py(1 hunks)airbyte_cdk/sources/declarative/declarative_component_schema.yaml(1 hunks)airbyte_cdk/sources/declarative/models/declarative_component_schema.py(1 hunks)airbyte_cdk/sources/declarative/parsers/model_to_component_factory.py(2 hunks)
🧰 Additional context used
🧠 Learnings (3)
📓 Common learnings
Learnt from: pnilan
Repo: airbytehq/airbyte-python-cdk PR: 0
File: :0-0
Timestamp: 2024-12-11T16:34:46.319Z
Learning: In the airbytehq/airbyte-python-cdk repository, ignore all `__init__.py` files when providing a recommended reviewing order.
📚 Learning: 2024-12-11T16:34:46.319Z
Learnt from: pnilan
Repo: airbytehq/airbyte-python-cdk PR: 0
File: :0-0
Timestamp: 2024-12-11T16:34:46.319Z
Learning: In the airbytehq/airbyte-python-cdk repository, the `declarative_component_schema.py` file is auto-generated from `declarative_component_schema.yaml` and should be ignored in the recommended reviewing order.
Applied to files:
airbyte_cdk/sources/declarative/models/declarative_component_schema.pyairbyte_cdk/sources/declarative/declarative_component_schema.yamlairbyte_cdk/sources/declarative/parsers/model_to_component_factory.py
📚 Learning: 2024-11-18T23:40:06.391Z
Learnt from: ChristoGrab
Repo: airbytehq/airbyte-python-cdk PR: 58
File: airbyte_cdk/sources/declarative/yaml_declarative_source.py:0-0
Timestamp: 2024-11-18T23:40:06.391Z
Learning: When modifying the `YamlDeclarativeSource` class in `airbyte_cdk/sources/declarative/yaml_declarative_source.py`, avoid introducing breaking changes like altering method signatures within the scope of unrelated PRs. Such changes should be addressed separately to minimize impact on existing implementations.
Applied to files:
airbyte_cdk/sources/declarative/declarative_component_schema.yaml
🧬 Code graph analysis (2)
airbyte_cdk/sources/declarative/async_job/job_orchestrator.py (1)
airbyte_cdk/sources/declarative/async_job/status.py (1)
AsyncJobStatus(9-25)
airbyte_cdk/sources/declarative/parsers/model_to_component_factory.py (1)
airbyte_cdk/sources/declarative/async_job/status.py (1)
AsyncJobStatus(9-25)
⏰ Context from checks skipped due to timeout of 90000ms. You can increase the timeout in your CodeRabbit configuration to a maximum of 15 minutes (900000ms). (13)
- GitHub Check: Check: source-shopify
- GitHub Check: Check: destination-motherduck
- GitHub Check: Check: source-pokeapi
- GitHub Check: Check: source-intercom
- GitHub Check: Check: source-hardcoded-records
- GitHub Check: Pytest (All, Python 3.11, Ubuntu)
- GitHub Check: Pytest (All, Python 3.12, Ubuntu)
- GitHub Check: Pytest (All, Python 3.13, Ubuntu)
- GitHub Check: Pytest (All, Python 3.10, Ubuntu)
- GitHub Check: Pytest (Fast)
- GitHub Check: SDM Docker Image Build
- GitHub Check: Manifest Server Docker Image Build
- GitHub Check: Analyze (python)
🔇 Additional comments (5)
airbyte_cdk/sources/declarative/async_job/status.py (1)
14-14: LGTM! Clean addition of IGNORE status.The new IGNORE enum member follows the exact same pattern as the other terminal statuses, and the
is_terminal()method will handle it correctly. Nice and consistent implementation.airbyte_cdk/sources/declarative/models/declarative_component_schema.py (1)
1209-1209: Good addition of the optional ignore field.Making the
ignorefield optional (while the other status fields are required) is the right choice for backward compatibility. This allows existing configurations to continue working without specifying ignore status mappings.Note: Based on learnings, this file is auto-generated from
declarative_component_schema.yaml, so any manual edits would be overwritten.airbyte_cdk/sources/declarative/declarative_component_schema.yaml (1)
3866-3869: Schema maps IGNORE statesAppreciate the extra
ignorebucket here—this lets manifest authors cleanly route API statuses to the new terminal state without workarounds.airbyte_cdk/sources/declarative/async_job/job_orchestrator.py (1)
103-105: Orchestrator short-circuits IGNORE partitions cleanlyNice touch treating
IGNOREas a terminal outcome and freeing resources immediately; this keeps the retry/error paths untouched while honoring the new status.Also applies to: 371-374
airbyte_cdk/sources/declarative/parsers/model_to_component_factory.py (1)
3564-3565: LGTM!The new case for "ignore" status is implemented correctly and consistently with the existing status cases. It properly returns
AsyncJobStatus.IGNOREas defined in the enum.
airbyte_cdk/sources/declarative/parsers/model_to_component_factory.py
Outdated
Show resolved
Hide resolved
PyTest Results (Full)3 820 tests 3 808 ✅ 11m 9s ⏱️ Results for commit 48880e3. |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Summary by CodeRabbit