2 min lesson
Ingestion patterns: batch, streaming, CDC
Use two rows in "Ingestion patterns: batch, streaming, CDC" to state the practical decision rules.
Step 1 of 2
The pattern follows the source, not the other way around. High-volume product events want a stream; SaaS systems of record want a connector. Pick wrong and you either melt a connector or build a streaming stack for a nightly Salesforce pull.
Assume two source families that pull in opposite directions. First-party product telemetry is a firehose of editor and client events that arrives continuously. Third-party business data such as billing, CRM and support lives in SaaS systems you read on a schedule or via change capture.
- Source
- Product telemetry events
- Pattern
- Streaming or micro-batch
- Why
- Continuous high volume; you want it in bronze within seconds or minutes
- Tooling
- Kafka/Kinesis → Spark Structured Streaming
- Source
- SaaS systems of record such as CRM and billing
- Pattern
- Connector batch + CDC
- Why
- Source is a database/API, not a stream; you want low-effort, reliable pulls
- Tooling
- Fivetran/Airbyte connectors, scheduled or log-based CDC
- Source
- Operational app databases
- Pattern
- Log-based CDC
- Why
- Capture inserts/updates/deletes without full re-scans
- Tooling
- Debezium-style log capture → stream → bronze
| Source | Pattern | Why | Tooling |
|---|---|---|---|
| Product telemetry events | Streaming or micro-batch | Continuous high volume; you want it in bronze within seconds or minutes | Kafka/Kinesis → Spark Structured Streaming |
| SaaS systems of record such as CRM and billing | Connector batch + CDC | Source is a database/API, not a stream; you want low-effort, reliable pulls | Fivetran/Airbyte connectors, scheduled or log-based CDC |
| Operational app databases | Log-based CDC | Capture inserts/updates/deletes without full re-scans | Debezium-style log capture → stream → bronze |
Match the pattern to the source's nature: streams for firehoses, connectors/CDC for systems of record.
Before you stand up a connector to a system of record, check that the source is actually supported instead of guessing. When you hit an "is this even supported" question, chat with the Cursor docs in-product.
You can actually chat with them directly and ask questions like, 'Is GitHub Enterprise supported?' And then it will actually go through, look through our docs, and give you an answer using the same underlying models that our agent uses.