2 min lesson
Ingestion patterns: batch, streaming, CDC
Use two rows in "Ingestion patterns: batch, streaming, CDC" to state the practical decision rules.
Step 1 of 2
The pattern follows the source, not the other way around. High-volume product events want a stream; SaaS systems of record want a connector. Pick wrong and you either melt a connector or build a streaming stack for a nightly Salesforce pull.
Cursor's two source families pull in opposite directions. First-party product telemetry is a firehose of editor and client events that arrives continuously. Third-party business data - billing, CRM, support - lives in SaaS systems you read on a schedule or via change capture.
- Source
- Product telemetry (events)
- Pattern
- Streaming or micro-batch
- Why
- Continuous high volume; you want it in bronze within seconds/minutes
- Tooling
- Kafka/Kinesis → Spark Structured Streaming / Auto Loader
- Source
- SaaS systems of record (CRM, billing)
- Pattern
- Connector batch + CDC
- Why
- Source is a database/API, not a stream; you want low-effort, reliable pulls
- Tooling
- Fivetran/Airbyte connectors, scheduled or log-based CDC
- Source
- Operational app databases
- Pattern
- CDC (log-based)
- Why
- Capture inserts/updates/deletes without full re-scans
- Tooling
- Debezium-style log capture → stream → bronze
| Source | Pattern | Why | Tooling |
|---|---|---|---|
| Product telemetry (events) | Streaming or micro-batch | Continuous high volume; you want it in bronze within seconds/minutes | Kafka/Kinesis → Spark Structured Streaming / Auto Loader |
| SaaS systems of record (CRM, billing) | Connector batch + CDC | Source is a database/API, not a stream; you want low-effort, reliable pulls | Fivetran/Airbyte connectors, scheduled or log-based CDC |
| Operational app databases | CDC (log-based) | Capture inserts/updates/deletes without full re-scans | Debezium-style log capture → stream → bronze |
Match the pattern to the source's nature: streams for firehoses, connectors/CDC for systems of record.
When you're standing up a connector to a system of record, two Cursor moves save a round of guesswork. Microsoft SQL Server has a SQL Server VS Code extension, and you can also connect it over MCPModel Context Protocol. A standard that lets an AI agent pull in context from outside the repo, like Jira tickets or internal docs. Press Enter for the full definition. so the agent runs queries against the source directly while you design the pull - ask the agent how to set either up. And when you hit an "is this even supported" question, chat with the Cursor docs in-product rather than guessing.
You can actually chat with them directly and ask questions like, 'Is GitHub Enterprise supported?' And then it will actually go through, look through our docs, and give you an answer using the same underlying models that our agent uses.