Loaders API

Loaders are inbound ELT (Extract, Load, Transform) configurations that pull data from external SaaS tools and databases into your data warehouse (Source). They complement Sources by bringing data into the warehouse, while Sources provide read access from the warehouse.

Endpoints

List Loaders

GET /api/v1/workspaces/{id}/loaders

Returns all loaders in the workspace.

Response

[
  {
    "id": "550e8400-e29b-41d4-a716-446655440000",
    "workspace_id": "660e8400-e29b-41d4-a716-446655440000",
    "source_id": "770e8400-e29b-41d4-a716-446655440000",
    "name": "Service Cloud",
    "loader_type": "salesforce_service_cloud",
    "config": {
      "instance_url": "https://mycompany.my.salesforce.com",
      "api_version": "v59.0"
    },
    "target_schema": "SALESFORCE_RAW",
    "selected_streams": ["Case", "CaseHistory", "CaseTeamMember", "Contact"],
    "sync_mode": "incremental",
    "stream_sync_modes": {
      "Case": "incremental_merge",
      "CaseTeamMember": "full_refresh",
      "Contact": "incremental_merge"
    },
    "primary_keys": {},
    "change_detection": {},
    "schedule": "0 */6 * * *",
    "status": "active",
    "status_error": "",
    "created_by": "880e8400-e29b-41d4-a716-446655440000",
    "created_at": "2024-01-15T09:30:00Z",
    "updated_at": "2024-01-15T09:30:00Z",
    "last_run_at": "2024-01-15T15:00:00Z",
    "last_run_id": "990e8400-e29b-41d4-a716-446655440000",
    "last_run_status": "completed"
  }
]

In this example CaseHistory has no entry in stream_sync_modes, so it runs in the loader-wide sync_mode (incremental, append); the other three streams run in their own mode. See Sync modes.

Example

curl -X GET https://agentic.zeotap.com/api/v1/workspaces/{id}/loaders \
  -H "Authorization: Bearer <token>" \
  -H "X-Workspace-ID: <workspace-id>"

Create Loader

POST /api/v1/workspaces/{id}/loaders

Creates a new loader to pull data from an external source into a warehouse.

Request Body

Supported Loader Types

GET /api/v1/loader-types returns the full catalog, including each type’s configuration schema. Commonly used types:

Supported Loader Types

Example

curl -X POST https://agentic.zeotap.com/api/v1/workspaces/{id}/loaders \
  -H "Authorization: Bearer <token>" \
  -H "X-Workspace-ID: <workspace-id>" \
  -H "Content-Type: application/json" \
  -d '{
    "name": "Service Cloud",
    "loader_type": "salesforce_service_cloud",
    "source_id": "770e8400-e29b-41d4-a716-446655440000",
    "config": {
      "instance_url": "https://mycompany.my.salesforce.com",
      "api_version": "v59.0"
    },
    "credentials": {"credential_ref": "<ref from the OAuth connect flow>"},
    "target_schema": "SALESFORCE_RAW",
    "selected_streams": ["Case", "CaseHistory", "CaseTeamMember", "Contact"],
    "sync_mode": "incremental",
    "schedule": "0 */6 * * *"
  }'

The request omits stream_sync_modes, so each selected stream gets the Service Cloud loader’s recommended mode, and the response stores only the streams whose mode differs from sync_mode:

{
  "id": "550e8400-e29b-41d4-a716-446655440000",
  "sync_mode": "incremental",
  "stream_sync_modes": {
    "Case": "incremental_merge",
    "CaseTeamMember": "full_refresh",
    "Contact": "incremental_merge"
  },
  "...": "remaining loader fields"
}

To choose the modes yourself, send stream_sync_modes explicitly — it then replaces the recommendations entirely:

{
  "sync_mode": "incremental_merge",
  "stream_sync_modes": {"CaseHistory": "incremental"}
}

Returns 201 Created with the loader object.

Get Loader

GET /api/v1/workspaces/{id}/loaders/{loaderId}

Returns a single loader by ID.

Update Loader

PUT /api/v1/workspaces/{id}/loaders/{loaderId}

Updates an existing loader configuration.

Request Body

Connector-recommended modes are applied only on create. An update never fills stream_sync_modes from the connector, so a loader’s tables only change mode when you change it. An entry for a stream you deselect is dropped along with the stream.

Example

Run CaseHistory as append and leave every other setting as stored:

curl -X PUT https://agentic.zeotap.com/api/v1/workspaces/{id}/loaders/{loaderId} \
  -H "Authorization: Bearer <token>" \
  -H "X-Workspace-ID: <workspace-id>" \
  -H "Content-Type: application/json" \
  -d '{
    "name": "Service Cloud",
    "config": {
      "instance_url": "https://mycompany.my.salesforce.com",
      "api_version": "v59.0"
    },
    "target_schema": "SALESFORCE_RAW",
    "selected_streams": ["Case", "CaseHistory", "CaseTeamMember", "Contact"],
    "stream_sync_modes": {
      "Case": "incremental_merge",
      "CaseHistory": "incremental",
      "CaseTeamMember": "full_refresh",
      "Contact": "incremental_merge"
    }
  }'

stream_sync_modes is replaced as a whole, so send every entry you want to keep.

Returns 200 OK with the loader object.

Sync Modes

A stream’s effective sync mode is stream_sync_modes[stream] if the map has an entry for it, otherwise the loader’s sync_mode.

Validation, applied on create and update (a violation returns 400):

• every key in stream_sync_modes, primary_keys and change_detection must be one of selected_streams;
• every value in stream_sync_modes must be one of the three modes above;
• change_detection rules that depend on the sync mode are checked against each stream’s effective mode.

A stream that cannot be read incrementally (no cursor — supports_incremental: false on Discover Streams) is read in full on every run whatever its mode. Under incremental that appends a complete copy of the stream every run, so give such streams full_refresh (or incremental_merge, which keeps one row per key but never removes rows deleted at the source).

Loaders created before per-stream modes existed have stream_sync_modes: {} and run every stream in sync_mode.

Change Detection

change_detection maps a stream to an object describing how its incremental reads detect change. Streams with no entry read on a timestamp cursor, which is the default.

native_cdc needs an incremental effective mode, and with incremental_merge it also needs primary_keys for the stream.

Delete Loader

DELETE /api/v1/workspaces/{id}/loaders/{loaderId}

Deletes a loader.

Response

{
  "status": "deleted"
}

Trigger Sync Run

POST /api/v1/workspaces/{id}/loaders/{loaderId}/trigger

Manually triggers a loader sync run.

Response

Returns the created loader run object with status 201 Created.

Example

curl -X POST https://agentic.zeotap.com/api/v1/workspaces/{id}/loaders/{loaderId}/trigger \
  -H "Authorization: Bearer <token>" \
  -H "X-Workspace-ID: <workspace-id>"

List Loader Runs

GET /api/v1/workspaces/{id}/loaders/{loaderId}/runs

Returns the execution history for a loader.

Response

[
  {
    "id": "aae8400-e29b-41d4-a716-446655440000",
    "loader_id": "550e8400-e29b-41d4-a716-446655440000",
    "workspace_id": "660e8400-e29b-41d4-a716-446655440000",
    "status": "completed",
    "started_at": "2024-01-15T15:00:00Z",
    "completed_at": "2024-01-15T15:05:30Z",
    "rows_loaded": 15420,
    "streams_total": 2,
    "streams_failed": 0,
    "error_message": "",
    "triggered_by": "880e8400-e29b-41d4-a716-446655440000",
    "created_at": "2024-01-15T15:00:00Z"
  }
]

Loader Run Statuses

Get Loader Run

GET /api/v1/workspaces/{id}/loaders/{loaderId}/runs/{runId}

Returns details of a specific loader run.

Discover Streams

POST /api/v1/workspaces/{id}/loaders/discover (body: loader_type, config, credentials) or POST /api/v1/workspaces/{id}/loaders/{loaderId}/discover (stored configuration)

Returns the streams the loader can read with these credentials.

{
  "streams": [
    {
      "name": "CaseHistory",
      "label": "Case History",
      "columns": [
        {"name": "Id", "type": "string", "nullable": false},
        {"name": "CreatedDate", "type": "timestamp", "nullable": true}
      ],
      "cursor_field": "CreatedDate",
      "supports_incremental": true,
      "primary_key": ["Id"],
      "default_sync_mode": "incremental"
    }
  ]
}

List Loader Types

GET /api/v1/loader-types

Returns the list of available loader connector types with their configuration schemas and available streams. This endpoint is global and does not require the X-Workspace-ID header.

Loader Object

Loader Run Object