configuring-imports — Claude Code skill for Celigo imports
A skill for configuring Celigo imports, the destination step that writes records to external systems. Useful when you pick an adaptor type, set up lookups or wire an upsert.
npx skills add celigo/ai --skill configuring-imports -a claude-codeWhat configuring-imports does
An import is the destination in a Celigo integration. It takes records from an upstream step and writes them to a REST API, database, ERP, file server or AI model. Every import has one adaptor type and, apart from the connection-less kinds, exactly one connection. The skill covers six areas: field mapping, operation logic (create, update, upsert, delete, attach/detach), JavaScript hooks, one-to-many fan-out, response mapping and the postResponseMap hook.
The import pipeline runs in a strict order: input filter, preMap, field mapping, postMap, submit, postSubmit, response transform, response mapping, postResponseMap. Response mapping lives on the flow's pageProcessors[] entry, not on the import resource, so plan it while designing the flow. Fields added in steps 2–4 never reach the next step.
There are traps. http.relativeURI and header templates render against the record before mapping, and an undefined path fails the request with handlebars_template_parse_error. That is missing data, so a retry fails the same way. A field holding "" is defined and renders empty; a field holding null renders the text null. Mapping rules belong to the writing-mappings skill, AI agent steps to configuring-ai-agents.
What it helps with
- pick an adaptor type for a new import
- set up an upsert or an exactly-once write
- fan out a child array with one-to-many
- pull the created record's id out of the API response
- debug a URL template that fails on an undefined field
Example prompts for the agent
- Set up a NetSuite import that upserts on the external id
- Fan out
lineItemswith one-to-many and reference{{record._PARENT.id}} - Add a
postSubmithook that logs the status code
How to install configuring-imports
1Skills CLI (Vercel)
npx skills add celigo/ai --skill configuring-imports -a claude-code2Manually: put the files in place
~/.claude/skills/configuring-imports/SKILL.mdCopy the whole skill folder; for one project use .claude/skills/configuring-imports/
1Skills CLI (Vercel)
npx skills add celigo/ai --skill configuring-imports -a codex2Manually: put the files in place
.agents/skills/configuring-imports/SKILL.mdFor all projects use ~/.agents/skills/
1Skills CLI (Vercel)
npx skills add celigo/ai --skill configuring-imports -a cursor2Manually: put the files in place
.cursor/skills/configuring-imports/SKILL.mdCursor also reads .agents/skills/ and .claude/skills/
1GitHub CLI
gh skill install celigo/ai configuring-imports2Manually: put the files in place
.github/skills/configuring-imports/SKILL.md1Skills CLI (Vercel)
npx skills add celigo/ai --skill configuring-imports -a gemini-cli2Platform CLI
gemini skills install https://github.com/celigo/ai --path skills/configuring-importsManually: put the files in place
.devin/skills/configuring-imports/SKILL.mdThe legacy .windsurf/skills/ path still works
Skills CLI (Vercel)
npx skills add celigo/ai --skill configuring-imports -a opencodeClaude.ai
Zip the folder and upload it in Settings → Capabilities → Skills
Step by step, with commands for each platform: How to install skills in Claude Code: five ways
Questions about configuring-imports
How do I install configuring-imports in Claude?
Run npx skills add celigo/ai --skill configuring-imports -a claude-code. The skill lands in ~/.claude/skills/configuring-imports/SKILL.md.
What does the configuring-imports skill do?
It helps you configure Celigo imports: adaptor type, lookups, upsert, exactly-once writes, file-based imports, hooks and response mapping.
How is response mapping different from field mapping?
Field mapping shapes only the request the import sends. Response mapping is configured on the flow's pageProcessors[] entry and carries data from the response back into the record for downstream steps.
Source text
SKILL.md · MIT
---
name: configuring-imports
description: Configure Celigo imports -- the destination step that writes records to external systems. Use when creating or editing imports, choosing the adaptor type, setting up lookups, upsert or exactly-once writes, or file-based imports. For mapping rules, use writing-mappings; for AI agent steps, configuring-ai-agents.
---
<!-- TIER:1 -->
# Configuring Imports
An import is the **data destination** in a Celigo integration. It takes records from an upstream step and writes them to an external system -- REST APIs, databases, ERPs, file servers, or AI models. Every import has one adaptor type and, apart from the connection-less kinds in [Minimum Required Fields](#minimum-required-fields), is bound to exactly one connection.
Imports handle six concerns:
- **Field mapping** -- transforming source fields into the destination system's expected format (including value resolution via static maps and lookup tables). Uses Mapper 2.0 (`mappings[]` array) by default, including `NetSuiteHTTPImport`. `SalesforceImport` and `WrapperImport` support only Mapper 1.0 (`mapping.fields[]` / `mapping.lists[]`), and `NetSuiteDistributedImport` uses its own Mapper 1.0 dialect at `netsuite_da.mapping`
- **Operation logic** -- create, update, upsert, delete, attach/detach
- **Hooks** -- JavaScript pre/post processing at various pipeline stages (preMap, postMap, postSubmit). File-based imports that generate files from records also support postAggregate; an HTTP records import refuses it (422 `invalid_hook`)
- **One-to-many** -- fan out a child array already on the input record. Set `oneToMany: true` and `pathToMany` to that path (e.g., `"lineItems"`) when one source record should create one import operation per child. After fan-out the child is `record` (`{{record.sku}}`) and the parent (without the array) is `record._PARENT` (`{{record._PARENT.id}}`). Leave `pathToMany` blank when the arriving record is itself an array (grouped rows from a `groupByFields` export or a file group) -- each element then becomes the record, with no `_PARENT`. A path on an array record processes zero children and still reports success; a blank path on an object record fails with `oneToMany_error`. `oneToMany` doesn't batch by itself: templates see `batch_of_records` only when `http.batchSize` is above 1 (the default varies by API) or an RDBMS query runs `per_page`
- **Response mapping** -- extract fields from the import's API response back into the record for downstream steps. Configured on the flow's `pageProcessors[]` entry, but planned when building the import. Extracts read the response envelope: `id` (the destination-assigned record ID), `statusCode`, `ignored`, `errors`, `_headers`, `dataURI` (the record's link, when the import sets `dataURITemplate`), and `_json` (the response body after any response transform, whose top-level fields are also flattened onto the envelope). Use `id` for a created record's ID and `_json.fieldName` for other response fields (e.g., `_json.output.1.content.0.text` for OpenAI responses) -- `_json.id` works only when the body itself has a top-level `id`. Flow test runs never send imports, so `id` is empty there. Response mapping uses Transformation 1.0 syntax (extract/generate pairs), not the newer expression-based transforms
- **postResponseMap hook** -- JavaScript processing after response mapping merges the response back into the record. Configured on the flow's `pageProcessors[]` entry, but planned when building the import. Use to transform or enrich the merged record before downstream steps
Imports are used across flows, APIs, and tools.
## Import Execution Pipeline
When records arrive at an import step, this pipeline executes in strict order:
1. **Input filter** (optional) -- records that don't match skip this import (see [Gotchas](#gotchas)). Configured as `filter` on the import resource itself -- an `inputFilter` key on an import is silently discarded. See [configuring-filters](../configuring-filters/SKILL.md#import-filter-import-filter)
2. **preMap hook** (optional) -- JavaScript processing before field mapping
3. **Field mapping** -- Mapper 2.0 or 1.0 maps source fields to destination fields, including lookups and hardcoded values
4. **postMap hook** (optional) -- JavaScript processing after field mapping, before submission
5. **Submit to destination** -- writes the mapped record to the external system
6. **postSubmit hook** (optional) -- JavaScript processing after the destination responds (access response data, log results, trigger side effects)
7. **Response transform** (optional, `responseTransform`) -- reshapes the response body (`_json`) after postSubmit. Its expression rules are Transformation 1.0 only (`expression.version: "1"`; `"2"` is refused with 422), and it keeps only the fields it maps, so a field it drops is gone for response mapping. Its execution-log stage is `responseTransformation`
8. **Response mapping** (optional) -- carries data from the destination response back into the record for downstream steps. Configured on the flow's `pageProcessors[]` entry, not on the import itself
9. **postResponseMap hook** (optional) -- JavaScript processing after response mapping merges response data back into the record
**Key distinction:** Response mapping lives on the flow's `pageProcessors[]` entry, not on the import resource. preMap, the mapping and postMap shape only the request this import sends: the next step receives the record as it arrived plus what response mapping and postResponseMap add, so a field added in steps 2-4 never reaches it. When building an import that needs to pass data downstream, plan the response mapping at flow design time.
**What an HTTP import's templates see.** `http.relativeURI` and header values render against the record before mapping, and an undefined path fails the request: a field the record lacks in the URI -- or a `{{lookup.<name>}}` whose lookup found nothing -- fails the record before anything is sent, with `handlebars_template_parse_error` (`Failed to generate request url from template: … Details: "<field>" not defined in the model.`). That is missing data, not a template bug, so a retry fails the same way; an update import that some records reach without the destination id needs a composite UPDATE/CREATE or `ignoreMissing` (Common Errors). A field holding `""` is defined: it renders empty, and the request goes out without the id. A field holding null renders the text `null` instead (the request goes to `…/null`), and a field under an object that is null fails the record like a missing one (`"<field>" not defined in null`) -- see writing-handlebars, gotcha 16. `http.body` renders against the mapped record, where an undefined path renders empty and the record still succeeds (`strictHandlebarEvaluation` doesn't change that). Leave `http.body` unset to send the mapped record; when you template it, use the mapped (destination) field names, and double braces for string values in a JSON body (gotcha 11).
## Categories of Import
### Record-Based Imports
Submit structured records to APIs, databases, or ERPs. The vast majority of imports.
- `NetSuiteDistributedImport` -- high-performance SuiteApp writes (add, update, addupdate, delete, attach, detach). The default for NetSuite when the Celigo SuiteApp or bundle is installed. `netsuite_da.restletVersion` is fixed when the step is created, so author new steps on `suiteapp2.0`; a hook's SuiteScript version must match the step's
- `NetSuiteHTTPImport` -- NetSuite's REST APIs on a `netsuite` connection (`nsDomainType: "suitetalk"`: `POST /services/rest/record/v1/<type>` to insert, `PATCH .../<type>/{id}` to update; `"restlet"` for a custom RESTlet). Use it when neither the SuiteApp nor the bundle is installed, or the user asks for REST
- `HTTPImport` -- REST/GraphQL APIs (POST, PUT, PATCH, DELETE). Supports connector-assisted (`formType: "assistant"`) and GraphQL (`graph_ql`, `assistant_graphql`) modes; see [GraphQL imports](#graphql-imports)
- `SalesforceImport` -- Salesforce writes through `salesforce.api`: `soap` (the default), `rest`, or `compositerecord` (Composite Graph, a tree of related records in one request); `metadata` deploys configuration, not records. `salesforce.api` has no Bulk value
- `RDBMSImport` -- SQL databases (Snowflake, PostgreSQL, MySQL, SQL Server, Oracle). `rdbms.queryType` is a one-element array whose value fixes the companion field: `per_record`, `per_page` (renders against `batch_of_records`) and `first_page` (runs once, on the first page) take `rdbms.query`, an array of SQL strings; `bulk_insert` takes `rdbms.bulkInsert` (pure INSERT, with no upsert or ignore-existing logic); `bulk_load` takes `rdbms.bulkLoad` (a staged load that becomes a MERGE with `primaryKeys`, or runs your own SQL with `overrideMergeQuery`). Prefer `bulk_load` where the database supports it, then `bulk_insert` for pure INSERTs, then `per_page`, and `per_record` only when the logic can't be batched. The legacy uppercase `INSERT` (and the retired `MERGE`, `COPY` and `BULK INSERT`) fail any save with 422 "Import query types … are deprecated"; `UPDATE` is legacy too, so don't use it. Other strings, such as lowercase `insert`, fail as an invalid enum value. A `bulk_load` import's GET also returns a `file` block describing its staging file; the platform derives it on every read, so a save that changes it is accepted and discarded. The SQL itself: [writing-sql](../writing-sql/SKILL.md)
- `MongodbImport`, `DynamodbImport`, `JDBCImport` -- other databases
#### GraphQL imports
A GraphQL import is an `HTTPImport` with `http.formType` `graph_ql` (or `assistant_graphql`, the connector-assisted GraphQL form). Each `http.body[]` entry is a string holding the `{"query", "operationName", "variables"}` envelope, one per operation on an update/create pair -- never an object, which is stored as-is and doesn't render -- and the mutation's input comes from the record through Handlebars in `variables`.
- **Endpoint.** The import posts to the connection's base URI -- its form has no relative-URI field and writes `relativeURI: ["/"]` on every save -- so the connection must supply the GraphQL endpoint: a `graph_ql` connection, or a connector connection bound to the app's GraphQL API. On a connection bound to the app's REST API every record fails (HTTP 406 on Shopify); add a second connection bound to the GraphQL API, re-point the GraphQL steps to it, and leave the shared REST connection unchanged. The form type is set on the step, so a plain `http` connection whose base URI is the endpoint works too.
- **Variables.** Under `graph_ql` or `assistant_graphql`, a `variables` value stored as a JSON string is sent as an object; under `http` it goes out as a string. An object takes expressions inside string values only: `"variables": {"input": {"email": "{{record.email}}"}}`. A string can also carry bare values, but its string values sit two JSON levels deep, so give each one `jsonEncode` in double braces: `"variables": "{\"input\": {\"note\": \"{{jsonEncode record.note}}\", \"quantity\": {{record.qty}} } }"`. An object or array goes in as `{{jsonSerialize record.items}}`, also in double braces. With `{{record.note}}`, or triple braces on either helper, a value holding a quote fails its record with 422 `invalid_json` before anything is sent.
- **Errors come back as HTTP 200.** A GraphQL server usually answers a failed mutation with 200 and an `errors` array, and with no response settings the import counts the record as written. Set `http.response.resourceIdPath` to the id only a successful mutation returns (`["data.customerCreate.customer.id"]`): any other answer then fails the record with `missing_id`, top-level `errors` and the mutation's own `userErrors` alike. For the server's message on top-level errors, also set `failPath: ["errors[0].message"]` and `errorPath: "errors[0].message"`; `failPath: ["errors"]` catches a response with exactly one error and lets two or more through.
### File-Based Imports
Write or upload files to remote storage. Require the `file{}` configuration block. Two modes:
- **Record-to-file** -- aggregates incoming records into a file (CSV, JSON, XML, XLSX). The `file{}` block defines the output format, and `file.aggregation.type` how a run's records become files: `all` (one file per run), `page` (one per page) or `key` (EDI file definitions only, with B2B Manager). Send the legacy `file.skipAggregation` as its pair -- `true` with `page`, `false` with `all` or `key` -- because validation reads the value you send: an EDI file definition on `page` without `true`, or a `file.batchSize` without it, fails with 422 `not_supported`. The format is `file.type` (`csv`, `json`, `xml`, `xlsx`), with its settings under that name: a CSV's `file.csv.includeHeader` defaults to `true`, so set it `false` for a file with no header row. The columns are the import's `mappings` rows, in order, each row's `generate` naming its column; there is no `file.columns`.
- **Blob passthrough** -- transfers a file as-is from an upstream blob export or blob lookup. Set `blob: true`, and `blobKeyPath` to the record field holding the `blobKey` that step put there (FTP and Google Drive imports fall back to the record's root `blobKey` when it's unset). Base64 content in a record is record data, not a blobKey. AI agent, guardrail and tool imports reject both fields with 422 `not_supported`.
Adaptor types:
- `HTTPImport` with `http.type: "file"` -- the folder mode of the cloud file providers (Google Drive, Box, Dropbox, Azure Blob Storage, Google Cloud Storage, Azure Data Lake Storage Gen2, Celigo Storage). Address the folder by path in `http.relativeURI` (a folder, never an API endpoint) or by the provider's folder id in `file.directory` (`pathMode: "directoryId"`), and name the file in `file.fileName`; provider-specific location rules are in [file.yml](references/schemas/file.yml)
- `FTPImport` -- CSV, XML, JSON, XLSX, EDI files to FTP/SFTP. The folder goes in `ftp.directoryPath` and the file's name in `ftp.fileName`, which takes no directory separators (the UI stores the same name in `file.fileName` too); a dated name is a Handlebars template ([writing-handlebars > File paths and names](../writing-handlebars/SKILL.md#file-paths-and-names))
- `S3Import` -- objects to Amazon S3
- `AS2Import` -- AS2 EDI file transmission
- `FileSystemImport` -- local/on-premise filesystem writes
### AI Imports
Invoke AI models for classification, extraction, or safety checks. No `_connectionId` required unless using BYOK.
- `AiAgentImport` -- LLM invocations (any provider in the `aiAgent.provider` enum) with structured output, tool use, and reasoning
- `GuardrailImport` -- PII detection, content moderation, or custom AI-based validation
### Stack and Tool Imports
- `WrapperImport` -- custom connector code, typically a function on a stack (`wrapper.function`)
- `ToolImport` -- runs the Celigo Tool named by `tool._toolId`. It needs no `_connectionId`; instead it must bind each connection the tool's steps use in `tool.overrides.connections` as `{_abstractId, _id}` -- set `_id` equal to `_abstractId` to keep the tool's own connection. A connection left unbound fails the run with `connection_not_overridden`
### Topic Imports (Event Streams)
- `TopicImport` -- publishes each record as a message to an Event Streams topic, which topic listeners (`TopicExport`) read. Set `topic._topicId` to the topic's `_id`, never its name (`celigo topics list`; an unknown id fails the save with 422 `invalid_ref`), and optionally `topic.batchSize` (default 1000). There's no connection and no create/update logic: each record becomes one message, reshaped by the mapping if there is one; with one-to-many, each element is its own message carrying the parent under `_PARENT`. Register the topic on the flow's integration so its flows can bind to it (`celigo integrations register-topics <integrationId> <topicId>`); the account needs the Event Streams entitlement. Test runs publish nothing; an invoke publishes for real
## Import Matching (create / update / upsert)
The core decision on most record-based imports is the **operation** -- what happens to each record at the destination. Create requires no match key; update and delete require one; upsert checks first and does whichever applies. Users describe the intent in business terms ("look up the customer and update them", "match by email and upsert", "skip the ones that already exist") that resolve into five behaviors:
- **Always create** -- every record is submitted as new. No matching, no checks.
- **Create only if missing** -- match-check first; submit as create if not found, skip silently if found.
- **Update only if exists** -- match-check first; submit as update if found, skip silently if not.
- **Update only if exists, fail if missing** -- strict variant; no-match records error out instead of skipping.
- **Upsert** -- submit a create when no match is found, an update when matched. The catch-all, and the right default when the user is vague.
When matching applies, decide four things: the **matching behavior**, the **match key field(s)** (`email`, `external_id`, `customer.id` -- required for anything other than always-create), the **on-match action** (update, skip, fail), and the **on-no-match action** (create, skip, fail). A lookup on a cross-reference field (NetSuite `externalid`, or a custom field that holds the source id) finds only records whose field holds that id, so the same import must write the source id into the field when it creates them; without that mapping no later run finds them, and each run adds the record again.
How a destination implements matching is adaptor-specific -- there is no single "matching mode" field. A destination might expose: a native **upsert** keyed off an external ID (Salesforce upsert, NetSuite upsert, RDBMS `ON CONFLICT`); a distinct `addupdate` operation that handles both paths in one call; an `ignoreExisting` flag paired with a lookup that probes before writing (lock it when repeats can arrive concurrently -- see [Writing Exactly Once](#writing-exactly-once)); two separate create and update endpoints with no upsert variant (see composite imports below); or a lookup endpoint plus separate create and update endpoints, where the lookup runs pre-write to drive the create-vs-update decision. Some destinations have no matching concept at all -- writing a CSV to FTP, sending an email, posting to a webhook -- so every record goes out as-is.
**Prefer import-level matching over a separate lookup step.** Imports natively support this pre-write check, so a single import that does the matching and the write together means fewer steps, fewer round-trips, and no glue logic to maintain. A standalone lookup earns its place only when the looked-up data has a consumer *beyond* the write -- a router branching on something other than "does this exist", an AI agent reasoning over the result, or multiple downstream steps reading different fields. If the only consumer is the destination call itself, the work belongs inside the import. It's also the only check that runs under the import's concurrency lock, which matters whenever the same record can arrive twice ([Writing Exactly Once](#writing-exactly-once)). An import's own lookups refuse an ambiguous match only when told to: an HTTP lookup that finds more than one match fails the record only with `allowFailures: false`, and otherwise sends the first match with no error (gotcha 14); a NetSuite DA lookup with `allowFailures: true` also resolves to one of the matching records unless `useDefaultOnMultipleMatches` sends it to the `default` ([netsuitedistributed.yml](references/schemas/netsuitedistributed.yml)). So search on a key that identifies one record. When an update's key must come from a separate lookup that can match several records, guard it -- fail or skip on multiple matches, or check for exactly one ([configuring-lookup-caches > Safe Lookup-to-Update](../configuring-lookup-caches/SKILL.md#safe-lookup-to-update----never-write-off-an-unguarded-first-match)).
### Composite (two-endpoint) imports
When an HTTP destination has no native upsert but exposes separate create and update endpoints, a **composite import** pins both endpoints on a single import node, role-tagged create and update. The runtime picks per record via a match-key check -- the same way a native upsert would -- so the flow stays one step. Prefer this over two separate imports driven by an upstream lookup or router; reach for separate imports only when the create and update paths must diverge beyond endpoint selection (different mappings, different downstream consumers, or different hook chains).
The operations sit in positional arrays, one entry per operation: `http.requestType` (`["UPDATE", "CREATE"]`), `http.method` and `http.relativeURI`. Exactly one decider picks per record: `http.existingExtract`, a field the record already carries (non-empty means UPDATE, whose URI reads it as `{{{data.0.<field>}}}`), or `http.existingLookupName`, a lookup in `http.lookups` that probes the destination (the UPDATE URI reads `{{lookup.<name>}}`). Never add `ignoreExisting` to a composite import, and give every `http.response.*` array one entry per operation.
`existingExtract` must hold the destination's id (put there by an upstream step's response mapping, or kept as a cross-reference), never the source record's own id: every record carries one, so every record would go to UPDATE. With `existingLookupName`, a lookup that comes back empty sends the record to CREATE, with no error -- and so does one whose `extract` resolves nothing, so a `$.`-prefixed `extract` sends every record to CREATE. A templated `http.body` has one entry per operation too; left unset, both requests send the mapped record. A search-driven upsert against an API that finds contacts by email:
```json
"http": {
"requestType": ["UPDATE", "CREATE"],
"method": ["PATCH", "POST"],
"relativeURI": ["/contacts/{{lookup.findContact}}", "/contacts"],
"existingLookupName": "findContact",
"lookups": [{
"name": "findContact",
"method": "POST",
"relativeURI": "/contacts/search",
"body": "{\"email\": \"{{email}}\"}",
"extract": "results[0].id"
}]
}
```
The lookup sits in `http.lookups` (a `lookups` array sent only at the top level is dropped on save), its request template is in `body`, and the template names the record's fields at the top level (`{{email}}`, never `{{record.email}}`, which renders empty, so every search misses and every record is created again; gotchas 10 and 11). Its `extract` is the id itself, because that is what the UPDATE URI receives. An HTTP lookup's `extract` is a path into the response written without `$.` -- `results[0].id`, or `ids` for an array of ids -- unlike a Mapper 2.0 row's `extract`, which needs `$.`. A `$.` prefix, `[*]`, or a dot path through an array (`results.id`) resolves nothing, so the lookup always comes back empty.
### Writing Exactly Once
Every source can hand an import the same record twice: a sender fires a webhook again, a failed or canceled run is re-run over the same delta window, an error is retried, a record that wrote this step but failed a later one comes back on the next run, or another flow writes the same destination. **A trace key never de-duplicates** -- it correlates a record's errors and is the default lock key below; a record that arrives twice is written twice. Make the write itself idempotent, in this order of preference:
| Rank | Pattern | Why it holds under concurrency |
|---|---|---|
| 1 | **A key the destination enforces** -- an upsert or insert on a primary key or unique constraint (`ON CONFLICT`, `MERGE` -- see [writing-sql](../writing-sql/SKILL.md)), a unique external ID, an API that rejects a duplicate reference | The destination refuses the second copy |
| 2 | **`ignoreExisting: true` + the existence check on this import + `idLockTemplate` on the same business key.** `ignoreExisting` (like `ignoreMissing`) is a top-level import field, not part of `http`. The check is `http.ignoreLookupName` (a lookup on the key) or `http.ignoreExtract` (the record already carries the destination ID) -- exactly one -- or the adaptor's equivalent (NetSuite `internalIdLookup`, Salesforce `idLookup.whereClause`) | The existence check runs inside the per-key lock |
| 3 | **A ledger step** in front of a destination that can't be queried -- [configuring-lookup-caches > Deduplication Ledger](../configuring-lookup-caches/SKILL.md#deduplication-ledger) | The ledger's claim is itself rank 2 |
An insert on a unique key refuses a repeat as an error (Salesforce `DUPLICATE_VALUE`, a unique-constraint violation). That keeps a repeated record out, but it refuses a changed one too: when changes to existing records should arrive, upsert on that key.
A match that is really a search run before the write -- a lookup-driven composite create/update, an `addupdate` found by search -- is a check-then-write, not rank 1, unless the destination also rejects the duplicate key. A natural id is unique only within its source: when two sources can produce the same one (two stores, two accounts), namespace the key per source (`{{record.storeId}}-{{record.orderId}}`), or one source's record collides with or overwrites the other's. Rank 2 is only as good as its lookup: a lookup that fails counts as "not found", so the record is written (gotcha 12).
**The concurrency ID lock.** `idLockTemplate` renders a key per record; records that render the same key are never submitted at the same time, and different keys run in parallel.
- **Default:** the record's trace key (a source's `traceKeyTemplate` arrives on the record as `io_custom_trace_key`). Without a template, the trace key is the first top-level id field the record has -- `id`, `internalId`, `_id`, `internal_id` or `identifier`, in a few spellings each -- and a nested id is never used. A blank trace key means **no lock**, so set `idLockTemplate` when the identity is nested or composite. A template replaces the default rather than adding to it.
- **Render time:** from the record as it arrives, before this import's own lookups and mapping. A key built from `{{lookup.<name>}}` or a mapped field renders empty -- and an empty key is no lock at all. Key it on a field the record carries (`{{{record.orderId}}}`), or one an upstream lookup step added through response mapping.
- **Scope:** this import across every page, every concurrent run, and every flow that uses it (`celigo imports dependencies <id>`) -- not the connection: two imports rendering the same key still run in parallel. So two imports that both create the same entity (an order flow that creates missing customers, and a separate customer sync) can each miss the other's write and create it twice: create the entity through one import, or schedule the flows so they don't overlap. Records within one page are sent one at a time anyway, so a one-page test never shows the race.
- **Coverage:** the `ignoreExisting` existence check runs inside the lock. Nothing documents the same for `ignoreMissing`, the composite create/update decision, or one-to-many children -- where those must write once, connection `concurrencyLevel: 1` closes the check-then-write race (the import's pages run one at a time), at the cost of serializing everything on that connection.
## Quick Reference
### Adaptor Decision Matrix
| Your data goes to... | Use adaptorType | Category | Read schema |
|---|---|---|---|
| REST or GraphQL API | `HTTPImport` | Record-based | [http.yml](references/schemas/http.yml) |
| NetSuite, Celigo SuiteApp or bundle installed | `NetSuiteDistributedImport` | Record-based | [netsuitedistributed.yml](references/schemas/netsuitedistributed.yml) |
| NetSuite, neither installed (or REST requested) | `NetSuiteHTTPImport` + `nsDomainType` | Record-based | [http.yml](references/schemas/http.yml) |
| Salesforce objects | `SalesforceImport` | Record-based | [salesforce.yml](references/schemas/salesforce.yml) |
| SQL database (Snowflake, PostgreSQL, etc.) | `RDBMSImport` | Record-based | [rdbms.yml](references/schemas/rdbms.yml) |
| MongoDB | `MongodbImport` | Record-based | [mongodb.yml](references/schemas/mongodb.yml) |
| DynamoDB | `DynamodbImport` | Record-based | [dynamodb.yml](references/schemas/dynamodb.yml) |
| JDBC database (non-built-in) | `JDBCImport` | Record-based | [jdbc.yml](references/schemas/jdbc.yml) |
| Files to cloud storage (Google Drive, Box, Dropbox, Azure Blob, Google Cloud Storage, Celigo Storage) | `HTTPImport` with `http.type: "file"` | File-based | [http.yml](references/schemas/http.yml) |
| Files to FTP/SFTP | `FTPImport` | File-based | [ftp.yml](references/schemas/ftp.yml) |
| Files to S3 | `S3Import` | File-based | [s3.yml](references/schemas/s3.yml) |
| AS2 EDI transmission | `AS2Import` | File-based | [as2.yml](references/schemas/as2.yml) |
| Local filesystem | `FileSystemImport` | File-based | [filesystem.yml](references/schemas/filesystem.yml) |
| An LLM (any provider in the `aiAgent.provider` enum) | `AiAgentImport` | AI | [aiagent.yml](references/schemas/aiagent.yml) |
| PII detection / content moderation | `GuardrailImport` | AI | [guardrail.yml](references/schemas/guardrail.yml) |
| Celigo Tool | `ToolImport` | Tool | [tool.yml](references/schemas/tool.yml) |
| Custom connector code on a stack | `WrapperImport` | Stack | [wrapper.yml](references/schemas/wrapper.yml) |
| An Event Streams topic | `TopicImport` | Topic | [topic.yml](references/schemas/topic.yml) |
**Raw HTTP is the fallback, not the default.** Pick the most specific match, in order:
1. **Native adaptor** -- if the application has its own row (NetSuite, Salesforce, databases, FTP/S3), use it. Do not build an `HTTPImport` against that app's REST API.
2. **Pre-built HTTP connector** -- for any other REST/GraphQL app, check the 550+ connector catalog before writing HTTP config (see [Check...
© celigo · MIT · celigo/ai