YAML Reference
This page lists all components, interpolation variables and interpolation macros that can be used when defining a low code YAML file.
For the technical JSON schema definition that low code manifests are validated against, see here.
Components
DeclarativeSource object
An API source that extracts data according to its declarative components.
Properties:- check- #/definitions/CheckStream
- streams- array- #/definitions/DeclarativeStream
- version- string
- schemas- #/definitions/Schemas
- spec- #/definitions/Spec
- metadata- object- For internal Airbyte use only - DO NOT modify manually. Used by consumers of declarative manifests for storing related metadata. 
AddedFieldDefinition object
Defines the field to add on a record.
Properties:- path- array- List of strings defining the path where to add the value on the record. Examples:- [ "segment_id" ] - [ "metadata", "segment_id" ] 
- value- string- Value of the new field. Use {{ record['existing_field'] }} syntax to refer to other fields in the record. Available variables:
 Examples:- "{{ record['updates'] }}"- "{{ record['MetaData']['LastUpdatedTime'] }}"- "{{ stream_partition['segment_id'] }}"
- value_type- #/definitions/ValueType- Type of the value. If not specified, the type will be inferred from the value. 
- $parameters- object- Set parameters that are inherited to all children. See the section in the advanced topics for more details. 
AddFields object
Transformation which adds field to an output record. The path of the added field can be nested.
Properties:- fields- array- List of transformations (path and corresponding value) that will be added to the record. 
- $parameters- object- Set parameters that are inherited to all children. See the section in the advanced topics for more details. 
ApiKeyAuthenticator object
Authenticator for requests authenticated with an API token injected as an HTTP request header.
Properties:- api_token- string- The API key to inject in the request. Fill it in the user inputs. Available variables:
 Examples:- "{{ config['api_key'] }}"- "Token token={{ config['api_key'] }}"
- header- string- The name of the HTTP header that will be set to the API key. This setting is deprecated, use inject_into instead. Header and inject_into can not be defined at the same time. Available variables:
 Examples:- "Authorization" - "Api-Token" - "X-Auth-Token" 
- inject_into- #/definitions/RequestOption- Configure how the API Key will be sent in requests to the source API. Either inject_into or header has to be defined. Examples:- { "inject_into": "header", "field_name": "Authorization" }- { "inject_into": "request_parameter", "field_name": "authKey" }
- $parameters- object- Set parameters that are inherited to all children. See the section in the advanced topics for more details. 
AuthFlow object
Additional and optional specification object to describe what an 'advanced' Auth flow would need to function.
- A connector should be able to fully function with the configuration as described by the ConnectorSpecification in a 'basic' mode.
- The 'advanced' mode provides easier UX for the user with UI improvements and automations. However, this requires further setup on the server side by instance or workspace admins beforehand. The trade-off is that the user does not have to provide as many technical inputs anymore and the auth process is faster and easier to complete.
- auth_flow_type- string- The type of auth to use 
- predicate_key- array- JSON path to a field in the connectorSpecification that should exist for the advanced auth to be applicable. Example:- [ "credentials", "auth_type" ] 
- predicate_value- string- Value of the predicate_key fields for the advanced auth to be applicable. Example:- "Oauth" 
- oauth_config_specification- #/definitions/OAuthConfigSpecification
BasicHttpAuthenticator object
Authenticator for requests authenticated with the Basic HTTP authentication scheme, which encodes a username and an optional password in the Authorization request header.
Properties:- username- string- The username that will be combined with the password, base64 encoded and used to make requests. Fill it in the user inputs. Available variables:
 Examples:- "{{ config['username'] }}"- "{{ config['api_key'] }}"
- password- string- The password that will be combined with the username, base64 encoded and used to make requests. Fill it in the user inputs. Available variables:
 Examples:- "{{ config['password'] }}"- "" 
- $parameters- object- Set parameters that are inherited to all children. See the section in the advanced topics for more details. 
BearerAuthenticator object
Authenticator for requests authenticated with a bearer token injected as a request header of the form Authorization: Bearer <token>.
- api_token- string- Token to inject as request header for authenticating with the API. Available variables:
 Examples:- "{{ config['api_key'] }}"- "{{ config['token'] }}"
- $parameters- object- Set parameters that are inherited to all children. See the section in the advanced topics for more details. 
SelectiveAuthenticator object
Authenticator that selects concrete authenticator based on config property.
Properties:- authenticator_selection_path- array- Path of the field in config with selected authenticator name Examples:- [ "auth" ] - [ "auth", "type" ] 
- authenticators- object- Authenticators to select from. Example:- { "authenticators": { "token": "#/definitions/ApiKeyAuthenticator", "oauth": "#/definitions/OAuthAuthenticator" } }
- $parameters- object- Set parameters that are inherited to all children. See the section in the advanced topics for more details. 
CheckStream object
Defines the streams to try reading when running a check operation.
Properties:- stream_names- array- Names of the streams to try reading from when running a check operation. Examples:- [ "users" ] - [ "users", "contacts" ] 
CompositeErrorHandler object
Error handler that sequentially iterates over a list of error handlers.
Properties:- error_handlers- array- List of error handlers to iterate on to determine how to handle a failed response. 
- $parameters- object- Set parameters that are inherited to all children. See the section in the advanced topics for more details. 
ConstantBackoffStrategy object
Backoff strategy with a constant backoff interval.
Properties:- backoff_time_in_seconds- Backoff time in seconds. Type:- number
- string
 
 Available variables:
 Examples:- 30 - 30.5 - "{{ config['backoff_time'] }}"
- $parameters- object- Set parameters that are inherited to all children. See the section in the advanced topics for more details. 
CursorPagination object
Pagination strategy that evaluates an interpolated string to define the next page to fetch.
Properties:- cursor_value- string- Value of the cursor defining the next page to fetch. Available variables:
 Examples:- "{{ headers.link.next.cursor }}"- "{{ last_records[-1]['key'] }}"- "{{ response['nextPage'] }}"
- page_size- integer- The number of records to include in each pages. Example:- 100 
- stop_condition- string- Template string evaluating when to stop paginating. Available variables:
 Examples:- "{{ response.data.has_more is false }}"- "{{ 'next' not in headers['link'] }}"
- decoder- #/definitions/JsonDecoder- Component decoding the response so records can be extracted. 
- $parameters- object- Set parameters that are inherited to all children. See the section in the advanced topics for more details. 
CustomAuthenticator object
Authenticator component whose behavior is derived from a custom code implementation of the connector.
Properties:- class_name- string- Fully-qualified name of the class that will be implementing the custom authentication strategy. Has to be a sub class of DeclarativeAuthenticator. The format is Example:- source_<name>.<package>.<class_name>.- "source_railz.components.ShortLivedTokenAuthenticator" 
- $parameters- object- Set parameters that are inherited to all children. See the section in the advanced topics for more details. 
CustomBackoffStrategy object
Backoff strategy component whose behavior is derived from a custom code implementation of the connector.
Properties:- class_name- string- Fully-qualified name of the class that will be implementing the custom backoff strategy. The format is Example:- source_<name>.<package>.<class_name>.- "source_railz.components.MyCustomBackoffStrategy" 
- $parameters- object- Set parameters that are inherited to all children. See the section in the advanced topics for more details. 
CustomErrorHandler object
Error handler component whose behavior is derived from a custom code implementation of the connector.
Properties:- class_name- string- Fully-qualified name of the class that will be implementing the custom error handler. The format is Example:- source_<name>.<package>.<class_name>.- "source_railz.components.MyCustomErrorHandler" 
- $parameters- object- Set parameters that are inherited to all children. See the section in the advanced topics for more details. 
CustomIncrementalSync object
Incremental component whose behavior is derived from a custom code implementation of the connector.
Properties:- class_name- string- Fully-qualified name of the class that will be implementing the custom incremental sync. The format is Example:- source_<name>.<package>.<class_name>.- "source_railz.components.MyCustomIncrementalSync" 
- cursor_field- string- The location of the value on a record that will be used as a bookmark during sync. 
- $parameters- object- Set parameters that are inherited to all children. See the section in the advanced topics for more details. 
CustomPaginationStrategy object
Pagination strategy component whose behavior is derived from a custom code implementation of the connector.
Properties:- class_name- string- Fully-qualified name of the class that will be implementing the custom pagination strategy. The format is Example:- source_<name>.<package>.<class_name>.- "source_railz.components.MyCustomPaginationStrategy" 
- $parameters- object- Set parameters that are inherited to all children. See the section in the advanced topics for more details. 
CustomRecordExtractor object
Record extractor component whose behavior is derived from a custom code implementation of the connector.
Properties:- class_name- string- Fully-qualified name of the class that will be implementing the custom record extraction strategy. The format is Example:- source_<name>.<package>.<class_name>.- "source_railz.components.MyCustomRecordExtractor" 
- $parameters- object- Set parameters that are inherited to all children. See the section in the advanced topics for more details. 
CustomRequester object
Requester component whose behavior is derived from a custom code implementation of the connector.
Properties:- class_name- string- Fully-qualified name of the class that will be implementing the custom requester strategy. The format is Example:- source_<name>.<package>.<class_name>.- "source_railz.components.MyCustomRecordExtractor" 
- $parameters- object- Set parameters that are inherited to all children. See the section in the advanced topics for more details. 
CustomRetriever object
Retriever component whose behavior is derived from a custom code implementation of the connector.
Properties:- class_name- string- Fully-qualified name of the class that will be implementing the custom retriever strategy. The format is Example:- source_<name>.<package>.<class_name>.- "source_railz.components.MyCustomRetriever" 
- $parameters- object- Set parameters that are inherited to all children. See the section in the advanced topics for more details. 
CustomPartitionRouter object
Partition router component whose behavior is derived from a custom code implementation of the connector.
Properties:- class_name- string- Fully-qualified name of the class that will be implementing the custom partition router. The format is Example:- source_<name>.<package>.<class_name>.- "source_railz.components.MyCustomPartitionRouter" 
- $parameters- object- Set parameters that are inherited to all children. See the section in the advanced topics for more details. 
CustomTransformation object
Transformation component whose behavior is derived from a custom code implementation of the connector.
Properties:- class_name- string- Fully-qualified name of the class that will be implementing the custom transformation. The format is Example:- source_<name>.<package>.<class_name>.- "source_railz.components.MyCustomTransformation" 
- $parameters- object- Set parameters that are inherited to all children. See the section in the advanced topics for more details. 
DatetimeBasedCursor object
Cursor to provide incremental capabilities over datetime.
Properties:- cursor_field- string- The location of the value on a record that will be used as a bookmark during sync. To ensure no data loss, the API must return records in ascending order based on the cursor field. Nested fields are not supported, so the field must be at the top level of the record. You can use a combination of Add Field and Remove Field transformations to move the nested field to the top. Available variables:
 Examples:- "created_at" - "{{ config['record_cursor'] }}"
- datetime_format- string- The datetime format used to format the datetime values that are sent in outgoing requests to the API. Use placeholders starting with "%" to describe the format the API is using. The following placeholders are available: - %s: Epoch unix timestamp - 1686218963
- %ms: Epoch unix timestamp (milliseconds) - 1686218963123
- %a: Weekday (abbreviated) - Sun
- %A: Weekday (full) - Sunday
- %w: Weekday (decimal) - 0(Sunday),6(Saturday)
- %d: Day of the month (zero-padded) - 01,02, ...,31
- %b: Month (abbreviated) - Jan
- %B: Month (full) - January
- %m: Month (zero-padded) - 01,02, ...,12
- %y: Year (without century, zero-padded) - 00,01, ...,99
- %Y: Year (with century) - 0001,0002, ...,9999
- %H: Hour (24-hour, zero-padded) - 00,01, ...,23
- %I: Hour (12-hour, zero-padded) - 01,02, ...,12
- %p: AM/PM indicator
- %M: Minute (zero-padded) - 00,01, ...,59
- %S: Second (zero-padded) - 00,01, ...,59
- %f: Microsecond (zero-padded to 6 digits) - 000000
- %z: UTC offset - (empty),+0000,-04:00
- %Z: Time zone name - (empty),UTC,GMT
- %j: Day of the year (zero-padded) - 001,002, ...,366
- %U: Week number of the year (starting Sunday) - 00, ...,53
- %W: Week number of the year (starting Monday) - 00, ...,53
- %c: Date and time - Tue Aug 16 21:30:00 1988
- %x: Date standard format - 08/16/1988
- %X: Time standard format - 21:30:00
- %%: Literal '%' character
 - Some placeholders depend on the locale of the underlying system - in most cases this locale is configured as en/US. For more information see the Python documentation. Examples:- "%Y-%m-%dT%H:%M:%S.%f%z" - "%Y-%m-%d" - "%s" - "%ms" 
- %s: Epoch unix timestamp - 
- start_datetime- The datetime that determines the earliest record that should be synced. Type:
 Available variables:
 Examples:- "2020-01-1T00:00:00Z" - "{{ config['start_time'] }}"
- cursor_datetime_formats- array- The possible formats for the cursor field, in order of preference. The first format that matches the cursor field value will be used to parse it. If not provided, the - datetime_formatwill be used.
- cursor_granularity- string- Smallest increment the datetime_format has (ISO 8601 duration) that is used to ensure the start of a slice does not overlap with the end of the previous one, e.g. for %Y-%m-%d the granularity should be P1D, for %Y-%m-%dT%H:%M:%SZ the granularity should be PT1S. Given this field is provided, Example:- stepneeds to be provided as well.- "PT1S" 
- end_datetime- The datetime that determines the last record that should be synced. If not provided, Type:- {{ now_utc() }}will be used.
 Available variables:
 Examples:- "2021-01-1T00:00:00Z" - "{{ now_utc() }}"- "{{ day_delta(-1) }}"
- end_time_option- #/definitions/RequestOption- Optionally configures how the end datetime will be sent in requests to the source API. 
- is_data_feed- boolean- A data feed API is an API that does not allow filtering and paginates the content from the most recent to the least recent. Given this, the CDK needs to know when to stop paginating and this field will generate a stop condition for pagination. 
- lookback_window- string- Time interval before the start_datetime to read data for, e.g. P1M for looking back one month. Available variables:
 Examples:- "P1D" - "P{{ config['lookback_days'] }}D"
- partition_field_end- string- Name of the partition start time field. Example:- "ending_time" 
- partition_field_start- string- Name of the partition end time field. Example:- "starting_time" 
- start_time_option- #/definitions/RequestOption- Optionally configures how the start datetime will be sent in requests to the source API. 
- step- string- The size of the time window (ISO8601 duration). Given this field is provided, Examples:- cursor_granularityneeds to be provided as well.- "P1W" - "{{ config['step_increment'] }}"
- $parameters- object- Set parameters that are inherited to all children. See the section in the advanced topics for more details. 
OAuthAuthenticator object
Authenticator for requests using OAuth 2.0 authorization flow.
Properties:- client_id- string- The OAuth client ID. Fill it in the user inputs. Examples:- "{{ config['client_id }}"- "{{ config['credentials']['client_id }}"
- client_secret- string- The OAuth client secret. Fill it in the user inputs. Examples:- "{{ config['client_secret }}"- "{{ config['credentials']['client_secret }}"
- refresh_token- string- Credential artifact used to get a new access token. Examples:- "{{ config['refresh_token'] }}"- "{{ config['credentials]['refresh_token'] }}"
- token_refresh_endpoint- string- The full URL to call to obtain a new access token. Example:- "https://connect.squareup.com/oauth2/token" 
- access_token_name- string- The name of the property which contains the access token in the response from the token refresh endpoint. Example:- "access_token" 
- expires_in_name- string- The name of the property which contains the expiry date in the response from the token refresh endpoint. Example:- "expires_in" 
- grant_type- string- Specifies the OAuth2 grant type. If set to refresh_token, the refresh_token needs to be provided as well. For client_credentials, only client id and secret are required. Other grant types are not officially supported. Examples:- "refresh_token" - "client_credentials" 
- refresh_request_body- object- Body of the request sent to get a new access token. Example:- { "applicationId": "{{ config['application_id'] }}", "applicationSecret": "{{ config['application_secret'] }}", "token": "{{ config['token'] }}" }
- scopes- array- List of scopes that should be granted to the access token. Example:- [ "crm.list.read", "crm.objects.contacts.read", "crm.schema.contacts.read" ] 
- token_expiry_date- string- The access token expiry date. Examples:- "2023-04-06T07:12:10.421833+00:00" - 1680842386 
- token_expiry_date_format- string- The format of the time to expiration datetime. Provide it if the time is returned as a date-time string instead of seconds. Example:- "%Y-%m-%d %H:%M:%S.%f+00:00" 
- refresh_token_updater- When the token updater is defined, new refresh tokens, access tokens and the access token expiry date are written back from the authentication response to the config object. This is important if the refresh token can only used once. 
- $parameters- object- Set parameters that are inherited to all children. See the section in the advanced topics for more details. 
DeclarativeStream object
A stream whose behavior is described by a set of declarative low code components.
Properties:- retriever- Component used to coordinate how records are extracted across stream slices and request pages. Type:
- incremental_sync- Component used to fetch data incrementally based on a time field in the data. Type:
- name- string- The stream name. 
- primary_key- #/definitions/PrimaryKey- The primary key of the stream. 
- schema_loader- Component used to retrieve the schema for the current stream. Type:
- transformations- array- A list of transformations to be applied to each output record. 
- $parameters- object- Set parameters that are inherited to all children. See the section in the advanced topics for more details. 
DefaultErrorHandler object
Component defining how to handle errors. Default behavior includes only retrying server errors (HTTP 5XX) and too many requests (HTTP 429) with an exponential backoff.
Properties:- backoff_strategies- array- List of backoff strategies to use to determine how long to wait before retrying a retryable request. 
- max_retries- integer- The maximum number of time to retry a retryable request before giving up and failing. Examples:- 5 - 0 - 10 
- response_filters- array- #/definitions/HttpResponseFilter- List of response filters to iterate on when deciding how to handle an error. When using an array of multiple filters, the filters will be applied sequentially and the response will be selected if it matches any of the filter's predicate. 
- $parameters- object- Set parameters that are inherited to all children. See the section in the advanced topics for more details. 
DefaultPaginator object
Default pagination implementation to request pages of results with a fixed size until the pagination strategy no longer returns a next_page_token.
Properties:- pagination_strategy- Strategy defining how records are paginated. Type:- #/definitions/CursorPagination
- #/definitions/CustomPaginationStrategy
- #/definitions/OffsetIncrement
- #/definitions/PageIncrement
 
- decoder- #/definitions/JsonDecoder- Component decoding the response so records can be extracted. 
- page_size_option- #/definitions/RequestOption
- page_token_optionType:
- $parameters- object- Set parameters that are inherited to all children. See the section in the advanced topics for more details. 
DpathExtractor object
Record extractor that searches a decoded response over a path defined as an array of fields.
Properties:- field_path- array- List of potentially nested fields describing the full path of the field to extract. Use "*" to extract all values from an array. See more info in the docs. Examples:- [ "data" ] - [ "data", "records" ] - [ "data", "{{ parameters.name }}" ]- [ "data", "*", "record" ] 
- decoder- #/definitions/JsonDecoder- Component decoding the response so records can be extracted. 
- $parameters- object- Set parameters that are inherited to all children. See the section in the advanced topics for more details. 
ExponentialBackoffStrategy object
Backoff strategy with an exponential backoff interval. The interval is defined as factor * 2^attempt_count.
Properties:- factor- Multiplicative constant applied on each retry. Type:- number
- string
 
 Available variables:
 Examples:- 5 - 5.5 - "10" 
- $parameters- object- Set parameters that are inherited to all children. See the section in the advanced topics for more details. 
SessionTokenAuthenticator object
Properties:- login_requester- #/definitions/HttpRequester- Description of the request to perform to obtain a session token to perform data requests. The response body is expected to be a JSON object with a session token property. Example:- { "type": "HttpRequester", "url_base": "https://my_api.com", "path": "/login", "authenticator": { "type": "BasicHttpAuthenticator", "username": "{{ config.username }}", "password": "{{ config.password }}" } }
- session_token_path- array- The path in the response body returned from the login requester to the session token. Examples:- [ "access_token" ] - [ "result", "token" ] 
- expiration_duration- string- The duration in ISO 8601 duration notation after which the session token expires, starting from the time it was obtained. Omitting it will result in the session token being refreshed for every request. Examples:- "PT1H" - "P1D" 
- request_authentication- Authentication method to use for requests sent to the API, specifying how to inject the session token. Type:- #/definitions/SessionTokenRequestApiKeyAuthenticator
- #/definitions/SessionTokenRequestBearerAuthenticator
 
- $parameters- object- Set parameters that are inherited to all children. See the section in the advanced topics for more details.