Skip to main content

Decodable setup

Community plugin

Some core functionality may be limited. If you're interested in contributing, see the source code for the repository listed below.

  • Maintained by: Decodable
  • Authors: Decodable Team
  • GitHub repo: decodableco/dbt-decodable
  • PyPI package: dbt-decodable
  • Slack channel: #general
  • Supported dbt Core version: 1.3.1 and newer
  • dbt Cloud support: Not supported
  • Minimum data platform version: n/a

Installing dbt-decodable

Use pip to install the adapter. Before 1.8, installing the adapter would automatically install dbt-core and any additional dependencies. Beginning in 1.8, installing an adapter does not automatically install dbt-core. This is because adapters and dbt Core versions have been decoupled from each other so we no longer want to overwrite existing dbt-core installations. Use the following command for installation:

Configuring dbt-decodable

For Decodable-specific configuration, please refer to Decodable configs.

Connecting to Decodable with dbt-decodable

Do the following steps to connect to Decodable with dbt.

Prerequisites

In order to properly connect to Decodable, you must have the Decodable CLI installed and have used it to login to Decodable at least once. See Install the Decodable CLI for more information.

Steps

To connect to Decodable with dbt, you'll need to add a Decodable profile to your profiles.yml file. A Decodable profile has the following fields.

~/.dbt/profiles.yml
dbt-decodable:       
target: dev
outputs:
dev:
type: decodable
database: None
schema: None
account_name: [your account]
profile_name: [name of the profile]
materialize_tests: [true | false]
timeout: [ms]
preview_start: [earliest | latest]
local_namespace: [namespace prefix]

Description of Profile Fields

OptionDescriptionRequired?Example
typeThe specific adapter to useRequireddecodable
databaseRequired but unused by this adapter.Required
schemaRequired but unused by this adapter.Required
account_nameThe name of your Decodable account.Requiredmy_awesome_decodable_account
profile_nameThe name of your Decodable profile.Requiredmy_awesome_decodable_profile
materialize_testsSpecify whether to materialize tests as a pipeline/stream pair. Defaults to false.Optionalfalse
timeoutThe amount of time, in milliseconds, that a preview request runs. Defaults to 60000.Optional60000
preview_startSpecify where preview should start reading data from. If set to earliest, then preview will start reading from the earliest record possible. If set to latest, preview will start reading from the latest record. Defaults to earliest.Optionallatest
local_namespaceSpecify a prefix to add to all entities created on Decodable. Defaults to none, meaning that no prefix is added.Optionalnone

Supported features

NameSupportedNotes
Table materializationYesOnly table materialization are supported. A dbt table model translates to a pipeline/stream pair in Decodable, both sharing the same name. Pipelines for models are automatically activated upon materialization. To materialize your models, run the dbt run command which does the following:
  1. Create a stream with the model's name and schema inferred by Decodable from the model's SQL.
  2. Create a pipeline that inserts the SQL's results into the newly created stream.
  3. Activate the pipeline.
By default, the adapter does not tear down and recreate the model on Decodable if no changes to the model have been detected. Invoking dbt with the --full-refresh flag or setting that configuration option for a specific model causes the corresponding resources on Decodable to be destroyed and built from scratch.
View materializationNo
Incremental materializationNo
Ephemeral materializationNo
SeedsYesRunning the dbt seed command performs the following steps for each specified seed:
  1. Create a REST connection and an associated stream with the same name as the seed.
  2. Activate the connection.
  3. Send the data stored in the seed’s .csv file to the connection as events.
  4. Deactivate the connection.
After the dbt seed command has finished running, you can access the seed's data on the newly created stream.
TestsYesThe dbt test command behaves differently depending on the materialize_tests option set for the specified target.

If materialize_tests = false, then tests are only run after the preview job has completed and returned results. How long a preview job takes as well as what records are returned are defined by the timeout and preview_start configurations respectively.

If materialize_tests = true, then dbt persists the specified tests as pipeline/stream pairs in Decodable. Use this configuration to allow for continuous testing of your models. You can run a preview on the created stream with the Decodable CLI or web interface to monitor the results.
SourcesNoSources in dbt correspond to Decodable source connections. However, the dbt source command is not supported.
Docs generateNoFor details about your models, check your Decodable account.
SnapshotsNoSnapshots and the dbt snapshot command are not supported.

Additional operations

dbt-decodable provides a set of commands for managing the project’s resources on Decodable. Those commands can be run using dbt run-operation {name} --args {args}.

For example, the following command runs the delete_streams operation

dbt run-operation delete_streams --args '{streams: [stream1, stream2], skip_errors: True}'
stop_pipelines(pipelines)
  • pipelines: An optional list of pipeline names to deactivate. Defaults to none.

Deactivate all pipelines for resources defined within the project. If the pipelines argument is provided, then only the specified pipelines are deactivated.



delete_pipelines(pipelines)
  • pipelines: An optional list of pipeline names to delete. Defaults to none.

Delete all pipelines for resources defined within the project. If the pipelines argument is provided, then only the specified pipelines are deleted.



delete_streams(streams, skip_errors)
  • streams: An optional list of stream names to delete. Defaults to none.
  • skip_errors: Specify whether to treat errors as warnings. When set to true, any stream deletion failures are reported as warnings. When set to false, the operation stops when a stream cannot be deleted. Defaults to true.

Delete all streams for resources defined within the project. If a pipeline is associated with a stream, then neither the pipeline nor stream are deleted. See the cleanup operation for a complete removal of stream/pipeline pairs.

cleanup(list, models, seeds, tests)
  • list: An optional list of resource entity names to delete. Defaults to none.
  • models: Specify whether to include models during cleanup. Defaults to true.
  • seeds: Specify whether to include seeds during cleanup. Defaults to true.
  • tests: Specify whether to include tests during cleanup. Defaults to true.



Delete all Decodable entities resulting from the materialization of the project’s resources, i.e. connections, streams, and pipelines. If the list argument is provided, then only the specified resource entities are deleted. If the models, seeds, or test arguments are provided, then those resource types are also included in the cleanup. Tests that have not been materialized are not included in the cleanup.

0