How to Create Webhook Notifications in dbt Cloud Services

Using dbt Cloud services, you can set up outbound webhooks capable of communicating with additional devices and sending notifications or event updates, which a system can subscribe to. This is a useful feature to help automate your workflows or trigger a workflow from these webhooks.

In this blog, we will cover the basics of what a webhook really is, how it work, and how to create one.

Before we dive into the blog, let’s examine what a webhook is essentially.

What is a Webhook?

A webhook is a simple HTTP-based callback function. It assists in event-triggered communication between multiple systems using different web apps. Webhooks address the need for real-time updates about your dbt jobs running on your systems.

Without webhooks, you would need to repeatedly make API calls to receive the same updates, which saves us from polling repeatedly. Webhooks are very important for production systems and infrastructure development.

How Does a Webhook Work?

Simply put, dbt Cloud sends a JSON payload to the app’s endpoint URL once your webhook is triggered. For example, if your dbt jobs fail, you can automate it to send a Slack or Microsoft Teams notification.

You can setup custom webhooks for the following events from the dbt Cloud web-based UI and by using the dbt Cloud API:

job.run.started — Run started.
job.run.completed — Run completed. This can be a run that has failed or succeeded.
job.run.errored — Run errored.

dbt Cloud has several failsafe methods in case your webhook delivery fails. In such a case, dbt Cloud will retry the webhook delivery five times. There’s also a provision for webhook delivery logs for up to 30 days. Every webhook you set up has a section called ‘Recent Deliveries’ where you can check the delivery logs at a glance.

A webhook in dbt Cloud has a run time of 10 seconds, after which it will time out. If the app’s endpoint URL does not respond in those 10 seconds, then the webhook processor will timeout and retry again later.

It’s important to note that this may cause a gap in status staging as you might connect successfully after 10 seconds, and dbt Core will register it as a success. However, dbt Cloud will log it as a failure as it is not timed out at the end of those 10 seconds. dbt also has a free self-paced learning course on webhooks if you’re interested in learning more.

Now that we have some background information on webhooks, it’s time to create one in dbt!

Prerequisites of Creating a Webhook

A few of the must-haves for setting up webhooks are as follows:

Have access to a dbt Cloud account that is associated with a Team or Enterprise plan.
Ensure that you have write access to the webhooks and that it differs for both account types:
- Enterprise Plan Account – The same permission sets work on both the dbt Cloud UI and API service token, allowing you to access them. The Permission sets are Developer, Admin, and Account Admin, which have write access to the webhooks.
- Team Plan Account – Someone with a Developer license can access the dbt Cloud UI, but you need Member or Account Admin permission set assigned to the service token to have write access on the webhooks.
Access to a multi-tenant deployment model in dbt Cloud, and you can find more information about tenancy here.

How to Create a Webhook Notification in dbt Cloud Services

Step 1: Create a Webhook Subscription

You should first find the correct dbt Cloud URL for your respective region and plan at this link of Regions and IP addresses. After logging into dbt Cloud:

Select Account.
Click on Settings (gear icon in the top right corner).
Click on Create New Webhook in the Webhooks section.

The steps to configure your new webhook are as follows:

Name — Choose and enter a name for your webhook.
Description — Enter a description to convey brief details of the webhook.
Events — Select the event that you want the webhook to trigger. Multiple events can also be subscribed to from the same webhook.
Jobs — In this step, you need to specify the job or jobs that you want the webhook to be triggered for. To trigger the webhook for all the jobs running on your system, all you need to do is leave this field empty. dbt Cloud configures the webhook at an account level by default.
Endpoint — Then you must enter your URL endpoint where dbt Cloud needs to send the event or events.

When you are done with the setup, click Save. dbt Cloud also has built-in procedures to authenticate webhook requests using secret tokens, which can be used to check how authentic the webhook is in order to avoid falling prey to spoofed requests.

Step 2: Possible Webhook Completion Status

The webhook can end in job.run.completed and job.run.errored events. The errored event is a subset of the completed events. Subscribing to both events would give you an error, but those events would be triggered at different times.

Completed – This event is triggered when artifacts and the job metadata are ingested and available via the discovery API or dbt Cloud Admin.
Errored – This event is fired immediately upon encountering an error, and the artifacts and metadata of the job might not have been ingested and cannot be accessed.

Step 3: How to Validate a Webhook

The secret token that you can generate from dbt Cloud can be used to validate the webhook. Any official webhook will always include the Authorization header, which has the SHA256 hash of the request body and will use the secret token as the key.

The following code snippet in Python will help you validate the webhook’s authenticity:

				
					
def validate():
    auth_header = request.headers.get('authorization', None)
    app_secret = os.environ['MY_DBT_CLOUD_AUTH_TOKEN'].encode('utf-8')
    signature = hmac.new(app_secret, request_body, hashlib.sha256).hexdigest()
    return signature == auth_header

Step 4: Properly Inspect HTTP Requests

Following the best practices and regulations, you should use tools to analyze your response payloads, headers, and HTML requests. Before incorporating any webhook into your system, you should properly debug and test them. Given below are some examples of payloads.

JSON Payloads

This is how a webhook payload starts to run:

An example of a completed run:

An example of an Errored run:

Step 5: dbt Cloud API for Webhooks

You can use the dbt Cloud API to create, modify, and delete webhooks. You can subscribe to those webhooks, manage them, and get detailed information about your webhooks. The following API endpoints are available to perform the tasks:

1. List all Webhook Subscriptions

You can use the following endpoint to list all the webhook subscriptions that are available in the specific dbt Cloud account:

Endpoint

				
					GET request at 
'https://cloud.getdbt.com/api/v3/accounts/{account_id}/webhooks/subscriptions'

The required parameters of the URL are accountId

2. Retrieve Details About a Webhook Subscription

You can use the following endpoint to get the specific details about a webhook:

Endpoint

				
					GET request at 
'https://cloud.getdbt.com/api/v3/accounts/{account_id}/webhooks/subscription/{webhook_id}'

The required parameters of the URL are accountId and webhook_id of the webhook you want to retrieve.

3. Create a new Webhook Subscription

You can create a new webhook subscription and mention the URL that will be listening to the webhook events.

Endpoint

				
					POST request at 
'https://cloud.getdbt.com/api/v3/accounts/{account_id}/webhooks/subscriptions'

The required parameters of the URL are accountId which you want to subscribe to. The body of the request needs various parameters, which are as follows:

event_types: The event type where you want to trigger the webhook, which can also be more than one. The possible values are a job.run.started, completed, and errored.
name: Provide the name of the webhook.
client_url: You need to provide the application’s endpoint URL to where dbt Cloud will send the events.
active: Provide a boolean value to toggle your webhook on or off. The acceptable values are true and false.
description: You need to enter a description for the webhook.
job_ids: You need to enter the specific jobs on which you want the webhook to get triggered. If you leave this list of numeric values empty, it will be triggered for all the jobs in the account by default.

4. Edit or Update a Webhook

You can edit and update a webhook from this endpoint:

Endpoint

				
					POST request at 
'https://cloud.getdbt.com/api/v3/accounts/{account_id}/webhooks/subscription/{webhook_id}'

The required parameters of the URL are accountId and webhook_id which you want to update.

The body of the request has various parameters, which are as follows:

event_types: Update the event type where you want to trigger the webhook. The possible values are a job.run.started, completed, and errored.
name: Change the name of the webhook.
client_url: Update the application’s endpoint URL where dbt Cloud will send the events.
active: Change the boolean value to toggle your webhook on or off. The acceptable values are true and false.
description: Update the description for the webhook.
job_ids: Change the jobs on which you want the webhook to get triggered. If you leave this list of numeric values empty, it will be triggered for all the jobs in the account by default.

5. Test or Validate a Webhook

You can use this endpoint to test or validate a webhook:

Endpoint

				
					GET request at 
'https://cloud.getdbt.com/api/v3/accounts/{account_id}/webhooks/subscription/{webhook_id}/test'

The required parameters of the URL are accountId and webhook_id which you want to test.

6. Remove or Delete a Webhook

You can use this endpoint to delete or remove a webhook:

Endpoint

				
					DELETE request at 
'https://cloud.getdbt.com/api/v3/accounts/{account_id}/webhooks/subscription/{webhook_id}'

The required parameters of the URL are accountId and webhook_id which you want to delete.

Step 6: dbt Cloud Job Notifications

Now it’s time to set up notifications in dbt Cloud, allowing you to receive updates via email, Teams message, or a Slack channel for when a job succeeds, is canceled, or fails.

1. Notifications via Email

As a User and an Admin, you can set up email notifications for your dbt Cloud account. To be more specific:

User – Can set email notifications for themselves under the profile tab.
Admin – Can set up notifications on behalf of their team members.

The steps to set up an email notification are:

First, click on the gear menu at the top right corner and go into Notification Settings.
Then, select the Edit button to start editing the email notifications in settings.
- Users: Select the notification type, which can be Canceled, Success, or Failure for the jobs you want to receive notification for.
- Admin: The same as users, except you can choose one or more users from the dropdown for which you would be setting up the email notification
Click on Save

2. Notifications via Slack

You can receive Slack notifications from dbt Cloud for the success, cancellation, or failure of a dbt Cloud job. You need to be an admin of the Slack workspace, and one Slack channel notification can be only one user and not multiple Slack users for that channel. First, you need to set up the integration and then enable notifications. The steps are as follows:

Setup the Slack integration
- Select the gear icon on the top right and select the Profile part.
- Select the integrations on the left side.
- Select the link to connect your Slack profile and complete the process
- Allow Access to Slack
Configure the Slack notifications
- Select the gear icon on the top right and select Account Settings from the dropdown.
- Select Slack Notifications on the left and select Edit.
- You will find the Slack notification settings at the bottom of the page.

Note: To disable the Slack integration, return to the integrations page and disconnect your Slack from dbt Cloud.

Conclusion

We hope that this blog was beneficial in your journey to set up dbt Cloud webhooks and job notifications. If you need more help setting up a webhook or job notification, feel free to contact us! Our dbt experts would be more than happy to work with you to solve your problem.

FAQs

What is a webhook notification?

Webhooks are messages from applications without polling any API when an event happens. The messages are sent to a specified URL. They are efficient, faster than polling, and act like notifications for various applications.

What is the use of webhook?

Webhooks have a variety of tasks, from receiving data to triggering workflows in DevOps or other environments.