This article was co-written by Mayank Singh & Ayush Kumar Singh
Your organization’s data pipelines will inevitably run into issues, ranging from simple permission errors to significant network or infrastructure incidents. To improve visibility and maintain your data SLAs, it’s crucial to have a robust monitoring system that not only logs errors but also can alert the relevant personnel.
In this blog, we’ll show you how Fivetran allows you to do this specifically via triggering a Slack notification when a pipeline fails.
What is an Alert in Fivetran?
Usually, alerts and notifications can be used interchangeably, but when it comes to Fivetran, they serve different functions. Let’s go over some of the most significant differences between the two:
Alerts are messages that are automatically created on the Fivetran dashboard.
If something goes wrong with your Fivetran account (like an incomplete sync or damaged connector), Fivetran alerts you and provides instructions on how to fix it.
An alert is not an account-level issue but more a Fivetran issue mentioned as an alert.
There are two types of alerts: errors and warnings. Errors describe a problem that keeps Fivetran from syncing your data, while warnings describe a problem that you may need to fix but that does not keep Fivetran from syncing your data.
What is Notification in Fivetran?
In Fivetran, a notification is defined as an automatically generated email. It quickly informs you of certain kinds of alerts in your Fivetran account so that you can resolve any potential issues as soon as possible. You can customize notification settings on the notifications page to only receive the notifications that you want.
Previously, Fivetran could only send notifications to Fivetran users, but with recent updates, Fivetran now has the ability to send emails to anyone, even if they’re not Fivetran users. This opens the door to sending notifications to third-party apps like Slack, Teams, etc.
In Fivetran, Depending on your notification settings, we may send you email notifications about new alerts. Here’s how you can do it:
Go to the Notifications tab in Fivetran.
Customize your notification settings to only receive the notifications that you want.
You can also enable or disable alert reminder emails. These emails, whose subject lines always begin with “Reminder,” notify you when your connector has an unresolved Error or delay.
Fivetran sends notifications to the email address that you use to log in to your Fivetran dashboard. You can see this email address at the top of the page.
NOTE: Your settings are unique to your user, so changing them does not affect the settings for other users on your shared account. Depending on your notification settings, Fivetran may send you email notifications about new alerts.
What Are the Types of Notifications in Fivetran?
1. Connector Notifications
MAR Estimate Available
Notifies all subscribed users about Monthly Active Rows (MAR) estimates during the connector’s free trial period.
Connector Trial Ended
Notifies all subscribed users when the free trial of a connector ends.
New Error
In case a connector breaks, Fivetran generates an error in the dashboard and sends the potential resolution in the notification.
Failed Syncs
If three failed syncs are due to an unknown cause, Fivetran sends a notification.
New Source Data
Detects and notifies all subscribed users of any new data object in the source.
Sync Delays
Notification sent out in case of sync delays.
NOTE: The type of notifications you receive depends on your user role and notification subscription.
2. Transformation Notifications
If a transformation fails, Fivetran generates an error in the dashboard and promptly notifies the subscribed users about the details of the error and potential resolution. Additionally, Fivetran can now utilize dbt Cloud for transformations, which are available in the transformation tab.
To learn more about this exciting update, check out our blog, which explores the dbt Cloud integration with Fivetran in more detail.
3. Account Notifications
Monthly Spend Warning
Notification is sent to the Account Administrator or Account Billing Role when the user exceeds the monthly spend threshold.
Low Spend Warning
Notification is sent to the Account Administrator or Account Billing Role 45 days before the account balance runs out.
System Outages
Receive notification in case of Fivetran Outage to a subscribed user/non-user.
Failed Webhooks
If webhooks are configured and the webhook event fails, a notification will be sent out.
Why Use Slack for Fivetran Notifications?
Centralized Communication: By connecting your data pipeline alerts with Slack, you can remove the need to constantly check other platforms for changes, as Slack is frequently the primary communication channel for many teams.
Proactive Monitoring & Faster Troubleshooting: Teams may easily monitor and debug operations by using Slack to receive rapid notifications on pipeline events like task completions and errors. This facilitates faster issue resolution and response times.
Visibility and Collaboration: Messages are arranged by subject in Slack channels and are highly visible to interested parties. Additionally, a message posted to a Slack group will be visible to all group members (including any prior messages made to the group).
Integration with Other Solutions: Slack is a flexible option for a range of tech stacks since it can be coupled with other data pipeline solutions, including Bitbucket Pipelines, Azure Pipelines, and Apache Airflow.
Accessibility: Slack can be viewed from your phone while you’re not at your desk, much like email, so you can always be aware of the status of your data pipeline.
How to Trigger a Slack Notification From Fivetran
With Fivetran, you can send notifications to any email, and the good news is that you can create new channels on Slack, each with a unique email address. Even better news is that it’s easy to assemble; here’s how you do it:
Step 1: Configure Slack for Notifications
Open the Slack desktop application.
Go to the Channels section and click Create, then from the drop-down, select Create channel.
Enter a name and privacy settings for the channel.
Add the members that you want to add to the channel.
Go to the channel and select Send emails to the channel.
If you like, you can customize how emails sent in the channel will appear.
Copy the email address and try sending a test email to it.
If the test is successful, emails sent to that address will now appear in the Slack channel.
Configure your ETL tool to send emails to that address and invite people to join the Slack channel.
Step 2: Configuring Fivetran to Send Notifications to Slack
Go to your Slack channel email address.
From the Fivetran dashboard, click User and Notification.
From there, click Switch Recipient to add the Slack channel’s email address (or any other email address).
Add the desired email addresses and click Create.
Select the type of notification you want to subscribe to and then click Add. In this example, we are only considering connector notifications.
From there, you can filter on source and destinations, select any connectors you want to subscribe to, and click Apply Changes.
From the Notifications panel, we can see all the active notification subscriptions.
In case of a connector failure, you will receive a Slack notification from Fivetran describing the pipeline failure and potential steps to fix the connector.
This is how the failed connector should look in the connector dashboard.
Closing
Monitoring, reviewing errors, and configuring notifications and alerts are crucial in managing any data pipeline.
Slack offers a centralized platform for stakeholders and project tools to communicate and track notifications and alerts in real time, ensuring transparency and traceability. Fivetran’s ability to easily configure ETL pipelines and automatically send failure notifications with possible resolutions to anyone subscribed makes it one of the best tools available in the market.
As Fivetran’s 2024 Partner of the Year, phData boasts a celebrated track record of helping organizations succeed with the Fivetran platform. If your company seeks expert advice, smooth deployment, and continuous assistance with Fivetran, contact the team at phData today!
We’re dedicated to supporting you at every stage of your Fivetran journey!
FAQs
Does Fivetran delete records?
Fivetran soft-deletes data from the destination when it is deleted from the source. Fivetran creates a new column called _fivetran_deleted in the destination table and sets its value to TRUE for the rows that were removed from the source. Fivetran doesn’t delete data from the source and only does a soft delete on the target.
What is an error in the Fivetran dashboard?
An error in the Fivetran dashboard indicates a sync issue with the data. Possible causes include server-side faults, incomplete syncs, unauthorized access, and connection problems. Troubleshooting is aided by the error message displayed on the dashboard. Efficient resolution guarantees continuous data flow for prompt analytics and reporting.
