The modern data stack (MDS) has seen massive changes over the past few decades, fueled by technological advances and new platforms. As a result, we are presented with specialized data platforms, databases, and warehouses. With the rise of cloud computing, the MDS has evolved even further to include cloud-based storage and tools for analysis. Today, the MDS is composed of multiple players. All of which have a specific role used to collect, store, process, and analyze data.
Introduced late last year, Sigma Computing now has a new collaborator. dbt!
This blog will hone in on the new collaboration, how to implement it into your workbooks, and why Sigma users should be excited about the feature.
Please keep in mind this new feature in Sigma is currently in beta. It is subject to change, with the goal of maximizing its efficiency and ease of use.
To request access, reach out to your Sigma Account Representative today!
Platform and More
dbt is a database deployment & development platform. It is version-controlled and scalable, maintains referential integrity, and tests/deploys database objects. Think of it as a transformation tool; dbt is the T in the extract, load, transform process (ELT). It is a compiler and a runner. Using SQL-centric transformations to model data to be deployed.
dbt is also great for data lineage and documentation to empower business analysts to make informed decisions on their data. dbt’s vision is to popularize creating data as a product with software engineering best practices, with team collaboration as its cornerstone.
New to using dbt with Snowflake? Go check out our Beginners Guide to Using dbt With Snowflake.
Sigma is a true data exploration platform. Specializing in spreadsheet formulas and calculations to create dashboards and visuals (elements). It can also implement embedded data product applications. Sigma shares a similar vision as dbt, focused on making collaboration as simple as possible. Their design is intentional, built for efficiency, and to make data-driven decisions. As a result, this leads its users to answer more and more questions about their data.
Interested in learning more about Sigma Computing? Everything You Need to Know is just a click away.
The Modern Data Stack
Above, we have an image of the Modern Data Stack (MDS). It is composed of layers, with each platform/tool having its own function.
Data Ingestion with Fivetran
Fivetran is used to move your source(s) into a centralized space for storage.
Data Storage with Snowflake
Snowflake is the main data warehouse, the foundation. Storing all the collected data sent from Fivetran Once in Snowflake, the data is ready to be accessed and analyzed.
Data Transformation with dbt
Once the data is moved to storage, it now needs to be transformed into a more user-friendly model. The transformation job allows analysts to query data and extract insights, build dashboards/visuals, and easily make more informed decisions.
Data Consumption with Sigma
Here we are using Sigma to explore our data, create visuals, and make business decisions.
Benefits of the Integration
Data Freshness
Ever get stuck and wonder, “Is this data fresh?” Or “When was this last updated? How long have we been using old data?” I know I have! Now that your data is loaded in using dbt, one can see the data displayed in Sigma itself, allowing the user to verify how up-to-date their data is.
Data Quality
View dbt quality tests on columns and models, providing precision and transparency into your data quality questions and concerns – What a relief. Now we have one spot to check if the data is accurate. Brilliant!
Data Cataloging
Descriptions of your columns from your dbt table are displayed, leading to more exploration of your data – Instead of jumping back and forth between Snowflake and Sigma, all of your data details are in one, easy to understand space.
- Data Lineage
See exactly how your data is connected, edit on the fly, and follow your outputs as they flow downstream to your final dashboard and elements.
Is dbt an Ideal Fit for YOUR Organization’s Data Stack?
Ideal
No centralized code repository or collaboration
Prefer SQL for model definition
Existing raw data sources for the data platform
You have tried to use Snowflake’s native Tasks and Scheduling and are experiencing pain points around visibility and troubleshooting.
An existing manual change management plan
No one person would do a manual “release”
No process on execution
Not Ideal
Prefer low-code/no-code route
Dbt is code and engineer centric. The focus is on SQL, which is easier to learn but can still prove to be a barrier.
Having a singular ELT/ETL solution is a firm requirement
Dbt does not handle ingestion well, it works best when paired with another ingestion tool such as Fivetran.
How to Add dbt to Your Workbooks
To get the add-on, you must be your organization’s Admin and can grant access to Creator roles enabling them to view the dbt data.
Steps
At this point, you have your dbt models ready to load into Sigma.
As the admin, begin the process by opening your admin portal by selecting Administration at the top right. Next, on the Account tab, go down to the Integrations section, find dbt, and click Add.
Next, go to the Deploy dropdown menu, search, and select the Job IDs that you want to bring into Sigma.
Once selected, on the dbt Integration page, enter your dbt API key. In the section below, enter your Job IDs from earlier. Make sure to format the field(s) as a comma-separated list of integers.
Then click Create.
At this point, you can view the metadata for the successful dbt job, tables, and columns – all with descriptions, quality test status, and job execution data. In the details section, you can see the integrated metadata from your dbt model(s).
In the Details section, looking at the Last Modified field, one can now verify how fresh the dbt job is and if it ran successfully.
Once your dbt job has been integrated successfully, you will be prompted with the option to remove the integration. Proceed as you see fit.
Closing
The integration of dbt into Sigma has made data modeling and analytics more accessible, efficient, and accurate for organizations. As a result, Sigma has empowered its users to leverage dbt’s robust modeling capabilities to create reliable, scalable, and reusable models that drive business insights.
The collaboration aspect of the integration will enable teams to work more collaboratively by creating a seamless workflow from data modeling to analysis. With Sigma’s user-friendly, low-code interface and dbt’s power, data teams can work together to create and share analyses and dashboards that help drive informed decision-making across the organization.
dbt’s addition of data freshness, quality, and cataloging is just another example of Sigma’s vision.
As a platform in an ever-changing landscape, this collaboration will play a pivotal role in the success of data-driven organizations in the future.
Interested in learning more about Sigma Computing? This 6 Things to Know blog is an informative read, sure to make you even more excited about Sigma.
