The Snowflake Data Cloud is a powerful platform that helps businesses gain more value from their data, but if your business isn’t careful with monitoring usage costs, the overall expense of the leading data cloud platform can skyrocket.
Luckily, that won’t happen to your company because you have phData to guide you.
In this blog, we’ll help you understand the costs of Snowflake, establish how to set a budget, explore how to create resource monitors, and properly leverage Snowflake alerts to ensure that your Snowflake usage remains efficient and cost-effective.Â
As a bonus, we’ll show you how our powerful (and free) Advisor Tool from the phData Toolkit can further curb costs and ensure your Snowflake account runs as efficiently as possible.
Understanding the Cost of Snowflake
Like any other cloud computing tool, costs can quickly add up if not kept in check. The total cost of using Snowflake is the aggregate of the cost of using data transfer, storage, and computing resources. Snowflake’s innovative cloud architecture separates the cost of accomplishing any task into one of these usage types.
Compute Resources
Using compute resources within Snowflake consumes Snowflake credits. The billed cost of using compute resources is calculated by multiplying the number of consumed credits by the price of a credit.
There are three types of compute resources that consume credits:
Virtual Warehouse Compute
Serverless Compute
Snowflake Features such as Search Optimization and Snowpipe
Cloud Services Compute
Behind-the-scenes tasks such as metadata management and authentication
This is only charged if it exceeds 10% of daily warehouse usage
Storage
The monthly cost for storing data in Snowflake is based on a flat rate per terabyte. Storage is calculated monthly based on the average number of on-disk bytes stored each day in your Snowflake account.
Data Transfer
Snowflake does not charge data ingress fees to bring data into your account but does charge for data egress. Snowflake charges a per-byte fee when you transfer data from a Snowflake account into a different region on the same or completely different cloud platform. This per-byte fee for data egress depends on the region where your Snowflake account is hosted.Â
The Snowflake edition being used, the cloud provider, and the region also affect the cost of each credit, as seen in this example for a Snowflake account in AWS Region US-EAST1:
While separating the costs of these usage types is very beneficial to Snowflake consumers, trying to monitor them all can become confusing. Luckily, there are several tools in place to monitor these costs in Snowflake.
Snowflake Internal Tools
Budget
A budget defines a monthly spending limit on the compute costs for a Snowflake account or a custom group of Snowflake objects. When the spending limit is projected to be exceeded, a daily notification email is sent to the email addresses designated for budget notifications.
The following objects are supported for monitoring by a budget:
Tables
Materialized Views
Schemas
Databases
Warehouses
Pipes
Tasks
Budgets monitor the serverless feature usage for the supported objects contained in a budget. Credit usage for the following serverless features is supported:
AUTO_CLUSTERINGMATERIALIZED_VIEWPIPEQUERY_ACCELERATIONREPLICATIONSEARCH_OPTIMIZATIONSERVERLESS_TASKSNOWPIPE_STREAMINGWAREHOUSE_METERINGWAREHOUSE_METERING_READER
To help you get started with what spending limit to set, Snowflake provides you with some guidance on what your credit spend has been for the past three months and an estimate of what your current spend might be (in case you are activating the account-level budget in the middle of the month).
Resource Monitor
Snowflake offers resource monitoring capabilities to aid in cost management and prevent unexpected credit consumption from virtual warehouse operations. Each virtual warehouse consumes Snowflake credits while active. A resource monitor enables tracking credit usage for virtual warehouses and the associated cloud services.Â
If desired, users can pause the warehouse when it reaches a specified credit threshold. The credit consumption is influenced by the warehouse’s size and runtime. Users can establish limits for specific time intervals or date ranges. When these limits are met or approaching, the resource monitor can initiate various actions, including sending alert notifications and suspending user-managed warehouses.
A resource monitor consists of the following properties:
Credit Quota
Number of Snowflake credits where the resource monitor will take a specified action
Includes both user-managed warehouses and cloud services compute
Monitor Level
Determines whether the resource monitor is set to monitor all the warehouses on the account or a subset of them
Schedule
Sets the frequency with which the credit quota resets (Daily, Weekly, Monthly, etc.) along with start and end dates of the resource monitor if desired
Actions
Determines what action the resource monitor should take at specified percentages of the credit quota
Notify and Suspend: Emails a notification and suspends all assigned warehouses after executing current running statements
Notify and Suspend Immediately: Emails a notification and suspends all assigned warehouses ending all currently running statements
Notify: Sends an email notification
Resource monitors can be created, altered, and dropped like other first-class objects in Snowflake. Here is an example of a weekly monitor on an entire account:
CREATE OR REPLACE RESOURCE MONITOR ACCOUNT_CREDIT_LIMIT
WITH CREDIT_QUOTA=1000
NOTIFY_USERS = (JDOE, "JANE SMITH")
FREQUENCY = WEEKLY
TRIGGERS
ON 75 PERCENT DO NOTIFY
ON 100 PERCENT DO SUSPEND
ON 110 PERCENT DO SUSPEND_IMMEDIATE;
Alerts
Alerts are a relatively new feature in Snowflake that allows users to become notified when data changes in a table. This can be especially useful in monitoring costs as the alerts can be used to monitor data in the INFORMATION_SCHEMA, essentially allowing for a finer detail resource monitor.Â
For example, an alert can notify administrators when a warehouse’s sizing has been changed. In Snowflake, each warehouse size doubles the number of credits being consumed, so an admin being notified of a potentially unauthorized size change can be crucial.Â
The following alert will run every two hours to determine if a warehouse has been changed in size by someone other than the WAREHOUSE_ADMIN and will send an email notification:
CREATE OR REPLACE ALERT WAREHOUSE_DDL_CHANGE
WAREHOUSE = PROD_LOAD_WH
SCHEDULE = '120 minute'
IF (EXISTS(
select *
from snowflake.account_usage.warehouse_events_history
where event_name not in ('RESUME_WAREHOUSE', 'RESUME_CLUSTER', 'SPINDOWN_CLUSTER', 'SUSPEND_WAREHOUSE', 'SPINUP_CLUSTER', 'SUSPEND_CLUSTER')
AND USER <> 'WAREHOUSE_ADMIN'
AND TIMESTAMP
BETWEEN SNOWFLAKE.ALERT.LAST_SUCCESSFUL_SCHEDULED_TIME() AND SNOWFLAKE.ALERT.SCHEDULED_TIME()
))
THEN CALL SYSTEM$SEND_EMAIL(
'email_alert',
'jane.doe@example.com; john.doe@example.com',
'EMAIL ALERT: Snowflake Warehouse DDL Change',
'At least one warehouse''s ddl has been changed in Snowflake.
Run the following query to view the warehouses that changed:
select *
from snowflake.account_usage.warehouse_events_history
where event_name not in (''RESUME_WAREHOUSE'', ''RESUME_CLUSTER'', ''SPINDOWN_CLUSTER'', ''SUSPEND_WAREHOUSE'', ''SPINUP_CLUSTER'', ''SUSPEND_CLUSTER'')
and USER_NAME <> ''WAREHOUSE_ADMIN''
and TIMESTAMP >= current_date-1;'
);
Other cost monitoring use cases include notifying if a query is running long or if a warehouse has become queued. Having alerts for these will allow the admin to take action on specific queries/warehouses before excessive credits are consumed.
phData Advisor Tool
Utilizing all of the Snowflake internal tools for monitoring costs can be very beneficial for consumers, but also having a pre-configured dashboard and alert system can take monitoring your Snowflake account to the next level. That’s why phData created the Advisor Tool, which monitors not only cost efficiency but also the security and performance of your Snowflake account.
With the Advisor Tool, you can quickly and easily identify opportunities to improve the configuration, security, performance, and efficiency of your Snowflake environment. We have worked closely with our Snowflake data and operations experts to ensure that our Snowflake Advisor Tool delivers actionable recommendations based on best practices and phData’s vast real-world experience on the Snowflake Platform.
The Advisor Tool comes with a pre-configured set of checks to identify opportunities and provide recommendations that ensure your Snowflake environment is following best practices and industry standards across a variety of categories, including:
Configuration Best Practices: Recommendations for your Snowflake environment configuration based on best practices and industry standards.
Security Enhancement: Identify and fix potential security risks in your account and maintain a compliant security posture.
Operational Risks: Uncover operational risks such as data loss or failures in the event of an unforeseen outage or disaster.
Performance Optimization: Locate and fix bottlenecks in your data pipelines so that you can get the most out of your Snowflake investment.
Resource Utilization: Optimize the use of resources, such as computing and storage, to ensure that you are not overpaying for resources you don’t need.
Environment Scale and Capacity: View critical metrics about the scale and features used in your account to help you understand your current state and plan for growth.Â
The check results are provided in a rich interactive report for users to view and use to address them. The default set of checks run in a report can be customized, and new checks can be defined using simple configuration options to tailor the reports to your organization’s needs.Â
Once all the checks pass, the Advisor Tool can be run on a schedule and automatically alert if checks fail going forward.
Each check in the report can be drilled into to find a clear description of the check, including any recommended actions that should be taken based on the result. A customizable graph is also included on the check details page for more informational checks or multiple rows to help visualize and understand the presented results quickly.
This tool is accessible to everyone and is now available for free on the Snowflake Marketplace.
Ready to see the Advisor Tool in action?
Visit the Snowflake Marketplace and revolutionize your data strategy today.
Closing
Like any other cloud-based service, Snowflake costs can get out of control quickly if not monitored and kept in check. With the help of Snowflake’s built-in tools and the strategies covered in this blog, your business can stay on top of it before it gets out of hand.Â
If you have additional questions on monitoring costs in Snowflake, the Snowflake experts at phData can help! Contact us today for advice, best practices, actionable strategies, and more!
FAQs
Yes, Data Dog and New Relic provide tools with templated dashboards and pre-configured alerts to monitor Snowflake accounts. For even more advanced use cases, phData created the Platform Monitoring tool for its customers to monitor costs as well as efficiency within Snowflake. Users can also utilize their own BI tooling, Power BI, or Sigma Computing to create monitoring dashboards.
Here is an example of a credit monitoring dashboard phData created for it’s own account in PowerBI:
There are best practices for setting data retention policies and managing historical storage costs in Snowflake. Effectively managing historical data storage is crucial for cost optimization and maintaining the performance of your Snowflake environment. Some best practices include utilizing Time Travel and Data Sharing to avoid data duplication, working with your data stewards to understand historical data significance, and automating cleanup of unused data.
