Last updated:

Migrating PySpark to Snowflake with CoCo

By Andy Bouts

AI Overview
  • Migration from PySpark to Snowflake has historically meant weeks of manual rewrites; Snowflake Cortex Code (CoCo) compresses that work into a single prompt by acting as an autonomous engineer inside the Snowsight UI.

  • CoCo handles the full migration sequence — analyzing source notebooks, mapping tables, converting Spark loads to native Snowflake SQL, and migrating Python logic — and self-heals when it hits runtime errors, such as duplicate bin edges in RFM scoring, without engineer intervention.

  • A real-world CLV migration from a 7-billion-record TPC-DS dataset produced 174,554 customer records in a functional Snowflake Notebook in minutes, with engineers spending their time on final validation rather than line-by-line rewrites.

  • Human review remains necessary: CoCo gets teams 90%+ of the way through a migration, but engineers still need to validate self-healed logic against business definitions and confirm SQL execution plans for their warehouse configuration.

The “migration tax” is real. When teams move workloads between platforms, the cost lands on engineers who spend days refactoring PySpark logic, re-mapping dependencies, and debugging transformations that ran fine in Spark but choke in a new environment. It’s time that doesn’t show up in a project plan until it’s already gone.

CoCo changes that calculus. CoCo is Snowflake’s native AI coding agent, built to plan, execute, and self-correct data engineering tasks directly inside the Snowsight UI. In this post, I’ll walk through a live PySpark-to-Snowflake notebook migration using CoCo, including the moment it fixed its own bug without any prompting. 

This post walks through that migration end to end, from upload to a validated Snowflake Notebook. 

The use case: customer lifetime value on TPC-DS

The source notebook is a .py file built for the TPC-DS dataset. It processes three tables: WEB_SALES (over 7 billion records), CUSTOMER, and WEB_RETURNS. The goal is to calculate Recency, Frequency, and Monetary (RFM) scores to produce a Customer Lifetime Value (CLV) model.

Traditionally, moving this notebook to Snowflake would mean a manual rewrite into Snowpark, Snowflake’s Python library for writing data pipelines natively in the platform. RFM scoring logic, Spark DataFrame operations, and visualization code would all need to be converted by hand. For a 300+ line notebook over a multi-billion-row dataset, that’s a material time investment.

Here’s what the agentic path looks like instead.

Step 1: Upload the source file

The process starts in Snowflake Workspaces: no staging, no pipeline, no pre-processing. Use the + button to upload PySpark_Notebook_CLV_RFM.py directly. At this point, the file is raw Python sitting in your workspace. CoCo knows what to do with it.

Step 2: Prompt CoCo

The CoCo icon (a blue diamond) lives in the bottom-right of the Snowsight UI. Select the uploaded file, then send a prompt. Here’s exactly what I typed — typos included:

				
					migrate this attached pyspark notebook to a snowflake notebook that can run in workspaces. The purpose of the notebook it do to customer lifetime value.
				
			

CoCo parses intent, not just syntax. The typos didn’t matter.

How CoCo plans and executes the migration

CoCo runs a structured multi-step orchestration rather than generating code in a single pass. After analyzing all 300+ lines of the notebook, it produced this execution plan:

  • Initialize: Create the .ipynb file and configure the environment.

  • Data mapping: Identify TPCDS_SF10TCL tables in the sample data.

  • SQL transformation: Convert Spark loads into native Snowflake SQL cells.

  • Python logic: Migrate data cleaning and RFM scoring.

  • Visualization: Generate distribution and scatter plots.

  • Persistence: Save results to a permanent Snowflake table.

Each step runs in sequence. CoCo reports back on progress and surfaces issues as they arise.

The self-heal in action

This is the most practically useful part of CoCo for migration work. During execution, CoCo hit a runtime error: duplicate bin edges during RFM scoring. This is a common distribution problem when quantile-cutting imbalanced data. The qcut function throws an error if bucket boundaries overlap.

CoCo identified the failure, diagnosed the cause, applied the fix (duplicates=’drop’ to the qcut call), and re-ran the cell successfully. No engineer intervention. No Stack Overflow. The agent reasoned through the error and corrected it in the same session.

That kind of failure handling is where agentic migration tools earn their keep. A human engineer would have caught it too, but CoCo caught it faster and documented the fix inline.

Human review still matters

CoCo gets teams 90%+ of the way through a migration in minutes. The final 10% is where engineering judgment earns its value.

As the person responsible for validating this output, I needed to confirm two things: that the self-healed RFM logic matched our business definitions for scoring segments, and that the final SQL execution plans were tuned for our specific warehouse size. CoCo produced correct, functional code. Validating that the code is right for your context is still a human task.

This is a feature, not a limitation. Agentic tools move engineers off boilerplate and onto the decisions that require domain knowledge.

The result

Within minutes, the migration produced a functional Snowflake Notebook with:

  •  Native SQL cells for high-performance data loading from the TPC-DS tables

  •  Pandas DataFrames returned directly from SQL cells (native in Snowflake)

  •  174,554 customer records written to a permanent Snowflake table, ready for downstream marketing use

That output came from a 7-billion-row dataset. The migration required one prompt.

What's next

This walkthrough covers the Snowsight UI path, fast, accessible, and good for individual notebooks. For teams migrating dozens of notebooks at once, the same agentic approach extends to the CoCo CLI alongside the Snowflake CLI for automating these pushes at the object level.

Ready to accelerate your PySpark-to-Snowflake migration?

Explore our data engineering services to see how CoCo can transform your data workloads.

FAQs

CoCo is Snowflake’s native AI coding agent, available inside the Snowsight UI. For PySpark migration, it analyzes source notebooks, maps tables to Snowflake equivalents, converts Spark DataFrame operations to native Snowflake SQL, and migrates Python logic, all from a single natural language prompt. It also self-heals runtime errors during execution without requiring engineer intervention.

In practice, a 300+ line PySpark notebook processing multi-billion-row TPC-DS tables migrated to a functional Snowflake Notebook in minutes using CoCo. Final validation — confirming business logic and SQL execution plans — adds additional time depending on notebook complexity.

Yes. CoCo handles roughly 90% of the migration work automatically, including code conversion and error correction. Engineers are still needed to validate that self-healed logic aligns with business definitions and that SQL execution plans are optimized for the target warehouse size. CoCo removes the boilerplate; engineers handle the judgment calls.

Snowpark is Snowflake’s Python library for writing data pipelines that run natively on Snowflake compute. Snowflake Notebooks are an interactive, cell-based environment inside Snowsight that supports both SQL and Python cells. For notebook-style PySpark code, migrating to Snowflake Notebooks preserves the interactive workflow while shifting execution to Snowflake’s engine.

Yes. The migration described in this post ran against a TPC-DS dataset with over 7 billion records in the WEB_SALES table. CoCo planned and executed the migration at the code level; the resulting Snowflake Notebook processes the full dataset using Snowflake’s native compute.

More to explore

Claude partner network
Anthropic

phData Raises New Funding to Turn Enterprise Data Into Trusted AI Answers and Reliable Agents with Claude

phData

·

October 28, 2026

Read article

Read article
aws golden jacket
AWS

The road to the AWS Golden Jacket: 13 exams, one foundation

Ryan Leonard

·

October 26, 2026

Read article

Read article
engineering productivity
AI & ML

How AI affects engineering productivity

Daniel Pierce

·

September 28, 2026

Read article

Read article