Databricks Migration Services

Section 02

Go / No-Go Assessment

✅ Migrate to Databricks if:

• Hadoop hardware EOL + $500K+ annual support costs
• AWS EMR clusters idle >40% of time (cost opportunity)
• ML teams blocked by legacy infrastructure (data scientists extracting to S3)
• Snowflake ETL costs >$200K/year (9x cost difference vs. Databricks)
• Multi-cloud data unification (Unity Catalog governance)

⚠️ Caution if:

• Source system <50TB data (migration effort may not justify ROI)
• Team lacks Spark/Python skills + no training budget ($50K-$100K for certification)
• On-prem compliance requirements (rare, but happens in government/defense)
• BI-only workloads with no AI/ML plans (Snowflake may be better fit)

❌ Don't migrate if:

• Annual infrastructure costs <$200K (ROI break-even hard to achieve)
• No executive buy-in for cloud migration
• Organization unwilling to adopt Delta Lake format (vendor lock-in concern)

---

Section 03

Top 3 Failure Modes

40% of Databricks migrations fail. Click for prevention strategies.

1. DBU Cost Explosion (60% of projects)

What Happens:

• DBU consumption spikes 3-5x vs. initial estimates
• Monthly bills hit $50K-$100K+ in first 3 months
• CFO demands explanation, project credibility damaged

Root Causes:

• Idle clusters (no auto-suspend configured)
• Oversized clusters (porting Hadoop configs directly: "16-node MapReduce cluster → 16-node Spark cluster")
• Unoptimized Spark code (using Parquet instead of Delta Lake)
• No FinOps monitoring (resource tagging, cost allocation)

Prevention:

• Auto-Suspend Policy: 5-minute idle timeout (default is 120 minutes!)
• Right-Size Clusters: Start with 1/3 of Hadoop cluster size, scale up based on metrics
• Delta Lake Migration: Convert Parquet → Delta Lake (3-5x faster queries, lower scan costs)
• Resource Monitors: Hard limits on warehouse spending (trigger alerts at 80% budget)
• Photon Engine: Leverage for SQL workloads (2x faster, 40% cheaper)

Real Example: Fintech company migrated from Snowflake. Initial budget: $15K/month DBU. Actual: $45K/month. Root cause: No auto-suspend, oversized clusters. Post-optimization: $18K/month.

---

2. Unity Catalog Misconfiguration (35% of projects)

What Happens:

• Data governance gaps emerge 6 months post-migration
• Unauthorized access to sensitive data (GDPR/CCPA violations)
• Audit failures (no lineage tracking, no access logs)

Root Causes:

• "Lift-and-shift" approach: Migrating Hive metastore as-is without Unity Catalog transformation
• External tables bypassing governance controls
• Incorrect RBAC configuration (too permissive or too restrictive)

Prevention:

• Transformational Approach: Redesign metastore structure for Unity Catalog (catalogs → schemas → tables)
• Managed Tables: Convert external tables → managed tables for full governance
• RBAC from Day 1: Define role-based access before cutover
• Phased Rollout: Pilot with 1 business unit, validate governance, then scale

Real Example: Healthcare org migrated Hadoop → Databricks. Used external tables for "quick win." 3 months later: HIPAA audit failure (no access control on external S3 buckets).

---

3. Incomplete Inventory (30% of projects)

What Happens:

• Undocumented Spark jobs discovered mid-migration
• Production dashboards break (dependencies not identified)
• Timeline slips by 3-6 months

Root Causes:

• Poor discovery process (only documented jobs included)
• Ad-hoc notebooks not tracked (data scientists running "side projects")
• Lack of lineage mapping (upstream/downstream dependencies unknown)

Prevention:

• Automated Discovery: Use Databricks metadata APIs to inventory ALL jobs/notebooks
• Lineage Analysis: Map data flows (source → transformation → destination)
• Stakeholder Interviews: Survey ALL data consumers (not just IT)
• Buffer Timeline: Add 20-30% contingency for undocumented workloads

Real Example: Retailer migrated AWS EMR. Discovered 47 undocumented Spark streaming jobs (real-time inventory system). Had to extend migration by 4 months.

---

Section 04

5 Technical Traps

1. The "MapReduce Translation" Trap

Trap: Direct 1:1 translation of Hadoop MapReduce jobs to PySpark.

Why It Fails:

• MapReduce is disk-based (writes intermediate results to HDFS)
• Spark is memory-based (in-memory RDDs)
• Delta Lake adds indexing, Z-ordering, data skipping → 3x faster than naive PySpark

Example:

# ❌ WRONG: Direct MapReduce translation
rdd = sc.textFile("s3://bucket/data/*.txt")
mapped = rdd.map(lambda line: parse(line))
reduced = mapped.reduceByKey(lambda a, b: a + b)
reduced.saveAsTextFile("s3://output/")

# ✅ RIGHT: Delta Lake + DataFrame API
df = spark.read.format("delta").load("s3://bucket/data/")
result = df.groupBy("key").agg(sum("value"))
result.write.format("delta").mode("overwrite").save("s3://output/")

Impact: 60% faster, 40% cheaper. Delta Lake uses statistics, indexing, caching.

---

2. Cluster Sizing Anti-Patterns

Trap: Porting Hadoop cluster configs directly to Databricks.

Example:

• Hadoop: 16-node cluster (256GB RAM/node, 4TB total)
• Naive migration: 16-node Databricks cluster

Why It Fails:

• Spark is more memory-efficient than MapReduce
• Photon engine adds 2x performance → need fewer nodes

Right-Sizing Formula:

• Start with 1/3 of Hadoop cluster size
• Example: 16-node Hadoop → 5-node Databricks
• Monitor CPU/memory metrics, scale up if needed

Hidden Cost: Oversized cluster = 3x DBU burn rate.

---

3. Snowflake to Databricks SQL Syntax Differences

Trap: Assuming Snowflake SQL works on Databricks SQL without changes.

Common Incompatibilities:

Solution: Use automated SQL translation tools (Deloitte Migration Factory, phData toolkit) + manual validation.

---

4. Data Quality Exposure

Trap: Assuming source data is clean.

What Happens:

• Migration exposes data quality issues that were masked by legacy systems
• Example: Hadoop tolerates schema drift (no enforcement). Databricks Delta Lake enforces schema → migration fails.

Common Issues:

• Null values in "NOT NULL" columns
• Schema mismatches (source has 10 columns, target expects 12)
• Duplicate primary keys

Prevention:

• Pre-Migration Data Profiling: Analyze source data for completeness, accuracy
• Data Quality Framework: Implement Great Expectations or similar validation
• Incremental Cutover: Migrate 10% of data first, validate before full cutover

---

5. Streaming Jobs Performance Ceiling

Trap: Running 100+ Delta Live Tables (DLT) streaming jobs simultaneously.

Why It Fails:

• Each streaming job holds cluster resources
• 100+ jobs → system crashes, slow processing

Solution:

• Batch Aggregation: Combine multiple sources into fewer DLT pipelines
• Z-Ordering: Optimize Delta tables for read performance (OPTIMIZE table Z-ORDER BY (col))
• Auto Loader: Use for efficient ingestion (cheaper than DLT for simple cases)

Performance Target: <50 concurrent streaming jobs per workspace.

---

Section 05

Migration Roadmap

Phase 1: Discovery & Assessment (Months 1-3)

Activities:

• Inventory ALL jobs, tables, notebooks (automated + manual discovery)
• Analyze source system usage patterns (query logs, job frequency)
• Define Unity Catalog architecture (catalogs → schemas mapping)
• Calculate TCO (current infrastructure vs. Databricks DBU projections)

Deliverables:

• Migration complexity scorecard
• Unity Catalog design (external vs. managed tables strategy)
• Cost projection with ±20% buffer

Key Decision: Lift-and-Shift vs. Transformational Unity Catalog migration?

---

Phase 2: Pilot & Foundation (Months 4-6)

Activities:

• Set up Databricks workspace (multi-cloud account structure)
• Configure Unity Catalog (metastore, storage credentials, external locations)
• Migrate "vertical slice" (end-to-end data pipeline: source → transformation → BI dashboard)
• Establish FinOps baseline (resource tagging, cost allocation, auto-suspend policies)

Deliverables:

• Production Databricks account with Unity Catalog
• Pilot use case live (prove migration viability)
• FinOps runbook (cost monitoring, optimization playbook)

Validation: DBU costs within ±10% of projection for pilot workload.

---

Phase 3: Code Conversion & Data Migration (Months 7-12)

Activities:

• Code: Convert ETL jobs (MapReduce → PySpark, Hive → Databricks SQL)
  • Use automated tools (60-80% conversion rate)
  • Manual refactoring for complex logic (remaining 20-40%)
• Data: Migrate historical data
  • Small datasets (<10TB): Direct load via COPY INTO
  • Large datasets (>10TB): AWS Snowball, Direct Connect, or Cirata automation
• Testing: Row count validation, hash comparisons, performance testing

Deliverables:

• 90% of data in Databricks Delta Lake
• Converted codebase (PySpark notebooks, Databricks SQL queries)
• Test results (data accuracy validation)

Risk Mitigation: Dual-run period (run jobs in parallel on source + Databricks for 1-2 months).

---

Phase 4: Cutover & Optimization (Months 13-18)

Activities:

• Point BI tools (Tableau, Power BI, Looker) to Databricks SQL Warehouse
• Monitor DBU consumption (identify top 10 expensive queries, optimize)
• Decommission source system (Hadoop, EMR, Snowflake)
• Knowledge transfer (train data team on Databricks best practices)

Deliverables:

• Retired legacy system
• Optimized Databricks environment (cost within budget)
• Trained team (Databricks Certified Data Engineer certifications)

Success Metric: 40-60% cost reduction vs. legacy infrastructure.

---

Section 06

Total Cost of Ownership

Hidden Costs NOT Included:

• Dual-Run Period: Source system + Databricks for 6-12 months (budget +30%)
• DBU Consumption Overages: First 3 months typically exceed estimates by 2-3x (stabilizes after optimization)
• Network Egress: On-prem → cloud data transfer ($0.08-$0.12/GB)

Break-Even Analysis:

• Median Investment: $800K
• Annual Savings: $300K-$500K (infrastructure + ops costs)
• Break-Even: 18-24 months

---

Section 07

How to Choose a Databricks Migration Partner

If you need AI/ML-first migration: Tiger Analytics. 2025 Databricks Enterprise AI Partner of the Year. 600+ certified professionals. MLOps accelerators (Tiger MLCore).

If you need Hadoop → Databricks automation: phData or Cirata. phData has SQL translation toolkit. Cirata offers zero-downtime Hadoop migrations with automated Unity Catalog metadata migration.

If you need large-scale enterprise migration: Accenture or Deloitte. Accenture handles 200PB+ Hadoop migrations. Deloitte's Migration Factory offers automated code conversion.

If you need Unity Catalog expertise: Slalom or Databricks Professional Services. Slalom won 2025 Partner of the Year (multiple categories). Databricks PS offers direct access to product team.

If you need Snowflake → Databricks migration: Credencys or MSRcosmos. Credencys has end-to-end Snowflake migration experience. MSRcosmos excels at multi-cloud migrations.

If you need Oracle/Teradata → Databricks: Krish TechnoLabs. Deep expertise in legacy database migration patterns (financial services, retail).

Red flags when evaluating partners:

• No Unity Catalog case studies (critical for 2025+ migrations)
• Promising 100% automation (realistic is 60-80%)
• No FinOps/cost optimization plan in SOW (leads to DBU bill shock)
• Offshore-only delivery for complex migrations (communication/timezone issues)
• No published cost ranges ("Contact Us" opacity)

---

How We Select Implementation Partners

We analyzed 60+ Databricks migration companies based on:

• Databricks Partner Status: Elite/Select tier preferred (direct access to product team)
• Unity Catalog Experience: Migration case studies with metrics (timeline, cost, data volume)
• Code Conversion Tools: Proprietary accelerators for ETL translation (60-80% automation rate)
• Cost Transparency: Published cost ranges vs. "Contact Us" opacity

Vetting Process:

1. Analyze partner case studies for technical depth (100PB Hadoop → Databricks, 45% cost reduction)
2. Verify Databricks certifications (Certified Data Engineer, Certified ML Professional)
3. Map specializations to buyer use cases (Hadoop vs. Snowflake vs. EMR migrations)
4. Exclude firms with red flags (no Unity Catalog expertise, unrealistic automation claims)

---

Section 08

When to Hire Databricks Migration Services

1. The Hadoop Hardware Refresh Decision

Your Hadoop cluster is EOL. Renewal costs: $2M+ for new hardware.

Trigger: "Do we really want to buy another box, or move to the cloud?"

Business Case: Databricks Lakehouse = 40-60% cheaper than on-prem Hadoop refresh.

---

2. AWS EMR Cost Optimization

Your AWS EMR clusters are idle >40% of time (weekend/overnight). You're paying for unused capacity.

Trigger: "Why is our AWS bill $30K/month when we only run jobs during business hours?"

Business Case: Databricks auto-scaling + auto-suspend = 50% cost reduction.

---

3. AI/ML Infrastructure Bottleneck

Data scientists are extracting data to S3, running ML models on SageMaker. Data is duplicated across systems.

Trigger: "We need a unified platform for data + ML."

Business Case: Databricks = built-in MLOps (MLflow), feature store, AutoML (no data movement).

---

4. Snowflake ETL Cost Explosion

Your Snowflake bill hit $200K/year, mostly from ETL workloads (not BI queries).

Trigger: "ETL costs 9x more on Snowflake vs. Databricks."

Business Case: Consolidate ETL + ML on Databricks, keep Snowflake for BI (federated query).

---

5. Multi-Cloud Data Unification

You have data in AWS (S3), Azure (ADLS), GCP (GCS). No centralized governance.

Trigger: "We need one catalog for all our data, across clouds."

Business Case: Unity Catalog = multi-cloud governance, single metastore.

---

Section 09

Architecture Transformation

---

Section 10

Post-Migration: Best Practices

Months 1-3: FinOps Optimization

• Auto-Suspend: Reduce idle timeout to 5 minutes (default 120 min)
• Right-Size Clusters: Downsize clusters by 30-50% based on CPU/memory metrics
• Resource Monitors: Set hard limits (alert at 80% budget, block at 100%)
• Top 10 Expensive Queries: Identify via Query History, rewrite for Delta Lake

Target: Reduce DBU costs by 30-40% vs. months 1-3.

---

Months 4-6: Unity Catalog Governance

• Data Classification: Tag PII/sensitive data for access control
• Lineage Tracking: Enable for regulatory compliance (GDPR, CCPA)
• External Table Migration: Convert to managed tables for full governance

Target: 100% compliance with data classification policies.

---

Months 7-12: AI/ML Acceleration

• MLflow Integration: Centralize model tracking, versioning
• Feature Store: Build reusable feature pipelines
• AutoML: Experiment with Databricks AutoML for rapid prototyping

Target: Deploy first production ML model within 6 months.

---

Section 11

Risk Factors

Unity Catalog Migration Complexity

Migrating to Unity Catalog is mandatory by 2026. Most orgs choose 'lift-and-shift' approach and regret it 6 months later when governance gaps emerge.

DBU Cost Explosion

DBU consumption spikes 3-5x post-migration due to idle clusters, oversized configurations, and unoptimized Spark code. FinOps planning is critical.

MapReduce Translation Trap

Direct 1:1 translation of Hadoop MapReduce to PySpark ignores Delta Lake optimizations, leading to 3x slower performance and 2x higher costs.

---

Section 12

Feasibility Analysis

Ideal candidates

• Hadoop hardware EOL + $500K+ annual support costs
• AWS EMR cost optimization (40-60% savings vs. self-managed Spark)
• AI/ML teams blocked by legacy infrastructure
• Snowflake consolidation for ETL + ML workloads (9x cost difference)
• Multi-cloud data unification

Break-even cost: $300K/year in infrastructure + ops costs Talent risk: High. Spark/Python data engineers expensive ($150K-$220K). Certifications: Databricks Certified Data Engineer.