Listing

Cloudary SAS

Data Platform Engineer (102-08SENG-01)

  • Remote
  • remote
  • Contractor

Posted

Apply on Cloudary SAS’s site

Free to apply. You’ll finish on the employer’s own site.

Skills mentioned
  • Azure
  • AWS
  • Spark
  • SQL
  • Python
  • Terraform
+4 more
  • Airflow
  • dbt
  • Kafka
  • SAS

This role converts a large Azure Data Factory estate into Databricks workflows on AWS. The scope for year one: 2,094 ADF pipelines to migrate — built as reusable templates rather than one-by-one — 9,859 pipeline activities to translate (some map directly, others need rewriting as Lambda or Step Functions), 471 Spark dataflows to move onto Databricks on AWS, and 4 Databricks workspaces (Dev, QA, Pre-prod, Prod) to rehost, including notebook paths and Unity Catalog rewiring. This is a regulated environment, so reconciling migrated data against source systems is part of the definition of done, not an afterthought.

What you will do

  • Convert Azure Data Factory pipelines into Databricks workflows on AWS, building reusable templates rather than migrating one at a time.

  • Rehost Databricks workspaces onto AWS and migrate ADLS Gen2 storage to S3.

  • Rewrite ADF Web Activities as Lambda functions or Step Functions tasks, and replace ADF-specific scaling with native Databricks mechanisms.

  • Build and tune PySpark transformations for production data volumes.

  • Replace Azure Synapse Serverless with Databricks SQL Warehouse.

  • Reconcile migrated data against source systems as part of the definition of done.

Required

  • Production experience with Databricks: workspaces, jobs and workflows. The central skill for this role.

  • Strong Spark and PySpark experience for real data volumes, including tuning.

  • Production-grade Python.

  • Experience building or migrating Azure Data Factory pipelines, with a solid understanding of the ADF activity model.

  • AWS data services: S3, Glue, Athena, Lambda and Step Functions.

  • Advanced SQL, including reading and reasoning about stored procedures.

  • Professional written and spoken English.

Nice to have

Delta Lake, Unity Catalog, Azure Synapse, Terraform, Airflow/MWAA, dbt, Kafka, Databricks certification, data modeling, Great Expectations, SAS/analytics platform integration, CRM data.

Engagement details

  • Estimated start date: December 2026

  • Duration of project: 9 months

  • EST Time Zone

  • Open to candidates from all LATAM

Apply on Cloudary SAS’s site

Not the right role?

Search jobs from 47,000+ company career pages, many not on LinkedIn or Indeed.

Data Platform Engineer (102-08SENG-01)Cloudary SAS
Apply now