Senior Data Engineer - Databricks Lakehouse & Streaming Ingestion

About the role

An early-stage healthcare technology company is looking for a Senior Data Engineer to build and maintain the data infrastructure behind its healthcare AI platform, which includes voice agents and document-processing agents for specialty medical practices. The role is remote, full-time (40 hours per week), and open to candidates across the MENA region with a preference for Palestine-based candidates. It reports to the Data Engineering Lead / CTO and requires at least 4 hours of daily overlap with US Mountain Time.

Key responsibilities

• Build and maintain ingestion from third-party vendor REST APIs, handling pagination schemes, row and rate caps, schema drift and contract checks.
• Own streaming ingestion from Azure Event Hubs into Databricks using Structured Streaming / Auto Loader, with checkpoints, watermarking and deduplication, and run source-parity reconciliation.
• Develop Delta Lake pipelines in a medallion architecture (bronze, silver, gold) using Spark SQL and PySpark, including idempotent MERGE upserts, Delta Live Tables and materialized views.
• Build entity resolution over free-text, normalizing practice, provider and payer names with regex, canonical dictionaries, fuzzy matching and confidence scores.
• Maintain the semantic and metrics layer with precise metric definitions and correct handling of non-additive aggregates.
• Instrument data quality end-to-end: freshness, parity, contract checks and alerting.

Required profile

• 4+ years in data engineering with substantial production Databricks experience.
• Demonstrated ability to debug source-vs-warehouse discrepancies and explain the root cause.
• Experience integrating with third-party REST APIs and resolving entity resolution on messy text.
• Strong SQL and Python, able to own pipelines end-to-end; written and spoken English for async collaboration with a US team.
• Nice to have: healthcare data experience, voice-agent or telephony data, Holistics AML/AQL, and Azure beyond Event Hubs.

Required skills

• Databricks, Spark SQL, PySpark, Delta Lake, Delta Live Tables
• Structured Streaming or equivalent (Event Hubs, Kafka, Kinesis)
• Entity resolution and metrics/semantic layers (Holistics AML/AQL, dbt metrics, LookML)

The engagement is handled by Apricot, a nonprofit recruiter, and may include a practical assessment. Candidates provide their own laptop and equipment and a reliable remote setup.