Data Engineer

<h2><b>Career Category</b></h2>Information Systems<h2></h2><h2><b>Job Description</b></h2><p><b>Role Summary</b></p><ul><li>Build and operate large-scale healthcare data pipelines across batch workflows, metadata-driven ingestion, and data service publishing.</li><li>Own end-to-end engineering from source ingestion to conformed data products, with strong focus on reliability, data quality, and operational observability.</li><li>Partner with analytics, business, and platform teams to deliver trusted datasets for sales, claims, activity, patient, and rare disease use cases.</li></ul><p><b>Key Responsibilities</b></p><ul><li>Design and maintain PySpark/SQL pipelines in Databricks for landing, unified, unstitched, and published data layers.</li><li>Build and support Airflow DAGs for scheduling, dependencies, retries, and production operations.</li><li>Implement metadata/config-driven frameworks for ingestion, transformation, and rule-based processing.</li><li>Develop robust data quality controls, DQ summaries, failure handling, and alerting workflows.</li><li>Manage batch/process audit logs, run status tracking, release flags, and operational reporting.</li><li>Integrate multi-source data (files, APIs, cloud storage, and relational systems) into governed Delta/Spark tables.</li><li>Optimize pipeline performance using partitioning, parallelization, and query tuning.</li><li>Collaborate on schema evolution, business-rule onboarding, and production support.</li></ul><p><b>Required Skills</b></p><ul><li>Bachelor’s degree in Computer Science, Information Technology, or a related field with 5-9 years of experience </li><li>Advanced Python, PySpark, and SQL (window functions, complex joins, MERGE patterns, optimization).</li><li>Hands-on Databricks and Airflow experience in enterprise environments.</li><li>Experience with cloud data platforms (AWS), object storage, and secure secret handling.</li><li>Strong data quality engineering, monitoring, and troubleshooting in regulated data contexts.</li><li>Solid understanding of ETL orchestration, dependency management, and SLA-driven delivery.</li></ul><p style="text-align:inherit"></p><p style="text-align:inherit"></p><p style="text-align:inherit"></p>.