Snr. Spec. Data Engineering Expert

Job Description Summary

location: Hyderabad

The Data Engineering and Quality team builds data and platform capabilities that enables analytics team across organization to build data ingestion, transformation pipelines. We are looking for a Data Engineering Solution Specialist who can design, build, and optimize scalable, reusable, and high-performing data engineering solutions across the Novartis data platform ecosystem.

The ideal candidate will work across products and platforms, think horizontally, and create unified engineering capabilities that improve standardization, scalability, and speed of delivery. This role requires strong ownership, independent problem solving, and the ability to collaborate effectively with product leads, architects, DevOps, and domain teams in a fast-paced global environment.

We are looking for a strong data engineer first with hands-on expertise in building robust pipelines, performance tuning, ETL and ELT frameworks, platform engineering, and CI CD enablement. In addition, the role requires the ability to strategically apply AI, ML, and Agentic AI capabilities to accelerate software development, improve engineering productivity, and deliver intelligent automation and end-user focused solutions.

 

Job Description

Major Accountabilities:

• Design, develop, and maintain scalable data engineering solutions, pipelines, and workflows using Databricks, PySpark, Lakehouse, Unity Catalog, SQL, and AWS services such as S3 and Lambda.

• Apply strong knowledge of data warehousing, ER modeling, fact and dimension data models, ETL and ELT patterns, pipeline orchestration, and data quality controls to build robust and reliable solutions.

• Design and implement data observability and monitoring capabilities across pipelines and workflows, including freshness, completeness, schema drift, volume anomalies, pipeline health, alerting, and controls to detect, diagnose, and reduce data incidents across batch and near real-time pipelines.

• Drive engineering excellence through code management, peer reviews, unit testing, integration testing, documentation, standardization, and reusable design and development practices to enable maintainable and cost-effective platform delivery.

• Optimize solution performance through tuning of pipelines, Spark jobs, SQL workloads, and platform components.

• Implement and enforce data quality checks, validation rules, and monitoring controls within ETL pipelines to improve trust and reliability. Collaborate with DevOps teams to meet CI CD requirements and ensure reliable, automated, and smooth deployment of data engineering solutions.

• Build shared engineering assets such as APIs, SDKs, utilities, libraries, and reusable platform components that can be adopted across teams. Leverage AI, ML, and Agentic AI capabilities to improve engineering productivity, accelerate code development, automate repetitive tasks, support test case generation and pipeline validation, and enhance SDLC practices.

• Apply Agentic AI and intelligent automation patterns to build smart workflows, platform automations, end-user productivity solutions, and enhanced observability use cases such as anomaly detection, incident summarization, intelligent alert correlation, and guided issue resolution.

• Establish engineering standards for the responsible adoption of AI-generated code, including review controls, testing guardrails, explainability, traceability, and governance.

• Evaluate and adopt relevant platform and AI capabilities such as MLflow, Databricks AI Functions, Snowflake Cortex AI, and Dataiku to build scalable, future-ready platform solutions. Work within Agile delivery teams, actively participating in Scrum ceremonies and ensuring delivery to agreed timelines, quality standards, and release schedules.

• Follow data security, compliance, and governance requirements with a security-first mindset in all engineering activities.

Minimum Requirements

• Bachelor’s or Master’s degree in Computer Science, Engineering, Information Technology, or a related field

• 6 to 10 years of experience in data engineering, including data ingestion, CI/CD,ETL and ELT, pipeline development, orchestration, platform engineering, and building scalable, maintainable, production-grade data pipelines.

• Strong hands-on experience in AWS, Databricks, Spark, Python, and SQL.

• Experience implementing data quality checks, controls, and observability frameworks including monitoring, alerting, and logging for enterprise data pipelines.

AI and Agentic AI Expectations

• Practical understanding of AI and ML concepts and their application in data and platform engineering contexts.

• Experience using AI-enabled capabilities in Databricks platforms. Familiarity with Agentic AI patterns such as autonomous agents, tool-using AI systems, multi-agent workflows, and AI orchestration frameworks.

• Ability to identify where AI adds meaningful value versus where conventional automation is more appropriate. Experience applying AI to improve developer productivity, SDLC acceleration, workflow automation, and end-user productivity solutions.

• Strong interest in building intelligent automation and platform services that create measurable business value. Strong ownership mindset with accountability from design through production and continuous improvement.

• Ability to work independently with minimal supervision, with strong analytical and problem-solving skills. Excellent verbal and written communication skills in a global environment.

• Agile mindset with active participation in Scrum ceremonies and team delivery processes.Open to learning and adapting to new technologies, tools, and engineering practices.

Why consider Novartis?
Our purpose is to reimagine medicine to improve and extend people’s lives and our vision is to become the most valued and trusted medicines company in the world. How can we achieve this? With our people. It is our associates that drive us each day to reach our ambitions. Be a part of this mission and join us!
Learn more here:

https://www.novartis.com/about/strategy/people-and-culture

Commitment to Diversity and Inclusion:
Novartis is committed to building an outstanding, inclusive work environment and diverse teams' representative of the patients and communities we serve.
Join our Novartis Network: If this role is not suitable to your experience or career goals but you wish to stay connected to hear more about Novartis and our career opportunities, join the Novartis Network here:
https://talentnetwork.novartis.com/network

 

Skills Desired

Agentic AI, Amazon Web Services (AWS), data bricks, Data Engineering, python, spark