Software Engineer - Data Engineer Azure

Gartner is seeking a Data Engineer to design, build, and optimize data ingestion and transformation pipelines from multiple sources using Azure Data Factory and Databricks notebooks. You will partner with architects and business/data science teams to design scalable, business-aligned solutions on Azure.

Key Responsibilities

• Design, build, and optimize data ingestion and transformation pipelines from multiple sources using Azure Data Factory and Databricks notebooks.

• Partner with architects and business/data science teams to design scalable, business-aligned solutions on Azure.

• Build and maintain processes for data transformation, data structures, metadata management, and workload/dependency orchestration.

• Design data models and warehouse schemas that enable fast, intuitive analytics.

• Provide technical expertise and thought leadership on cloud architecture and infrastructure strategy within Microsoft Azure.

• Maintain and evolve existing Azure infrastructure, proactively planning for scale and future needs.

• Author clear specification documents covering design, testing, and support processes.

• Translate business requirements into development plans, effort estimates, and project timelines.

• Work cross-functionally with project teams to identify and mitigate risks across design, implementation, and delivery.

Requirements

• 3+ years of professional IT experience, with a strong foundation in data engineering.

• 3+ years of hands-on experience with core Azure data services: ADLS, Data Factory, Databricks, SQL DW, Azure Functions.

• Strong proficiency in Python for data engineering workflows.

• Proven experience designing and building solutions on Azure Cloud, particularly Azure SQL Data Warehouse/DB, Data Factory, Data Lake Store/Blob, and Databricks.

• Hands-on expertise with Databricks for data engineering workloads, including notebooks, Delta Lake, cluster management, and job orchestration.

• Solid grasp of lakehouse architecture principles, with experience implementing them in production.

• Strong SQL skills across relational and NoSQL databases, with particular depth in T-SQL and Azure SQL DW.

• Experience building, debugging, and migrating data pipelines using Azure Data Factory and Databricks notebooks.

• Ability to develop and maintain conceptual, logical, and physical data models.

Preferred Qualifications

• Familiarity with LLMs and RAG architecture.

• Experience in HR analytics.