Data Engineer

Overview

We are hiring a Mid–Senior Data Engineer with strong expertise in Azure cloud, Databricks, PySpark, and orchestration of data pipelines. The ideal candidate must hold a minimum of an Honours degree in an IT-related field and be experienced in building scalable, automated data solutions on Azure.

Key Responsibilities

• Design, develop, and maintain scalable and automated data pipelines using Azure Data Factory, Azure Databricks, and Azure Data Lake.
• Build and optimize PySpark-based data workflows in Azure Databricks for batch and streaming workloads.
• Develop ETL/ELT processes and ensure reliable orchestration and scheduling of data workflows.
• Implement CI/CD practices and DevOps methodologies for data pipeline deployments.
• Ensure high availability, performance, and security of data pipelines in production environments.
• Monitor, troubleshoot, and resolve issues in data workflows and pipeline performance.
• Document data pipelines, architectures, processes, and operational procedures.
• Collaborate with architects and engineering teams to design efficient and scalable data solutions.
• Mentor junior data engineers and contribute to best practice adoption.

Minimum Qualifications

• Honours Degree in Computer Science, Information Technology, Computer Engineering, or a related field (minimum NQF Level 8).

Experience Required

• 5+ years’ experience in data engineering within a cloud-based environment.
• Hands-on experience with Azure Databricks, PySpark, Azure Data Factory, and Azure Data Lake.
• Experience in designing and orchestrating end-to-end data pipelines.
• Experience working with CI/CD, DevOps practices, and version control for data engineering workflows.

Core Technical Skills

• Advanced SQL and strong Python programming with a focus on PySpark.
• Azure Databricks for building, transforming, and optimizing data workflows.
• Azure Data Factory for pipeline orchestration, scheduling, and monitoring.
• Experience with Azure Data Lake (Gen2) for data storage and management.
• Designing, deploying, and managing ETL/ELT pipelines.
• Troubleshooting, performance tuning, and monitoring data pipelines.
• Ability to write clear technical documentation and architectural diagrams.

Soft Skills

• Strong analytical and problem-solving ability.
• Excellent communication and interpersonal skills.
• Ability to work in cross-functional teams and mentor junior engineers.