Senior Data Engineer
A leading multinational technology services company is seeking a talented senior Data Engineer with strong Azure skills to work on dynamic data challenges.
Key Responsibilities:
* Design and manage data engineering infrastructure using cloud-based tools like Azure Data Factory, Databricks, and Azure Data Lake.
* Build and maintain secure, scalable data pipelines for data ingestion and exposure (e.g., APIs).
* Implement and enforce data quality standards across all workflows.
* Automate infrastructure deployment and management using Terraform and other Infrastructure-as-Code (IaC) tools.
* Manage Azure cloud environments for production and non-production workloads.
* Identify and resolve security and compliance vulnerabilities in cloud environments.
* Troubleshoot and resolve production issues and implement long-term solutions.
* Collaborate with cross-functional teams to establish best practices for analytics pipeline development.
* Mentor junior team members to foster growth and knowledge sharing.
* Document processes, standards, and architecture to ensure operational efficiency.
Requirements:
* Hands-on experience designing and developing data pipelines with a strong foundation in software engineering principles.
* Proficiency with Azure services (e.g., Blobs, Functions, Data Factory, Service Principal, Containers, Key Vault).
* Expertise in real-time and batch data processing, with the ability to select the appropriate approach based on requirements.
* Strong programming skills in Python, SQL, and Spark.
* Proven experience developing and deploying APIs.
* In-depth knowledge of designing and implementing best practices for data architecture, quality, and security.
* Experience with Terraform for automating and managing cloud infrastructure.
* Critical thinking skills and the ability to solve complex problems in dynamic environments.
* Flexibility and adaptability to contribute to various aspects of a project.
Preferred Skills:
* Experience with Dev Ops tools, CI/CD pipelines, and Git workflows.
* Familiarity with agile/scrum methodologies.
* Understanding of artificial intelligence, machine learning, and their applications.
* Experience working on proof-of-concept projects for Big Data and Data Science.