Job Title:
Senior Data Engineer
Overview:
Cpl is supporting a dynamic client in their search for a Senior Data Engineer.
Key Responsibilities:
* Collaborate with cross-functional teams to enhance our Azure-based data platform.
* Lead the creation of a reference data system for securities and pricing, with future expansion into other areas.
* Design, implement, and maintain scalable data pipelines and storage solutions.
* Utilize Azure tools like Data Factory, Lake Gen 2, and Databricks for efficient data processing.
* Develop and integrate APIs using frameworks such as Fast API.
* Ensure security, scalability, and performance best practices in cloud solutions.
* Align data solutions with business goals and evolving requirements.
* Continuously optimize data architecture and engineering processes.
Requirements:
* 10+ years of experience with deep expertise in Python and Py Spark.
* Proven experience as a Data Engineer with extensive knowledge of Azure Cloud services.
* Proficiency with Azure Data Factory, Lake Gen 2, Databricks, Functions, and API Gateway.
* Hands-on experience with SQL across relational and No SQL databases.
* Proven skills in developing APIs with Fast API or similar frameworks.
* Expertise in ETL/ELT processes and Dev Ops tools (e.g., Git, Jenkins).
* Ability to thrive in a fast-paced, collaborative environment.