Principal Data Engineer and Azure Cloud Infrastructure
Dublin, Ireland
About the Role:
As a Principal Data Engineer, you will manage Azure related data architecture and cloud infrastructure supporting complex risk models (statistical/AI/ML). The focus for this position will be the design, development and maintenance of the data architecture components (including ETL data pipelines) required to ingest data from sources and expose data to downstream consumers.
Responsibilities:
* Design, configure, manage data engineering infrastructure on the cloud (Databricks/Azure DataFactory/Azure Data Lake)
* Maintain data ingestion and exposure (including via APIs) with high security and availability
* Ensure data quality standards are established and enforced
* Build infrastructure using code and repeatable designs in Terraform
* Maintain Azure cloud infrastructure across prod and non-prod environments
* Identify and resolve security and compliance vulnerabilities
* Troubleshoot production support issues post-deployment and come up with solutions as required
* Identify opportunities for improvement, optimize performance and scalability of existing solutions
* Setup and maintain related documentation and standards
* Prioritize and manage work to critical project timelines in a fast-paced environment
* Partner with Data Science and Product leaders to design best practices and standards for developing and productionalizing analytic pipelines
* Mentor and train junior members of the team
Requirements:
* Extensive hands-on experience developing data pipelines that demonstrate a strong understanding of software engineering principles
* Excellent knowledge of Azure Services such as – Blobs, Functions, Azure Data Factory, Service Principal, Containers, Key Vault etc.
* Experience working with both real-time and batch data, knowing the strengths and weaknesses of both and when to apply one over another
* Experience developing and deploying APIs
* Expert skills in Python, SQL, Spark
* Experience designing best practice Data Architecture, Data Quality and Data Security process and implementing those utilizing the latest tools and technologies
* Experience working with Terraform for cloud infrastructure automation and management
* Bring innovative ideas to help continuously improve the application portfolio including the design, build, deploy, test and runtime within cloud ecosystems
* Proven critical thinking skills and persistence to solve complex problems
* Flexibility to take on various responsibilities in a start-up like team atmosphere
Estimated Salary:
$120,000 - $150,000 per annum
Work Schedule:
Flexible work arrangements, with 3 days per week in Dublin office and 2 days per week telecommuting from a home-based office