Primary responsibilities:
- Oversee the design, development, and deployment of end-to-end ETL pipelines, ensuring alignment with business and technical objectives.
- Lead and mentor a team of engineers, fostering skill development and encouraging innovation in data engineering practices.
- Act as the technical point of contact for cross-functional teams, collaborating with Product, Design, QA, and Operations in an Agile environment following Scrum principles.
- Drive the adoption of best practices for data security, governance, and scalability across workflows.
- Manage the architecture and development of complex distributed systems, ensuring high performance and reliability.
- Support the team in delivering actionable insights through visualization tools like Power BI.
- Continuously evaluate and integrate emerging tools and technologies such as Spark, Kafka, StarRocks, MinIO, Kerberos, and Iceberg to enhance efficiency.
- Ensure timely and high-quality delivery of projects while balancing stakeholder requirements and technical feasibility.
Required Qualifications:
- 8-12 years of experience in software development, including 2+ years in a leadership role managing teams and complex projects.
- Strong expertise in ETL pipeline development and data engineering technologies such as Spark, Kafka, StarRocks, Apache Polaris, and Data Warehousing concepts.
- Proficiency in data visualization tools like Power BI.
- Thorough understanding of Agile methodologies and hands-on experience with Scrum principles.
- Proven ability to design and implement scalable and secure data solutions, including familiarity with Kerberos, MinIO, and Iceberg.
- Excellent problem-solving and decision-making skills, with a track record of delivering high-impact solutions.
- Strong leadership and communication skills, capable of guiding teams and interacting with technical and non-technical stakeholders.
Preferred Qualifications:
- Experience in a product development environment.
- Certifications in Agile or relevant data engineering technologies.
- Familiarity with managing distributed teams across different geographies.