Primary responsibilities:
- Design and develop end-to-end ETL pipelines, working from data source integration to visualization.
- Architect scalable data workflows and manage complex distributed systems to ensure efficient data processing.
- Collaborate cross-functionally in an Agile environment, adhering to Scrum principles to ensure iterative and incremental delivery.
- Mentor and lead a team of engineers, providing technical guidance and fostering a culture of continuous improvement.
- Partner with stakeholders across Product, Design, QA, and Operations teams to gather requirements, align priorities, and deliver high-quality solutions.
- Implement best practices for data security, governance, and performance optimization, including the use of technologies like Kerberos and Iceberg.
- Continuously explore and adopt new tools and technologies to enhance the pipeline’s efficiency and scalability.
Required Qualifications:
- Strong expertise in ETL development, including proficiency in Spark, Kafka, StarRocks, and Apache Polaris.
- Hands-on experience with Data Warehousing concepts and tools.
- Proficient in visualization tools like Power BI, with a deep understanding of creating actionable insights from data.
- Familiarity with data storage and security frameworks, including MinIO, Kerberos, and Iceberg.
- Experience working in an Agile model and a thorough understanding of Scrum principles and practices.
- 8+ years of experience in software development, with at least 3 years in a lead or architect role involving ETL and data pipeline development.
- Excellent communication and collaboration skills to interact with both technical and non-technical stakeholders.
- Strong analytical and problem-solving skills, with the ability to address complex challenges efficiently.