go to searchlead data engineer
Lead Data Engineer
We are seeking an experienced remote Lead Big Data Engineer to help us migrate the Data Products pipelines from Oracle workloads to Databricks.
The successful candidate will be responsible for developing, monitoring, and operating the most critical curated data pipeline in the Sales Order Data system. The pipeline processes hundreds of millions of records to provide high-quality datasets for analytical and machine-learning use cases.
The candidate will work closely with analysts, data scientists, and product managers to build and continuously improve the "Single Source of Truth" KPI for business steering.
responsibilities
- Develop, monitor, and operate the Sales Order Data pipeline
- Consult with analysts, data scientists, and product managers to build and continuously improve the "Single Source of Truth" KPI for business steering
- Redevelop old legacy pipelines to new, advanced, and standard versions that are easy to maintain and scalable for future demands
- Leverage and improve a cloud-based tech stack that includes AWS, Databricks, Kubernetes, Spark, Airflow, Python, and Scala
- Build Apache Airflow pipelines
- Ensure data quality, data security, and data privacy
- Collaborate with other teams to design and implement data solutions that meet business requirements
- Create and maintain technical documentation
requirements
- 5+ years of working experience in Data Engineering
- 1+ years of relevant leadership experience
- Expertise in Apache Spark along with Spark streaming
- Good hands-on experience with Databricks and delta-lake
- Fluency in Scala programming language
- Expertise in SQL
- Good understanding & hands-on experience with CI/CD
- Extensive working experience with GitHub
- Fluency working with AWS landscape
- Experience in building Apache Airflow pipelines
- Team player (easy & respectful communications, shares responsibilities for the team's overall success)
- Organizational skills
- English B2+ (is able to communicate fluently with English-speaking stakeholders, able to share ideas and provide reasoning)
nice to have
- Presto
- Superset
- Starburst
- Oracle & Exasol
benefits for locations
Greece
For you
- Paid time off
- Paid sick leave days
- Stable income
- Meal and home office compensation
For your comfortable work
- 100% remote work
- Hybrid work opportunities
- Free licensed software
- Possibility to work on your own device (BYOD)
- Stable workload
- Relocation opportunities
- Flexible engagement models
For your growth
- Free trainings for technical and soft skills
- Free access to LinkedIn Learning platform
- Free access to internal and external e-Libraries
- Certification opportunities
- Skill advisory service
- Language courses
get job alerts in your inboxHundreds of open jobs for Software Engineers, QA, DevOps, Business Analysts and other tech professionals
looking for something else?
Find a vacancy that works for you. Send us your CV to receive a personalized offer.