senior data engineer (spark + aws)
We are looking for a skilled Senior Data Engineer with experience in Spark and AWS who can design, develop, and maintain data pipelines in our project. The ideal candidate must have an understanding of structured and unstructured data, with advanced skills in Spark, SQL, and No-SQL. They should also have experience in Apache Airflow, Kafka Streaming and Connect, and Kubernetes.
- Design, develop and maintain data pipelines
- Delivers robust, scalable, and reusable solutions
- Automate data extraction, transformation, and provisioning irrespective of the type of source and target
- Understanding of structured & unstructured data, big data
- Knowledge of Core Engineering (versioning, CI/CD, quality, automated testing)
- Experience with Kubernetes (high level), Docker
- SQL Advanced complex data merging techniques including windowing functions etc.
- Understanding of Big-data Concepts (HDFS and overall distributed architecture understanding)
- Work experience in one NoSQL database
- Strong experience with Spark, Spark Streaming, and Performance Tuning in Spark
- Advanced Python programming concepts
- Familiar with Airflow - Managed Airflow Platform
- AWS Cloud Concepts knowledge
- Experience with Snowflake data warehouse
nice to have
- Good experience in test automation and relative tools and frameworks, like Pytest, Behave, Lettuce, etc.