go to searchsenior data engineer (spark + aws)
Data Software Engineering, Apache Airflow, Big Data, Kafka Streaming and Connect, Kubernetes, NoSQL in BigData, Python, SQL, Spark Streaming, Amazon Web Services
Sorry the job is no longer available.
Senior Data Engineer (Spark + AWS)
We are looking for a skilled Senior Data Engineer with experience in Spark and AWS who can design, develop, and maintain data pipelines in our project. The ideal candidate must have an understanding of structured and unstructured data, with advanced skills in Spark, SQL, and No-SQL. They should also have experience in Apache Airflow, Kafka Streaming and Connect, and Kubernetes.
responsibilities
- Design, develop and maintain data pipelines
- Delivers robust, scalable, and reusable solutions
- Automate data extraction, transformation, and provisioning irrespective of the type of source and target
requirements
- Understanding of structured & unstructured data, big data
- Knowledge of Core Engineering (versioning, CI/CD, quality, automated testing)
- Experience with Kubernetes (high level), Docker
- SQL Advanced complex data merging techniques including windowing functions etc.
- Understanding of Big-data Concepts (HDFS and overall distributed architecture understanding)
- Work experience in one NoSQL database
- Strong experience with Spark, Spark Streaming, and Performance Tuning in Spark
- Advanced Python programming concepts
- Familiar with Airflow - Managed Airflow Platform
- AWS Cloud Concepts knowledge
- Experience with Snowflake data warehouse
nice to have
- Good experience in test automation and relative tools and frameworks, like Pytest, Behave, Lettuce, etc.
These jobs are for you
benefits for locations
Colombia
For you
- Prepaid Medicine with Colsanitas for you and your legal dependents
- MetLife Life Insurance for you
- Thousands of projects for top brands
- Stable income
For your comfortable work
- 100% remote work forever
- Free licensed software
- Possibility to work on your own device (BYOD)
- Stable workload
- Relocation opportunities
- Flexible engagement models
For your growth
- Free trainings for technical and soft skills
- Free access to LinkedIn Learning platform
- Support from a personal Skill Advisor
- Language courses
- Free access to internal and external e-Libraries
- Access to internal communities and competency centers
- Certification opportunities
get job alerts in your inboxHundreds of open jobs for Software Engineers, QA, DevOps, Business Analysts and other tech professionals
