backgo to search

senior big data developer for healthcare company

Apache AirflowApache Spark, Big Data, Scala
Sorry, the job is expired

Currently, we are looking for a remote Senior Big Data Developer to join our team.

  • Maintaining the Log and Audit information in Hive/SQL tables, Experienced in providing Logging, Error Handling
    • Analyzing large amounts of data sets to determine optimal way to aggregate and report on it
      • Configuring Spark Streaming to receive real time data from the Apache Kafka and store the stream data to HDFS using Scala
        • Scheduling jobs using Airflow/ESP
          • Developing scripts for doing transformations using Scala
            • Create the required technical tasks in backlog and update status regularly
              • Participate in all Agile ceremony meetings and update task status and adher to timelines and quality standards for the task
                • Developing Shell scripts to orchestrate execution of all other scripts and move the data files within and outside of HDFS
                  • Using Kafka on publish-subscribe messaging as a distributed commit log, have experienced in its fast, scalable and durability
                    • Implementing Kerberos Security Authentication protocol for existing cluster
                      • Understanding existing business logic & implement the business changes
                        • 3+ years of relevant experience
                          • In-depth understanding/knowledge of Hadoop & spark Architecture and its components such as HDFS, Job Tracker, Task Tracker,executor cores and memory params
                            • Experience in Hadoop development, working experience on SPARK, SCALA is mandatory and Database exposure
                              • Hands on experience in Spark and Spark Streaming creating RDD's, applying operations -Transformation and Actions
                                • Experience in code optimize to fine tune the applications
                                  • Expertise in writing Hadoop/spark Jobs for analyzing data using Spark, Scala, Hive, Kafka and Python
                                    • Experienced with streaming work flow operations
                                      • Experience with developing large-scale distributed applications and developing solutions to analyze large data sets efficiently
                                        • Integration with Hadoop/HDFS, Real-Time Systems, Data Warehouses, and Analytics solutions
                                          • Experience in Data Warehousing and ETL processes
                                            • Strong database, SQL, ETL and data analysis skills
                                              • Experienced in Agile development methodology

                                                benefits for locations


                                                For you

                                                • Insurance Coverage 
                                                • Paid Leaves – including maternity, bereavement, paternity, and special COVID-19 leaves. 
                                                • Financial assistance for medical crisis 
                                                • Retiral Benefits – VPF and NPS 
                                                • Customized Mindfulness and Wellness programs 
                                                • EPAM Hobby Clubs

                                                For your comfortable work

                                                • Hybrid Work Model 
                                                • Soft loans to set up workspace at home 
                                                • Stable workload 
                                                • Relocation opportunities with ‘EPAM without Borders’ program

                                                For your growth

                                                • Certification trainings for technical and soft skills 
                                                • Access to unlimited LinkedIn Learning platform 
                                                • Access to internal learning programs set up by world class trainers 
                                                • Community networking and idea creation platforms 
                                                • Mentorship programs 
                                                • Self-driven career progression tool

                                                subscribe to EPAM Anywhere vacancies!Hundreds of open jobs for Software Engineers, QA, DevOps, Business Analysts and other tech professionals
                                                Girl in front of laptop

                                                looking for something else?

                                                Find a vacancy that works for you. Send us your CV to receive a personalized offer.