Lead Big Data Engineer

We are currently looking for a remote Lead Big Data Engineer with 5+ years of experience in software development with Big Data technologies (administration, configuration management, monitoring, debugging, and performance tuning) to join our team.

Please note that even though you are applying for this position, you may be offered other projects to join within EPAM Anywhere.

We accept CVs only in English.

Join EPAM Anywhere to quickly and easily find projects that match your knowledge and experience, while working with Forbes Global 2000 clients, building a successful IT career, and earning competitive rewards. The platform provides additional perks, including a flexible schedule, professional development opportunities, and access to a community of experts.

Responsibilities

• Lead, design and implement innovative analytical solution using Hadoop, NoSQL and other Big Data related technologies, evaluating new features and architecture in Cloud/on premise/Hybrid solutions
• Work with product and engineering teams to understand requirements, evaluate new features and architecture to help drive decisions
• Build collaborative partnerships with architects and key individuals within other functional groups.
• Perform detailed analysis of business problems and technical environments and use this in designing quality technical solution
• Actively participate in code review and test solutions to ensure it meets best practice specifications
• Build and foster a high performance engineering culture, mentor team members and provide team with the tools and motivation
• Write project documentation

Requirements

• More than 5 years of experience in software development with Big Data technologies (e.g. administration, configuration management, monitoring, debugging and performance tuning)
• Engineering experience and practice in Data Management, Data Storage, Data Visualization, Disaster Recovery, Integration, Operation, Security
• Strong experience building data ingestion pipelines (simulating Extract, Transform, Load workload), Data Warehouse or Database architecture
• Strong experience with data modeling; hands-on development experience with modern Big Data components
• Cloud: experience in designing, automation, provisioning, deploying and administering scalable, available and fault tolerant systems
• Good understanding of CI/CD principles and best practices
• Analytical approach to problem-solving with an ability to work at an abstract level and gain consensus; excellent interpersonal, leadership and communication skills
• Data-oriented personality and possessing compliance awareness, such as PI, GDPR, HIPAA
• Motivated, independent, efficient and able to handle several projects; work under pressure with a solid sense for setting priorities
• Ability to work in a fast-paced (startup like) agile development environment
• Strong experience in high load and IoT Data Platform architectures and infrastructures
• Vast experience with Containers and Resource Management systems: Docker, Kubernetes, Yarn
• Experience in direct customer communications
• Experience in technology/team leading of data oriented projects
• Solid skills in infrastructure troubleshooting, support and practical experience in performance tuning and optimization, bottleneck problem analysis
• Experienced in different business domains
• English proficiency – B2 and higher
• Advanced understanding of distributed computing principles
• 1+ year of relevant leadership experience

Technologies

• Programming Languages: Java/Scala/Python/SQL/Bash
• Big Data stack: Hadoop, Yarn, HDFS, MapReduce, Hive, Spark, Kafka, Flume, Sqoop, ZooKeeper
• NoSQL: Cassandra/Hbase/MongoDB
• Queues and Stream processing: Kafka Streams, Flink, Spark Streaming, Storm, Event Hub, IOT Hub MQTT, Storage Queues, Service Bus, Stream Analytics
• Data Visualization: Tableau/QlikView
• ETL & Streaming Pipelines: Pentaho, Talend, Apache Oozie, Airflow, NiFi, Streamsets
• Operation: Cluster operation, Cluster planning
• Search: Solr, Elasticsearch/ELK
• InMemory: Ignite, Redis
• Cloud (AWS/Azure/GCP): Storage, Compute, Networking, Identity and Security, NoSQL, RDBMS and Cubes, Big Data Processing, Queues and Stream Processing, Serverless, Data Analysis and Visualization, ML as a service (SageMaker, Tensorflow)
• Enterprise Design Patterns (ORM, Inversion of Control etc.)
• Development Methods (TDD, BDD, DDD)
• Version Control Systems (Git, SVN)
• Testing: Component/Integration Testing, Unit testing (JUnit)
• Deep understanding of SQL queries, joins, stored procedures, relational schemas; SQL optimization
• Messaging systems, such as Kafka, ZeroMQ/RabbitMQ
• Rest, Thrift, GRPC, SOAP
• Build Systems: Maven, SBT, Ant, Gradle
• Docker, Kubernetes, Yarn, Mesos

We offer

• Competitive compensation depending on experience and skills
• Work on enterprise-level projects on a long-term basis
• You will have 100% remote full-time job
• Unlimited access to learning resources (EPAM training courses, English classes, Internal Library)
• Community of 38,000+ industry's top professionals