Big Data Developer Jobs

EPAM Anywhere is looking for Big Data Developers.

Competitive compensation

We back all payments and operate in compliance with country-specific labor regulations. Your annual paycheck can range from $15k to $200k.

Numerous benefits

Make use of 1,500+ online and location-specific benefits across 30+ countries that also include healthcare and sports programs for you and your family.

Limitless career growth opportunities

Get access to a large-scale ecosystem that consists of educational and career development services created for your growth. A personal skill advisor will support you on each step of this journey.

looking for a big data engineer job?

Let us find one for you! Send us your CV and we'll get back with your personalized best-match job at Anywhere

find me a job

lead big data developer (python)

Data Software Engineering

Amazon Web Services, PySpark, Python

40 hrs/week

12+ months

Colombia

We are seeking a highly skilled remote Lead Big Data Engineer to join our team, working on a cutting-edge project that leverages computer vision and machine learning to provide innovative solutions for our clients.

In this position, you will be responsible for designing and implementing data pipelines, developing and maintaining data infrastructure, and collaborating with cross-functional teams to ensure seamless integration with the company's products and services. If you are passionate about data engineering and have a strong background in computer vision and machine learning, we invite you to apply for this exciting opportunity.

responsibilities

Design and implement data pipelines, data infrastructure, and data storage solutions, ensuring scalability and performance
Collaborate with cross-functional teams to integrate data solutions with the company's products and services
Develop and maintain data infrastructure, ensuring data quality, consistency, and reliability
Lead and mentor a team of data engineers, providing guidance and support to ensure high-quality deliverables
Optimize data processing and analysis using PySpark and Python, and develop and deploy computer vision and machine learning models
Conduct code reviews and ensure adherence to best practices and coding standards

requirements

A minimum of 5 years of experience in Data Software Engineering, showcasing your expertise in designing and implementing data pipelines, data infrastructure, and data storage solutions
1+ years of relevant leadership experience
In-depth knowledge of Amazon Web Services, including experience with core services such as EC2, S3, and Lambda
Expertise in PySpark and Python, with a solid understanding of data processing and analysis
Strong background in computer vision and machine learning engineering, with experience in building and deploying models
Experience with distributed computing frameworks such as Apache Spark and Hadoop, and the ability to optimize performance and scalability
Excellent communication skills and the ability to collaborate effectively with cross-functional teams, including data scientists, product managers, and software engineers
Ability to lead and mentor a team of data engineers, providing guidance and support to ensure high-quality deliverables
Fluent spoken and written English at an upper-intermediate level or higher

nice to have

Experience with data visualization tools such as Tableau or Power BI
Knowledge of natural language processing and text analytics
Experience with containerization and orchestration technologies such as Docker and Kubernetes

lead big data engineer

Data Software Engineering

Apache Spark, Databricks, Python

40 hrs/week

12+ months

Chile, Colombia

We are seeking a highly skilled remote Lead Big Data Engineer to join our team, working on a cutting-edge project that leverages the latest technologies to deliver innovative solutions in the field of data engineering.

In this role, you will lead a team of engineers in the development, implementation, and maintenance of large-scale data processing systems, ensuring the highest levels of performance, reliability, and scalability. If you are a creative problem solver with a passion for data engineering and a proven track record of delivering complex projects, we invite you to apply for this exciting opportunity.

responsibilities

Lead a team of engineers in the design, development, and maintenance of large-scale data processing systems
Collaborate with stakeholders to identify business requirements and translate them into technical specifications
Develop and implement data processing pipelines using technologies such as Apache Spark and Databricks
Optimize data processing performance and scalability, ensuring high levels of reliability and availability
Provide technical guidance and mentorship to team members, fostering a culture of continuous learning and development
Conduct code reviews and ensure adherence to coding standards and best practices
Stay up-to-date with emerging trends and technologies in data engineering, and make recommendations for their adoption

requirements

A minimum of 5 years of experience in Data Software Engineering, with expertise in designing and building large-scale data processing systems
A minimum of 1 year of experience in leading a team of engineers, providing technical guidance and mentorship, and driving successful project outcomes
Expertise in Apache Spark and Databricks, showcasing your proficiency in distributed computing and data processing
In-depth knowledge of programming languages such as Python and Scala, enabling you to write efficient and effective code
Strong understanding of data modeling and database design principles, including experience with SQL and NoSQL databases
Experience with cloud-based data processing platforms such as AWS, Azure, or Google Cloud Platform
Excellent communication skills and the ability to work collaboratively with cross-functional teams
Fluent spoken and written English at an Upper-Intermediate level or higher
Spanish language proficiency is a must-have

nice to have

Completed Databricks Data Engineering Professional certification
Experience with machine learning and data analytics tools and frameworks such as TensorFlow, PyTorch, or Apache Flink
Experience with data visualization tools such as Tableau, Power BI, or QlikView

lead big data engineer

Data Software Engineering

Apache Spark, Databricks, Python

40 hrs/week

12+ months

Chile, Colombia

responsibilities

Lead a team of engineers in the design, development, and maintenance of large-scale data processing systems
Collaborate with stakeholders to identify business requirements and translate them into technical specifications
Develop and implement data processing pipelines using technologies such as Apache Spark and Databricks
Optimize data processing performance and scalability, ensuring high levels of reliability and availability
Provide technical guidance and mentorship to team members, fostering a culture of continuous learning and development
Conduct code reviews and ensure adherence to coding standards and best practices
Stay up-to-date with emerging trends and technologies in data engineering, and make recommendations for their adoption

requirements

A minimum of 5 years of experience in Data Software Engineering, with expertise in designing and building large-scale data processing systems
A minimum of 1 year of experience in leading a team of engineers, providing technical guidance and mentorship, and driving successful project outcomes
Expertise in Apache Spark and Databricks, showcasing your proficiency in distributed computing and data processing
In-depth knowledge of programming languages such as Python and Scala, enabling you to write efficient and effective code
Strong understanding of data modeling and database design principles, including experience with SQL and NoSQL databases
Experience with cloud-based data processing platforms such as AWS, Azure, or Google Cloud Platform
Excellent communication skills and the ability to work collaboratively with cross-functional teams
Fluent spoken and written English at an Upper-Intermediate level or higher
Spanish language proficiency is a must-have

nice to have

Completed Databricks Data Engineering Professional certification
Experience with machine learning and data analytics tools and frameworks such as TensorFlow, PyTorch, or Apache Flink
Experience with data visualization tools such as Tableau, Power BI, or QlikView

big data solution architect

Data Software Engineering

Apache Spark, Databricks, Python

40 hrs/week

12+ months

Chile, Colombia

We are seeking a skilled Big Data Solution Architect to join our remote team, working on a cutting-edge project in the software architecture industry.

We are seeking a skilled Databricks Solution Architect to join our remote team, working on a cutting-edge project in the software architecture industry. As a Big Data Solution Architect, you will be responsible for designing and implementing data engineering solutions using Apache Spark and Databricks. The ideal candidate will have completed the Databricks Data Engineering Professional certification and have hands-on experience in data engineering and programming with Databricks. Spanish fluency is a must-have for this role.

responsibilities

Design and implement data engineering solutions using Apache Spark and Databricks
Collaborate with cross-functional teams to understand project requirements and provide technical expertise
Develop and execute data engineering workflows and pipelines
Optimize and troubleshoot Apache Spark on Databricks for performance and scalability
Provide technical guidance and mentorship to junior team members
Create and maintain technical documentation for data engineering solutions

requirements

Minimum of 7 years of practical hands-on experience in Data Software Engineering
At least 2+ projects in a technical team lead role with constant hands-on work during the last 3 years
1+ years of experience as a Data Solution Architect
Completed Databricks Data Engineering Professional certification
Proficiency in programming languages such as Python and Scala
Strong communication skills and the ability to work effectively in a team environment
Fluent spoken and written English at an Upper-Intermediate level or higher (B2+)
Fluent spoken and written Spanish

nice to have

Experience in cloud-based data engineering solutions, such as AWS or Azure
Familiarity with data warehousing and ETL processes
Knowledge of machine learning and data science concepts and tools

senior big data engineer (python)

Data Software Engineering

Amazon Web Services, PySpark, Python

40 hrs/week

12+ months

Colombia

We are looking for a highly skilled remote Senior Big Data Engineer to join our team and work on complex data engineering projects.

In this position, you will be responsible for building and maintaining data pipelines and infrastructure, integrating data within Kubeflow and Databricks using PyTorch and its ecosystem, and creating data tools for analytics and data scientist team members. The ideal candidate will have extensive experience in AWS, Python, Computer Vision, Machine Learning Engineering, SQL and NoSQL databases, and Terraform, and will be comfortable working in a fast-paced, dynamic environment.

responsibilities

Create and maintain optimal data pipelines for ingesting machine logs, image data, and various other types of datasets
Assemble large, complex data sets that meet functional and non-functional business requirements
Identify, design, and implement internal process improvements, such as automating manual processes, optimizing data delivery, and re-designing infrastructure for greater scalability
Build the infrastructure required for optimal extraction, transformation, and loading of data from a wide variety of data sources using Python, SQL, Databricks Spark, and AWS 'big data' technologies
Build and enhance CVML pipelines, integrate data within Kubeflow and Databricks using PyTorch and its ecosystem
Work with stakeholders from the Product, Data, and Infra teams to assist with data-related technical issues and support their data infrastructure needs
Create data tools for analytics and data scientist team members that assist them in building and optimizing their workflows
Triage and fix support issues related to data anomalies

requirements

A minimum of 3 years of experience in Data Software Engineering, with a proven track record of delivering high-quality results
Extensive experience in Amazon Web Services (AWS), including EC2, S3, Lambda, and EMR
Expertise in PySpark, Python, and SQL databases, with a deep understanding of data structures and algorithms
Strong knowledge of Computer Vision and Machine Learning Engineering, with experience in building and enhancing CVML pipelines
Experience with Apache Kafka and NoSQL databases, such as MongoDB or Cassandra
Proficiency in Terraform for infrastructure as code
Excellent problem-solving skills and ability to work independently
Strong communication skills and ability to collaborate effectively with cross-functional teams
Fluent spoken and written English at an Upper-Intermediate level or higher

nice to have

Experience with other big data technologies, such as Hadoop or Hive
Expertise in data visualization tools, such as Tableau or Power BI
Knowledge of DevOps practices and tools, such as Jenkins or GitLab
Experience with containerization and orchestration tools, such as Docker or Kubernetes
Familiarity with data privacy and security regulations, such as GDPR or CCPA

senior big data engineer with databricks

Data Software Engineering

Amazon Web Services, Apache Airflow, Apache Spark

40 hrs/week

12+ months

India

We are looking for a remote Senior Big Data Engineer with Databricks expertise to join our team.

The ideal candidate should have a Bachelor's/Master's Degree in Computer Science, Information Systems, or equivalent with at least 3 years of experience in Data Software Engineering, DSE Python, and AWS Databricks.

responsibilities

Develop and implement solutions utilizing Data Lake
Build event-driven scalable and resilient systems
Thorough understanding of information systems, business processes, and key drivers and measures of success while choosing the proper methodologies and policies to support broad business goals
Contribution to disaster recovery planning and execution
Participate in the design and implementation of data warehousing solution

requirements

Bachelor's/Master's Degree in Computer Science, Information Systems or equivalent
At least 3 years of experience in Data Software Engineering, DSE Python, and AWS Databricks
Proficient with software development lifecycle (SDLC) methodologies like Agile, Test-driven development
Experience with SQL and NoSQL databases, advanced SQL is required
Experience with Big Data platforms such as Apache Hadoop and Apache Spark, Databricks
B2+ English level

nice to have

Experience with Apache Airflow
Expertise in building event-driven scalable and resilient systems

lead big data engineer with databricks

Data Software Engineering

Amazon Web Services, Apache Airflow, Apache Spark

40 hrs/week

12+ months

India

We are looking for a skilled remote Lead Big Data Engineer with Databricks experience to join our team.

Your primary responsibility will be to lead the development of our ETL pipelines on Databricks using Spark/Scala, Databricks, and AWS. You must have extremely strong experience with Databricks, AWS, Python, and Elastic.

responsibilities

Lead the development of ETL pipelines on Databricks using Spark/Scala, Databricks, and AWS
Work closely with cross-functional teams to define data requirements
Develop and maintain scalable data infrastructure
Ensure the reliability, efficiency, and scalability of data pipelines
Coach and mentor other members of the team

requirements

At least 5 years of experience in Big Data Engineering
1+ years of relevant leadership experience
Expertise in Databricks, AWS, Python, and Elastic
Deep understanding of Spark/Scala and data architecture principles
Knowledge of data models, ETL design, implementation, and maintenance
Strong communication skills and ability to collaborate with cross-functional teams
B2+ English level

nice to have

Experience with Apache Airflow
Familiarity with Agile methodologies and SDLC practices

senior big data engineer with python

Data Software Engineering

Databricks, Amazon Web Services, Python

40 hrs/week

12+ months

India

We are looking for an experienced remote Senior Big Data Engineer with a passion for data to develop, monitor, and operate the curated data pipeline for our project.

The ideal candidate has a strong background in Data Software Engineering and extensive experience with DSE Python AWS Databricks and SQL. In this position, you will be responsible for redeveloping legacy pipelines into new, advanced, and scalable versions, consulting with data scientists and product managers, building and improving KPIs, and maintaining our cloud-based tech stack.

responsibilities

Develop, monitor, and operate the most critical curated data pipeline in the project
Consult with data scientists and product managers to improve KPIs for business steering
Redevelop legacy pipelines into advanced, scalable versions that are easy to maintain
Leverage and improve our cloud-based tech stack that includes AWS, Databricks, Kubernetes, Spark, Airflow, Python, and Scala
Build, monitor, and maintain Apache Airflow pipelines

requirements

Minimum of 3 years of experience in Data Software Engineering
Expertise in Apache Spark, Spark streaming, and Databricks
Fluency in Scala programming language and SQL
Experienced working with AWS landscape and Github
Ability to build Apache Airflow pipelines
B2+ English level

nice to have

Knowledge of Presto, Superset, and Starburst
Experience with Oracle & Exasol

senior big data engineer with databricks

Data Software Engineering

Amazon Web Services, Apache Airflow, Apache Spark

40 hrs/week

12+ months

India

We are seeking a highly proficient remote Senior Big Data Engineer with extensive experience in Databricks, Python, AWS, and Data Warehousing.

The ideal candidate should be well-versed in software development lifecycle methodologies and proficient in advanced SQL coding. As a Senior Big Data Engineer, you will be responsible for developing and implementing scalable and resilient systems, utilizing data lake solutions, and stream processing engines to meet business goals.

responsibilities

Develop and implement scalable, resilient systems utilizing Data Lake solutions
Participate in the full software development lifecycle (SDLC) utilizing Agile methodologies
Build and maintain data pipelines using stream processing engines such as Kafka, Spark, etc.
Perform advanced data analysis and modeling using relevant programming languages
Ensure data quality and integrity through proper testing and validation techniques

requirements

Bachelor's/Master's Degree in Computer Science, Information Systems, or equivalent
At least 3 years of relevant experience in Data Software Engineering
Proficient with software development lifecycle (SDLC) methodologies like Agile, Test-driven development
Extensive experience in Databricks, Python, AWS, and Data Warehousing
Advanced SQL coding skills
B2+ English level

nice to have

Experience with Apache Airflow and Apache Spark
Expertise in Relational Database Design

senior big data engineer with databricks

Data Software Engineering

Amazon Web Services, Apache Airflow, Apache Spark

40 hrs/week

12+ months

India

We are looking for an experienced remote Senior Big Data Engineer with expertise in Databricks to lead our team of data engineers and take responsibility for designing and implementing data management solutions.

The successful candidate will be familiar with cloud-based data pipeline development and have experience building optimal data infrastructure solutions.

responsibilities

Lead a team of data engineers, organize teamwork, and solve technical and non-technical challenges
Design key software components of the data platform, and build collaborative partnerships with architects and technical leads
Create and maintain optimal on-prem and cloud-based data pipeline architectures capable of ingesting structured and unstructured data using batch & streaming methods
Assemble large, complex data sets that meet functional and technical requirements, and develop data flows that can leverage both on-premise and cloud architectures
Work with stakeholders to assist with data-related technical issues and support their data infrastructure needs

requirements

Bachelor's degree in Computer Science, Information Systems, or another applicable field is preferred
At least 3 years of experience in a Data Engineer role
Advanced working knowledge of cloud-based data pipeline development, particularly on AWS
Experience building and optimizing data pipelines for data acquisition and management
Expertise in Databricks, Python, SQL, and AWS
B2+ English level

nice to have

Understanding of message queuing, stream processing, and Change Data Capture mechanisms

Big Data Developer Jobs at EPAM Anywhere

Big data developers create various products using programming and development processes. The purpose of the latter is to accommodate a huge data flow in the company’s business processes, which the usual databases are not able to cope with. Most often, big data developers work with the Hadoop platform and create software using various programming languages and tools. Big data developer jobs involve many different tasks and the use of various methods to achieve goals.

Main job description

In any business, there are processes that require reliable software. Big data developers are able to process a huge amount of data every day to achieve the necessary requirements for any project. These professionals need to be familiar with many technologies and be able to find the optimal solution for any problem. A huge benefit of the profession is that you can work from home.

Big data developer job responsibilities

Online and offline big data developer work involves the following responsibilities:

Produce products for processing a huge data stream
Analyze data
Ensure a quick response from the product to solve the problems of the company
Support for scalability and high performance of developed services
Detail complex projects
Develop and implement Hadoop applications
Ensure the security of data storage
Set up all automation processes and access data quickly

The number of responsibilities will depend on the skill level of the big data developer. Juniors will not have to do even half of the above. Their job is to perform routine tasks under the supervision of senior specialists. But for middle and senior professionals, the list can grow. For example, a senior big data developer often fills the role of a manager. They constantly monitor the performance of their subordinates, manage them, and distribute the entire scope of tasks.

Requirements for big data developer work

There are many requirements for a big data developer, but we have chosen the most common ones that are required for both office and online jobs:

2+ years of experience in a similar job position
Excellent knowledge of several programming languages
Ability to debug huge data streams
Experience with different visualization and data analysis tools
Deep knowledge of modeling and data structuring processes
Knowledge of streaming frameworks and automation tools
Familiarity with one scripting language
Bachelor's or master's degree in computer technology
Knowledge of ETL methodology and agile principles
Experience in creating high-volume applications based on open source systems
Knowledge of the basics of machine learning and the Hadoop ecosystem

Personal qualities of big data developers

Big data developer jobs require more than just technical skills. These specialists are often tied to a whole team of professionals from other IT areas. Big data developers must have a high level of communication, the ability to delegate tasks to their colleagues, and the desire to work together with other specialists.

There are additional requirements for working from home. If you are sure that you are ready for a remote job in the big data developer specialty, make sure you have the right qualities. You will need to be good at separating your work area from your place of rest and have excellent time management skills.

Big Data Developer Jobs

Big Data Developer Jobs at EPAM Anywhere

Main job description

Big data developer job responsibilities

Requirements for big data developer work

Personal qualities of big data developers

Related skills