Big Data Engineering Lead

7 days ago


Columbus, Ohio, United States Diverse Lynx Full time
Job Overview:

We are seeking a highly skilled Big Data Engineering Lead to join our team at Diverse Lynx LLC. As a seasoned professional in distributed systems, you will be responsible for designing and implementing data platforms using Python, Spark, and PySpark.

Key Responsibilities:
  • Develop and maintain scalable data platforms utilizing Python, Spark, and PySpark.
  • Migrate existing applications to PySpark on AWS cloud infrastructure.
  • Design and implement efficient data pipelines that cater to the needs of large-scale datasets.
  • Collaborate with cross-functional teams to integrate data solutions with SQL databases (Microsoft, Oracle, Postgres, MySQL).
  • Write Scaladoc-style documentation for codebase to ensure maintainability and scalability.
  • Optimize Spark queries for improved performance.

Required Skills and Qualifications:

To succeed in this role, you must possess a deep understanding of distributed systems concepts, including CAP theorem, partitioning, replication, consistency, and consensus. Your expertise should extend to proficiency in Python, Scala (with a focus on functional programming), and Spark APIs (RDD, Data Frame, MLlib, GraphX, Streaming). Additionally, experience working with HDFS, S3, Cassandra, and/or DynamoDB is essential. A strong background in building or maintaining cloud-native applications and serverless approaches using AWS Lambda is a significant advantage.

Compensation Package:

The estimated salary range for this position is between $150,000 and $250,000 per year, commensurate with experience and qualifications. This figure may vary based on your skills, location, and industry standards. Our company offers a competitive benefits package, including health insurance, retirement plans, and opportunities for growth and development within the organization.

  • Columbus, Ohio, United States Intuites, LLC Full time

    Job OverviewThis is an exciting opportunity to join Intuites LLC as a Big Data Engineer on our QA Delivery team.Salary and BenefitsThe estimated annual salary for this position is $100,000 - $150,000 depending on experience. In addition to competitive compensation, we offer a comprehensive benefits package that includes health insurance, 401(k) matching, and...


  • Columbus, Ohio, United States Intuites, LLC Full time

    Job DescriptionWe are seeking a skilled Big Data Engineer to join our QA Delivery team in Columbus, OH. This individual contributor position will focus on delivering high-quality test execution for the 2nd DC project.This role involves testing data integration between Active Omni and MapR/Hive, as well as report validations. The successful candidate will be...


  • Columbus, Ohio, United States Seamless Full time

    Job Title: Principal Data EngineerThe Opportunity:We are seeking a highly skilled and experienced Principal Data Engineer with expertise in Python, Spark, AWS Glue, and other ETL technologies to join our team at Seamless.AI.The ideal candidate will have a proven track record in data acquisition and transformation, as well as experience working with large...


  • Columbus, Ohio, United States TechnoGen Full time

    Job Title: Data Architect SpecialistJob Summary:We are seeking a highly skilled Data Architect Specialist to join our team at TechnoGen. The ideal candidate will have a strong background in data modeling, Talend, Snowflake, and GCP/Big Query. As a Data Architect Specialist, you will be responsible for designing and implementing data models, working with...


  • Columbus, Ohio, United States Diverse Lynx Full time

    Job DescriptionWe are seeking a highly skilled Distributed Data Engineer to join our team at Diverse Lynx LLC. As a key member of our data engineering team, you will be responsible for designing, developing, and maintaining large-scale data platforms using Python, Spark, and PySpark.Key Responsibilities:Develop and maintain data platforms using Python,...


  • Columbus, Ohio, United States TechnoGen Full time

    Job Title: Azure Databricks Technical LeadJob Summary:We are seeking a highly skilled Azure Databricks Technical Lead to join our team at TechnoGen. As a key member of our data engineering team, you will be responsible for designing, developing, and administering data management solutions that align with the business needs of our organization.Key...

  • Data Science Leader

    2 days ago


    Columbus, Ohio, United States BJSS Full time

    Senior Data Scientist Role at BJSSInnovative Tech Consultancy Seeks Experienced ExpertiseWe are an award-winning tech consultancy dedicated to innovative problem-solving. With a 30-year history of delivering sustainable solutions for leading organizations, our teams bring together diverse expertise and collaborative cultures. Our latest recognition in the...


  • Columbus, Ohio, United States Seamless Full time

    Job OverviewAt Seamless.AI, we're seeking a highly skilled Principal Data Engineer to lead our data engineering efforts.Key ResponsibilitiesDesign and develop scalable ETL pipelines using Python and AWS Glue.Collaborate with cross-functional teams to understand data requirements and develop efficient data acquisition and integration strategies.Implement data...


  • Columbus, Ohio, United States Maintec Technologies Full time

    Job Title: Data EngineerExperience: 11 to 13 YearsAt Maintec Technologies, we are seeking a highly skilled Data Engineer with 6-8 years of experience in PySpark and AWS. The ideal candidate will have a strong background in data acquisition, processing, and integration.Key Responsibilities:Data Acquisition and Processing: Develop and implement robust data...


  • Columbus, Ohio, United States American Chemical Society Full time

    About UsThe American Chemical Society is a leading scientific organization dedicated to advancing the chemical sciences.Job DescriptionWe are seeking an experienced Data Analytics Team Lead to join our Publications Technology - Data Engineering & Analytics Team. As a senior member of the team, you will be responsible for leading the design and development of...


  • Columbus, Ohio, United States Saxon Global Full time

    Job Description:Saxon Global seeks a skilled Data Architect to design and deliver enterprise-wide solutions on cloud platforms. This role collaborates with architects, data experts, and solution architects to design and implement data solutions that meet business priorities in alignment with technology strategy, standards, and best practices.The successful...


  • Columbus, Ohio, United States Experis Full time

    Experis, a leading workforce solutions company, is seeking a seasoned Data Engineer to join its team in Columbus, Ohio or remotely. This role offers a competitive pay rate of $90-100/hour.About the RoleThe successful candidate will work closely with business stakeholders to design and implement data analytics solutions using Power BI Service and cloud &...


  • Columbus, Ohio, United States American Chemical Society Full time

    OverviewA leading scientific information solutions division is seeking a skilled DevOps Software Engineer to join their global team. Located in Columbus, Ohio, this role will play a key part in developing automation and observability for builds, deployments, and ongoing operations.


  • Columbus, Ohio, United States Jobs for Humanity Full time

    Job DescriptionAbout UsJobs for Humanity is a collaborative platform that aims to build an inclusive and just employment ecosystem. We strive to provide opportunities for individuals from diverse backgrounds.About the RoleWe are seeking a highly skilled Data Engineer to join our team at Safelite, a leading auto glass company. As a Senior Data Analytics...


  • Columbus, Ohio, United States Abercrombie and Fitch Co. Full time

    Abercrombie & Fitch Co. is a global leader in the retail industry, operating five iconic lifestyle brands. We are seeking an experienced Data Engineering Leader to join our team and drive the development of cutting-edge data solutions.Job Description:The successful candidate will be responsible for leading the development of data engineering infrastructure,...


  • Columbus, Ohio, United States Beacon Hill Staffing Group Full time

    Beacon Hill Staffing Group is seeking a Senior Data Quality Engineer to support our clients in the healthcare industry.About the RoleThe Senior Data Quality Engineer will be responsible for ensuring the accuracy, completeness, and integrity of data sets for use in internal data systems and for delivery from internal systems. This includes designing,...

  • Data Scientist II

    7 days ago


    Columbus, Ohio, United States Nationwide Children's Hospital Full time

    Welcome to Nationwide Children's Hospital, where we're committed to improving the lives of children and families through innovative data-driven solutions. As a Data Scientist II in our Research Information Solutions and Innovation division, you'll have the opportunity to work on cutting-edge clinical informatics research projects that impact clinical care...


  • Columbus, Ohio, United States SysMind Tech Full time

    About the RoleWe are seeking an experienced Data Scientist and Engineering Leader to join our team at SysMind Tech. As a key member of our organization, you will be responsible for developing and productionizing Python and Java applications using machine learning frameworks like Keras or PyTorch.Key ResponsibilitiesDevelop high-quality software solutions...


  • Columbus, Ohio, United States Noblis Full time

    OverviewNoblis is a leading provider of innovative solutions to the US government. Our team of experts works on cutting-edge projects in data science, aerospace engineering, and software development.Job DescriptionWe are seeking an Aerospace Software Engineer for Aeronautics and Data Science to join our team. The ideal candidate will have a strong background...

  • Data Engineer

    6 days ago


    Columbus, Ohio, United States Resource Informatics Group Full time

    Job Description:We are seeking a highly skilled Senior Data Engineer to join our team at Resource Informatics Group. As a key member of our data engineering team, you will be responsible for designing, developing, and maintaining large-scale data pipelines using Azure Data Factory (ADF), Azure Data Lake (ADLS), Azure Databricks, SQL Server, and Data...