Data Engineering Expert for Databricks and AWS

7 days ago


New York, New York, United States The Custom Group of Companies Full time

We are seeking a seasoned Data Engineering expert to join our team at The Custom Group of Companies. As a Senior Data Engineer, you will be responsible for designing, developing, and maintaining data pipelines in Databricks on an AWS 6 ecosystem with Delta Lake, Python, SQL, and Starburst as the technology stack.

In this role, you will collaborate with cross-functional teams to understand data needs and translate them into effective data pipeline solutions. Your key responsibilities will include establishing data quality checks, ensuring data integrity and accuracy throughout the data lifecycle, automating testing of data pipelines, and optimizing data processing and query performance for large-scale datasets within AWS and Databricks environments.

You will also be responsible for documenting data engineering processes, architecture, and configurations, as well as troubleshooting and debugging data-related issues on the AWS Databricks platform. Additionally, you will integrate Databricks with other AWS products such as SNS, SQS, and MSK.

To succeed in this position, you must have a minimum of 5 years of experience in data engineering roles, with a focus on AWS and expert-level skills in coding and testing Databricks in an AWS 6 environment. You should have experience with Python, PySpark, SQL, Spark, GitLab with CI/CD, and AWS services like S3, RDS, Lambda, SQS, SNS, and MSK. Databricks Certified is highly desired.

A strong understanding of data analysis and source data is essential, as well as experience with data pipeline orchestration tools. Excellent communication, collaboration, experimentation, inquisitiveness, and attention to detail are also required.

The selected candidate will work on a team of 15 professionals and will be based in New York, NY, with a hybrid position that requires onsite presence two days a week. The estimated salary for this position is $105 per hour, with a contract duration of 1+ year.



  • New York, New York, United States Diverse Lynx Full time

    About the RoleDiverse Lynx LLC is seeking a highly skilled Senior Data Engineering Specialist to join our team as a Databricks/PySpark Developer. As a key member of our data engineering team, you will be responsible for designing, developing, and maintaining large-scale data processing systems using Databricks and PySpark.Key ResponsibilitiesDesign and...


  • New York, New York, United States Diverse Lynx Full time

    Company OverviewDiverse Lynx LLC is an Equal Employment Opportunity employer. We promote and support a diverse workforce across all levels in the company.SalaryThe estimated salary for this position is $120,000 - $180,000 per year, depending on experience.Job DescriptionWe are seeking a highly skilled Databricks / PySpark Developer to join our team. As a key...

  • Data Engineer

    7 days ago


    New York, New York, United States Diverse Lynx Full time

    Job Description:We are seeking a skilled Data Engineer with expertise in Databricks and PySpark to join our team at Diverse Lynx LLC.Key Responsibilities:Design, develop, and maintain large-scale data processing pipelines using Databricks and PySpark.Collaborate with cross-functional teams to integrate data solutions into business operations.Analyze complex...

  • Data Engineer Manager

    2 months ago


    New York, New York, United States DataBricks Full time

    Job DescriptionWe are seeking a highly experienced Data Engineer Manager to lead our Field Engineering team. As a key member of our organization, you will be responsible for building and leading a team of technical pre-sales professionals to drive business growth in the Financial Services vertical.Key Responsibilities:Lead a team of Solutions Architects to...


  • New York, New York, United States Publicis Sapient Full time

    Unlock Business Value with Databricks ExpertiseAbout the Role:We're seeking a highly skilled Data Engineering Architect to lead the build of global Databricks platforms, enabling real business outcomes for our enterprise clients. As a top-notch technologist, you'll create impact for some of the world's biggest brands by translating their needs into...


  • New York, New York, United States DataBricks Full time

    We are seeking an experienced Data Architect Lead to join our team at Databricks, a leading provider of cloud-based data analytics and machine learning platforms. In this role, you will be responsible for driving customer success by delivering high-quality solutions that meet the complex needs of our clients.About the RoleThis is a hybrid technical and...


  • New York, New York, United States Encora Full time

    About the RoleWe're seeking a skilled Data Engineer to join our team at Encora. As a Data Engineer, you will be responsible for ensuring that data is received, verified, transformed, stored, and accessible to the team.Key ResponsibilitiesBuild and promote data pipeline management and database maintenanceGather and prepare data to be analyzed and worked on,...

  • Data Engineer

    7 days ago


    New York, New York, United States Diverse Lynx Full time

    About the RoleWe are seeking an experienced Data Engineer to join our team as an Azure Databricks Specialist. This role will focus on working with Azure Databricks services, creating and managing data pipelines, and developing scalable data solutions.ResponsibilitiesDesign and implement data ingestion, transformation, and loading pipelines using Azure Data...


  • New York, New York, United States Databricks Full time

    Job Summary:Databricks is seeking an experienced Privacy Legal Counsel to join our expanding legal team. As a key member of the Databricks privacy team, you will play a crucial role in navigating global privacy laws and regulations.About the Role:This position involves providing comprehensive legal counsel to various business partners, supporting the...


  • New York, New York, United States RIT Solutions, Inc. Full time

    Job Requirements: We are seeking a highly skilled professional with a robust background in data engineering and a strong proficiency in AWS technologies.Key Responsibilities:Proficiency in AWS services such as AWS Glue, Amazon Redshift, and AWS Lambda.Strong knowledge of SQL.Experience with Python or Java for data processing.Benefits: Opportunity to work on...


  • New York, New York, United States DataBricks Full time

    At DataBricks, we're seeking a highly skilled Privacy Legal Counsel to join our rapidly expanding team. This is a unique opportunity to make a meaningful impact on the future of data and AI, while working with a talented group of professionals who share your passion for innovation.About the Role:We're looking for an experienced lawyer with a deep...


  • New York, New York, United States Infojini Full time

    **Job Description:**We are seeking a highly skilled Data Engineering Expert to join our team at Infojini. As a key member of our data engineering team, you will be responsible for designing and implementing scalable data pipelines that meet the needs of our business.**Key Responsibilities:**Create optimal data pipeline architecture using best practices and...


  • New York, New York, United States S&P Global Full time

    About the Role:S&P Global seeks a skilled Data Engineer to join its Data Lake team. The ideal candidate will have expertise in designing and building data pipelines with an emphasis on scale, performance, and reliability. This role is responsible for providing technical expertise in the areas of design and implementation of Data Lake solutions powered by...


  • New York, New York, United States Inizio Partners Full time

    Job Title: Data ScientistAbout the Job:We are seeking a skilled Data Scientist to join our team at Inizio Partners. This individual will work closely with cross-functional project teams to develop and apply models and statistical analysis within our enterprise data science group.Job Description:Develop and implement machine learning and deep learning models...


  • New York, New York, United States DataBricks Full time

    Unlocking AI PotentialWe are seeking a highly skilled Research Scientist to join our GenAI Team at Databricks. As a member of this team, you will be responsible for pushing the boundaries of deep learning and advancing the scientific frontier by creating innovative techniques that surpass current standards.Key Responsibilities:Stay up-to-date with the latest...


  • New York, New York, United States Diverse Lynx Full time

    About UsDiverse Lynx LLC is an Equal Employment Opportunity employer committed to fostering a diverse and inclusive workforce.Job SummaryWe are seeking an experienced Senior Big Data Engineer with expertise in cloud native architecture to join our team. As a key member of our engineering team, you will be responsible for designing and implementing scalable...


  • New York, New York, United States NYU Langone Health Full time

    Job Summary:NYU Langone Health is seeking a highly skilled Azure Databricks Administrator to join our team. The ideal candidate will have experience managing and maintaining Databricks platforms, ensuring optimal performance and security across workspaces, clusters, and jobs.Key Responsibilities:Manage and maintain Azure Databricks Platform workspaces,...


  • New York, New York, United States S&P Global Full time

    About the Role:At S&P Global, we're seeking a highly skilled Data Lake Architect to join our Data Lake team. The successful candidate will be responsible for designing and building data pipelines with an emphasis on scale, performance, and reliability.Key Responsibilities:Design and implement data pipelines using Databricks on AWS cloud.Partner with data...


  • New York, New York, United States Conde Nast Full time

    About the Role:Conde Nast is seeking a seasoned Data Solutions Engineer to lead our global data engineering & data warehousing practice. As a key member of our engineering team, you will drive the development of a customer 360 initiative across our revenue lines of business.Responsibilities:Lead and manage a diverse team of data engineers, fostering a...


  • New York, New York, United States Publicis Groupe Full time

    At Publicis Sapient, we are dedicated to helping our clients drive digital transformation and achieve their business goals. As a Manager Data Engineering, Databricks Platform Architecture, you will play a key role in leading the build of global Databricks platforms to enable real business outcomes for our enterprise clients.Company OverviewWe are a digital...