Data Engineering Expert

2 days ago


New York, New York, United States The Custom Group of Companies Full time
Job Description:

The Custom Group of Companies is seeking a highly skilled Data Engineering Expert to join our team. In this role, you will design, develop, and maintain data pipelines in Databricks on an AWS 6 ecosystem with Delta Lake, Python, SQL, and Starburst as the technology stack.

As a Senior Data Engineer, you will be responsible for establishing data quality checks, ensuring data integrity and accuracy throughout the data lifecycle, automating testing of the data pipelines, optimizing data processing and query performance for large-scale datasets within AWS and Databricks environments, documenting data engineering processes, architecture, and configurations, troubleshooting and debugging data-related issues on the AWS Databricks platform, and integrating Databricks with other AWS products such as SNS, SQS, and MSK.

Responsibilities:
  • Design, develop, and maintain data pipelines in Databricks on an AWS 6 ecosystem.
  • Establish data quality checks and ensure data integrity and accuracy throughout the data lifecycle.
  • Automate testing of the data pipelines and configure as part of CICD.
  • Optimize data processing and query performance for large-scale datasets within AWS and Databricks environments.
  • Document data engineering processes, architecture, and configurations.
  • Troubleshoot and debug data-related issues on the AWS Databricks platform.
  • Integrate Databricks with other AWS products such as SNS, SQS, and MSK.


  • New York, New York, United States Hudson Data Full time

    Hudson Data is seeking a highly skilled Cloud Data Engineer Expert to join our team.We are looking for an experienced professional with hands-on knowledge of various Azure services, including but not limited to Azure Data Factory, Databricks, Data Lake, and Power BI. The ideal candidate should be able to design, build, and maintain ETL pipelines, manage data...


  • New York, New York, United States Hudson Data Full time

    Hudson Data is a cutting-edge technology company that is seeking a highly skilled Azure Data Engineering Lead to join our team.We are looking for an experienced professional with hands-on knowledge of various Azure services, including but not limited to Azure Data Factory, Databricks, Data Lake, and Power BI. The ideal candidate should be able to design,...


  • New York, New York, United States Inizio Partners Full time

    About Inizio PartnersWe provide data-driven solutions to business problems through statistical data mining and cutting-edge analytics techniques. Our team of experts includes 2,000+ data scientists and analysts who assist client organizations with complex risk minimization methods and advanced marketing strategies.Job OverviewWe are seeking a talented Data...


  • New York, New York, United States Glow Networks Full time

    Job Title: Data Engineering ExpertWe are seeking a seasoned Data Engineering Expert to join our team at Glow Networks in New York City. This is a 6-month contract position with possible extensions, offering a salary of $140,000 per year.Job Description:The successful candidate will have expertise in designing and implementing big data architecture in cloud...


  • New York, New York, United States Capital One Services, LLC Full time

    Company OverviewAt Capital One Services, LLC, we are seeking a talented Data Engineer Expert to join our team. As a leading financial institution, we are committed to innovation and excellence in data engineering.


  • New York, New York, United States Lubin Talent Solutions Full time

    Job Title: Data Engineer ExpertAbout Us: At Lubin Talent Solutions, we are a dynamic team of professionals dedicated to delivering innovative solutions in the field of data engineering. Our expertise spans across designing and implementing efficient ETL workflows, ensuring data governance principles, and providing financial services experience.Role Summary:...


  • New York, New York, United States Tech Tammina Full time

    Job Title: Data Engineering ExpertLocation: New YorkSalary: $180,000 - $220,000 per yearAbout the Role:The ideal candidate should have extensive experience in IT and data engineering, particularly with GCP in the financial sector. They should be proficient in managing and optimizing data storage, ensuring data security, and using Python and SQL for data...


  • New York, New York, United States Bloomberg Full time

    Job SummaryWe are seeking a highly motivated and experienced Data Engineering Expert to join our team at Bloomberg. As a key member of our data management team, you will be responsible for developing strategies to optimize the value of Equity Corporate Actions data for our clients and improve data operations.In this role, you will be diving deep into complex...


  • New York, New York, United States Infojini Full time

    **Job Description:**We are seeking a highly skilled Data Engineering Expert to join our team at Infojini. As a key member of our data engineering team, you will be responsible for designing and implementing scalable data pipelines that meet the needs of our business.**Key Responsibilities:**Create optimal data pipeline architecture using best practices and...


  • New York, New York, United States Prophecy Full time

    About ProphecyWe're a data copilot company that helps Fortune 500 enterprises accelerate AI and analytics by delivering clean, trusted, and timely data.Our customers rely on us to develop, deploy, and observe data pipelines on cloud data platforms. We make all data users productive by helping them create, manage, and monitor massive volumes of data for AI...


  • New York, New York, United States Futran Tech Solutions Pvt. Ltd. Full time

    Job Title: Data Engineer Cloud ExpertAbout Us:Futran Tech Solutions Pvt. Ltd. is a leading provider of data engineering services, helping organizations unlock the full potential of their data.Salary: $120,000 - $180,000 per year based on experience and qualifications.Location: New York City, NY (day one onsite)Job Description:We are seeking an experienced...


  • New York, New York, United States The Custom Group of Companies Full time

    We are seeking a seasoned Data Engineering expert to join our team at The Custom Group of Companies. As a Senior Data Engineer, you will be responsible for designing, developing, and maintaining data pipelines in Databricks on an AWS 6 ecosystem with Delta Lake, Python, SQL, and Starburst as the technology stack.In this role, you will collaborate with...


  • New York, New York, United States Enterprise Engineering Full time

    Join us at Enterprise Engineering as a Senior DB2 DBA and take on a challenging role that requires expertise, creativity, and a drive to deliver exceptional results.About the PositionThis position involves providing expert-level support for our clients' DB2 environments, ensuring they meet their business objectives while maintaining optimal performance and...


  • New York, New York, United States Amazon Full time

    At Amazon, we're committed to delivering exceptional customer experiences. To achieve this, we're seeking a skilled data engineer to join our Worldwide Delivery Experience team. As a key contributor, you'll play a vital role in designing, developing, and enhancing systems and pipelines that optimize processes in reporting, planning, and controlling...


  • New York, New York, United States Gen II Fund Services Full time

    Gen II Fund ServicesWe are seeking an experienced Data Engineer to join our team in New York. This is a challenging role that requires strong technical skills and excellent communication abilities.The ideal candidate will have a strong background in data engineering and experience working with cloud-based data platforms, such as Snowflake and dbt. The...

  • Data Engineer Expert

    3 weeks ago


    New York, New York, United States Capital One Full time

    Join Capital One as a Sr. Data EngineerWe are seeking an experienced Data Engineer to join our team at Capital One in Plano, Texas. As a Sr. Data Engineer, you will be responsible for designing, developing, testing, implementing, and supporting technical solutions in full-stack development tools and technologies.Responsibilities:Collaborate with Agile teams...


  • New York, New York, United States Early Warning Services Full time

    About Our TeamWe are Early Warning Services, a leading provider of payment solutions. We are seeking a highly skilled Big Data Analytics Expert to join our team.The successful candidate will be responsible for developing, documenting, testing, debugging, and maintaining data engineering solutions for Paze.The ideal candidate will have expertise in software...


  • New York, New York, United States Quanta Search Full time

    **Job Description**Quanta Search is looking for an experienced Data Architecture and Engineering Expert to design and implement scalable database infrastructure for their Algo Trading Strategies and Technology department. The ideal candidate will have a strong background in database administration and software development.**Key Responsibilities:**Design,...


  • New York, New York, United States The Data Sherpas Full time

    Job Summary:We are seeking an experienced GCP Data Science Engineering Lead to join our team at The Data Sherpas. As a GCP Data Science Engineering Lead, you will be responsible for driving technology innovation by delivering world-class machine learning solutions on Google Cloud Platform (GCP).Key Responsibilities:Lead the design and deployment of...

  • Cloud Data Engineer

    17 hours ago


    New York, New York, United States Diverse Lynx Full time

    We are seeking a highly skilled Cloud Data Engineer - Pyspark and Databricks Expert to join our team at Diverse Lynx LLC. This is an excellent opportunity to work on challenging projects and develop your skills in data engineering.About the RoleWe are looking for someone with at least 5 years of experience working as a Databricks developer and 3 years of...