Big Data Engineer Lead

2 weeks ago

New York, United States InterSources Full time
Job Title: Big Data Engineer
Type: Contract
Location: New York, NY ( Day 1 onsite)

Job Description:
A Big Data Engineering lead handles the ETL jobs, Optimization, Automation of data processing for complex scenarios.

Responsibilities:
  • Data ingestion from Source
  • DQ checks
  • Handling ETL Automation
  • Building features / variable

Required Skills:
  • Should be subject matter expert in building and designing the ETL pipelines in ADF or ADLS Gen2.
  • Should be able to use Spark, Synapse to process data from Azure data lake
  • Experience in downloading data from UDL & ingesting in PDS. Good experience with ADF.
  • Expert in Spark for structured, semi-structured and un-structured data processing
  • Hands-on knowledge in Azure Databricks or Azure Purview, Synapse
  • Should be clear on designing constructs for different data sets and come up with reusable templates for the same
  • Should have experience of implementing Data Quality methodologies to pre-empt errors in patterns and to identify Data Quality issues.

About Us:
InterSources Inc, a Certified Diverse Supplier, was founded in 2007 and offers innovative solutions to help clients with Digital Transformations across various domains and industries. Our history spans over 16 years and today we are an Award-Winning Global Software Consultancy solving complex problems with technology. We recognize that our employees and our clients are our strengths as the diverse talents and opportunities they bring to the table enable us to grow as a global platform and they are causally linked with our success. We provide strategic and technical advice, and we have expertise in areas covering Artificial Intelligence, Cloud Migration, Custom Software Development, Data Analytics Infrastructure & Cloud Solutions, Cyber Security Services, etc. We make reasonable accommodations for clients and employees and we do not discriminate based on any protected attribute including race, religion, color, national origin, gender sexual orientation, gender identity, age, or marital status. We also are a Google Cloud partner company. We align strategy with execution and provide secure service solutions by developing and using the latest technologies that thrive our resources to deliver industry-leading capabilities to our clients and customers, making it convenient for our clients to do business with InterSources Inc. Our teams also drive growth by refining technology-driven client experiences that put the users first, providing an unparalleled experience. This results in strengthening the core technologies of clients, enabling them to scale with flexibility, create seamless digital experiences and build lifelong relationships.

  • Big Data Engineer

    7 days ago

    New York, United States E-Solutions Full time

    Role : Big Data EngineerLocation : NYC office Fully Onsite (Local only)Inperson Interview is mustTechnical Skills:• Big Data Ecosystem: HDFS, Hive, Oozie, Sqoop, Impala, Presto and Spark.• Cloud Ecosystem: AWS (EMR, S3, Athena), Snowflake• Languages: Python, SQL and Shell Scripting.• Databases: MySQL, Teradata, and SQL SERVER.• Orchestration/Tools:...

  • Lead Data Engineer

    2 months ago

    New York, United States Capital One Full time

    Center 1 (19052), United States of America, McLean, VirginiaLead Data EngineerDo you love building and pioneering in the technology space? Do you enjoy solving complex business problems in a fast-paced, collaborative,inclusive, and iterative delivery environment? At Capital One, you'll be part of a big group of makers, breakers, doers and disruptors, who...

  • Lead Data Engineer

    2 months ago

    New York, United States Capital One Full time

    11 West 19th Street (22008), United States of America, New York, New YorkLead Data EngineerDo you love building and pioneering in the technology space? Do you enjoy solving complex business problems in a fast-paced, collaborative,inclusive, and iterative delivery environment? At Capital One, you'll be part of a big group of makers, breakers, doers and...

  • Lead Data Engineer

    2 months ago

    New York, United States Capital One Full time

    Locations: VA - McLean, United States of America, McLean, VirginiaLead Data EngineerDo you love building and pioneering in the technology space? Do you enjoy solving complex business problems in a fast-paced, collaborative,inclusive, and iterative delivery environment? At Capital One, you'll be part of a big group of makers, breakers, doers and disruptors,...

  • Big Data Architect with Devops

    6 months ago

    New York, United States CyberTec Full time

    Role 3: Big Data Architect with Devops experience and local to NY Start Date: ASAP Rate - $90/hr. Maximum Start Date: ASAP Local to NY its onsite Client: Microsoft We'd like to see candidates at the Tech 3 level Description: Job Summary and Responsibilities: Hands on keyboard engineer with experience in the DevOps space and, preferably, experience in the...

  • Lead Data Engineer

    2 weeks ago

    New York, United States Radicle Health Full time

    Job DescriptionJob DescriptionRadicle Health acquires, invests, and operates mission critical human services software (SaaS) companies. We believe that human services agencies and the people they serve deserve functional, modern, and easy-to-use software. And we believe we're the ones to build it. Radicle Health is backed by Alpine Software Group (ASG),...

  • Lead Data Engineer

    1 month ago

    NEW YORK, United States Capital One Careers Full time

    11 West 19th Street (22008), United States of America, New York, New YorkLead Data EngineerDo you love building and pioneering in the technology space? Do you enjoy solving complex business problems in a fast-paced, collaborative, inclusive, and iterative delivery environment? At Capital One, you'll be part of a big group of makers, breakers, doers and...

  • Lead Data Engineer

    1 month ago

    NEW YORK, United States Capital One Careers Full time

    Center 1 (19052), United States of America, McLean, VirginiaLead Data EngineerDo you love building and pioneering in the technology space? Do you enjoy solving complex business problems in a fast-paced, collaborative, inclusive, and iterative delivery environment? At Capital One, you'll be part of a big group of makers, breakers, doers and disruptors, who...

  • Lead Data Engineer

    2 months ago

    New Morgan, United States Penske Truck Leasing Full time

    Summary Statement: Working with a team, you will lead the technical design, engineering, and support of complex data integrations between business-critical applications and systems. You will also participate in defining and implementing data engineering and analytics strategies; and mentor other data engineers on data engineering best practices to create...

  • Lead Data Engineer

    1 month ago

    NEW YORK, United States Capital One Careers Full time

    Locations: VA - McLean, United States of America, McLean, VirginiaLead Data EngineerDo you love building and pioneering in the technology space? Do you enjoy solving complex business problems in a fast-paced, collaborative, inclusive, and iterative delivery environment? At Capital One, you'll be part of a big group of makers, breakers, doers and disruptors,...

  • Senior Lead Data Engineer

    4 weeks ago

    New York, United States Capital One Full time

    130 5th Ave (22130), United States of America, New York, New YorkSenior Lead Data EngineerDo you love building and pioneering in the technology space? Do you enjoy solving complex business problems in a fast-paced, collaborative,inclusive, and iterative delivery environment? At Capital One, you'll be part of a big group of makers, breakers, doers and...

  • Data Engineer

    4 weeks ago

    New York, United States hackajob Full time

    Exciting Opportunity Alert! Join our Team as a Data Engineer!hackajob has partnered with an innovative company seeking a skilled Data Engineer to leverage data for impactful solutions. If you have expertise in big data tools like Spark and Kafka, and experience with cloud services like AWS, we want to hear from YOU!Role: Data EngineerLocation: Hybrid (New...

  • Big data Architect

    1 week ago

    New York, United States SysMind Tech Full time

    Hi, Hope you are doing well. Here is the position. Kindly revert me back if you are interested. Role: Big Data Architect Location: Remote Duration: 12+ Months Skills: 1. 8-10 Years of strong Data architect\modeling experience 2. Design and architect the tables\data marts for staging, facts and dimensions 3. Develop mapping specs for Fact and Dimension...

  • GCP Machine Learning Engineer

    4 weeks ago

    New York, United States The Data Sherpas Full time

    J ob Summary : As a GCP Machine Learning Engineer, you will drive technology innovation by delivering world-class machine learning solutions. You will integrate GCP-native and third-party technologies to build scalable data infrastructures, focusing on modernizing data pipelines and machine learning platforms. You will collaborate with implementation teams...

  • Big Data Solutions Developer

    4 weeks ago

    New York, New York, United States RiVi Consulting Group Full time

    Job SummaryAs a seasoned Spark Developer at RiVi Consulting Group, you will play a vital role in designing and implementing cutting-edge big data solutions. Your expertise in Spark and HDFS will enable us to deliver scalable and efficient data processing capabilities to drive business growth.Key Responsibilities• Collaborate with stakeholders to understand...

  • Generative AI Architect

    2 months ago

    New Brunswick, United States eProSoft Full time

    Generative AI Architect (Big Data & Java Expertise)Hybrid roleNew Brunswick, NJWe are seeking a skilled and innovative Generative AI Architect with a strong background in Big Data and Java to lead our AI-driven initiatives. The ideal candidate will design and implement cutting-edge generative AI models and systems that leverage large datasets, while ensuring...

  • Data Engineer

    4 weeks ago

    new york city, United States hackajob Full time

    Exciting Opportunity Alert! Join our Team as a Data Engineer!hackajob has partnered with an innovative company seeking a skilled Data Engineer to leverage data for impactful solutions. If you have expertise in big data tools like Spark and Kafka, and experience with cloud services like AWS, we want to hear from YOU!Role: Data EngineerLocation: Hybrid (New...

  • Data Engineer

    4 weeks ago

    new york city, United States hackajob Full time

    Exciting Opportunity Alert! Join our Team as a Data Engineer!hackajob has partnered with an innovative company seeking a skilled Data Engineer to leverage data for impactful solutions. If you have expertise in big data tools like Spark and Kafka, and experience with cloud services like AWS, we want to hear from YOU!Role: Data EngineerLocation: Hybrid (New...

  • Sr. Data Engineer

    1 month ago

    New York, United States iO Associates - US Full time

    Are you a seasoned data engineer looking to make a significant impact in a dynamic environment? We're seeking a highly skilled professional to join our team and drive our data initiatives forward.Responsibilities:Design, develop, and implement advanced data pipelines using industry-leading cloud platforms.Architect and manage scalable data solutions for...

  • Principal Data Engineer

    4 weeks ago

    New York, United States Ichor Strategies Full time

    Job SummaryIchor Strategies seeks a Principal Data Engineer to lead the data architecture of our nascent SaaS platform which connects companies to the communities they operate in so that they can understand different perspectives, build authentic relationships, and foster mutual success. The ideal candidate will have extensive experience in driving...