Sr. Data Pipeline Engineer

1 week ago


New York, United States NR Consulting Full time

BS or MS in Computer Science, a related field, or equivalent industry experience
. 3 years of professional experience engineering complex, high-volume data pipelines using SQL, Python, and Airflow
. 3 years of experience building cloud scalable and high-performance data lake / data warehouse solutions using AWS products - S3, Athena, Glue, and EMR
. Experience with binary data serialization formats such as Parquet
. Deep understanding of data structures and algorithms
. Understanding of code versioning tools such as GIT
. Have a passion for data solutions
Preferred Qualifications Nice to Have:
. Exposure to AWS cloud data pipeline tools such as Managed Airflow and Glue
. Experience integrating with Ad Tech platforms such as Operative and STAQ
. Exposure and opinions regarding alternate orchestration tooling beyond Airflow
. Understanding of SOX compliance needs and how they affect system design.
. Have worked with a variety of Airflow Operator types, including REST, Lambda, ECS
. Can flex between Python and Javascript/Typescript.

Technical Environment
. Aurora/Hive (databases)
. Spark (large-scale data processing)
. Airflow (workflow management)
. Docker (software packaging and delivery)
. AWS (development and hosting) Required Education BS/MS in Computer Science or sim. field Preferred Education Additional Information - This team is considered Hybrid so looking for someone that's close to a Hulu Office. So.Cal/Seattle/NYC - Can be remote to start but expectation is in office part time if needed.


  • Sr. Data Engineer

    2 weeks ago


    New York, United States Planet Technology USA Full time

    Sr. Data Engineer- SQL Server Expert with Investment experience Fulltime Job in NY - Hybrid-Please do not apply if not local to NY No C2C - Must be able to work without sponsorship Position: Sr Data Engineer - Strong SQL Application Development You will be working with a skilled team and help in defining the future of their data platform environment. You...

  • Sr. Data Engineer

    3 weeks ago


    New York, United States Planet Technology Full time

    Sr. Data Engineer- SQL Server Expert with Investment experienceFulltime Job in NY - Hybrid-Please do not apply if not local to NYNo C2C - Must be able to work without sponsorshipPosition: Sr Data Engineer - Strong SQL Application DevelopmentYou will be working with a skilled team and help in defining the future of their data platform environment. You will be...

  • Sr. Data Engineer

    3 days ago


    New York, United States Planet Technology Full time

    Sr. Data Engineer- SQL Server Expert with Investment experienceFulltime Job in NY - Hybrid-Please do not apply if not local to NYNo C2C - Must be able to work without sponsorshipPosition: Sr Data Engineer - Strong SQL Application DevelopmentYou will be working with a skilled team and help in defining the future of their data platform environment. You will be...

  • Sr. Data Engineer

    2 weeks ago


    New York, United States Planet Technology Full time

    Sr. Data Engineer- SQL Server Expert with Investment experienceFulltime Job in NY - Hybrid-Please do not apply if not local to NYNo C2C - Must be able to work without sponsorshipPosition: Sr Data Engineer - Strong SQL Application DevelopmentYou will be working with a skilled team and help in defining the future of their data platform environment. You will be...

  • Sr. Data Engineer

    3 weeks ago


    New York, United States Dexian Full time

    Job Summary:Dexian is seeking a Sr. Data Engineer for an opportunity with a client located in New York, NY. Responsibilities:Implement new mechanisms for modeling and predicting reader behaviorDesign and implement experiments and models to answer targeted questionsLead technical efforts to modernize and replace legacy reporting solutionsLead daily...


  • New York, United States Ampcus Full time

    Role: Sr. Data Platform Engineer Location: New York, NY JOB DESCRIPTION: "Responsibilities: Design and implement Azure cloud-based Data Warehousing and Governance architecture with Lakehouse paradigm Integrating technical functionality, ensuring data accessibility, accuracy, and security. Architect the Unity Catalog to provide centralized access control,...


  • New York, United States Optomi Full time

    Sr. Data Engineer - (Scala / Spark)100% Remote, must sit in hub area - New York, Orlando, San Jose, Seattle, Santa MonicaW2 Only - USC or GC Holder - Due to the nature of this opportunity, please do not apply for this position if you need to do so on a C2C basis or through a 3rd party.4 month contract, will likely extend past the fiscal yearOptomi, in...

  • Sr. Data Engineer

    3 weeks ago


    New York, New York, United States HealthFirst Full time

    Hello Thank you for considering Healthfirst for your next career opportunity. Who we are: Healthfirst was founded by a collection of major healthcare systems in the New York area to emphasize high quality of care, and better patient outcomes. We have grown into New York's largest not-for-profit health insurer, and cover over 1.7 million members in New York...

  • Sr. Data Engineer

    3 weeks ago


    New York, New York, United States HealthFirst Full time

    Hello Thank you for considering Healthfirst for your next career opportunity. Who we are: Healthfirst was founded by a collection of major healthcare systems in the New York area to emphasize high quality of care, and better patient outcomes. We have grown into New York's largest not-for-profit health insurer, and cover over 1.7 million members in New York...

  • Sr Data Engineer

    4 weeks ago


    New York, United States enexusglobal Full time

    Job DescriptionJob DescriptionWe are looking to hire a Sr. Data Engineer in New York City, NY Summary:We are looking to hire Sr. Data Engineer with a financial background. Who will play a crucial role in a cloud-based data warehousing platform, to manage and analyze financial data.Job ResponsibilitiesDesign and develop robust, scalable software solutions for...


  • New York, United States Sun Pharma Full time

    As a member of our Software Engineering Group, we look first and foremost for people passionate about solving business problems through innovation and engineering practices. You'll be required to apply your depth of knowledge and expertise to all aspects of the software development lifecycle and partner continuously with your many stakeholders daily to stay...

  • Sr. Data Architect

    2 days ago


    New York, New York, United States TWO95 International Full time

    Hi,Title – Sr.Data ArchitectPosition – Contract to Hire (3+ Months)Location – RemoteRate - $Open (Best Possible)ENVIRONMENTNeo4j, Gremlin , Java, Node.JS, AWS Lambda, AWS Kinesis, AWS lifecycle management with GitLab, Kubernetes, Other AWS servicesREQUIREMENTS7+ years of experience with Data Engineering/Architecture5+ years of experience engineering...


  • New York, United States MigrationIT Full time

    Requirements Cloud computing platforms: AWS/Azure/GCP Programming languages: Python/Java/Scala & Unix Shell scripting Data warehouse technologies: Redshift/Snowflake/Big Query Data lake technologies: Hadoop, Apache Beam, Spark, and Hive Orchestration: Airflow/Control-M/(Any Orchestration tool) Machine learning frameworks: TensorFlow, PyTorch, and...

  • Sr. Data Engineer

    3 weeks ago


    New Bern, North Carolina, United States Core Scientific Full time

    Who We Are ​​Bold. Unapologetic. Hardworking. We are building something special. Put simply: We power and secure the next global financial system. At Core Scientific, we host best-in-class enterprise level blockchain hardware in expansive, state-of-the-art data center facilities. Our proprietary software stack automates blockchain mining optimization,...

  • Sr. Data Engineer

    1 month ago


    New Bern, North Carolina, United States Core Scientific Full time

    Who We Are ​​Bold. Unapologetic. Hardworking. We are building something special. Put simply: We power and secure the next global financial system. At Core Scientific, we host best-in-class enterprise level blockchain hardware in expansive, state-of-the-art data center facilities. Our proprietary software stack automates blockchain mining optimization,...

  • Sr. Azure Engineer

    1 month ago


    New York, United States Creative Data Resources Full time

    Required 8+ years overall IT experience. 4+ years in a cloud engineering role using service and hosting solutions such as private/public cloud IaaS, PaaS and SaaS platforms. SME Level knowledge of: ASP.NET ,C#,XML,YAML,SQL,CSS, JavaScript Angular UI development experience Azure SQL Server Stored Procedure Development (TSQL) Azure Services Knowledge Azure...

  • Sr. Azure Engineer

    3 weeks ago


    New York, United States Creative Data Resources Full time

    Required 8+ years overall IT experience. 4+ years in a cloud engineering role using service and hosting solutions such as private/public cloud IaaS, PaaS and SaaS platforms. SME Level knowledge of: ASP.NET ,C#,XML,YAML,SQL,CSS, JavaScript Angular UI development experience Azure SQL Server Stored Procedure Development (TSQL) Azure Services Knowledge Azure...

  • Sr. Azure Engineer

    2 days ago


    New York, New York, United States Creative Data Resources Full time

    Required 8+ years overall IT experience. 4+ years in a cloud engineering role using service and hosting solutions such as private/public cloud IaaS, PaaS and SaaS platforms. SME Level knowledge of: ASP.NET ,C#,XML,YAML,SQL,CSS, JavaScript Angular UI development experience Azure SQL Server Stored Procedure Development (TSQL) Azure Services Knowledge Azure...

  • Sr. DevOps Engineer

    5 days ago


    New York, United States Diverse Lynx Full time

    Title: Sr. Devops Engineer (Cloud Implementation with GCP) (2 Roles) Location: Berkley Heights, NJ / Alpharetta, GA (5 Days Onsite) Job Description: 6-10 years of overall experience 3+ years of Devops experience on GCP Strong cloud engineering skills, experienced and certified on GCP Strong understanding of cloud security principles, architecture, and best...


  • New York, New York, United States Federal Reserve System Full time

    Come Work with Us at the Federal Reserve Bank of New YorkWorking at the Federal Reserve Bank of New York puts you right in the middle of all things finance, giving you a special view of both national and international markets and economies. You'll be part of a diverse team of seasoned professionals dedicated to the strength and prosperity of our economic and...