Data Engineer

4 weeks ago

New York, United States Randstad Digital Full time

Required: Strong OOP Python skills for Data Engineering

Desirable: MongoDB and Redis


Everything else below is a bonus


W2 only - sponsorship available

No c2c

Job Description Summary:

Generative AI (GenAI) presents an exciting opportunity to derive valuable insights from data and drive revenue growth, efficiencies, and improved business processes. Technology will collaborate with Global Markets Sales & Trading, Quantitative Strategies & Data Group (QSDG) & Platform teams to the design and buildout its global GenAI platform.

The platform will cater to a rapidly growing number of use cases that harness the power of GenAI. Both proprietary and open-source Large Language Models, and large structured and un-structured data sets will be leveraged to produce insights for Global Markets and its clients.

  • We are seeking a Data Engineer to build out data pipelines to source large volumes of structured (ex: KDB) & unstructured data (ex: Research documents, Term Sheets), classify, and store data to meet GenAI requirements.
  • The Data Engineer will design, develop, and engineer platform for high performance and scalability.

Key Responsibilities:

  • Collaborate with data scientists and software engineers to understand data requirements.
  • Develop and maintain scalable data ingestion pipelines from various data sources, including large databases (Hadoop, etc.), web services, and APIs.
  • Analyze and organize raw data from different sources: internal or external.
  • Implement data cleaning, transformation, and normalization processes to ensure data quality and consistency.
  • Implement automated unit tests and conduct reviews with other team members to make sure your code is rigorously designed, elegantly coded, and effectively tuned for performance.
  • Automate reports and processes to run with varying frequencies.
  • Monitor and troubleshoot data pipelines to ensure data accuracy and pipeline performance.
  • Explore ways to enhance data quality, reliability, and efficiency of data pipelines.

Experience required:

MUST HAVE Technical:

  • Proficient in data engineering practices and using design and architectural patterns. o
  • At least 4+ years' experience as a Data Engineer or in a similar role, in Extract, Transform, Load (ETL) and/or Extract, Load, Transform (ELT) processes. o
  • Proficiency in working with unstructured and structured data. o
  • Experience with data processing frameworks such as Hadoop, Spark, or similar technologies. o
  • Experience with data platforms like SQL, HDFS, and NoSQL (MongoDB) databases. o
  • Extensive experience with object-oriented programming (OOP)/Scripting languages (Python preferred). o
  • Technical expertise with data models, data mining, and segmentation techniques. o
  • Experience working in multiple technology deployment lanes (development through production). o
  • DevOps processes and CICD tooling (Jira, Git/Bitbucket, Jenkins, Datical, Artifactory, Ansible), orchestration & automation.
  • Containerization technologies such as Docker and Kubernetes
  • Must have minimum 5 years similar experience

Non-Technical

  • Ability to communicate effectively to a wide range of audience (business stakeholders, developer & support teams). o
  • Detail oriented & highly organized. o
  • Adaptable to shifting & competing priorities. o
  • Problem solving skills to diagnose & resolve complex issues. o
  • Committed and pro-active in ensuring high quality of service

Experience desired:

  • Familiarity in AI & Deep learning, modeling techniques, Generative AI application stack. o
  • Familiarity with containerization and orchestration technologies such as Docker, Kubernetes and OpenShift. o
  • Experience with creating visualization dashboards (Tableau). o
  • Experience with vector databases such as Redis.

  • Data Engineer

    2 weeks ago

    New York, United States NTT DATA Full time

    Req ID: 270954 NTT DATA strives to hire exceptional, innovative and passionate individuals who want to grow with us. If you want to be part of an inclusive, adaptable, and forward-thinking organization, apply now.We are currently seeking a Data Engineer to join our team in New York, New York (US-NY), United States (US).Responsibilities:- Expand and optimize...

  • Data Engineer

    4 weeks ago

    New York, New York, United States NTT DATA Services Full time

    Req ID: NTT DATA strives to hire exceptional, innovative and passionate individuals who want to grow with us. If you want to be part of an inclusive, adaptable, and forward-thinking organization, apply now.We are currently seeking a Data Engineer to join our team in New York, New York (US-NY), United States (US). Responsibilities:- Expand and optimize the...

  • Data Engineer

    3 weeks ago

    New York, United States NTT DATA Services Full time

    Req ID: 270954  NTT DATA strives to hire exceptional, innovative and passionate individuals who want to grow with us. If you want to be part of an inclusive, adaptable, and forward-thinking organization, apply now. We are currently seeking a Data Engineer to join our team in New York, New York (US-NY), United States...

  • Data Engineer

    3 months ago

    New York, United States Silver Search, Inc Full time

    Data Engineer New York, NY Senior Data Engineer--Redshift/AWS/Python/Scripting---**Will be hybrid onsite in January 2022 onwards** Contract-to-hire Our client is a cutting edge E-Learning company based in NYC and is looking to hire a Senior Data Engineer to join their data and analytics center of excellence. He/She will be building...

  • Data Engineer

    1 week ago

    New York, United States Insight Global Full time

    Required Skills & Experience3+ years of experience as a Data EngineerExtensive experience writing code and doing data manipulation in PythonExperience developing data workflowsUnderstanding of data operations, including processing, storage, quality and managementExperience building ETL pipelines Nice to Have Skills & ExperienceTech or data centric industry...

  • Senior Engineer

    4 days ago

    New York, New York, United States Acquired Data Solutions Full time

    Job SummaryWe are seeking a highly skilled Senior Engineer to join our Transportation Security Laboratory (TSL) Team in Egg Harbor, NJ as Key Personnel to support Development Test & Evaluation (DT&E) of existing and emerging technologies for airport security equipment.Key ResponsibilitiesData Collection and AnalysisAnalysis of Vendor Data PackagesTest Plan...

  • Data Engineer

    1 day ago

    New York, New York, United States Capstone Investment Advisors Full time

    Job Summary:Capstone Investment Advisors is seeking a highly skilled Data Engineer to join our team. As a Data Engineer, you will be responsible for designing, developing, and deploying data ingestion and processing pipelines and tools utilizing Python, Linux, and the AWS Cloud platform.Key Responsibilities:Contribute to the development of a cutting-edge...

  • Data Engineers

    5 days ago

    New York, United States Zortech Solutions Full time

    Job DescriptionJob DescriptionRole : Data Engineers Location : NYC (Hyrbid) Job Description Deep hands-on knowledge in big data engineering in a cloud environment, including: Excellent knowledge of SQL in a large scale data warehouse or data Lakehouse environment such as Spark, Databricks, Presto/Athena/Trino Experience working in cloud environment,...

  • Data Engineer

    3 months ago

    New York, New York, United States gpac Full time

    Data Engineer - Fintech Firm in the Healthcare Industry - Seed Funded Startup (Very Well Funded) - On Site (Financial District NYC) A Seed Funded Startup that's looking to build out a Healthcare Platform that will revolutionize the way you interact with your healthcare provider are looking for their first Data Engineer Currently at a Seed Stage and a step...

  • Data Engineer

    4 weeks ago

    New York, United States gpac Full time

    Data Engineer - Fintech Firm in the Healthcare Industry - Seed Funded Startup (Very Well Funded) - On Site (Financial District NYC)   A Seed Funded Startup that's looking to build out a Healthcare Platform that will revolutionize the way you interact with your healthcare provider are looking for their first Data Engineer   Currently at a Seed Stage...

  • Data Engineer

    3 months ago

    New York, New York, United States TopEdge Technology Full time

    Job DescriptionPosition Data Engineer Duration 12 months Location New Jersey Remote Work Permitted Due to COVID-19 the client has agreed to allow the selected candidate to work remotely for the time being. However the selected candidate must be available to report onsite as directed by the client.Position Data EngineerJob DescriptionThe Senior Data Engineer...

  • Data Engineer

    1 month ago

    New York, New York, United States Phaxis, LLC Full time

    Salary is 120k to 140k bonus2 days remote and 3 days onsiteWe are seeking a highly skilled Data Engineer with expertise in financial services to join our team. The ideal candidate will have a strong background in designing and implementing data pipelines, creating visually compelling reports and dashboards using Power BI, and optimizing data storage and...

  • Senior Computer Systems Engineer

    2 weeks ago

    New York, New York, United States Data Intelligence Full time

    Company Overview:Data Intelligence, LLC (DI) is a reputable small business dedicated to supporting the vital missions of government clients since its inception in 2005. We specialize in full life cycle system development, systems engineering, cybersecurity, and comprehensive analytical and logistics support for C4ISR and other intricate systems.Position...

  • Data Engineer

    3 months ago

    New York, United States Capstone Investment Advisors Full time

    RESPONSIBILITIES: Be part of Data Engineering team responsible for all market data in the firmDesign and implement data-ingestion and data-processing pipelines and tools for a diverse range of datasets using Python, Linux and AWS Cloud platform.Work with variety of backend database storage systems including MS SQL Server, AWS Redshift, Cassandra,...

  • Data Engineer

    1 month ago

    New York, United States Russell Tobin & Associates Full time

    What are we looking for in our Data Engineer? Russell Tobin's client, a multinational mass media and business information company is urgently seeking a Data Engineer to work remotely Our client is hiring a Data Engineer to help us organize, standardize, and gain insightsfrom the tremendous amount of data generated from our community and...

  • Data Engineer

    2 weeks ago

    New York, United States Bond Sports Full time

    ResponsibilitiesData Warehouse Development: Collaborate with senior data engineers and data architects to design, develop, and maintain our Snowflake-based data warehouse infrastructure. This includes schema design, table creation, and optimizing data loading processes. ETL Development with dbt: Utilize dbt (Data Build Tool) to develop and maintain ETL...

  • Data Engineer

    1 month ago

    New York, United States Bond Sports Full time

    ResponsibilitiesData Warehouse Development: Collaborate with senior data engineers and data architects to design, develop, and maintain our Snowflake-based data warehouse infrastructure. This includes schema design, table creation, and optimizing data loading processes. ETL Development with dbt: Utilize dbt (Data Build Tool) to develop and maintain ETL...

  • Data Engineer

    4 weeks ago

    New York, United States Selby Jennings Full time

    Role: Data EngineerSelby Jennings is working with a leading hedge fund that currently has over $45 billion in AUM. They are dedicated to harnessing cutting-edge technology and data innovations to achieve superior investment returns. Their dynamic and highly skilled investment team is looking for an exceptional Data Engineer. This role offers a fast-paced and...

  • Data Engineer

    3 months ago

    New York, United States Jane Street Full time

    About the PositionWe are looking for a Data Engineer who can help us understand, clean, manage, and share the data that guides our trading. At Jane Street, having a thorough and accurate understanding of data is at the core of the work we do.Using our mix of in-house and open-source software, you will analyze datasets gathered from a variety of sources,...

  • Data Engineer

    1 month ago

    New York, New York, United States Kargo Full time

    Kargo creates breakthrough cross-screen ad experiences for the world's leading brands and publishers. Everyday, our 600+ employees bring the power of their creativity and diversity to radically raising the bar on what mobile, CTV, AI, social, and eCommerce can do to wow consumers and build businesses. Now 20 years strong, Kargo has offices in NYC, Chicago,...