Data Engineer

4 weeks ago

New York, United States Blackbird Full time

Blackbird.AI helps organizations discover emergent threats and stay one step ahead of real-world harm through our AI-powered Narrative and Risk Intelligence Platform. Our commitment is to prioritize safety and security, providing the tools to identify potential risks and ensure a safer environment proactively. No matter the job or where it’s located, we’re all connected by a shared vision: To lead and enhance the landscape of risk intelligence.

Reporting to the Head of Data Engineering, the Data Engineer will optimize data pipelines for our cutting-edge real-time streaming cloud-hosted analytics platform. You will be the key driver of our collection, analysis, and visualization processes and will play a pivotal role in shaping the future of our platform.

The Data Engineer will encompass writing ETL processes, designing database systems, and developing tools for query and analytic processing, all focusing on real-time streaming applications. Additionally, you will elevate performance tradeoffs, own the database architecture, and lead build automation, continuous integration, and deployment efforts while ensuring adherence to security requirements.

As the Data Engineer, you’ll have the chance to:

  • Write ETL processes to support the ingestion and normalization of various data formats from social media, news, and web sources.
  • Design robust database systems and developing tools for query and analytic processing, especially for real-time streaming applications.
  • Conduct performance analysis and empirical studies to evaluate tradeoffs (e.g., cost vs. throughput/latency).
  • Develop, manage, and own the database architecture for our real-time streaming cloud-hosted analytics platform.
  • Lead build automation, continuous integration, deployment, and performance optimization efforts while ensuring compliance with security requirements.
  • Lead design of test suites and inline instrumentation as needed to ensure data correctness

What you’ll bring:

  • Bachelor’s degree in Computer Science or related field.
  • Proven success in deploying products in the cloud and SaaS model, with expertise in building optimized processing pipelines for streaming analytics applications and cloud-agnostic solutions (e.g., Kubernetes, Docker).
  • Expertise in databases and query optimization, including PostgresSQL, ElasticSearch, MongoDB, Redis, and Druid, with additional experience in NoSQL and graph databases being advantageous.
  • Experience in horizontally scaling databases.
  • Proficiency in Kafka and Airflow, with a deep understanding of runtime profiling tools to optimize throughput and latency, and able to establish comparative performance benchmarks.
  • Strong skills in build automation, continuous integration, and deployment (CI/CD) tools like Webpack, Buddy, Jenkins, and Docker.
  • Expert-level Python coding skills.
  • Experience collaborating with distributed teams.

Helpful to have:

  • Technical background in Artificial Intelligence (AI) and Machine Learning (ML).
  • Experience designing and implementing interactive query-driven main-machine intelligence systems.
  • Solid skills in Java.

We’ve outlined specific skills, experience, and requirements for this position, but don’t stress if you don’t meet every single one. Our Talent Team is dedicated to discovering exceptional individuals, and they might identify a relevant aspect of your background that suits this role or another opportunity within Blackbird.AI.

If you have passion for the role, please still apply.

What’s in it for you:

Blackbird.AI is embarking on an exciting growth journey with numerous opportunities for career development within the company. You will join a nurturing, inclusive, and experienced team.

Join us as we soar to new heights

Values:

At Blackbird.AI, our core values shape how we work and make decisions. Our values inspire us to be authentic and continue improving.

We embrace a strong sense of responsibility to society, recognizing the vital role our services play in empowering governments, communities, and individuals to foster critical thinking and empowerment. We believe in integrating personal and professional lives with societal needs, emphasizing the importance of creating an environment that attracts top talent and provides substantial growth opportunities. We are motivated by the potential of science and technology to impact humanity positively.

  • Project Engineer

    5 days ago

    New York, United States Engineer Rec Full time

    Our client are seeking Project Engineers for multiple projects along the East Coast, USA. Are you a Project Engineer currently looking for a new challenge and experienced in supporting offshore wind projects? If so please read on. The Project Engineer has the skills and experience to take a leading role related to land and underground installation of high...

  • Senior Systems Engineer

    5 days ago

    New York, United States VAST Data Full time

    VAST Data is looking for a Senior Systems Engineer to join our growing team! This is a great opportunity to be part of one of the fastest-growing infrastructure companies in history, an organization that is in the center of the hurricane being created by the revolution in artificial intelligence. "VAST's data management vision is the future of the...

  • Data Engineer

    2 months ago

    New York, United States Silver Search, Inc Full time

    Data Engineer New York, NY Senior Data Engineer--Redshift/AWS/Python/Scripting---**Will be hybrid onsite in January 2022 onwards** Contract-to-hire Our client is a cutting edge E-Learning company based in NYC and is looking to hire a Senior Data Engineer to join their data and analytics center of excellence. He/She will be building...

  • Data Engineer

    5 days ago

    New York, New York, United States rag & bone Full time

    We are seeking a highly skilled and motivated Data Engineer to join our team. The ideal candidate will have strong proficiency in Python programming language, extensive experience with Oracle PL/SQL, and a solid understanding of Azure Serverless compute resources. As a Data Engineer, you will be responsible for designing, implementing, and maintaining...

  • Data Engineer

    4 days ago

    New York, United States JK Partners Full time

    Key ResponsibilitiesCloud-First Data Ingestion: Develop and implement data ingestion processes using Python, SQL, and Spark in a cloud-first environment.Data Modeling and Infrastructure Engineering: Construct robust data models and infrastructure to accommodate a diverse array of market and alternative data sets.Dataset Operations: Maintain and monitor...

  • Data Engineer

    1 month ago

    New York, New York, United States Masterworks Full time

    Masterworks Overview:Masterworks is the only platform for investing in blue-chip, multi-million dollar artworks. Prior to Masterworks, the only way to allocate to art was to purchase a multi-million dollar painting, making the asset class inaccessible for most investors. The company has over 900,000 users signed-up purchasing shares in primary offerings...

  • Data Engineer

    1 month ago

    New York, New York, United States TopEdge Technology Full time

    Job DescriptionPosition Data Engineer Duration 12 months Location New Jersey Remote Work Permitted Due to COVID-19 the client has agreed to allow the selected candidate to work remotely for the time being. However the selected candidate must be available to report onsite as directed by the client.Position Data EngineerJob DescriptionThe Senior Data Engineer...

  • Data Engineer

    1 month ago

    New York, United States Capstone Investment Advisors Full time

    RESPONSIBILITIES: Be part of Data Engineering team responsible for all market data in the firmDesign and implement data-ingestion and data-processing pipelines and tools for a diverse range of datasets using Python, Linux and AWS Cloud platform.Work with variety of backend database storage systems including MS SQL Server, AWS Redshift, Cassandra,...

  • Data Engineer

    1 week ago

    New York, United States Jane Street Full time

    About the PositionWe are looking for a Data Engineer who can help us understand, clean, manage, and share the data that guides our trading. At Jane Street, having a thorough and accurate understanding of data is at the core of the work we do.Using our mix of in-house and open-source software, you will analyze datasets gathered from a variety of sources,...

  • Manufacturing Engineer

    2 weeks ago

    New York, United States Onset Data Loggers Full time

    Based in Bourne, Massachusetts, for more than 40 years Onset has been dedicated to designing and producing high-quality data loggers. Our manufacturing processes ensure precision and reliability in data logging technology. Onset's products are trusted worldwide for capturing critical data, making them an essential choice for professionals and organizations...

  • Data Engineer

    1 month ago

    New York, United States Open Systems Technologies Full time

    A financial firm is looking for a Data Engineer to join their team in New York, NY.Compensation: $150-200k Responsibilities:Build data pipelinesWork with data vendors such as Bloomberg, Refinitiv, etc.Take vendor data and normalize it for firm consumptionQualifications:A Bachelor's Degree in Computer Science or Computer Engineering4+ years of experience in...

  • Data Engineer

    1 month ago

    New York, United States Open Systems Technologies Full time

    A financial firm is looking for a Data Engineer to join their team in New York, NY.Compensation: $150-200k Responsibilities:Build data pipelinesWork with data vendors such as Bloomberg, Refinitiv, etc.Take vendor data and normalize it for firm consumptionQualifications:A Bachelor's Degree in Computer Science or Computer Engineering4+ years of experience in...

  • Data Engineer

    2 weeks ago

    New York, United States Open Systems Technologies Full time

    An international law firm is looking for a Data Engineer to join their team in NYC. Compensation: $115-160kThe Data Engineer is responsible for expanding and optimizing our data and data pipeline architecture, as well as optimizing data flow and collection. The Data Engineer is an experienced data pipeline builder who enjoys optimizing data systems and...

  • Data Engineer

    2 weeks ago

    New York, United States Open Systems Technologies Full time

    An international law firm is looking for a Data Engineer to join their team in NYC. Compensation: $115-160kThe Data Engineer is responsible for expanding and optimizing our data and data pipeline architecture, as well as optimizing data flow and collection. The Data Engineer is an experienced data pipeline builder who enjoys optimizing data systems and...

  • Data Engineer

    5 days ago

    New York, New York, United States Kustomer Full time

    About KustomerKustomer is the industry leading conversational CRM platform perfecting every customer experience. Built with intelligent tools such as AI and Automation, no code-configuration and a connected data platform that unifies data from multiple sources through a single timeline, Kustomer empowers businesses to operate with greater efficiency and...

  • Data Engineer

    1 month ago

    New York, United States Jane Street Full time

    About the PositionWe are looking for a Data Engineer who can help us understand, clean, manage, and share the data that guides our trading. At Jane Street, having a thorough and accurate understanding of data is at the core of the work we do.Using our mix of in-house and open-source software, you will analyze datasets gathered from a variety of sources,...

  • Senior Data Engineer

    3 days ago

    New York, United States Bayforce Full time

    Senior Data Engineer The Senior Data Engineer will be responsible for accelerating and solidifying the Data Engineering team’s development, deployment, and operational support practices by building tools leveraged by the entire Data Engineering team. In addition, the Senior Data Engineer will build new data pipelines and enhance and troubleshoot existing...

  • Senior Data Engineer

    1 month ago

    New York, United States Bayforce Full time

    The Senior Data Engineer will be responsible for accelerating and solidifying the Data Engineering team’s development, deployment, and operational support practices by building tools leveraged by the entire Data Engineering team. In addition, the Senior Data Engineer will build new data pipelines and enhance and troubleshoot existing ones.Requirements •...

  • Senior Data Engineer

    1 month ago

    New York, United States Bayforce Full time

    The Senior Data Engineer will be responsible for accelerating and solidifying the Data Engineering team’s development, deployment, and operational support practices by building tools leveraged by the entire Data Engineering team. In addition, the Senior Data Engineer will build new data pipelines and enhance and troubleshoot existing ones.Requirements •...

  • Sr Data Engineer

    1 month ago

    New York, United States Ticketing technology company Full time

    Role Description: The Senior Data Engineer will be responsible for accelerating and solidifying the Data Engineering team's development, deployment, and operational support practices by building tools leveraged by the entire Data Engineering team. In addition, the Senior Data Engineer will build new data pipelines and enhance and troubleshoot existing...