Big Data Engineer

6 days ago


Chicago, Illinois, United States Collabera Full time
Job Title: Big Data Engineer

We are seeking a highly skilled Big Data Engineer to join our team at Collabera. As a Big Data Engineer, you will be responsible for designing, building, and maintaining large-scale data processing systems.

Key Responsibilities:
  • Design and implement data pipelines using Kafka, Pyspark, and real-time data processing techniques
  • Develop and maintain large-scale data processing systems using Elastic Search and Kibana
  • Work with cross-functional teams to design and implement data ingestion solutions
  • Collaborate with data scientists to develop and deploy machine learning models
  • Ensure data quality, security, and compliance with company policies
Requirements:
  • Strong communication and collaboration skills
  • Experience with Kafka, Pyspark, and real-time data processing
  • Proficiency in SQL and Spark
  • Experience with data ingestion from message queues, file shares, REST APIs, and relational databases
  • Experience with data formats like JSON, CSV, and XML
  • Experience with Spark Structured Streaming and Hadoop/Big Data and Distributed Systems
Preferred Skills:
  • Experience with Elastic Search and Kibana
  • Experience with designing system from scratch and designing abstract layers
  • Experience with Linux operating system
  • Experience with Agile development process and Software Development Life Cycle
  • Experience with Source Code and Version Control systems like SVN, Git, and Bit Bucket
What We Offer:
  • Competitive salary and benefits package
  • Opportunities for professional growth and development
  • Collaborative and dynamic work environment

  • Big Data Engineer

    2 weeks ago


    Chicago, Illinois, United States Collabera Full time

    Job DescriptionJob Title: Big Data EngineerCompany: CollaberaLocation: Remote/HybridJob Type: Full-timeJob Category: EngineeringAbout the Role:We are seeking a highly skilled Big Data Engineer to join our team at Collabera. As a Big Data Engineer, you will be responsible for designing, developing, and maintaining large-scale data processing systems.Key...

  • Big Data Engineer

    2 weeks ago


    Chicago, Illinois, United States Collabera Full time

    About the RoleWe are seeking a highly skilled Big Data Engineer to join our team at Collabera. As a key member of our data engineering team, you will be responsible for designing, building, and maintaining large-scale data processing systems.Key ResponsibilitiesData Ingestion and Processing: Design and implement data ingestion pipelines to collect and...

  • Big Data Architect

    3 weeks ago


    Chicago, Illinois, United States Tiger Analytics Full time

    Job OverviewTiger Analytics, a leading advanced analytics consulting firm, is seeking a highly skilled Big Data Engineer to join our team. Our company provides expert analytics services to Fortune 500 companies, helping them unlock business value from data.The ideal candidate will have a strong background in traditional and modern Big Data technologies,...

  • Big Data Engineer

    3 weeks ago


    Chicago, Illinois, United States RSN GINFO SOLUTIONS LLC Full time

    Job Summary:We are seeking a highly skilled Senior Data Engineer to join our team at RSN GINFO SOLUTIONS LLC. As a Senior Data Engineer, you will be responsible for designing, developing, and maintaining large-scale data processing systems using PySpark, Spark, and SQL.Key Responsibilities:Design and implement data pipelines using PySpark and SparkDevelop...

  • Big Data Developer

    2 weeks ago


    Chicago, Illinois, United States Collabera Full time

    About the RoleWe are seeking a highly skilled Big Data Developer to join our team at Collabera. As a Big Data Developer, you will play a key role in designing and developing large-scale data processing systems.Key ResponsibilitiesDesign and develop data processing systems using Python and PysparkWork on individual modules and come up with design for a...

  • Lead Data Engineer

    6 hours ago


    Chicago, Illinois, United States Epsilon Full time

    Job Title: Lead Data EngineerWe are seeking a highly skilled Lead Data Engineer to join our team at Epsilon. As a key member of our engineering team, you will be responsible for designing, developing, and maintaining scalable, cloud-based software solutions.Key Responsibilities:Technical Expertise:Design and develop scalable, cloud-based software solutions...


  • Chicago, Illinois, United States Epsilon Data Interactive, Inc. Full time

    About the RoleWe are seeking a highly skilled Senior Software Engineer to join our team at Epsilon Data Interactive, Inc. as an AI/ML Expert. In this role, you will collaborate with decision scientists and cross-functional teams to develop innovative AI and machine learning applications using Spark and Scala.Key ResponsibilitiesCollaborate with decision...


  • Chicago, Illinois, United States Jobot Full time

    About the RoleWe are seeking a highly skilled Data Pipeline Engineer to join our team at Jobot. As a key member of our data infrastructure team, you will be responsible for designing, building, and deploying ETL and ELT flows for consuming data from disparate sources.Key ResponsibilitiesCreate and deploy ETL and ELT flows for consuming data from various...

  • Data Engineer

    1 week ago


    Chicago, Illinois, United States Capital One Full time

    About the RoleCapital One is seeking a highly skilled Data Engineer to join our team. As a Data Engineer, you will play a critical role in designing, developing, and implementing technical solutions that drive business value.Key ResponsibilitiesCollaborate with cross-functional teams to design and develop scalable data architecturesDevelop and maintain...

  • Data Engineer

    2 weeks ago


    Chicago, Illinois, United States McDonald's Full time

    About the RoleWe are seeking a highly skilled Data Engineer to join our Global Technology – Data & Analytics team at McDonald's. As a Data Engineer, you will play a critical role in building scalable and efficient data solutions to support our company's data products and analytics initiatives.Key ResponsibilitiesDesign and Develop Data Pipelines: Develop...


  • Chicago, Illinois, United States Magicforce Full time

    About This RoleWe are seeking a highly skilled Azure Data Engineer with expertise in Medallion Architecture to join our team at Magicforce.Key ResponsibilitiesData Platform Architecture: Design and implement large-scale data and analytics platforms on Azure, leveraging Medallion Architecture to drive business growth.Data Integration and Governance: Develop...

  • Data Engineer

    3 weeks ago


    Chicago, Illinois, United States United Airlines Full time

    Job SummaryWe are seeking a highly skilled Data Engineer to join our team at United Airlines. As a Data Engineer, you will be responsible for designing and building scalable and reliable data infrastructure and pipelines to support data for measurement and reporting across our digital platforms.Key ResponsibilitiesDesign and build data infrastructure and...

  • AWS Data Engineer

    6 hours ago


    Chicago, Illinois, United States Epsilon Full time

    Job Title: Sr. AWS Data EngineerWe are seeking a highly skilled Sr. AWS Data Engineer to join our team at Epsilon. As a key member of our interdisciplinary team, you will be responsible for designing, developing, and implementing cloud-based data solutions that drive business growth and customer engagement.Job Responsibilities:80% of time spent on actual...


  • Chicago, Illinois, United States CME Group Full time

    Job OverviewA critical role in CME Group's Cloud data transformation, the Data SRE will be aligned to data product pods ensuring the firm's data infrastructure is reliable, scalable, and efficient as the GCP data footprint expands rapidly.Key ResponsibilitiesOptimize data pipelines to ensure efficient data processing and reduce latency.Ensure data integrity...

  • Lead Data Engineer

    3 weeks ago


    Chicago, Illinois, United States Foursquare Full time

    About FoursquareFoursquare is a leading independent location technology and data cloud platform, dedicated to building meaningful bridges between digital spaces and physical places. Our proprietary technology unlocks the most accurate, trustworthy location data in the world, empowering businesses to answer key questions, uncover hidden insights, improve...

  • Senior Data Engineer

    3 weeks ago


    Chicago, Illinois, United States United Airlines Full time

    Job SummaryWe are seeking a highly skilled Data Engineer to join our team at United Airlines. As a Data Engineer, you will play a critical role in designing and building scalable and reliable data infrastructure and pipelines to support data for measurement and reporting across our digital platforms.Key ResponsibilitiesDesign and build scalable and reliable...


  • Chicago, Illinois, United States Epsilon Data Interactive, Inc. Full time

    About the RoleWe are seeking a highly skilled Senior Software Engineer to join our team at Epsilon Data Interactive, Inc. as an AI/ML Expert. In this role, you will collaborate with decision scientists and cross-functional teams to develop innovative AI and machine learning applications using Spark and Scala.Key Responsibilities:Collaborate with decision...

  • Data Scientist

    4 days ago


    Chicago, Illinois, United States Lorven technologies Full time

    Job DescriptionPosition: Data EngineerLocation: Chicago, ILDuration: ContractSkills: ETL concepts, GCP and its services, Big Query, Python, AirflowJob Requirements12+ Years of Overall IT ExperienceDesign and development of data ingestion and transformation processesProficiency in Python, BigQuery, DataProc, Cloud Storage and other GCP ServicesData Migration...


  • Chicago, Illinois, United States minden Full time

    About UsMinden is a technology venture that redefines the engagement between brands and consumers in Southeast Asia. We are a strategic partnership between Temasek, DFI Retail Group, and coalition partners.Our CultureWe value diversity, inclusivity, and collaboration. Our team is built on respect, authenticity, and transparency. We prioritize agility,...

  • Data Engineer

    2 days ago


    Chicago, Illinois, United States WHIZTEK CORP Full time

    Job Title: Data EngineerWe are seeking a highly skilled Data Engineer to join our team at Whiztek Corp. As a Data Engineer, you will play a critical role in designing, building, and maintaining large-scale data pipelines and architectures.Key Responsibilities:Design and implement data pipelines using AWS services such as Redshift, S3, and AthenaDevelop and...