Big data engineer

6 days ago


Chicago, United States Collabera Full time

Position Details:

Title: Big Data Engineer

Location: Chicago, IL/ Denver, CO- Hybrid - 3x/week onsite

Compensation: $65-$68.26/hour on W2

Interview Process: One Inperson round


Why open: New project


Scope of role: Experience with Kafka, Pyspark and real-time data processing, Elastic Search, Kibana (nice to have)


Project: Dealing with streaming data


Expected to accomplish:

  • Processing Data in Real-time
  • Building a system from scratch
  • Designing abstract layers


Must have:

  • Strong communication skills
  • Kafka
  • Pyspark
  • Real time data processing
  • SQL
  • Spark
  • Coding/ development in Python


Nice to have:

  • Elastic Search
  • Kibana
  • Designing system from scratch/ designing abstract layers


Required Skills:

  • Strong SQL Skills - one or more of MySQL, HIVE, Impala, SPARK SQL
  • Data ingestion experience from message queue, file share, REST API, relational database, etc. and experience with data formats like json, csv, xml
  • Experience working with SPARK Structured steaming
  • Experience working with Hadoop/Big Data and Distributed Systems
  • Working experience with Spark, Sqoop, Kafka, MapReduce, NoSQL Database like HBase, SOLR, CDP or HDP, Cloudera or Hortonworks, Elastic Search, Kibana, etc.
  • Hands on programming experience in at least one of Scala, Python, PHP, or Shell Scripting, to name a few
  • Performance tuning experience with spark /MapReduce or SQL jobs
  • Experience and proficiency with Linux operating system is a must
  • Experience in end-to-end design and build process of Near-Real Time and Batch Data Pipelines
  • Experience working in Agile development process and deep understanding of various phases of the Software Development Life Cycle
  • Experience using Source Code and Version Control systems like SVN, Git, Bit Bucket etc.
  • Experience working with Jenkins and Jar management
  • Self-starter who works with minimal supervision and the ability to work in a team of diverse skill sets
  • Ability to comprehend customer requests and provide the correct solution
  • Strong analytical mind to help take on complicated problems
  • Desire to resolve issues and dive into potential issues
  • Ability to adapt and continue to learn new technologies is important

  • Big Data Engineer

    3 days ago


    Chicago, Illinois, United States Collabera Full time

    Job DescriptionJob Title: Big Data EngineerCompany: CollaberaLocation: Remote/HybridJob Type: Full-timeJob Category: EngineeringAbout the Role:We are seeking a highly skilled Big Data Engineer to join our team at Collabera. As a Big Data Engineer, you will be responsible for designing, developing, and maintaining large-scale data processing systems.Key...

  • Big Data Engineer

    3 days ago


    Chicago, Illinois, United States Collabera Full time

    About the RoleWe are seeking a highly skilled Big Data Engineer to join our team at Collabera. As a key member of our data engineering team, you will be responsible for designing, building, and maintaining large-scale data processing systems.Key ResponsibilitiesData Ingestion and Processing: Design and implement data ingestion pipelines to collect and...

  • Big Data Engineer

    4 weeks ago


    Chicago, United States Tiger Analytics Full time

    Job DescriptionJob DescriptionTiger Analytics is a fast-growing advanced analytics consulting firm. Our consultants bring deep expertise in Data Science, Machine Learning and AI. We are the trusted analytics partner for multiple Fortune 500 companies, enabling them to generate business value from data. Our business value and leadership has been recognized by...

  • Big Data Engineer

    3 months ago


    Chicago, United States Tiger Analytics Full time

    Job DescriptionJob DescriptionTiger Analytics is a fast-growing advanced analytics consulting firm. Our consultants bring deep expertise in Data Science, Machine Learning and AI. We are the trusted analytics partner for multiple Fortune 500 companies, enabling them to generate business value from data. Our business value and leadership has been recognized by...

  • Big Data Architect

    7 days ago


    Chicago, Illinois, United States Tiger Analytics Full time

    Job OverviewTiger Analytics, a leading advanced analytics consulting firm, is seeking a highly skilled Big Data Engineer to join our team. Our company provides expert analytics services to Fortune 500 companies, helping them unlock business value from data.The ideal candidate will have a strong background in traditional and modern Big Data technologies,...

  • Sr. Manager, AI Cloud

    3 months ago


    Chicago, United States Abbott Laboratories Full time

    Abbott is a global healthcare leader that helps people live more fully at all stages of life. Our portfolio of life-changing technologies spans the spectrum of healthcare, with leading businesses and products in diagnostics, medical devices, nutritionals and branded generic medicines. Our 114,000 colleagues serve people in more than 160 countries.Working at...

  • Big Data Engineer

    7 days ago


    Chicago, Illinois, United States RSN GINFO SOLUTIONS LLC Full time

    Job Summary:We are seeking a highly skilled Senior Data Engineer to join our team at RSN GINFO SOLUTIONS LLC. As a Senior Data Engineer, you will be responsible for designing, developing, and maintaining large-scale data processing systems using PySpark, Spark, and SQL.Key Responsibilities:Design and implement data pipelines using PySpark and SparkDevelop...


  • Chicago, United States CME Group Full time

    Description Position Overview: Data System Reliability Engineer (dSRE) CME Group: Where Futures Are Made CME Group is the world's leading and most diverse derivatives marketplace. But who we are goes deeper than that, here you can impact markets worldwide, transform industries and build a career shaping tomorrow. We invest in your success and you own it, all...

  • Sr Data Engineer

    2 weeks ago


    Chicago, United States Abbott Laboratories Full time

    At Abbott, you can do work that matters, grow, and learn, care for yourself and family, be your true self and live a full life. You’ll also have access to:Career development with an international company where you can grow the career you dream of .Free medical coverage for employees* via the Health Investment Plan (HIP) PPOAn excellent retirement...


  • Chicago, United States CME Group Full time

    Position Overview: Data System Reliability Engineer (dSRE) CME Group: Where Futures Are Made CME Group is the world's leading and most diverse derivatives marketplace. Here you can impact markets worldwide, transform industries and build a career shaping tomorrow. We invest in your success and you own it, all while working alongside a team of leading experts...

  • Data Engineer

    3 weeks ago


    Chicago, United States McDonald's Full time

    McDonald’s is proud to be one of the most recognized brands in the world, with restaurants in over 100 countries that serve 70 million customers daily. We continue to operate from a position of strength. Our updated growth strategy is focused on staying ahead of what our customers want and realizing further growth potential. Our relentless ambition is why...

  • Data Engineer

    3 weeks ago


    Chicago, United States McDonald's Full time

    McDonald’s is proud to be one of the most recognized brands in the world, with restaurants in over 100 countries that serve 70 million customers daily. We continue to operate from a position of strength. Our updated growth strategy is focused on staying ahead of what our customers want and realizing further growth potential. Our relentless ambition is why...


  • Chicago, Illinois, United States Jobot Full time

    About the RoleWe are seeking a highly skilled Data Pipeline Engineer to join our team at Jobot. As a key member of our data infrastructure team, you will be responsible for designing, building, and deploying ETL and ELT flows for consuming data from disparate sources.Key ResponsibilitiesCreate and deploy ETL and ELT flows for consuming data from various...

  • Senior Data Engineer

    1 month ago


    Chicago, United States Capital One Full time

    77 West Wacker Dr (35012), United States of America, Chicago, IllinoisSenior Data Engineer Do you love building and pioneering in the technology space? Do you enjoy solving complex business problems in a fast-paced, collaborative, inclusive, and iterative delivery environment? At Capital One, you'll be part of a big group of makers, breakers, doers and...

  • Data Engineer

    3 days ago


    Chicago, Illinois, United States McDonald's Full time

    About the RoleWe are seeking a highly skilled Data Engineer to join our Global Technology – Data & Analytics team at McDonald's. As a Data Engineer, you will play a critical role in building scalable and efficient data solutions to support our company's data products and analytics initiatives.Key ResponsibilitiesDesign and Develop Data Pipelines: Develop...

  • Senior Data Engineer

    3 weeks ago


    Chicago, United States Capital One Full time

    Center 1 (19052), United States of America, McLean, VirginiaSenior Data Engineer Do you love building and pioneering in the technology space? Do you enjoy solving complex business problems in a fast-paced, collaborative,inclusive, and iterative delivery environment? At Capital One, you'll be part of a big group of makers, breakers, doers and disruptors, who...

  • AI Data Engineer

    3 weeks ago


    Chicago, United States Motion Recruitment Full time

    Chicago, IllinoisOpen to RemoteFull Time$165k - $185k AI Data Engineer As a leading industrial supply distributer dedicated to making businesses run smoothly, this company is searching for a full-time AI Data Engineer at their Chicago office. Offering over a million products and serving over 4.5 million customers, this company is well above the competition...

  • Data Engineer

    2 months ago


    Chicago, Illinois, United States United Airlines Full time

    There's never been a more exciting time to join United Airlines. We're on a path towards becoming the best airline in the history of aviation. Our shared purpose – Connecting People, Uniting the World – is about more than getting people from one place to another. It also means that as a global company that operates in hundreds of locations around the...


  • Chicago, United States Algo Capital Group Full time

    Quantitative Software Engineer - Data EngineeringOur client is a leading proprietary trading firm headquartered in Chicago seeking a Quantitative Software Engineer to join their Front Office Data Engineering team. The firm's team integrates innovative technology and trading strategies while utilizing a sophisticated research platform and development...


  • Chicago, United States Algo Capital Group Full time

    Quantitative Software Engineer - Data EngineeringOur client is a leading proprietary trading firm headquartered in Chicago seeking a Quantitative Software Engineer to join their Front Office Data Engineering team. The firm's team integrates innovative technology and trading strategies while utilizing a sophisticated research platform and development...