Big Data Engineer

3 days ago


Atlanta, Georgia, United States Open Systems Inc. Full time

Job Title: Big Data Engineer

Company: Open Systems Inc.

Job Summary:

We are seeking an experienced Big Data Engineer to join our team at Open Systems Inc. in Atlanta, GA. As a Big Data Engineer, you will be responsible for designing, developing, and maintaining large-scale data processing systems.

Key Responsibilities:

  • Design and implement data ingestion pipelines using Apache Kafka and Spark Streaming
  • Develop and maintain real-time data processing and streaming techniques using Spark structured streaming and Kafka
  • Troubleshoot and tune Spark applications to ensure optimal performance
  • Design and implement data architectures using Hadoop, Spark, Spark SQL, Kafka, Sqoop, Hive, S3, HDFS, or
  • Build, test, and optimize 'Big Data' data ingestion pipelines, architectures, and data sets
  • Develop and maintain data platforms on Azure/ AWS using cloud platforms e.g. AWS, GCP, etc.
  • Design and implement database solutions like Kudu/Impala, or Delta Lake or Snowflake or BigQuery
  • Develop and maintain NoSQL databases, including HBASE and/or Cassandra
  • Experience in successfully building and deploying a new data platform on Azure/ AWS
  • Experience in Azure / AWS Serverless technologies, like, S3, Kinesis/MSK, lambda, and Glue
  • Strong knowledge of Messaging Platforms like Kafka, Amazon MSK & TIBCO EMS or IBM MQ Series
  • Experience with Databricks UI, Managing Databricks Notebooks, Delta Lake with Python, Delta Lake with Spark SQL, Delta Live Tables, Unity Catalog
  • Knowledge of Unix/Linux platform and shell scripting is a must
  • Strong analytical and problem-solving skills

Requirements:

  • 6+ years of overall IT experience
  • 3+ years of experience with high-velocity high-volume stream processing: Apache Kafka and Spark Streaming
  • Experience with real-time data processing and streaming techniques using Spark structured streaming and Kafka
  • Deep knowledge of troubleshooting and tuning Spark applications
  • 3+ years of experience with data ingestion from Message Queues (Tibco, IBM, etc.) and different file formats across different platforms like JSON, XML, CSV
  • 3+ years of experience with Big Data tools/technologies like Hadoop, Spark, Spark SQL, Kafka, Sqoop, Hive, S3, HDFS, or
  • 3+ years of experience building, testing, and optimizing 'Big Data' data ingestion pipelines, architectures, and data sets
  • 2+ years of experience with Python (and/or Scala) and PySpark/Scala-Spark
  • 3+ years of experience with Cloud platforms e.g. AWS, GCP, etc.
  • 3+ years of experience with database solutions like Kudu/Impala, or Delta Lake or Snowflake or BigQuery
  • 2+ years of experience with NoSQL databases, including HBASE and/or Cassandra
  • Experience in successfully building and deploying a new data platform on Azure/ AWS
  • Experience in Azure / AWS Serverless technologies, like, S3, Kinesis/MSK, lambda, and Glue
  • Strong knowledge of Messaging Platforms like Kafka, Amazon MSK & TIBCO EMS or IBM MQ Series
  • Experience with Databricks UI, Managing Databricks Notebooks, Delta Lake with Python, Delta Lake with Spark SQL, Delta Live Tables, Unity Catalog
  • Knowledge of Unix/Linux platform and shell scripting is a must
  • Strong analytical and problem-solving skills

Preferred Qualifications:

  • Strong SQL skills with ability to write intermediate complexity queries
  • Strong understanding of Relational & Dimensional modeling
  • Experience with GIT code versioning software
  • Experience with REST API and Web Services
  • Good business analyst and requirements gathering/writing skills

Education:

Bachelor's Degree required. Preferably in Information Systems, Computer Science, Computer Information Systems or related field


  • Big Data Engineer

    2 months ago


    Atlanta, Georgia, United States Norfolk Southern Corp. Full time

    Norfolk Southern offers a unique opportunity to be part of our proud legacy that spans nearly 200 years. We are a customer-centric, operations-driven team dedicated to advancing safety, serving communities, and driving innovation for tomorrow's rail. As part of Norfolk Southern, you'll join a collaborative team where there are opportunities for growth across...


  • Atlanta, Georgia, United States Georgia Tech Research Institute Full time

    About the RoleWe are seeking a highly skilled Big Data Software Engineer to join our team at the Georgia Tech Research Institute. As a key member of our Health Emerging & Advanced Technologies (HEAT) Division, you will play a critical role in designing, developing, and implementing secure and scalable big data solutions for healthcare data analytics.Key...


  • Atlanta, Georgia, United States MasterCard Full time

    About the RoleWe are seeking a highly skilled Senior Data Engineer to join our Marketing Services (MS) Technology team at Mastercard. As a key member of our team, you will be responsible for designing, developing, and implementing data solutions using cutting-edge technologies such as Big Data, AI, and Machine Learning.Key ResponsibilitiesDevelop batch and...

  • Big Data Specialist

    1 week ago


    Atlanta, Georgia, United States Brooksource Full time

    About the RoleWe are seeking a highly skilled Data Engineer to join our team at Brooksource. As a Data Engineer, you will play a critical role in managing and optimizing data pipelines, ensuring the reliability and scalability of our data architecture.Key ResponsibilitiesData Pipeline Development: Design, build, and maintain scalable data pipelines to...

  • Data Engineer

    5 months ago


    Atlanta, Georgia, United States Boston Data Pro, LLC Full time

    We are looking for a Data Engineer to join our engineering team and help us develop and maintain various software products.The main responsibilities include writing and testing code debugging programs and integrating applications and transmission of the data from on-prem to AWS cloud.OPT CPT OK but must not be fresher.Key Skills Python Snowflake...


  • Atlanta, Georgia, United States Georgia Tech Research Institute Full time

    About the RoleWe are seeking a highly skilled Big Data Software Engineer to join our team at the Georgia Tech Research Institute. As a key member of our Health Emerging & Advanced Technologies (HEAT) Division, you will play a critical role in designing, developing, and implementing secure and scalable big data solutions for healthcare data analytics.Key...

  • Cloud Data Engineer

    13 hours ago


    Atlanta, Georgia, United States Voto Consulting Full time

    Job Title: Cloud Data EngineerJob Type: Full-timeLocation: Remote (Hybrid)Rate: $50/hourInterview: WebExAbout the Job:This is a long-term contract position with Voto Consulting LLC, a leading consulting firm in the industry. We are seeking an experienced Cloud Data Engineer to join our team and work on a project that involves integrating Cloud SQL, Big Query...

  • Data Engineer

    1 month ago


    Atlanta, Georgia, United States iVoyant Full time

    About the job Data Engineer iVoyant is looking for multiple Data Engineers to join our client Digital Team.The Data Engineers will work with the client functional SMEs within the Digital and the Business team to understand the requirements and prepare technical design.They will ensure the availability and validity of the data to ensure that it fits with the...


  • Atlanta, Georgia, United States Mueller Co. LLC Full time

    Job Summary : Mueller Co. LLC is seeking a highly skilled AWS-BI Developer to join our team. The ideal candidate will have extensive experience in data pipelines (ELT/ETL), data warehousing and dimensional modeling, and curation of data sets for Data Scientists and Business Intelligence users. Experience in building Data Lake is a must. This candidate will...


  • Atlanta, Georgia, United States Rose International Full time

    Job SummaryRose International is seeking a highly skilled Data Engineer Analytics Specialist to join our team. As a key member of our organization, you will be responsible for designing, developing, and implementing data analytics solutions to drive business growth and improvement.Key ResponsibilitiesRequirements Gathering and Analysis: Gather and analyze...

  • Key Data Engineer

    2 months ago


    Atlanta, Georgia, United States EPAM Systems Full time

    We are looking for a Key Data Engineer to join the project in the financial services industry. This role combines technical and people management skills for the data platform that require knowledge of financial data, big data processing and management, as well as cloud-based data warehouse solutions.What You'll DoBuild and maintain secure and compliant...


  • Atlanta, Georgia, United States Tata Consultancy Services Full time

    Job DescriptionWe are seeking a highly skilled Azure Sr Data Engineer to join our team at Tata Consultancy Services.Key Responsibilities:Cloud Application Development: Design and develop scalable and secure applications that can be hosted on the Azure cloud platform.Framework and Core Functionality: Design and develop frameworks and core functionality to...

  • Data Engineer II

    4 weeks ago


    Atlanta, Georgia, United States Bank of America Corporation Full time

    Job Description:At Bank of America, we are guided by a common purpose to help make financial lives better through the power of every connection. Responsible Growth is how we run our company and how we deliver for our clients, teammates, communities and shareholders every day.One of the keys to driving Responsible Growth is being a great place to work for our...

  • Data Scientist II

    7 days ago


    Atlanta, Georgia, United States Travelport Full time

    About TravelportTravelport is a leading travel retail platform that provides innovative solutions for businesses to connect with agencies and travelers. Our mission is to power the future of travel.Our CultureWe value diversity and inclusivity, and our workforce reflects the communities we serve. We're committed to building a culture that's welcoming and...


  • Atlanta, Georgia, United States Brooksource Full time

    About the RoleWe are seeking a skilled Data Engineer to join our team at Brooksource, a leading provider of innovative energy solutions. As a Data Engineer, you will play a critical role in managing and optimizing data pipelines, ensuring the reliability and scalability of our data architecture.Key ResponsibilitiesData Pipeline Development: Design, build,...

  • Senior Data Engineer

    1 month ago


    Atlanta, Georgia, United States Motion Recruitment Full time

    Every year, nearly 200 million travelers trust our client to get them where they're going. Take your career to new heights by working for this longstanding leader in air travel that services more worldwide destinations than any other airline.We are looking for a Senior Data Engineer to join our team for a 12 month contract in Atlanta, GA. Contract Duration:...


  • Atlanta, Georgia, United States Gordon Food Service Full time

    About Gordon Food ServiceGordon Food Service is a leading foodservice distributor in North America, serving customers across the continent. As a family-owned business, we prioritize our customers' success and strive to provide exceptional service and support.Job SummaryWe are seeking a highly motivated and detail-oriented Data Engineering Intern to join our...


  • Atlanta, Georgia, United States Dexian - Signature Consultants Full time

    Job Summary:Dexian - Signature Consultants is seeking a highly skilled Observability Data Pipeline Engineer to join our team. As a key member of our data engineering team, you will be responsible for designing, implementing, and maintaining data pipelines that drive business insights and inform strategic decision-making.Key Responsibilities:Design and...


  • Atlanta, Georgia, United States CMP Full time

    About the RoleCMP is seeking a highly skilled Data Engineering Lead to join our Marketing Technology team. As a key member of our team, you will be responsible for designing and building data solutions that capture, explore, transform, and utilize curated data to support our business unit's data-driven transformation journey and decision-making process.Key...

  • Senior Data Analyst

    7 days ago


    Atlanta, Georgia, United States Travelport Full time

    About TravelportTravelport is a leading travel retail platform that provides innovative solutions for businesses to connect with agencies and travelers worldwide.We're on a mission to power the future of travel, and we're looking for talented individuals to join our team.What We DoWe develop and deploy machine learning models to drive business growth and...