Big Data Engineer

14 hours ago

Atlanta, GA, United States Nava Software Solutions Full time
NAVA Software solutions is looking for a Big Data Engineer

Details:

Big Data Engineer

Location: Atlanta GA - Hybrid

Duration: 12 months

Job Description

The successful candidate must have Big Data engineering experience and must demonstrate an affinity for working with others to create successful solutions. Join a smart, highly skilled team with a passion for technology, where you will work on our state of the art Big Data Platforms. They must be a very good communicator, both written and verbal, and have some experience working with business areas to translate their business data needs and data questions into project requirements. The candidate will participate in all phases of the Data Engineering life cycle and will independently and collaboratively write project requirements, architect solutions and perform data ingestion development and support duties.

Skills and Experience:

Required:
  • 8+ years of overall IT experience
  • 3+ years of experience with high-velocity high-volume stream processing: Apache Kafka and Spark Streaming
  • Experience with real-time data processing and streaming techniques using Spark structured streaming and Kafka
  • Deep knowledge of troubleshooting and tuning Spark applications
  • 3+ years of experience with data ingestion from Message Queues (Tibco, IBM, etc.) and different file formats across different platforms like JSON, XML, CSV
  • 3+ years of experience with Big Data tools/technologies like Hadoop, Spark, Spark SQL, Kafka, Sqoop, Hive, S3, HDFS, or
  • 3+ years of experience building, testing, and optimizing 'Big Data' data ingestion pipelines, architectures, and data sets
  • 2+ years of experience with Python (and/or Scala) and PySpark/Scala-Spark
  • 3+ years of experience with Cloud platforms e.g. AWS, GCP, etc.
  • 3+ years of experience with database solutions like Kudu/Impala, or Delta Lake or Snowflake or BigQuery
  • 2+ years of experience with NoSQL databases, including HBASE and/or Cassandra
  • Experience in successfully building and deploying a new data platform on Azure/ AWS
  • Experience in Azure / AWS Serverless technologies, like, S3, Kinesis/MSK, lambda, and Glue
  • Strong knowledge of Messaging Platforms like Kafka, Amazon MSK & TIBCO EMS or IBM MQ Series
  • Experience with Databricks UI, Managing Databricks Notebooks, Delta Lake with Python, Delta Lake with Spark SQL, Delta Live Tables, Unity Catalog
  • Knowledge of Unix/Linux platform and shell scripting is a must
  • Strong analytical and problem-solving skills
Preferred (Not Required):
  • Strong SQL skills with ability to write intermediate complexity queries
  • Strong understanding of Relational & Dimensional modeling
  • Experience with GIT code versioning software
  • Experience with REST API and Web Services
  • Good business analyst and requirements gathering/writing skills

  • Lead Big Data Engineer

    1 week ago

    Atlanta, GA, United States Vimerse InfoTech Inc Full time

    Job Title: Lead Data Engineer Location: Atlanta, GA (Hybrid) - 2-3 days a week in office, Day 1 onsite. Job Description Your role will actively lead the Data Operations Team, in maintaining & operating our data platform capabilities & establishing frameworks to make our data products become optimized and scalable. You will play a critical role in ensuring...

  • Data Engineer

    2 weeks ago

    Atlanta, GA, United States NTT DATA North America Full time

    Data Engineer (ETL DataStage) - 25-06063 Location - Atlanta, GA (Hybrid/Onsite - Mon, Wed, Thur - 9am to 6pm EST) Duration - 1yr+ W2 or C2C *2 Interviews / 2nd is In-Person Job duties : Design, develop, automate, and support complex applications to extract, transform, and load data; Implement and automate complex processes; Plan and conduct ETL unit and...

  • Data Engineer

    1 week ago

    Atlanta, GA, United States NTT DATA North America Full time

    Data Engineer (ETL DataStage) - 25-06063 Location - Atlanta, GA (Hybrid/Onsite - Mon, Wed, Thur - 9am to 6pm EST) Duration - 1yr+ W2 or C2C *2 Interviews / 2nd is In-Person Job duties : Design, develop, automate, and support complex applications to extract, transform, and load data; Implement and automate complex processes; Plan and conduct ETL unit and...

  • Data Engineer

    7 days ago

    Atlanta, GA, United States NTT DATA North America Full time

    Data Engineer (ETL DataStage) - 25-06063 Location - Atlanta, GA (Hybrid/Onsite - Mon, Wed, Thur - 9am to 6pm EST) Duration - 1yr+ W2 or C2C *2 Interviews / 2nd is In-Person Job duties : Design, develop, automate, and support complex applications to extract, transform, and load data; Implement and automate complex processes; Plan and conduct ETL unit and...

  • Data Engineer

    3 days ago

    Atlanta, GA, United States Next Level Business Services, Inc. Full time

    Analyze large and complex datasets to identify trends, insights, and business opportunities within the payment domain. Develop, optimize, and maintain data pipelines to support business intelligence and analytics. Work extensively with Aurora DB (MySQL/PostgreSQL) for data modeling, querying, and performance tuning. Design and implement scalable data...

  • Sr Data Engineer

    1 day ago

    Atlanta, GA, United States Pacer Group Full time

    Description: Senior Palantir Foundry Engineer (Development) - Big Data Job Description: Develops and supports business application utilizing latest Big Data technologies in a Cloud based development environment. Helps to define and analyze business requirements for Enterprise scale reports. Analyzes and evaluates business use cases for data engineering...

  • Data Engineer

    1 week ago

    Atlanta, GA, United States iVoyant Full time

    About the job Data Engineer iVoyant is looking for multiple Data Engineers to join our client Digital Team. The Data Engineers will work with the client functional SMEs within the Digital and the Business team to understand the requirements and prepare technical design. They will ensure the availability and validity of the data to ensure that it fits with...

  • Senior Software Engineer, Data Engineering

    3 days ago

    Atlanta, GA, United States Omada Health Full time

    Senior Software Engineer, Data Engineering Omada Health is on a mission to inspire and engage people in lifelong health, one step at a time. Job overview: We are dedicated to leveraging data to drive strategic decision-making and operational efficiency. Our team is passionate about harnessing the power of data to solve complex problems and deliver impactful...

  • Sr. Azure Data Engineer

    1 week ago

    Atlanta, GA, United States Diverse Lynx Full time

    Role: Sr. Azure Data Engineer Location: tlanta, GA (must be onsite for this opportunity - NO starting off remote) Duration: Long-term contract to hire (On-Site 4 days a week.) Job Type: W2 Contract - Visa Sponsorship not available Experience: 8-10 Year MUST HAVES: 8-10 years of data engineering exp, hands on experience with Microsoft Azure...

  • Data Engineer with ML Engineering Experience

    2 weeks ago

    Atlanta, GA, United States Highbrow Full time

    Job Title :- Data Engineer with ML Engineering Experience Employment Type :- W2 Duration :- Long Term Visa Type :- All Visa applicable which are ready for W2 Location- Atlanta, GA (Day-1 Onsite) Job Description: Education and Experience Bachelors degree in Computer Science, Data Science, Engineering, or a related field. A Masters degree or relevant...