Lead Big Data Engineer

4 weeks ago


Rockville, United States Tential Full time

Lead Big Data Developer
Are you passionate in data? Do you like working in a challenging environment where massive volume of data is ingested and processed every day? Are you a continuous learner who wants to learn new tools and technologies in evolving big data and data science technology? Do you have passion in detecting data anomalies in large datasets? Do you expect the best from yourself and those around you?

We are looking for lead big data developer for ETL project in our enterprise Transparency Services group. Big data developer will work on designing, ingesting, storing, validating and disseminating, after transforming data in a consumable format, for business intelligence teams and data analysts to get deeper business insight from the data.

Job Responsibilities

  • Understand complex business requirements
  • Design and develop ETL pipeline for collecting, validating and transforming data according to the specification
  • Develop automated unit tests, functional tests and performance tests.
  • Maintain optimal data pipeline architecture
  • Design ETL jobs for optimal execution in AWS cloud environment
  • Reduce processing time and cost of ETL workloads
  • Lead peer reviews and design/code review meetings
  • Provide support for production support operations team
  • Implement data quality checks.
  • Identify areas where machine learning can be used to identify data anomalies

Experience & Qualifications
  • 7+ years of experience in programming language Java or Scala
  • 7+ years of experience in ETL projects
  • 5+ years of experience in big data projects
  • 3+ years of experience with API development (REST API's)
  • Believes in Scrum/Agile, and has deep experience delivering software when working on teams that use Scrum/Agile methodology
  • Strong and creative analytical and problem-solving skills

Required Technical Skills & Knowledge
  • Strong experience in Java or Scala
  • Strong experience in big data technologies like AWS EMR, AWS EKS, Apache Spark
  • Strong experience with serverless technologies like AWS Dynamo DB, AWS Lambda
  • Strong experience in processing with JSON and csv files
  • Must be able to write complex SQL queries
  • Experience in performance tuning and optimization
  • Familiar with columnar storage formats (ORC, Parquet) and various compression techniques
  • Experience in writing Unix shell scripts
  • Unit testing using JUnit or ScalaTest
  • Git/Maven/Gradle
  • Code Reviews
  • Experience with CI/CD pipelines
  • Agile

The following skills a plus:
  • AWS Cloud
  • BPM/ AWS Step Functions
  • Python scripting
  • Performance testing tools like Gatling or JMeter

#LI-WB
#Dice
  • Data Engineer

    1 month ago


    Rockville, United States ASCENDING Full time

    Job DescriptionJob DescriptionFull-time, Remote - Location Requirement: Must be located in the Eastern Time ZoneAvailable for W-2 or 1099 Individual. Our client, one of the largest Amazon Web Services (AWS) partners for data services, is looking for a true Mid level Big Data Engineer to contribute to join their team of technologists to build and contribute...

  • Data Engineer

    1 week ago


    Rockville, United States ASCENDING Full time

    Job DescriptionJob DescriptionFull-time, Remote - Location Requirement: Must be located in the Eastern Time ZoneAvailable for W-2 or 1099 Individual. Our client, one of the largest Amazon Web Services (AWS) partners for data services, is looking for a true Mid level Big Data Engineer to contribute to join their team of technologists to build and contribute...

  • Lead Data Engineer

    4 weeks ago


    Rockville, United States Optomi Full time

    Lead Data Engineer | RemoteOptomi, in partnership with a global leader in the financial services industry, is seeking a Lead Data Engineer to join their team! This candidate will be responsible for designing, ingesting, storing, validating and disseminating data in a consumable format, for business intelligence teams and data analysts to get deeper business...

  • Lead Data Engineer

    4 weeks ago


    Rockville, United States Optomi Full time

    Lead Data Engineer | RemoteOptomi, in partnership with a global leader in the financial services industry, is seeking a Lead Data Engineer to join their team! This candidate will be responsible for designing, ingesting, storing, validating and disseminating data in a consumable format, for business intelligence teams and data analysts to get deeper business...

  • Lead Data Engineer

    4 weeks ago


    Rockville, United States Optomi Full time

    Lead Data Engineer | RemoteOptomi, in partnership with a global leader in the financial services industry, is seeking a Lead Data Engineer to join their team! This candidate will be responsible for designing, ingesting, storing, validating and disseminating data in a consumable format, for business intelligence teams and data analysts to get deeper business...

  • Sr. Data Engineer

    1 day ago


    Rockville, United States ASCENDING LLC Full time

    Our client is a leading provider of financial and economic data to the investment banking industry. They are seeking an experienced Senior Data Engineer to join and growing team. - This role is only available for W2 or individual contracts. Please no C2C. - Remote 100%Responsibilities: Analyze system requirements and design responsive algorithms and...

  • Sr. Data Engineer

    2 months ago


    Rockville, United States ASCENDING Full time

    Job DescriptionJob DescriptionOur client is a leading provider of financial and economic data to the investment banking industry. They are seeking an experienced Senior Data Engineer to join and growing team.- This role is only available for W2 or individual contracts. Please no C2C. - Remote 100%Responsibilities:Analyze system requirements and design...

  • Sr. Data Engineer

    1 week ago


    Rockville, United States ASCENDING Full time

    Job DescriptionJob DescriptionOur client is a leading provider of financial and economic data to the investment banking industry. They are seeking an experienced Senior Data Engineer to join and growing team.- This role is only available for W2 or individual contracts. Please no C2C. - Remote 100%Responsibilities:Analyze system requirements and design...

  • Data Engineer

    2 weeks ago


    Rockville, United States GCB Services LLC Full time

    Job DescriptionJob DescriptionJob Description:Ensure analytics infrastructure and associated systems meet business requirements and industry best practices.Gather and process raw data from multiple disparate sources (including writing scripts, calling APIs, write SQL queries, etc.) into a form suitable for analysis.Gathers, analyzes, documents, and...

  • Data Engineer

    2 weeks ago


    Rockville, United States GCB Services LLC Full time

    Job DescriptionJob DescriptionJob DescriptionEnsure analytics infrastructure and associated systems meet business requirements and industry best practices.Gather and process raw data from multiple disparate sources (including writing scripts, calling APIs, write SQL queries, etc.) into a form suitable for analysis.Gathers, analyzes, documents, and translates...

  • Data Engineer

    1 week ago


    Rockville, United States GCB Services LLC Full time

    Job DescriptionJob DescriptionJob DescriptionEnsure analytics infrastructure and associated systems meet business requirements and industry best practices.Gather and process raw data from multiple disparate sources (including writing scripts, calling APIs, write SQL queries, etc.) into a form suitable for analysis.Gathers, analyzes, documents, and translates...

  • Data Engineer

    1 week ago


    Rockville, United States GCB Services LLC Full time

    Job DescriptionJob DescriptionJob Description:Ensure analytics infrastructure and associated systems meet business requirements and industry best practices.Gather and process raw data from multiple disparate sources (including writing scripts, calling APIs, write SQL queries, etc.) into a form suitable for analysis.Gathers, analyzes, documents, and...

  • Senior Data Engineer

    2 months ago


    Rockville, United States University of Maryland Medical System Full time

    Job DescriptionJob DescriptionCompany DescriptionRenowned as the academic flagship of the University of Maryland Medical System, our Magnet®-designated facility is a nationally recognized, academic medical center with opportunities across the continuum of care.  Come join UMMC and discover the atmosphere where talents and ideas come together to enhance...


  • Rockville, United States The University of Maryland Medical System Full time

    Renowned as the academic flagship of the University of Maryland Medical System, our Magnet-designated facility is a nationally recognized, academic medical center with opportunities across the continuum of care. Come join UMMC and discover the atmosphere where talents and ideas come together to enhance patient care and advance the science of nursing. Located...


  • Rockville, United States University of Maryland Medical System Full time

    Job DescriptionJob DescriptionCompany DescriptionRenowned as the academic flagship of the University of Maryland Medical System, our Magnet®-designated facility is a nationally recognized, academic medical center with opportunities across the continuum of care.  Come join UMMC and discover the atmosphere where talents and ideas come together to enhance...


  • Rockville, United States ConsultNet Full time

    Senior Data EngineerRemote- must work standard EST hoursLong-term contract  OverviewOur client is looking for a Senior Data Engineer to join their teamAs the data engineer on the team, you are responsible for building the reporting infrastructure by working with the engineering team, business users, product management, and the cross-functional partnersYou...

  • Full Stack Engineer

    3 weeks ago


    Rockville, United States Hire Talent Full time

    About the Role: As a Full-stack Engineer [SS1] on our team, you'll be expected to lead by example, write great code, solve difficult problems, set a high bar for the team around you and help them achieve engineering excellence. We want you to execute on "your" ideas, challenge the status quo and generally make things better. You will need the soft skills...

  • Full Stack Engineer

    1 week ago


    Rockville, United States Hire Talent Full time

    About the Role: As a Full-stack Engineer [SS1] on our team, you'll be expected to lead by example, write great code, solve difficult problems, set a high bar for the team around you and help them achieve engineering excellence. We want you to execute on "your" ideas, challenge the status quo and generally make things better. You will need the soft skills...

  • Full Stack Engineer

    3 days ago


    Rockville, United States Hire Talent Full time

    About the Role: As a Full-stack on our team, you'll be expected to lead by example, write great code, solve difficult problems, set a high bar for the team around you and help them achieve engineering excellence. We want you to execute on your ideas, challenge the status quo and generally make things better. You will need the soft skills too; Communication,...


  • Rockville, United States Dine Development Corporation Full time

    Job Summary: The Clinical Data Management/Database Architect Lead (CDDAL) is responsible for overseeing the design, implementation, and maintenance of clinical data management systems. The CDDAL role will involve collaborating with clinical research teams to ensure data is accurately captured, stored, and retrieved for analysis, which will include developing...