ETL Engineer III

Description * Design, develop, and maintain the ETL (Extract, Transform, Load) processes for master data management (MDM) system. * Build and optimize data pipelines to extract data from various sources, transform it into the required format, and load using Databricks and AWS services. * Collaborate with stakeholders to gather data requirements, understand data sources, and ensure data quality and integrity throughout the ETL process. * Implement data validation, cleansing, and enrichment techniques to improve the accuracy and completeness of data. * Monitor and troubleshoot ETL processes to identify and resolve issues in a timely manner. * Work in a team environment to design, develop, and support a software system which is undergoing a modernization. * Participate in developing new functionality and migrating the application into the cloud and introducing new technologies into the tech stack. * Participate in Agile Scrum SDLC activities. * Support developing Agile SDLC phase documentation. * Perform unit and integration testing of software/systems prior to release to the users for user acceptance testing. Qualifications * BS degree and twelve (8) years of experience with programming and software development including analysis, design, development, implementation, testing, maintenance, quality assurance, troubleshooting and/or upgrading of software systems. * Strong proficiency in programming languages such as Scala or Java. * Experience in designing and developing ETL workflows using tools like Apache Spark or AWS Glue. * In-depth knowledge of ETL best practices, data integration techniques, and data quality management. * Familiarity with different data storage technologies and databases, such as Amazon S3 or Amazon Redshift. * Understanding of concepts of Data Lakehouse architecture as well as OpenSearch is a plus. * A current Top Secret security clearance with SCI eligibility and the ability to obtain a polygraph. Preferred Qualifications: * Knowledge of Tika * Experience with AWS cloud services * Knowledge of data acquisition and ingestion of structured and unstructured data sources ensuring quality and data integrity * Experience with open source technologies like Docker, ElasticSearch, and NoSQL Databases, * Experience with AWS Lamda, SQS, or NiFi