Data Pipeline Engineer

3 weeks ago

North Charleston, United States CommIT Enterprises Full time
Overview

CommIT Enterprises, Inc. is seeking an Data Pipeline Engineer to join the team in Charleston, SC-*this contingent position is remote We are looking for a talented and forward-facing Data Pipeline Engineer who will be a critical part of a fast-moving Marine Corps product development team. This team is responsible for the design, development, testing, implementation and support for a wide range of cloud-based data integrations, data pipelines, data warehousing solutions, and delivery of Business Intelligence (BI) services to Marines globally.

Established in 2001, CommIT is a Certified Veteran-Owned Small Business (CVOSB) providing innovative technical engineering and data science services. Our enterprise systems support includes the Department of Defense's (DoD) GCSS-MC, CAC2S, TBMCS-MC, and the Department of Veteran's Affairs' (VA) telehealth communications. We offer acquisition management, systems engineering, Agile software development, cloud management, IT modernization, data analytics, cybersecurity, and training, including leading-edge DevSecOps, automated testing, and mobile application development.

Responsibilities

Your essential job functions will include but may not be limited to-
  • You will identify and deliver data pipeline solutions every day
  • In this role you will be working in a cross-functional team to build data pipeline solutions and processes to enable data warehousing, analytics and reporting for the USMC Logistics community.
  • You will be responsible for anything and everything related to existing and future Data Pipelines
  • Work with the Product Owner team and data analysts to develop efficient ETL processes using Apache Nifi, and other data pipeline tools.
  • Build performant data transformations for analysis and reporting.
  • Utilize modern ETL/ELT practices to move data into our Data Lake (AWS S3) and Warehouse (AWS Redshift).
  • Own the design, code development, code reviews, testing, data quality monitoring, deployment activities and operations support for all Nifi data flows.
  • Work alongside the infrastructure, data ingestion, data warehouse and analytics teams to manage and optimize ETL and data pipeline parameters, scheduling, monitoring and alerting.
  • Support development of data dictionary, connection guides, operations and user support SLAs.
  • Basic SQL operations required to Load data into the Data Warehouse (AWS Redshift)
  • Proven analytical, troubleshooting, and problem-solving skills.
  • Ability to communicate well cross-functionally, including non-technical users.
  • Must have a strong background in building ETL workflows. Apache Nifi is preferred, but other similar tools such as AWS Glue, Talend, DataBricks, etc
  • Ability to be self-starting, responsive and collaborative in a remote environment working with a distributed team.
  • Stay current with the latest design and technology trends and adopt modern techniques as appropriate.
  • Familiarity with operating in Cloud environments
  • Proficiency in one or more database platforms (e.g., MySQL, Oracle, SQL Server).
  • Strong understanding of SQL and ability to optimize queries for performance.
  • Familiarity with database monitoring tools to track performance metrics.
  • Experience with open-source software
  • Experience with AWS GovCloud
  • Familiarity with AWS Glue and AWS Redshift
  • Familiarity with BI tools, e.g. Microsoft Power BI and AWS QuickSight
  • Basic Linux command line experience
  • Experience with NNCompass functionality
Qualifications

Required Experience and Education:
  • Education: BS Degree in Electrical/ Electronic/ Computer /Software Engineering, Physics, or Computer Science from an accredited College or University.
  • Experience: Six (6) years of experience in DoD C4ISR systems, or information technologies to include: Systems Analysis, Systems Architecture, Systems/Equipment Support, Test and Evaluation, and Logistics support of C5ISR requirements. Three (3) years of technical experience in support of DoD C4ISR systems, or information technologies. Experience may be achieved simultaneously OR-
    • Education: Technical training in Computer Science, Information Systems, Engineering, Business, and Supply Chain related scientific or technical discipline or 10 years of Marine Corps experience in Maintenance, Communications, Logistics, or in the Retail Supply (SMU/Reparable Issue Point/RIP) and Organizational Supply field preferably.
    • Experience: Twelve (12) years of hands-on experience with a USMC supply, maintenance or logistics IT system, GCSS-MC, to includethree (3) of the following five (5) areas: Systems Requirements, Operational Requirements, Data Analysis, Test & Evaluation, and Training. Experience may be achieved simultaneously. Recognized expert who has demonstrated industry and/or public service (e.g., federal, state, or local) leadership in Defense Acquisitions University Level I or II (Life Cycle Logistics or Program Management) and experience with Major Automated Information System (MAIS).
Security Requirements:
  • Secret Clearance
  • U.S. Citizenship
  • DOD 8570 compliant - Security + and OS certification

Equal Opportunity Employer:

CommIT Enterprises, Inc. is an Equal Opportunity Employer. Employment decisions are made without regard to race, color, religion, national origin, gender, sexual orientation, gender identity, age, physical or mental disability, genetic factors, military/veteran status or other characteristics protected by law.

  • Senior Business Data Engineer

    5 days ago

    Charleston, United States Refyn Data Full time

    We are looking forward a Senior Business Data Engineer based in Clarksburg , MD, USA . In RefynData, making our work exciting, engaging, meaningful; ensuring safety, health, wellness; and being a model of inclusion & diversity are already embedded in who we are - it’s in our values, part of our vision, and our clearly defined aspirational goals. This...

  • Data Engineer

    1 month ago

    North Miami, United States SNI Technology Full time

    Data Engineer SNI Technology is currently hiring for a Data Engineer position with one of our esteemed healthcare clients. As a Data Engineer, you will play a pivotal role in designing and optimizing data solutions to support critical healthcare analytics and reporting. This role offers an exciting opportunity to work in a dynamic industry, leveraging...

  • Data Engineer III

    2 weeks ago

    North Kingstown, United States FM Global Full time

    FM Global is a leading property insurer of the world's largest businesses, providing more than one-third of FORTUNE 1000-size companies with engineering-based risk management and property insurance solutions. FM Global helps clients maintain continuity in their business operations by drawing upon state-of-the-art loss-prevention engineering and research;...

  • Data Engineer III

    4 days ago

    North Kingstown, United States FM Global Full time

    FM Global is a leading property insurer of the world's largest businesses, providing more than one-third of FORTUNE 1000-size companies with engineering-based risk management and property insurance solutions. FM Global helps clients maintain continuity in their business operations by drawing upon state-of-the-art loss-prevention engineering and research;...

  • IT Consultant

    24 hours ago

    Creedmoor, North Carolina, 27522, Graham County, NC, United States Duke Health Full time

    At Duke Health, we're driven by a commitment to compassionate care that changes the lives of patients, their loved ones, and the greater community. No matter where your talents lie, join us and discover how we can advance health together.About Duke Health Technology SolutionsPursue your passion for caring and innovation with Duke Heath Technology...

  • Associate Data Engineer

    1 day ago

    North Chicago, United States AbbVie Full time

    Job DescriptionAbbVie Information Research is seeking an Associate Data Engineer who would contribute to the architecture, design, and development of the Data & Analytics Platform, supporting world-class research and development at AbbVie. As an Associate Data Engineer, you will be a core member of a high-performance team of data engineers and architects...

  • Associate Data Engineer

    2 hours ago

    North Chicago, United States AbbVie Full time

    Company DescriptionAbbVie's mission is to discover and deliver innovative medicines and solutions that solve serious health issues today and address the medical challenges of tomorrow. We strive to have a remarkable impact on people's lives across several key therapeutic areas immunology, oncology, neuroscience, and eye care and products and services in...

  • Associate Data Engineer

    1 day ago

    North Chicago, IL, United States Abbvie Full time

    Company DescriptionAbbVie's mission is to discover and deliver innovative medicines and solutions that solve serious health issues today and address the medical challenges of tomorrow. We strive to have a remarkable impact on people's lives across several key therapeutic areas – immunology, oncology, neuroscience, and eye care – and products and services...

  • Associate Data Engineer

    24 hours ago

    North Chicago, United States Abbvie Full time

    Company DescriptionAbbVie's mission is to discover and deliver innovative medicines and solutions that solve serious health issues today and address the medical challenges of tomorrow. We strive to have a remarkable impact on people's lives across several key therapeutic areas – immunology, oncology, neuroscience, and eye care – and products and services...

  • Associate Data Engineer

    1 day ago

    North Chicago, United States AbbVie Full time

    Job DescriptionJob DescriptionCompany DescriptionAbbVie's mission is to discover and deliver innovative medicines and solutions that solve serious health issues today and address the medical challenges of tomorrow. We strive to have a remarkable impact on people's lives across several key therapeutic areas – immunology, oncology, neuroscience, and...

  • Engineer, Engineering

    1 day ago

    Charleston, United States UGI Utilities Full time

    Mountaineer Gas Company is the largest natural gas distribution company in West Virginia with over 220,000 customers and 450 employees. We service fifty (50) of the fifty-five (55) counties in West Virginia and maintain close to 6,000 miles of pipeline. Job Summary ENGINEER - CORPORATE OFFICE (CHARLETON, WV) We are currently recruiting in our Corporate...

  • Mid-Level Software/DSO Pipeline Engineer

    1 week ago

    North Logan, United States Utah State University Space Dynamics Laboratory Full time

    Job ID 17726A The Space Dynamics Laboratory (SDL) is seeking a software engineer to become a part of our team. We are the world leaders in developing frameworks that support high speed image processing and distribution of image products.Our team implements an iterative software development methodology, providing solutions to our customer at regular...

  • Senior Data Scientist

    20 hours ago

    North, United States Scope3 Full time

    Scope3 is the industry's leading collaborative sustainability platform. By modeling the complex digital advertising supply chain, we help brands measure and dramatically reduce digital waste to reduce carbon emissions from advertising and increase the efficacy of advertising investment.We're a well-funded startup with an accomplished leadership team — many...

  • Director, Global Pipeline Forecasting

    4 hours ago

    North Chicago, United States BioSpace Full time

    Job DetailsCompany DescriptionAbbVie's mission is to discover and deliver innovative medicines and solutions that solve serious health issues today and address the medical challenges of tomorrow. We strive to have a remarkable impact on people's lives across several key therapeutic areas immunology, oncology, neuroscience, and eye care and products and...

  • Engineer, Engineering

    1 day ago

    Charleston, United States Mountaineer Gas Full time

    Mountaineer Gas Company is the largest natural gas distribution company in West Virginia with over 220,000 customers and 450 employees. We service fifty (50) of the fifty-five (55) counties in West Virginia and maintain close to 6,000 miles of pipeline. Job Summary ENGINEER CORPORATE OFFICE (CHARLETON, WV) We are currently recruiting in our Corporate...

  • Engineer, Engineering

    18 hours ago

    Charleston, United States UGI Full time

    Mountaineer Gas Company is the largest natural gas distribution company in West Virginia with over 220,000 customers and 450 employees. We service fifty (50) of the fifty-five (55) counties in West Virginia and maintain close to 6,000 miles of pipeline. Job Summary ENGINEER - CORPORATE OFFICE (CHARLETON, WV) We are currently recruiting in our Corporate...

  • Data Center Commissioning Engineer

    5 days ago

    Charleston, United States Datacenter People Group Full time

    The Client Datacenter People is working with the leading data center commissioning and quality services firm in the world. The Role Provide engineering and technical support to plan, schedule, coordinate and execute full life cycle commissioning of critical facilities. Participate in the MEP design process. Witnesses the static and dynamic testing of complex...

  • Staff Security Engineer

    1 month ago

    North, United States inCode Full time

    Incode Incode offers the world's most advanced identity verification & authentication system. Power a world of trust with simple, secure & fast identity experiences. View company page Incode is the leading provider of world-class identity solutions that is reinventing the way humans authenticate and verify their identities online to power a world of digital...

  • Bioinformatics Engineering Data Analyst

    7 hours ago

    North Chicago, United States US Tech Solutions Full time

    Duration: 12 months contractWork schedule:• Hybrid or remote.Job Description: As a Data Analyst in Bioinformatics Engineering team, you will be responsible for developing and running workflows to standardize and load project datasets in a centralized environment. You will work closely with Bioinformatics Engineering, as well as therapeutic area facing...

  • Mechanical Engineer, Senior

    1 day ago

    North Charleston, United States Scientific Research Full time

    MINIMUM SKILLS & REQUIREMENTS: Secret Clearance Bachelor's Degree in a Mechanical Engineering discipline Five years' experience as a Mechanical Engineer investigating solutions to and/or process issues Experience in consolidating results of mechanical design elements Experience coordinating drafting and design efforts Must be able to conduct independent...