Data Engineer
Save this job and keep your search organized
Create a free account to save jobs, create alerts and return to this listing from your dashboard.
By continuing, you agree to our Terms & Privacy Policy.
GovCIO is seeking a Senior Data Engineer / Enterprise Telemetry & Data Pipeline SME to design, optimize, automate, and sustain large-scale data collection and streaming pipelines supporting Air Force Intelligence Community missions.
The environment includes a global Elastic deployment with distributed endpoint collectors, syslog and telemetry from Cisco and other network devices, Cribl-based routing and normalization of native and non-native sources, Splunk engineering and integration, and AWS-based infrastructure and services. The primary focus is the movement and optimization of machine-generated data from source to authorized analytics and SIEM destinations.
This is not primarily a data science or dashboard-development role. The successful candidate must be able to evaluate the complete data path, identify collection and transport inefficiencies, engineer scalable solutions, automate repeatable operations, and independently resolve complex ingestion, routing, performance, reliability, and integration problems. This position will be located on Lackland AFB and will be an onsite position.
ResponsibilitiesServe as the senior technical owner and subject-matter expert for AF IC enterprise telemetry collection, ingestion, transformation, normalization, routing, and delivery architectures.
Architect and optimize end-to-end data flows from distributed endpoints, servers, applications, network devices, security platforms, and cloud services into enterprise Elastic, Splunk, and other authorized analytics platforms.
Assess collection architectures for bottlenecks, duplicate or unnecessary data, inefficient routing, bandwidth consumption, data-loss risk, processing overhead, and modernization opportunities.
Engineer and troubleshoot Cribl Stream and Cribl Edge pipelines, including sources, destinations, routes, transformations, parsing, filtering, enrichment, normalization, buffering, and replay.
Engineer Elastic collection and ingestion capabilities using Elastic Agent, Beats, Logstash, ingest pipelines, APIs, data streams, Fleet/integrations, and related technologies.
Support Splunk ingestion and integration using forwarders, syslog, Event Collector, APIs, intermediate collection tiers, and other approved mechanisms.
Develop data schemas, field mappings, metadata standards, tagging, enrichment, and normalization practices that improve interoperability and downstream usability.
Design resilient pipeline patterns using buffering, queueing, retry, backpressure management, store-and-forward, failover, and recovery for geographically distributed and bandwidth-constrained environments.
Establish pipeline observability and data-quality controls to identify dropped, delayed, malformed, duplicated, or incorrectly routed events and measure throughput, latency, queue depth, and error rates.
Perform capacity and performance analysis across collectors, transport links, processing tiers, AWS services, and downstream analytics platforms; recommend changes that improve efficiency and scale.
Optimize data volume and routing to reduce unnecessary network, compute, storage, cloud, and SIEM ingestion cost without compromising mission or cybersecurity requirements.
Design and support AWS-based ingestion, streaming, processing, storage, and monitoring architectures appropriate to mission requirements.
Develop Python tooling and automation for data transformation, pipeline validation, API integration, configuration generation, health monitoring, testing, and troubleshooting.
Develop and maintain Infrastructure as Code and configuration automation using Terraform, CloudFormation, Ansible, or equivalent technologies.
Manage pipeline code, configuration, and infrastructure artifacts in Git and integrate changes into CI/CD and DevSecOps workflows with automated validation and controlled promotion.
Work with network, cloud, cybersecurity, systems, Elastic, Splunk, and DevSecOps teams to resolve cross-domain data-flow and infrastructure issues.
Ensure data transport and pipeline implementations meet applicable security, access-control, encryption, certificate, auditing, retention, and compliance requirements.
Lead root-cause analysis for data loss, latency, parsing, routing, ingestion, scaling, capacity, and integration failures and implement corrective actions that reduce recurrence.
Develop architecture and data-flow diagrams, implementation plans, engineering standards, runbooks, SOPs, and technical recommendations.
Provide technical leadership for telemetry modernization, platform integration, technology evaluation, migration, and enterprise scaling efforts.
QualificationsHigh School with 10+ years (or commensurate experience) Required Skills and ExperienceCandidates must demonstrate substantial hands-on experience engineering production data pipelines. Data analysis, dashboard development, SIEM use, or product administration without ownership of collection and transport architecture is not sufficient for this position.
Clearance: TS/SCI10+ years o