Data Pipeline Technical Lead

ECS is seeking a Data Pipeline Technical Lead to work in a hybrid setting out of our Fairfax, VA office.

ECS is seeking talented professionals to join our successful and growing team in building the next-generation Continuous Diagnostics and Mitigation (CDM) Cyber data solution. The CDM Program is the Cybersecurity and Infrastructure Security Agency's (CISA) dynamic approach to strengthening the cybersecurity of Federal networks and systems through better awareness and visibility into their security posture and cyber threats. ECS is responsible for designing, building, deploying, operating, and maintaining a complete 'Data Services' solution which includes the collection, normalization, visualization, and sharing of cyber data from more than 100 Federal agencies. The CDM Data Services product is an integrated suite of multiple Commercial Off the Shelf (COTS) products, software configuration packages, and custom code which work together to operate as an integrated solution tailored to meet Department of Homeland Security (DHS) requirements.

We are seeking professionals who thrive in a dynamic, fast-paced, and highly collaborative environment where problem-solving, critical thinking, and a holistic approach to serving the mission are key. Our program operates within the Scaled Agile Framework (SAFe). An aptitude and enthusiasm for continuous learning, improvement, and cyber security is a must

ECS is seeking a highly skilled Data Pipeline Technical Lead to design, build, and maintain large-scale, real-time data pipelines that provide mission critical information for our customers. You'll be responsible for leading technical design, guiding a cross functional team of engineers, and ensuring our pipeline is resilient, scalable, and efficient.

Key Responsibilities:

• Accountable for all technical facets in the delivery of our data pipeline product.

• Lead the architecture, design, development, and delivery of our data pipeline product, which consists of a team of 20+ members.

• Build and maintain microservices and data connectors that ingest, process, and distribute data across multiple systems and APIs.

• Ensure reliability, scalability, and performance of data pipelines through best practices in resiliency, fault tolerance, and eventual consistency.

• Optimize data processing and enrichment leveraging caching layers like Redis, ElastiCache, or Valkey.

• Guide data modeling decisions and define and enforce data format standards (JSON, Avro, Parquet), while leveraging a schema registry.

• Define technical work necessary for cross-functional team members (DevSecOps, Site Reliability Engineering, Security, Testing, Infrastructure) to ensure successful delivery of the data pipeline product.

• Work within a SAFe Agile framework to plan work, decompose Epics, develop Stories, and task team members appropriately.

• Represent the data pipeline team on a broader level within the program, whether that is across internal teams, with our customer and other stakeholders, provide demos, deliver product status, or generally answering technical questions about the data pipeline.

• Provide mentorship to team members to strengthen their skills, knowledge, and capabilities.

• Must be a US Citizen and able to acquire DHS Public Trust Suitability
• 10+ years of hands-on technical experience as a software developer.

• 6+ years experience leading engineering teams of at least 10 members, balancing technical direction with mentorship and skills development.

• 6+ years experience focused specifically on data engineering or data pipeline solutions and concepts.

• 6+ years developing microservice based applications utilizing components such as containers, Docker, Kubernetes, AWS ECR/ECS/EKS/Fargate, and container registries.

• 6+ years leveraging git repositories (Github, Gitlab, Bitbucket), with an advanced understanding of branching strategies, pull requests, commits, and other key concepts.

• Expert level Python experience, to include reviewing code commits and coaching developers on how to write better code.

• Demonstrated solution architect experience in designing highly performant, reliable, and scalable data pipeline solutions that include batch and stream processing concepts.

• Experience with stream processing technologies such as Kafka, Kafka Streams, Flink, or Spark.

• Proficiency working with diverse data formats (JSON, Avro, Parquet), and how a schema registry, schema evolution, and semantic validation can all work together to provide a mature solution.

• Experience developing custom data connectors and integrations with REST APIs.

• Expertise implementing and tuning pipeline caching layers (Redis, ElastiCache, Valkey).

• Experience designing and implementing data pipeline solutions that leverage cloud native services, particularly in AWS (Step Functions, EventBridge, Lambda, SNS, SQS, S3, etc.).

• Understanding of resiliency, idempotency, and eventual consistency in distributed pipelines.

• Strong analytical and problem-solving skills, with attention to detail.