Senior Data Engineer

3 weeks ago

Washington, United States Sparibis Full time
Location: 100% Remote

Years' Experience: 10+ years

Education: Bachelor's in IT related field

Work Authorization: Must show that applicant is legally permitted to work in the United States.

Clearance: Applicants must be able to meet the requirements to obtain an Public Trust security clearance. NOTE: United States Citizenship is required to be eligible to obtain this security clearance.

Key Skills:
  • 10+ years of IT experience focusing on enterprise data architecture and management
  • Experience with Databricks required
  • 8+ years experience in Conceptual/Logical/Physical Data Modeling & expertise in Relational and Dimensional Data Modeling
  • Experience with Great Expectations or other data quality validation frameworks
  • Experience with ETL and ELT tools such as SSIS, Pentaho, and/or Data Migration Services
  • Advanced level SQL experience (Joins, Aggregation, Windowing functions, Common Table Expressions, RDBMS schema design, Postgres performance optimization)
  • Experience with AWS environment, CI/CD pipelines, and Python (Python 3)
Responsibilities
  • Plan, create, and maintain data architectures, ensuring alignment with business requirements
  • Obtain data, formulate dataset processes, and store optimized data
  • Identify problems and inefficiencies and apply solutions
  • Determine tasks where manual participation can be eliminated with automation.
  • Identify and optimize data bottlenecks, leveraging automation where possible
  • Create and manage data lifecycle policies (retention, backups/restore, etc)
  • In-depth knowledge for creating, maintaining, and managing ETL/ELT pipelines
  • Create, maintain, and manage data transformations
  • Maintain/update documentation
  • Create, maintain, and manage data pipeline schedules
  • Monitor data pipelines
  • Create, maintain, and manage data quality gates (Great Expectations) to ensure high data quality
  • Support AI/ML teams with optimizing feature engineering code
  • Expertise in Spark/Python/Databricks, Data Lake and SQL
  • Create, maintain, and manage Spark Structured Steaming jobs, including using the newer Delta Live Tables and/or DBT
  • Research existing data in the data lake to determine best sources for data
  • Create, manage, and maintain ksqlDB and Kafka Streams queries/code
  • Data driven testing for data quality
  • Maintain and update Python-based data processing scripts executed on AWS Lambdas
  • Unit tests for all the Spark, Python data processing and Lambda codes
  • Maintain PCIS Reporting Database data lake with optimizations and maintenance (performance tuning, etc)
  • Streamlining data processing experience including formalizing concepts of how to handle lake data, defining windows, and how window definitions impact data freshness.
Qualifications
  • 10+ years of IT experience focusing on enterprise data architecture and management
  • Experience in Conceptual/Logical/Physical Data Modeling & expertise in Relational and Dimensional Data Modeling
  • Experience with Databricks, Structured Streaming, Delta Lake concepts, and Delta Live Tables required
    • Additional experience with Spark, Spark SQL, Spark DataFrames and DataSets, and PySpark
    • Data Lake concepts such as time travel and schema evolution and optimization
    • Structured Streaming and Delta Live Tables with Databricks a bonus
  • Experience leading and architecting enterprise-wide initiatives specifically system integration, data migration, transformation, data warehouse build, data mart build, and data lakes implementation / support
    • Advanced level understanding of streaming data pipelines and how they differ from batch systems
    • Formalize concepts of how to handle late data, defining windows, and data freshness
    • Advanced understanding of ETL and ELT and ETL/ELT tools such as SSIS, Pentaho, Data Migration Service etc
    • Understanding of concepts and implementation strategies for different incremental data loads such as tumbling window, sliding window, high watermark, etc.
    • Familiarity and/or expertise with Great Expectations or other data quality/data validation frameworks a bonus
    • Understanding of streaming data pipelines and batch systems
    • Familiarity with concepts such as late data, defining windows, and how window definitions impact data freshness
  • Advanced level SQL experience (Joins, Aggregation, Windowing functions, Common Table Expressions, RDBMS schema design, Postgres performance optimization)
    • Indexing and partitioning strategy experience
  • Debug, troubleshoot, design and implement solutions to complex technical issues
  • Experience with large-scale, high-performance enterprise big data application deployment and solution
  • Understanding how to create DAGs to define workflows
  • Familiarity with CI/CD pipelines, containerization, and pipeline orchestration tools such as Airflow, Prefect, etc a bonus but not required
  • Architecture experience in AWS environment a bonus
    • Familiarity working with Kinesis and/or Lambda specifically with how to push and pull data, how to use AWS tools to view data in Kinesis streams, and for processing massive data at scale a bonus
    • Experience with Docker, Jenkins, and CloudWatch
    • Ability to write and maintain Jenkinsfiles for supporting CI/CD pipelines
    • Experience working with AWS Lambdas for configuration and optimization
    • Experience working with DynamoDB to query and write data
    • Experience with S3
  • Knowledge of Python (Python 3 desired) for CI/CD pipelines a bonus
    • Familiarity with Pytest and Unittest a bonus
  • Experience working with JSON and defining JSON Schemas a bonus
  • Experience setting up and management Confluent/Kafka topics and ensuring performance using Kafka a bonus
    • Familiarity with Schema Registry, message formats such as Avro, ORC, etc.
    • Understanding how to manage ksqlDB SQL files and migrations and Kafka Streams
  • Ability to thrive in a team-based environment
  • Experience briefing the benefits and constraints of technology solutions to technology partners, stakeholders, team members, and senior level of management


About Sparibis

Sparibis LLC is a professional solution firm that Clients rely on to access the best talent to drive their business success.

Sparibis is an equal opportunity employer that values diversity at all levels. All individuals, regardless of personal characteristics, are encouraged to apply.

  • Senior Implementation Data Engineer

    3 weeks ago

    Washington, Washington, D.C., United States Altana Technologies Full time

    The Opportunity at AltanaThe Implementation team is looking for talented Senior Implementation Data Engineers to help execute on our vision. In this role, you will manage, architect, and contribute to successful customer software deployments on premise in secured environments, design and configure integrations and other customizations based on customer...

  • Senior Implementation Data Engineer

    1 week ago

    Washington, United States Altana AI Full time

    Altana provides the world's only dynamic, intelligent map of the global supply chain - the Altana Atlas - using AI and machine learning models to connect with and learn from massive sets of public and private data. Through the Atlas, companies and governments can understand the distant origins of products well beyond their own direct suppliers; discover...

  • Senior Engineer Chief Engineer u0026 Data

    3 weeks ago

    Washington, United States CLevelCrossing Full time

    Location Washington, DC, United States Employment Type Full-Time Industry Clevel, Engineering, Executive, Security Clearance, It, 100k Posted on Jun 11, 2021 Apply for this job your email: upload resume: Profile :text,ActualValueFromSolar:null},{QuestionName:Recruiter,AnswerValue:John...

  • Senior Data Engineer Pentaho

    1 week ago

    Washington, United States Mindlance Full time

    Position Summary: Title: Senior Data Engineer - Pentaho Duration: 6 Months - Long Term Location: Washington, DC 20005 Hybrid Onsite: 2/3 days per week from Day1 Candidate should be comfortable with picking up official laptop from Client place during the onboarding process once offered. Summary/Objective: The Senior Data Engineer will execute on ...

  • Data Engineer

    3 weeks ago

    Washington, United States ITR Full time

    Job DescriptionJob DescriptionSenior Data Engineer – Top Secret Clearance RequiredWill implement large-scale data ecosystems including data management, governance and the integration of structured and unstructured data to generate insights leveraging cloud-based platforms. • Leverage automation, cognitive and science-based techniques to manage data,...

  • Senior Data Engineer

    1 week ago

    Washington, United States Sparibis Full time

    Location: 100% Remote Years' Experience: 10+ years Education: Bachelor's in IT related field Work Authorization: Must show that applicant is legally permitted to work in the United States. Clearance: Applicants must be able to meet the requirements to obtain an Public Trust security clearance. NOTE: United States Citizenship is required to be eligible to...

  • Senior Network Security Engineer

    2 weeks ago

    Washington, United States MDS (Micro-Data Systems) Full time

    Senior Security EngineerRemote, but prefer candidates to be located in the Washington, DC Metro AreaJob DescriptionYou will provide guidance and technical support to clients deploying security integrations. You'll act as the technical partner, providing strategic guidance around complex systems to secure a digital environment. Interacting directly with the...

  • Senior Network Security Engineer

    21 hours ago

    Washington, United States MDS (Micro-Data Systems) Full time

    Senior Security EngineerRemote, but prefer candidates to be located in the Washington, DC Metro AreaJob DescriptionYou will provide guidance and technical support to clients deploying security integrations. You'll act as the technical partner, providing strategic guidance around complex systems to secure a digital environment. Interacting directly with the...

  • Senior Network Security Engineer

    2 weeks ago

    Washington, United States MDS (Micro-Data Systems) Full time

    Senior Security EngineerRemote, but prefer candidates to be located in the Washington, DC Metro AreaJob DescriptionYou will provide guidance and technical support to clients deploying security integrations. You'll act as the technical partner, providing strategic guidance around complex systems to secure a digital environment. Interacting directly with the...

  • Senior Data Analyst

    1 week ago

    Washington, United States Macpower Digital Assets Edge Full time

    Benefits: Medical, Dental, Vision, Life insurance, Paid time off, Matching 401k, Tuition reimbursement. Role Overview: Our client seeks a Senior Data Analyst with a strong background in framing and conducting empirical and qualitative assessments of large enterprises to include data visualization, formulating data collection schemas from diverse sources,...

  • Data Analyst/Data Engineer

    4 weeks ago

    Ft. Washington, Maryland, United States ENSCO Inc. Full time

    ENSCO Mission Systems Group (MSG) is recruiting senior engineering and support staff supporting the Office of the Undersecretary of Defense - Intelligence & Security Branch (OUSD/I&S). We are currently seeking a Data Analyst/Data Engineer who will interface with functional and technical experts. Successful candidates will support the modernization,...

  • Senior Strategy and Change Management Data Engineer

    3 weeks ago

    Washington, United States Grant Leading Technology Full time

    Job DescriptionJob DescriptionSenior Strategy and Change Management Data EngineerGrant Leading Technology is seeking a candidate for a Senior Strategy and Change Management Data Engineer to join our dynamic team. The candidate will be responsible for assisting the FAA to develop an Organizational Strategic Vision and Framework, including change...

  • Senior Data Scientist

    6 days ago

    Washington, United States iO Associates - US Full time

    An exciting Bio-Medical start up is looking for a talented Senior Data Scientist to join their team on a Perm basis. The firm is headquartered in Washington DC but this role is open to fully remote.The firm are looking for a Data scientist with advanced expertise in statistical modeling, predictive analytics, and Machine Learning. This role will be...

  • Senior Data Scientist

    6 days ago

    Washington, United States iO Associates - US Full time

    An exciting Bio-Medical start up is looking for a talented Senior Data Scientist to join their team on a Perm basis. The firm is headquartered in Washington DC but this role is open to fully remote.The firm are looking for a Data scientist with advanced expertise in statistical modeling, predictive analytics, and Machine Learning. This role will be...

  • Data Analyst

    1 week ago

    Washington, United States INNOVER GLOBAL INC Full time

    Data Analyst with MongoDB/No SQL Senior Data Engineer to join our dynamic team in the technology industry. As a Senior Data Engineer, you will be responsible for designing, building, and maintaining our data infrastructure to support our growing business needs

  • AWS Senior Data Engineer

    4 weeks ago

    Washington, United States Atechstar Full time

    Job description Skills Required- Proficient in languages Python.- Experience in AWS Stack - Glue Athena Quick sight RDS Redshift Kafka pySpark.- Experience setting up data pipelines archiving data data lakes.- Bachelors or Master's degree in computer science Maths statistics or related field.- Expertise with designing complex Data Models and...

  • AWS Senior Data Engineer

    4 weeks ago

    Washington, Washington, D.C., United States Atechstar Full time

    Job description Skills RequiredProficient in languages Python.Experience in AWS StackGlue Athena Quick sight RDS Redshift Kafka pySpark.Experience setting up data pipelines archiving data data lakes.Bachelors or Master's degree in computer science Maths statistics or related field.Expertise with designing complex Data Models and Data Engineering...

  • Data Engineer

    4 weeks ago

    Washington, United States Atechstar Full time

    Key Responsibilities Design implement and support applications that provide structured and timely access to actionable business information addressing stakeholder needs. Interface directly with stakeholders gathering requirements and owning automated end-to-end reporting solutions. Partner with analysts data engineers business intelligence engineers ...

  • Data Engineer

    4 weeks ago

    Washington, Washington, D.C., United States Non-Departmental Agency Full time

    Summary Data Engineers work with data consumers to create and populate optimal data architectures, structures, and systems to meet CIA's business needs. Duties As a Data Engineer for CIA, you will focus on the design, implementation, and operation of data management systems to meet the CIA's business needs. This includes designing how the data will be...

  • Data Engineer

    4 weeks ago

    Washington, United States Non-Departmental Agency Full time

    Summary Data Engineers work with data consumers to create and populate optimal data architectures, structures, and systems to meet CIA's business needs. ...