Cloud Data Engineer with Databricks

1 month ago

Maryland Line, United States Diaconia LLC Full time
Diaconia is looking for a talented Cloud Data Engineer to join our Amazing team

If you're looking to join a company that truly appreciates you and your talents, look no further At Diaconia, we are committed to serving and caring for our colleagues, our clients and our community. Our team is made up of talented individuals who appreciate having the opportunity to contribute their knowledge and experience to further the growth and development of our industry. Our ideal candidates embrace diverse thinking, enjoy partnering with others and are seeking to make a difference

We are currently searching for a new, full-time member for our team for the position of:

Cloud Data Engineer with Databricks

**U.S. Citizenship is REQUIRED**

A Cloud Data Engineer with Databricks experience and U.S. Citizenship required per Federal Requirements for the Consumer Financial Protection Bureau (CFPB) is responsible for managing, developing, and maintaining data infrastructure and pipelines, with a specific focus on Databricks, to support the agency's data-driven initiatives and regulatory responsibilities. The CFPB is a U.S. government agency tasked with regulating and overseeing financial products and services to protect consumers.

Job Title: Cloud Data Engineer with Databricks

Job Summary: The Cloud Data Engineer with Databricks at the Consumer Financial Protection Bureau (CFPB) is responsible for designing, building, and maintaining data infrastructure and ETL pipelines using Databricks and cloud-based technologies. The role plays a crucial part in enabling data-driven decision-making and ensuring data accuracy and accessibility for regulatory purposes.

Key Responsibilities:
  1. Collaborate & contribute to the architecture, design, development, and maintenance of large-scale data & analytics platforms, system integrations, data pipelines, data models & API integrations.
  2. Prototype emerging business use cases to validate technology approaches and propose potential solutions.
  3. Data Pipeline Development: Design, develop, and maintain data pipelines using Databricks, Apache Spark, and other cloud-based technologies to ingest, transform, and load data from various financial institutions and sources.
  4. Data Transformation: Implement data transformation processes to ensure data quality, integrity, and consistency, meeting regulatory standards. Create transformation path for data to migrate from on-prem pipelines and sources to AWS.
  5. Data Integration: Integrate data from diverse sources, including financial databases, APIs, regulatory reporting systems, and internal data stores, into the CFPB's data ecosystem.
  6. Data Modeling: Develop and optimize data models for regulatory analysis, reporting, and compliance, following data warehousing and data lake principles.
  7. Performance Optimization: Monitor and optimize data pipelines for efficiency, scalability, and cost-effectiveness while ensuring data privacy and security.
  8. Data Governance: Ensure data governance and regulatory compliance, maintaining data lineage and documentation for audits and reporting purposes.
  9. Collaboration: Collaborate with cross-functional teams, including data analysts, legal experts, and regulatory specialists, to understand data requirements and provide data support for regulatory investigations.
  10. Documentation: Maintain comprehensive documentation for data pipelines, code, and infrastructure configurations, adhering to regulatory compliance standards.
  11. Troubleshooting: Identify and resolve data-related issues, errors, and anomalies to ensure data reliability and compliance with regulatory requirements.
  12. Continuous Learning: Stay updated with regulatory changes, industry trends, cloud technologies, and Databricks advancements to implement best practices and improvements in data engineering.

Disclaimer "The responsibilities and duties outlined in this job description are intended to describe the general nature and level of work performed by employees within this role. However, they are not exhaustive and may be subject to change or modification at any time to meet the evolving needs of the organization

Minimum Qualifications:
  • U.S. Citizens ONLY as per Federal Requirements
  • Bachelor's or higher degree in computer science, data engineering, or a related field.
  • The Databricks Certified Data Engineer Professional certification is required. NO EXCEPTIONs
  • U.S. Citizenship is required. NO EXCEPTIONS
  • Minimum of 3 years of experience in the following:
    • Strong understanding of data lake, lakehouse, and data warehousing architectures in a cloud-based environment.
    • Hands-on experience with Databricks including data ingestion, transformation, and analysis
    • Proficiency in Python for data manipulation, scripting, and automation
    • In-depth knowledge of AWS services relevant to data engineering such as Amazon S3, EC2, Database Migration Service (DMS), DataSync, EKS, CLI, RDS, Lambda, etc.
    • Understanding of data integration patterns and technologies.
    • Proficiency designing and building flexible and scalable ETL processes and data pipelines using Python and/or PySpark and SQL.
    • Proficiency in data pipeline automation and workflow management tools like Apache Airflow or AWS Step Functions.
    • Knowledge of data quality management and data governance principles.
    • Strong problem-solving and troubleshooting skills related to data management challenges.
    • Experience managing code in GitHub or other similar tools.
    • Experience leveraging Postgres in a parallel processing environment.
  • Hands-on experience migrating from an on-premise data platform(s) to a modern cloud environment (e.g. AWS, Azure, GCP).
  • Excellent problem-solving and communication skills.
  • Strong attention to detail and the ability to work independently and collaboratively.
Preferred Skills:
  • Experience with financial data or regulatory data management.
  • Experience working in Agile or DevSecOps environments and using related tools for collaboration and version control.
  • Knowledge of regulatory frameworks in the financial industry.
  • Familiarity with DevOps and CI/CD practices.
  • Experience with machine learning and AI technologies.
Clearance requirements:
  • Must be able to obtain and maintain a Public Trust clearance.
  • Must be a verifiable a US Citizen for this Federal support position.

A Cloud Data Engineer with Databricks at the CFPB plays a vital role in ensuring data accuracy, integrity, and compliance with regulatory standards, supporting the agency's mission to protect consumers in the financial sector. The role demands expertise in Databricks, cloud technologies, and a deep understanding of data engineering principles within a regulatory context.

Applicant selected will be subject to a government security investigation and must meet eligibility requirements for access to classified information. Diaconia is an Equal Opportunity Employer, Minorities/Females/Veterans/Disabled. All qualified applicants will receive consideration for employment without regard to race, color, religion, sex, sexual orientation, gender identity, or national origin.

MUST be willing to consent to a background check including a criminal record check, employment, and education verification. No Exceptions

  • Azure Databricks Data Engineer

    1 month ago

    Center Line, Michigan, United States CyberTec Full time

    Job Title: Azure Databricks Data Engineer - HybridLocation: Remote OpportunityJob Type: HybridDuration: 12 monthsOpening: 1 OpeningVisa Acceptable: All Visa TypesDuties: We seek a highly skilled Data Engineer with 5+ years of experience in PySpark optimization and development. The ideal candidate will have expertise in designing and developing data ingestion...

  • Cloud Systems Engineer

    3 months ago

    Maryland Line, United States SW Complete Full time

    Company Description Synergy ECP is a Service Disabled Veteran-Owned Small Business SD(VOSB) that was formed in July 2007 with Headquarters in Columbia, MD and is made up of talented, dedicated staff to provide a broad range of services to the defense, intelligence and health care industries. In an ultra-competitive environment, Synergy ECP has thrived by...

  • Cloud Data Architect

    3 weeks ago

    Maryland Heights, Missouri, United States NetEffects Full time

    Job Title: Cloud Data ArchitectWe are seeking a talented Cloud Data Architect to join our team at NetEffects. As a key member of our data engineering team, you will be responsible for designing and implementing scalable and secure cloud-based data pipelines and architectures.Your primary focus will be on building and maintaining cloud infrastructure,...

  • Senior Cloud FinOps Program Manager

    3 weeks ago

    Maryland Line, United States C4 Technical Services Full time

    Senior Cloud FinOps Consultant Remote Deliver new and iterative features to the Cloud FinOps reporting and data platform Leverage automation and scripting to democratize and deliver meaningful alerts, insights, anomalies, workflow, and other tools to be leveraged by multiple stakeholder personas to manage and optimize their cloud spend Senior Cloud FinOps...

  • AWS Cloud Engineer

    4 months ago

    Maryland Line, United States Barrow Wise Consulting, LLC Full time

    Enjoy problem-solving, need a venue to display your creativity, and emerging technologies pique your interest; if so, Barrow Wise Consulting, LLC is for you. As a multi-disciplined leader, you understand the gifts that set you apart from everyone else. Demonstrate innovative solutions to our clients. Join Barrow Wise Consulting, LLC today. Responsibilities: ...

  • Cloud Engineer

    4 weeks ago

    Maryland, United States Lockheed Martin Full time

    Unlock Your Potential as a Cloud EngineerJob SummaryWe are seeking a skilled Cloud Engineer - Java & Big Data Developer to join our team at Lockheed Martin. As a Cloud Engineer, you will be responsible for designing, developing, and deploying cloud-based systems and applications.Key ResponsibilitiesDevelop and maintain complex software systems, including...

  • Cloud Network Engineer

    1 month ago

    Maryland, United States LCG Full time

    Job SummaryThe Cloud Network Engineer, Junior is responsible for designing and implementing network systems and cloud architecture solutions that meet organizational requirements for performance, security, and compliance. This entry-level position involves supporting the design and implementation of cloud services, ensuring alignment with strategic goals and...

  • Data Scientist

    2 weeks ago

    Maryland Line, United States LCG Full time

    This job opportunity is part of an RFP process; candidates are invited to submit their resumes detailing relevant experience. Location: Bethesda, MD (Hybrid) LCG is a minority-owned technology consulting firm that has been a trusted partner to more than 40 federal agencies, including 21 of the 27 Institutes and Centers (ICs) at the National Institutes of...

  • Cloud Reliability Engineer

    4 weeks ago

    Maryland, United States LCG Full time

    Job Title: Cloud Reliability EngineerThis role involves managing and maintaining optimal platform infrastructure performance, reliability, and security through CI/CD tools and practices. The ideal candidate will have a background in cloud computing, automation, and performance optimization.Key Responsibilities:Manage and optimize platform infrastructure...

  • Cloud Cyber Security Specialist

    2 weeks ago

    Maryland Line, United States GovStaff Full time

    CLOUD CYBER SECURITY SPECIALIST - HYBRID, ABERDEEN PROVING GROUND, MD Candidates MUST be within commuting distance of Aberdeen Proving Ground, MD. Hybrid work arrangement with one to two days per week expected in the office at APG, MD. Now on to the details.....As part of several openings of ours in Aberdeen Proving Ground, MD, GovStaff is seeking a...

  • Senior Cloud Engineer

    4 weeks ago

    Maryland, United States LCG Full time

    Job Title: Senior Cloud EngineerThis is a unique opportunity to join LCG as a Senior Cloud Engineer. As a key member of our team, you will be responsible for designing, developing, and maintaining our cloud infrastructure. Your expertise in cloud computing, automation, and security will be instrumental in ensuring the performance, security, and reliability...

  • Data Engineer

    4 weeks ago

    Center Line, Michigan, United States FIS Full time

    About the Role:We are seeking a highly skilled Data Engineer to join our team at FIS. As a Data Engineer, you will be responsible for designing, developing, and maintaining cost-effective and high-performance data models, data architecture, and data integration pipelines in Snowflake.You will work with bleeding-edge technology to build innovative fintech...

  • Golang Data Engineer

    1 month ago

    maryland heights, United States neteffects Full time

    Golang Data Engineer (with Data intensive environment experience)Fully REMOTEW2 Candidates Only Please.......we can not use C2C for this positionVery Long Term Contract position (starts as 12+ months, but will most likely be a few years in durationRate: $85-90/hr. W2 with benefits/PTOGolang Data Engineer (with Data intensive environment experience)Team:...

  • Golang Data Engineer

    1 month ago

    Maryland Heights, United States neteffects Full time

    Golang Data Engineer (with Data intensive environment experience)Fully REMOTEW2 Candidates Only Please.......we can not use C2C for this positionVery Long Term Contract position (starts as 12+ months, but will most likely be a few years in durationRate: $85-90/hr. W2 with benefits/PTOGolang Data Engineer (with Data intensive environment experience)Team:...

  • Golang Data Engineer

    3 weeks ago

    maryland heights, United States neteffects Full time

    Golang Data Engineer (with Data intensive environment experience)Fully REMOTEW2 Candidates Only Please.......we can not use C2C for this positionVery Long Term Contract position (starts as 12+ months, but will most likely be a few years in durationRate: $85-90/hr. W2 with benefits/PTOGolang Data Engineer (with Data intensive environment experience)Team:...

  • Senior Cloud Software Engineer

    4 weeks ago

    Center Line, Michigan, United States Fast Switch Full time

    Job Title: Senior Cloud Software EngineerFast Switch is seeking a highly skilled Senior Cloud Software Engineer to join our team. As a key member of our cloud engineering team, you will be responsible for designing, developing, and deploying cloud-based applications using Java and cloud platforms such as GCP/PCF.Key Responsibilities:Design and develop...

  • Data Scientist I

    2 weeks ago

    Maryland Line, United States Chiron Technology Services Full time

    OverviewChiron Technology Services, Inc. currently has a Data Scientist I career opportunity in the Annapolis Junction, MD area.Category: Data Scientist I Clearance: Top Secret/SCI with Polygraph Capabilities:Produce data visualizations that provide insight into dataset structure and meaning Develop machine learning, data mining, statistical and graph-based...

  • DevOps Engineer

    3 weeks ago

    Maryland Line, United States Infinite Computer Solutions Full time

    Job descriptionDevOps Engineer will be working on setting up the CI/CD Pipeline and workflows for all of the development, test automation teams using Azure DevOps build, release pipelines. Extensive knowledge of Terraform, AWS EKS, Azure DevOps pipelines. AWS Cloud Expertise, AWS Networking, Cloud Security, Maven, Monitoring, Cloud Cost Management Git,...

  • GCP Engineer

    4 months ago

    Maryland Line, United States Photon Full time

    Job DescriptionResponsibilities: Infrastructure as Code (IaC): Design, implement, and manage infrastructure as code using Terraform for GCP environments. Ensure infrastructure configurations are scalable, reliable, and follow best practices. GCP Platform Management: Architect and manage GCP environments, including compute, storage, and networking...

  • Cloud FinOps Program Manager

    4 weeks ago

    Maryland, United States C4 Technical Services Full time

    Cloud FinOps Program ManagerAt C4 Technical Services, we are seeking a highly skilled Cloud FinOps Program Manager to join our team. The ideal candidate will have a strong background in cloud cost management and experience with public cloud service providers. Key responsibilities include delivering new and iterative features to the cloud FinOps reporting and...