Data Engineer

18 hours ago

San Diego, United States LanceSoft Full time
Title: Senior Software Engineer/Data Engineer
Location: Remote
Duration: 6 Months (Temp to perm)

Description:

About the role:
  • As the Senior Software Engineer, you will lead a team of data engineers in designing, building, and maintaining high-performance software system to manage analytical data pipelines that fuel the organization's data strategy using software engineering best practices. Beyond technical expertise, you will also serve as a change leader, guiding teams through adopting new tools, technologies, and workflows to improve data management and processing.
  • This position requires extensive hands-on data system design and coding experience, as well as the development of modern data pipelines (AWS Step functions, Prefect, Airflow, Luigi, Python, Spark, SQL) and associated code in AWS.
  • You will work closely with stakeholders across the business to understand their data needs, ensure scalability, and foster a culture of innovation and learning within the data engineering team and beyond.

Key Responsibilities:
• Be responsible for the overall architecture of a specific module within a product (e.g., Data-ingestion, near-real-time-data-processor, etc.), perform design and assist implementation considering system characteristics to produce optimal performance, reliability and maintainability.
• Provide technical guidance to team members, ensuring they are working towards the product's architectural goals.
• Create and manage RFCs (Request for Comments) and ADRs (Architecture Decision Records), Design notes and technical documentation for your module, following the architecture governance processes.
• Lead a team of data engineers, providing mentorship, setting priorities, and ensuring alignment with business goals.
• Architect, design, and build scalable data pipelines for processing large volumes of structured and unstructured data from various sources.
• Collaborate with software engineers, architects, and product teams to design and implement systems that enable real-time and batch data processing at scale.
• Be the go-to person for PySpark-based solutions, ensuring optimal performance and reliability for distributed data processing.
• Ensure that data engineering systems adhere to the best data security, privacy, and governance practices in line with industry standards.
• Perform code reviews for the product, ensuring adherence to company coding standards and best practices.
• Develop and implement monitoring and alerting systems to ensure timely detection and resolution of data pipeline failures and performance bottlenecks.
• Act as a champion for new technologies, helping ease transitions and addressing concerns or resistance from team members.

Ideal Candidate:
• Experience leading a data engineering team with a strong focus on software engineering principles such as KISS, DRY, YAGNI etc.
• Candidate MUST have experience in owning large, complex system architecture and hands-on experience designing and implementing data pipelines across large-scale systems.
• Experience implementing and optimizing data pipelines with AWS is a must.
• Production delivery experience in Cloud-based PaaS Big Data related technologies (EMR, Snowflake, Data bricks etc.)
• Experienced in multiple Cloud PaaS persistence technologies, and in-depth knowledge of cloud- based ETL offerings and orchestration technologies (AWS Step Function, Airflow etc.)
• Experienced in stream-based and batch processing, applying modern technologies
• Working experience with distributed file systems (S3, HDFC, ADLS), table formats (HUDI, Iceberg), and various open file formats (JSON, Parquet, Csv, etc.)
• Strong programming experience in PySpark, SQL, Python, etc.
• Database design skills including normalization/de-normalization and data warehouse design
• Knowledge and understanding of relevant legal and regulatory requirements, such as SOX, PCI, HIPAA, Data Protection
• Experience in the healthcare industry, a plus
• A collaborative and informative mentality is a must

Toolset:
• AWS, preferably AWS certified Data Engineer and AWS certified Solutions Architect.
• Proficiency in at least one programming language C#, GoLang, JavaScript or ReactJs
• Spark / Python / SQL
• Snowflake/ Databricks / Synapse / MS SQL Server
• ETL / Orchestration Tools (Step Function, DBT etc.)
• ML / Notebooks

Education and experience required
• Bachelors or Master's in Computer Science, Information Systems, or an engineering field or relevant experience.
• 10+ years of related experience in developing data solutions and data movement.

  • Senior Data Engineer

    4 weeks ago

    san diego, United States BioPhase Full time

    Sr. Data EngineerNo C2COn-site/hybrid role, must be in San Diego, CA. Position OverviewThe Sr. Data Engineer will play a pivotal role in designing, implementing, and maintaining data pipelines that process and analyze large volumes of instrument data, clinical biomarker inputs and patient information. The Sr. Data Engineer work will directly contribute to...

  • Senior Data Engineer

    1 month ago

    san diego, United States BioPhase Full time

    Sr. Data EngineerNo C2COn-site/hybrid role, must be in San Diego, CA. Position OverviewThe Sr. Data Engineer will play a pivotal role in designing, implementing, and maintaining data pipelines that process and analyze large volumes of instrument data, clinical biomarker inputs and patient information. The Sr. Data Engineer work will directly contribute to...

  • Senior Data Engineer

    1 month ago

    San Diego, United States BioPhase Full time

    Sr. Data EngineerNo C2COn-site/hybrid role, must be in San Diego, CA. Position OverviewThe Sr. Data Engineer will play a pivotal role in designing, implementing, and maintaining data pipelines that process and analyze large volumes of instrument data, clinical biomarker inputs and patient information. The Sr. Data Engineer work will directly contribute to...

  • Data Engineer

    4 months ago

    San Diego, United States Diverse Lynx Full time

    Role: Data Engineer - GCP, python core data engineer Contract Location : San Diego, CA Onsite day 1 Description: Essential Skills: Strong SQL and PythonExperience with ETL/ELT, data integrationsExperience with Big Query and handling large data sets Desirable Skills: Strong SQL and PythonExperience with ETL/ELT, data integrationsExperience with Big Query and...

  • Data Engineer

    14 hours ago

    San Diego, United States SteerBridge Full time

    Position OverviewSteerBridge is seeking a Data Engineer for its AI/ML Spares Project, which focuses on transforming supply chain management in the aerospace industry using advanced analytics. This role involves optimizing data systems and collaborating with squadron-level Marines to improve data entry, management, and extraction.Qualifications U.S....

  • Data Engineer

    2 weeks ago

    San Diego, United States SteerBridge Full time

    Position OverviewSteerBridge is seeking a Data Engineer for its AI/ML Spares Project, which focuses on transforming supply chain management in the aerospace industry using advanced analytics. This role involves optimizing data systems and collaborating with squadron-level Marines to improve data entry, management, and extraction.Qualifications• U.S....

  • Data Engineer

    2 weeks ago

    san diego, United States SteerBridge Full time

    Position OverviewSteerBridge is seeking a Data Engineer for its AI/ML Spares Project, which focuses on transforming supply chain management in the aerospace industry using advanced analytics. This role involves optimizing data systems and collaborating with squadron-level Marines to improve data entry, management, and extraction.Qualifications• U.S....

  • Senior Data Engineer

    1 month ago

    San Diego, California, United States AppFolio, Inc Full time

    About the RoleWe are seeking a highly skilled Sr. Data Engineer to join our growing Data Engineering and Operations team at AppFolio, Inc.The successful candidate will design, build, and operate next-generation data pipeline infrastructure based on Apache Kafka and its ecosystem.Key Responsibilities:Design and implement scalable and fault-tolerant data...

  • Data Engineer II

    4 weeks ago

    San Diego, United States Pelican Staffing Remote Work Freelance Full time $78 - $123

    Summary:We are a product led organization transforming the way we work from how easy it is to use our IT services to the applications we develop that help us scale seamlessly in face of incredible growth.The Data Engineer contributes in technological decisions for business teams' future data and analysis needs. You will support the business's daily...

  • Data Engineer II

    3 weeks ago

    San Diego, United States The Employee Connect Full time

    Summary We are a product led organization transforming the way we work from how easy it is to use our IT services to the applications we develop that help us scale seamlessly in face of incredible growth. The Data Engineer contributes in technological decisions for business teams future data and analysis needs. You will support the businesss daily...

  • Principal Data Engineer/ Architect

    19 hours ago

    San Diego, United States LanceSoft Full time

    Principal Data Engineer/ Architect Temp to perm Full time - Pay $70 ot $100/hr based on experience Remote As the Senior Software Engineer, you will lead a team of data engineers in designing, building, and maintaining high-performance software system to manage analytical data pipelines that fuel the organization's data strategy using software engineering...

  • Data Engineer with Scalable Data Pipeline Expertise

    4 weeks ago

    San Diego, California, United States Empirical Full time

    About the RoleWe are seeking a highly motivated Senior Data Engineer to join our client’s fast-paced data team. The ideal candidate will work closely with cross-functional teams to develop scalable data pipelines, optimize data workflows, and ensure data quality and reliability.The OpportunityDevelop scalable data pipelines and optimize data...

  • Senior Data Engineer

    2 weeks ago

    san diego, United States Zenith Services Inc. Full time

    Hi,Job Title: Lead Data Engineer (PST Preferred)W2/FTC/FTE OnlyExp: 12+ YearsLocation: San Diego, CA (Remote)Visa: Except OPT/CPTReach Me: maheshp@zenithcad.com

  • Senior Data Engineer

    2 weeks ago

    san diego, United States Zenith Services Inc. Full time

    Hi,Job Title: Lead Data Engineer (PST Preferred)W2/FTC/FTE OnlyExp: 12+ YearsLocation: San Diego, CA (Remote)Visa: Except OPT/CPTReach Me: maheshp@zenithcad.com

  • Senior Data Engineer

    2 weeks ago

    San Diego, United States Zenith Services Inc. Full time

    Hi,Job Title: Lead Data Engineer (PST Preferred)W2/FTC/FTE OnlyExp: 12+ YearsLocation: San Diego, CA (Remote)Visa: Except OPT/CPTReach Me: maheshp@zenithcad.com

  • SRX - Sr Data Engineer

    13 hours ago

    San Diego, United States Empirical Full time

    IMPORTANT: At the moment we are only considering candidates located in Latin America. Please do not apply if you are not located in Latin America. About Empirical Empirical empowers early and growth stage tech companies to create the right products in the right way by providing two pivotal services: Product & Technology Leadership: Our clients entrust...

  • DC Design Manager for Region, Data Center Engineering

    2 weeks ago

    San Francisco, United States Amazon Data Services, Inc. Full time

    This unique vantage point allows the Design Managers to maintain a close connection with project details and drive project outcomes.You will have an impact on the design direction and ability to improve design delivery for an entire region, while establishing design procedures and protocols for the development of future systems.Responsibilities...

  • Software Engineer with Proficiency in Data Structures

    4 weeks ago

    San Diego, California, United States Mossy Ford Full time

    Job Title: Software Engineer with Proficiency in Data StructuresWe are seeking a skilled Software Engineer with expertise in data structures to join our team at {company}. As a key member of our development team, you will be responsible for designing and implementing efficient data structures to support our software applications.Your primary duties will...

  • Experienced Data Scientist Machine Learning

    6 minutes ago

    San Francisco, United States Data Masked Full time

    Experienced Data Scientist Machine Learning - Identity Verification We believe that the way people interact with their finances will drastically improve in the next few years. We’re dedicated to empowering this transformation by building the tools and experiences that thousands of developers use to create their own products. Plaid powers the tools millions...

  • Azure Data Engineer

    1 month ago

    san diego, United States Avantiico Full time

    The RoleThe Data Engineer's primary responsibility will be to provide technical expertise and coordinate day-to-day deliverables with the consulting team both for Avantiico clients as well as internal initiatives. As an independent contributor, this position should be able to develop and support Microsoft Fabric initiatives using Microsoft tools such as...