Principal Data Engineer/ Architect

1 week ago


San Diego, United States LanceSoft Full time

Principal Data Engineer/ Architect
Temp to perm
Full time - Pay $70 ot $100/hr based on experience
Remote

As the Senior Software Engineer, you will lead a team of data engineers in designing, building, and maintaining high-performance software system to manage analytical data pipelines that fuel the organization's data strategy using software engineering best practices. Beyond technical expertise, you will also serve as a change leader, guiding teams through adopting new tools, technologies, and workflows to improve data management and processing.

This position requires extensive hands-on data system design and coding experience, as well as the development of modern data pipelines (AWS Step functions, Prefect, Airflow, Luigi, Python, Spark, SQL) and associated code in AWS.

You will work closely with stakeholders across the business to understand their data needs, ensure scalability, and foster a culture of innovation and learning within the data engineering team and beyond.

Key Responsibilities:
• Be responsible for the overall architecture of a specific module within a product (e.g., Data-ingestion, near-real-time-data-processor, etc.), perform design and assist implementation considering system characteristics to produce optimal performance, reliability and maintainability.
• Provide technical guidance to team members, ensuring they are working towards the product's architectural goals.
• Create and manage RFCs (Request for Comments) and ADRs (Architecture Decision Records), Design notes and technical documentation for your module, following the architecture governance processes.
• Lead a team of data engineers, providing mentorship, setting priorities, and ensuring alignment with business goals.
• Architect, design, and build scalable data pipelines for processing large volumes of structured and unstructured data from various sources.
• Collaborate with software engineers, architects, and product teams to design and implement systems that enable real-time and batch data processing at scale.
• Be the go-to person for PySpark-based solutions, ensuring optimal performance and reliability for distributed data processing.
• Ensure that data engineering systems adhere to the best data security, privacy, and governance practices in line with industry standards.
• Perform code reviews for the product, ensuring adherence to company coding standards and best practices.
• Develop and implement monitoring and alerting systems to ensure timely detection and resolution of data pipeline failures and performance bottlenecks.
• Act as a champion for new technologies, helping ease transitions and addressing concerns or resistance from team members.

Ideal Candidate:
• Experience leading a data engineering team with a strong focus on software engineering principles such as KISS, DRY, YAGNI etc.
• Candidate MUST have experience in owning large, complex system architecture and hands-on experience designing and implementing data pipelines across large-scale systems.
• Experience implementing and optimizing data pipelines with AWS is a must.
• Production delivery experience in Cloud-based PaaS Big Data related technologies (EMR, Snowflake, Data bricks etc.)
• Experienced in multiple Cloud PaaS persistence technologies, and in-depth knowledge of cloud- based ETL offerings and orchestration technologies (AWS Step Function, Airflow etc.)
• Experienced in stream-based and batch processing, applying modern technologies
• Working experience with distributed file systems (S3, HDFC, ADLS), table formats (HUDI, Iceberg), and various open file formats (JSON, Parquet, Csv, etc.)
• Strong programming experience in PySpark, SQL, Python, etc.
• Database design skills including normalization/de-normalization and data warehouse design
• Knowledge and understanding of relevant legal and regulatory requirements, such as SOX, PCI, HIPAA, Data Protection
• Experience in the healthcare industry, a plus
• A collaborative and informative mentality is a must

Toolset:
• AWS, preferably AWS certified Data Engineer and AWS certified Solutions Architect.
• Proficiency in at least one programming language C#, GoLang, JavaScript or ReactJs
• Spark / Python / SQL
• Snowflake/ Databricks / Synapse / MS SQL Server
• ETL / Orchestration Tools (Step Function, DBT etc.)
• ML / Notebooks

Education and experience required
• Bachelors or Master's in Computer Science, Information Systems, or an engineering field or relevant experience.
• 10+ years of related experience in developing data solutions and data movement.

This role can be REMOTE



  • San Diego, United States ASML Full time

    The Principal Data Science/AI Architect will be part of the EUV Source Performance team within ASML San Diego. Due to high complexity of the EUV light source, we, as Source Performance, are a dedicated team with the mission to validate and ensure that EUV sources meet or exceed customer performance requirements and expectations. In order to execute this...


  • San Diego, United States Intuit Full time

    Overview Intuit is hiring a Principal Data Science Architect to join our Intuit AI team. Come join our collaborative and creative group of data scientists and machine learning engineers and design and build AI systems that directly affect hundreds of thousands of our customers. In this role, you will be designing, building, and deploying machine learning...

  • Data Architect

    7 days ago


    San Diego, California, United States LanceSoft Full time

    {"title": "Senior Software Engineer/Data Architect", "description": "LanceSoftAbout the Role:We are seeking a highly skilled Senior Software Engineer/Data Architect to lead our data engineering team in designing, building, and maintaining high-performance software systems to manage analytical data pipelines that fuel our organization's data strategy using...


  • San Francisco, California, United States Gridware Full time

    At Gridware, we're seeking a skilled Principal Data Infrastructure Architect to join our team. This role plays a pivotal part in designing, building, and optimizing data pipelines and infrastructure to support real-time grid monitoring systems.The successful candidate will be responsible for handling large-scale sensor data, ensuring data integrity, and...


  • San Jose, California, United States Adobe Full time

    About the RoleAdobe is seeking a highly skilled Principal Data Scientist to join our AI Engineering team. This individual will play a key role in designing, architecting, and building data insight dashboards and visualizations.The ideal candidate will have expertise in one or more data science tools such as Pandas, Numpy, Octave, R, and experience building...


  • San Diego, California, United States MILLENNIUMSOFT Full time

    MILLENNIUMSOFT is seeking a highly skilled Principal Database Architect to join our team in San Diego, CA.The estimated salary for this position is $125,000 per year, based on industry standards and location.Job Description:We are looking for a talented Principal Database Architect to lead the design and development of database systems that meet the needs of...


  • San Diego, California, United States CoStar Group, Inc. Full time

    CoStar Group, Inc., a leading provider of commercial and residential real estate information, analytics, and online marketplaces, is seeking an experienced Principal Software Architect to join its Product Data Services team.The ideal candidate will have a strong background in designing and developing scalable, distributed systems using Amazon Web Services...

  • Cloud Architect

    7 days ago


    San Diego, California, United States LanceSoft Full time

    Job Title: Cloud Architect - Data SpecialistWe are seeking a highly skilled Cloud Architect - Data Specialist to join our team at LanceSoft.Key Responsibilities:Design and implement scalable data pipelines for processing large volumes of structured and unstructured data from various sources.Collaborate with software engineers, architects, and product teams...


  • San Diego, California, United States Northrop Grumman Full time

    We are seeking a highly skilled Principal Systems Engineer or Senior Principal Systems Engineer to join our team in San Diego, CA. This role will play a critical part in defining, developing, implementing, and transitioning new technology solutions to increase efficiency, reduce cost, and improve quality.About the RoleThis is an exciting opportunity for a...

  • AWS Data Architect

    4 weeks ago


    San Diego, United States Indotronix International Corporation Full time

    Job Title: AWS Data EngineerLocation: Remote (San Diego, CA 92130)Duration: 6 months (possibility of extension or conversion)Job Description:As the Senior Software Engineer, you will lead a team of data engineers in designing, building, and maintaining high-performance software system to manage analytical data pipelines that fuel the organization’s data...

  • AWS Data Architect

    4 weeks ago


    san diego, United States Indotronix International Corporation Full time

    Job Title: AWS Data EngineerLocation: Remote (San Diego, CA 92130)Duration: 6 months (possibility of extension or conversion)Job Description:As the Senior Software Engineer, you will lead a team of data engineers in designing, building, and maintaining high-performance software system to manage analytical data pipelines that fuel the organization’s data...


  • San Diego, United States Intuit Full time

    Overview Intuit is looking for a highly motivated and experienced Principal Software Engineer to join the AI Synapse team. Our charter is to build AI/Machine Learning solutions for Intuit’s suite of financial products that drive quantifiable customer benefit through the state-of-the-art Large Language Models (LLMs) and Multimodal Language Models. You...


  • San Jose, United States Analog Devices, Inc. Full time

    Analog Devices, Inc. (NASDAQ: ADI) is a global semiconductor leader that bridges the physical and digital worlds to enable breakthroughs at the Intelligent Edge. ADI combines analog, digital, and software technologies into solutions that help drive advancements in digitized factories, mobility, and digital healthcare, combat climate change, and reliably...


  • San Francisco, California, United States USM Business Systems Full time

    USM Business SystemsSenior Data Modeler/Data Architect + Big Data/Hadoop OpportunityLocation: Nationwide (Remote)Contract Length: 6+ MonthsEstimated Salary Range: $120,000 - $180,000 per yearAbout the Role:The Senior Data Modeler/Data Architect will be an integral member of the Digital Insights & Architecture team within USM Business Systems. This highly...


  • San Ramon, California, United States AHEAD Full time

    Principal Technical ArchitectAHEAD is looking for a highly skilled Principal Technical Architect to lead the development of innovative platform engineering solutions. This role requires a strong technical background, leadership abilities, and excellent communication skills.Key ResponsibilitiesDevelop and implement complex technical solutions for...


  • San Jose, California, United States Tik Tok Full time

    About the RoleAs a Principal Software Architect for Risk Engineering at TikTok, you will play a crucial role in building industry-leading risk control systems that prioritize user experience and establish trust between customers and business service providers.Job ResponsibilitiesDesign and develop highly scalable systems, including ad fraud detection, risk...


  • San Diego, United States Itility Full time

    WHO WE AREWe believe in merging technology and data to drive our customers one step beyond. Itility digital consultants are experts in data, cloud, software and IT infrastructure. Acting as the ‘digital twin’ of customers, we work shoulder-to-shoulder to exceed business goals and push the boundaries of what you thought was possible. We combine an agile...

  • Data Engineer

    1 week ago


    San Diego, United States LanceSoft Full time

    Title: Senior Software Engineer/Data Engineer Location: Remote Duration: 6 Months (Temp to perm) Description: About the role: As the Senior Software Engineer, you will lead a team of data engineers in designing, building, and maintaining high-performance software system to manage analytical data pipelines that fuel the organization's data strategy using...


  • San Francisco, United States Oleria Corp. Full time

    OverviewOleria is an enterprise cybersecurity startup founded by industry veterans, backed by over $43M in funding. We're on a mission to transform access control for enterprise cloud applications, using cutting-edge AI and graph technology to combat identity-based attacks and data breaches.Oleria was founded by notable industry senior leaders Jim Alkove and...


  • San Diego, United States Eli Lilly and Company Full time

    At Lilly, we unite caring with discovery to make life better for people around the world. We are a global healthcare leader headquartered in Indianapolis, Indiana. Our employees around the world work to discover and bring life-changing medicines to those who need them, improve the understanding and management of disease, and give back to our communities...