Lead Data Engineer

3 days ago


Philadelphia, Pennsylvania, United States Softcrylic Full time
About the Role

We are seeking a highly skilled Technical Lead - Data Engineering with a strong background in building and optimizing data pipelines and transformation processes using Azure Databricks, Python, and Spark.

The ideal candidate will have the ability to understand and convert Scala programs to PySpark within Databricks and possess hands-on experience with Azure Data Factory.

Key Responsibilities
  • Design, develop, and optimize data pipelines and transformation processes using Azure Databricks and ADF.
  • Understand and convert existing Scala programs to PySpark, ensuring seamless integration within Azure Databricks.
  • Collaborate with cross-functional teams to gather and translate business requirements into technical solutions.
  • Manage and utilize Azure Data Lake Gen 2 for efficient data storage and processing, ensuring data integrity and security.
  • Implement and optimize data warehousing solutions using Synapse Analytics, focusing on dedicated SQL pools.
  • Leverage Delta Lake to manage large-scale data workflows, ensuring high performance and reliability.
  • Provide technical leadership and guidance to junior engineers, fostering a collaborative and innovative environment.
  • Ensure data solutions align with best practices, compliance requirements, and industry standards, particularly in the healthcare domain, if applicable.
Primary Skills
  • Azure Databricks: Expertise in building and managing data pipelines using Azure Databricks, with a strong focus on leveraging Python and Spark.
  • Data Transformation: Proven experience in developing data transformation processes within Azure Databricks.
  • Scala to PySpark Conversion: Ability to understand and convert Scala programs to PySpark programs within Databricks.
  • Azure Data Factory (ADF): Hands-on experience in designing and implementing ETL pipelines using Azure Data Factory.
Secondary Skills
  • Azure Data Lake Gen 2: Proficiency in managing and utilizing Azure Data Lake Gen 2 for data storage and processing.
  • Synapse Analytics (Dedicated SQL Pool): Experience with Synapse Analytics, particularly with dedicated SQL pools for data warehousing and analytics.
  • Delta Lake: Expertise in working with Delta Lake for managing and optimizing big data workflows.
  • PySpark: Advanced skills in PySpark for data processing and transformation tasks.

  • Lead Data Engineer

    2 days ago


    Philadelphia, Pennsylvania, United States Cadent Full time

    Cadent Powers the Evolution of TV Brand AdvertisingCadent is a leading provider of data-driven solutions for buying and selling TV advertising. Our technology improves efficiencies and boosts the results of linear, addressable, and cross-screen campaigns.We are seeking a self-motivated and talented Lead Data Engineer to join our team responsible for...

  • GCP Data Lead

    4 weeks ago


    Philadelphia, Pennsylvania, United States Marshall Technologies Full time

    Job Title: GCP Data LeadWe are seeking an experienced professional to lead our team of data engineers in leveraging Google Cloud Platform (GCP) services to develop and deploy scalable data solutions. The ideal candidate will have expertise in GCP tools, including BigQuery, Dataflow, and Apache Airflow, as well as experience in data migration, data warehouse...

  • Senior Data Engineer

    3 weeks ago


    Philadelphia, Pennsylvania, United States InsideHigherEd Full time

    Job Title: Senior Data EngineerWe are seeking a highly skilled Senior Data Engineer to join our team at the University of Pennsylvania. As a Senior Data Engineer, you will be responsible for leading multi-disciplinary data analysis efforts and providing expertise in data engineering and analytics.About the Role:Develop and apply advanced data analysis...

  • Data Engineering Lead

    4 weeks ago


    Philadelphia, Pennsylvania, United States Avance Consulting Full time

    Job DescriptionAvance Consulting is seeking a highly skilled Technical Lead to join our team.Key Responsibilities:Design and implement data pipelines using Azure Databricks, leveraging Python and Spark.Develop data transformation processes in ADB.Convert Scala programs to PySpark programs.Understand and work with Scala programs in DataBricks.Secondary...

  • Senior Data Engineer

    3 weeks ago


    Philadelphia, Pennsylvania, United States PENN Interactive Full time

    About the RolePenn Interactive is a leading interactive gaming company that is revolutionizing the gaming industry with its innovative and immersive experiences. We are seeking a highly skilled Senior Data Engineer to join our team and contribute to the development of high-quality and sustainable data-driven solutions.As a key member of our Data Engineering...


  • Philadelphia, Pennsylvania, United States Publicis Health Media Full time

    Job Title: Associate Director, Data Engineering, Technology and DataAt Publicis Health Media, we're seeking an experienced Associate Director, Data Engineering, Technology and Data to join our team. As a key member of our data engineering team, you will play a critical role in designing and developing big data solutions built on the Databricks and Azure...


  • Philadelphia, Pennsylvania, United States Publicis Health Media Full time

    Job Title: Data Engineering ManagerPublicis Health Media is seeking a highly skilled Data Engineering Manager to join our Technology and Data Department. As a key member of our team, you will be responsible for overseeing the processing and maintenance of all data that flows into our organization.Key Responsibilities:Lead a portfolio of data projects across...

  • Data Engineer

    3 weeks ago


    Philadelphia, Pennsylvania, United States rockITdata Full time

    Job Title: Full Stack Data EngineerWe are seeking a highly skilled Full Stack Data Engineer to join our team at rockITdata. As a key member of our team, you will be responsible for designing, implementing, and maintaining scalable data solutions that meet the needs of our clients.Key Responsibilities:Data Ingestion and Integration:Design and implement data...

  • Data Engineer

    4 weeks ago


    Philadelphia, Pennsylvania, United States Constellation Search Group Full time

    Data Engineer Job DescriptionWe are seeking a skilled Data Engineer to join our dynamic team at Constellation Search Group. As a Data Engineer, you will play a crucial role in building and maintaining scalable data pipelines and ensuring seamless data integration, storage, and processing using the latest cloud technologies.Key Responsibilities:Develop and...

  • Project Manager

    3 days ago


    Philadelphia, Pennsylvania, United States Data Intelligence, LLC Full time

    Job Title: Project ManagerData Intelligence, LLC is seeking a highly skilled Project Manager to support our government clients in Philadelphia, PA. This is a full-time position that may be remote.Job Responsibilities:Develop and execute project plans to ensure timely completion of tasksAssess resource requirements and estimate costsCreate project schedules...


  • Philadelphia, Pennsylvania, United States Data Intelligence Full time

    Job Title: Senior Cloud Software EngineerData Intelligence, LLC is seeking a highly skilled Senior Cloud Software Engineer to join our team in Philadelphia, PA. As a key member of our software development team, you will be responsible for designing, developing, and deploying cloud-based software systems.Job Responsibilities:Provide full software development...


  • Philadelphia, Pennsylvania, United States Publicis Health Media Full time

    Job Title: Data Engineering ManagerPublicis Health Media is seeking a highly skilled Data Engineering Manager to join our Technology and Data Department. As a key member of our team, you will be responsible for overseeing the processing and maintenance of all data that flows into our organization.The ideal candidate will have a strong background in data...


  • Philadelphia, Pennsylvania, United States Pennsylvania Medicine Full time

    Penn Medicine is a leading academic medical center dedicated to providing the highest level of care to patients, conducting innovative research, and educating future leaders in the field of medicine.As a Data Engineer, you will work as part of a team to design, develop, and maintain robust data pipelines, ETL/ELT processes, and data transformation...

  • Data Engineer

    25 minutes ago


    Philadelphia, Pennsylvania, United States Mondo Full time

    Job OverviewMondo is seeking a highly skilled Data Engineer to join our experienced data engineering team. As a key contributor, you will design and develop enterprise-level applications using Snowflake and AWS tools.Key ResponsibilitiesDesign and develop scalable data applications using Snowflake and AWS tools.Take ownership of new data connection...

  • Senior Data Engineer

    4 hours ago


    Philadelphia, Pennsylvania, United States ARCADIS Full time

    Role OverviewAs a Senior Data Engineer at Arcadis, you will play a key role in designing and developing data solutions that drive business value. You will work closely with cross-functional teams to deliver high-quality data migration and integration projects, leveraging your expertise in data architecture, data modeling, and data engineering.Key...

  • Senior Data Engineer

    3 weeks ago


    Philadelphia, Pennsylvania, United States InsideHigherEd Full time

    Job Title: Senior Data EngineerJoin the Perelman School of Medicine at the University of Pennsylvania as a Senior Data Engineer and contribute to the development of innovative data analysis techniques for large-scale neuroimaging studies. As a member of the Lifespan Informatics and Neuroimaging Center (PennLINC), you will work closely with a...


  • Philadelphia, Pennsylvania, United States Generis Tek Inc Full time

    Job Title: Data Analytics EngineerWe are seeking a highly skilled Data Analytics Engineer to join our team at Generis Tek Inc. As a Data Analytics Engineer, you will be responsible for designing, developing, and maintaining large-scale data systems and analytics solutions.Key Responsibilities:Design and develop data architectures and analytics solutions...

  • Data Engineer

    4 hours ago


    Philadelphia, Pennsylvania, United States Perpay - Career's Page Full time

    About Us:We're a financial technology company dedicated to creating inclusive financial products that help our members improve their financial futures. Our mission is to build simple and accessible financial solutions that promote healthy financial habits and economic stability.Our Products:We've developed innovative products like Perpay Marketplace,...

  • Azure Data Engineer

    3 weeks ago


    Philadelphia, Pennsylvania, United States Tetrahed INC Full time

    Job Title: Azure Data EngineerWe are seeking a highly skilled Azure Data Engineer to join our team at Tetrahed INC. As a key member of our data engineering team, you will be responsible for designing, building, and maintaining large-scale data pipelines using Azure Data Factory.Key Responsibilities:Design and implement data pipelines using Azure Data...

  • Senior Data Engineer

    3 weeks ago


    Philadelphia, Pennsylvania, United States Comcast Corporation Full time

    Job Title: Senior Data EngineerComcast Corporation is seeking a highly skilled Senior Data Engineer to join our team in Philadelphia. As a Senior Data Engineer, you will be responsible for designing, developing, and maintaining large-scale data platforms using Neo4j and other technologies.Key Responsibilities:Design and develop big data ETL pipelines using...