Lead Data Engineer

13 hours ago


philadelphia, United States Softcrylic Full time

Who We Are

For more than 20 years, we have been working with organizations large and small to help solve business challenges through technology. We bring a unique combination of engineering and strategy to Make Data Work for organizations.


Our clients range from the travel and leisure industry to publishing, retail and banking. The common thread between our clients is their commitment to making data work as seen through their investment in those efforts.


In our quest to solve data challenges for our clients, we work with large enterprise, cloud based and marketing technology suites. We have a deep understanding of these solutions so we can help our clients make the most of their investment in an efficient way to have a data driven business.


Softcrylic now joins forces with Hexaware to Make Data Work in bigger ways


Why Work at Softcrylic?

Softcrylic provides an engaging, team-focused, and rewarding work environment where people are excited about the work they do and passionate about delivering creative solutions to our clients.


We are looking to add a Lead Data Engineer - Azure Databricks & ADF to our team


Job Summary:


We are looking for a highly skilled Technical Lead - Data Engineering with a strong background in building and optimizing data pipelines and transformation processes using Azure Databricks, Python, and Spark. The ideal candidate will have the ability to understand and convert Scala programs to PySpark within Databricks and possess hands-on experience with Azure Data Factory.


Key Responsibilities:


  • Lead the design, development, and optimization of data pipelines and transformation processes using Azure Databricks and ADF.
  • Understand and convert existing Scala programs to PySpark, ensuring seamless integration within Azure Databricks.
  • Collaborate with cross-functional teams to gather and translate business requirements into technical solutions.
  • Manage and utilize Azure Data Lake Gen 2 for efficient data storage and processing, ensuring data integrity and security.
  • Implement and optimize data warehousing solutions using Synapse Analytics, focusing on dedicated SQL pools.
  • Leverage Delta Lake to manage large-scale data workflows, ensuring high performance and reliability.
  • Provide technical leadership and guidance to junior engineers, fostering a collaborative and innovative environment.
  • Ensure data solutions align with best practices, compliance requirements, and industry standards, particularly in the healthcare domain, if applicable.


Primary Skills:


  • Azure Databricks: Expertise in building and managing data pipelines using Azure Databricks, with a strong focus on leveraging Python and Spark.
  • Data Transformation: Proven experience in developing data transformation processes within Azure Databricks.
  • Scala to PySpark Conversion: Ability to understand and convert Scala programs to PySpark programs within Databricks.
  • Azure Data Factory (ADF): Hands-on experience in designing and implementing ETL pipelines using Azure Data Factory.


Secondary Skills:


  • Azure Data Lake Gen 2: Proficiency in managing and utilizing Azure Data Lake Gen 2 for data storage and processing.
  • Synapse Analytics (Dedicated SQL Pool): Experience with Synapse Analytics, particularly with dedicated SQL pools for data warehousing and analytics.
  • Delta Lake: Expertise in working with Delta Lake for managing and optimizing big data workflows.
  • PySpark: Advanced skills in PySpark for data processing and transformation tasks.


  • Lead Data Engineer

    2 days ago


    Philadelphia, Pennsylvania, United States Cadent Full time

    Cadent Powers the Evolution of TV Brand AdvertisingCadent is a leading provider of data-driven solutions for buying and selling TV advertising. Our technology improves efficiencies and boosts the results of linear, addressable, and cross-screen campaigns.We are seeking a self-motivated and talented Lead Data Engineer to join our team responsible for...


  • Philadelphia, United States US Tech Solutions Full time

    Role: Lead Databricks Data Engineer Location: Philadelphia, PA Duration: Months We are seeking a skilled and experienced Lead Data Engineer to join our team and take a pivotal role in building, optimizing, and maintaining our data platform. As a technical leader, you will leverage your expertise in cloud technologies and data engineering to ensure...

  • GCP Data Lead

    4 weeks ago


    Philadelphia, Pennsylvania, United States Marshall Technologies Full time

    Job Title: GCP Data LeadWe are seeking an experienced professional to lead our team of data engineers in leveraging Google Cloud Platform (GCP) services to develop and deploy scalable data solutions. The ideal candidate will have expertise in GCP tools, including BigQuery, Dataflow, and Apache Airflow, as well as experience in data migration, data warehouse...

  • Data Engineer

    5 days ago


    Philadelphia, United States VySystems Full time

    Job Description:We are seeking a skilled and experienced Lead Data Engineer to join our team and take a pivotal role in building, optimizing, and maintaining our data platform. As a technical leader, you will leverage your expertise in cloud technologies and data engineering to ensure high performance and scalability of our data solutions. This position...

  • Lead Data Engineer

    1 week ago


    Philadelphia, United States Softcrylic Full time

    Who We Are For more than 20 years, we have been working with organizations large and small to help solve business challenges through technology. We bring a unique combination of engineering and strategy to Make Data Work for organizations. Our clients range from the travel and leisure industry to publishing, retail and banking. The common thread between our...

  • Senior Data Engineer

    3 weeks ago


    Philadelphia, Pennsylvania, United States InsideHigherEd Full time

    Job Title: Senior Data EngineerWe are seeking a highly skilled Senior Data Engineer to join our team at the University of Pennsylvania. As a Senior Data Engineer, you will be responsible for leading multi-disciplinary data analysis efforts and providing expertise in data engineering and analytics.About the Role:Develop and apply advanced data analysis...

  • Data Engineering Lead

    4 weeks ago


    Philadelphia, Pennsylvania, United States Avance Consulting Full time

    Job DescriptionAvance Consulting is seeking a highly skilled Technical Lead to join our team.Key Responsibilities:Design and implement data pipelines using Azure Databricks, leveraging Python and Spark.Develop data transformation processes in ADB.Convert Scala programs to PySpark programs.Understand and work with Scala programs in DataBricks.Secondary...

  • Lead Data Engineer

    3 days ago


    Philadelphia, Pennsylvania, United States Softcrylic Full time

    About the RoleWe are seeking a highly skilled Technical Lead - Data Engineering with a strong background in building and optimizing data pipelines and transformation processes using Azure Databricks, Python, and Spark.The ideal candidate will have the ability to understand and convert Scala programs to PySpark within Databricks and possess hands-on...

  • Senior Data Engineer

    3 weeks ago


    Philadelphia, Pennsylvania, United States PENN Interactive Full time

    About the RolePenn Interactive is a leading interactive gaming company that is revolutionizing the gaming industry with its innovative and immersive experiences. We are seeking a highly skilled Senior Data Engineer to join our team and contribute to the development of high-quality and sustainable data-driven solutions.As a key member of our Data Engineering...

  • Data Engineer

    1 day ago


    Philadelphia, United States Robert Half Full time

    Job DescriptionJob DescriptionWe are offering a contract to permanent employment opportunity for a Data Engineer in Philadelphia, Pennsylvania. As a Data Engineer, you will work in the healthcare industry, focusing on designing, developing, and delivering data solutions to support enterprise analytics. This role involves a strong emphasis on SQL and data...


  • Philadelphia, Pennsylvania, United States Publicis Health Media Full time

    Job Title: Associate Director, Data Engineering, Technology and DataAt Publicis Health Media, we're seeking an experienced Associate Director, Data Engineering, Technology and Data to join our team. As a key member of our data engineering team, you will play a critical role in designing and developing big data solutions built on the Databricks and Azure...


  • Philadelphia, Pennsylvania, United States Publicis Health Media Full time

    Job Title: Data Engineering ManagerPublicis Health Media is seeking a highly skilled Data Engineering Manager to join our Technology and Data Department. As a key member of our team, you will be responsible for overseeing the processing and maintenance of all data that flows into our organization.Key Responsibilities:Lead a portfolio of data projects across...

  • Data Engineer

    3 weeks ago


    Philadelphia, Pennsylvania, United States rockITdata Full time

    Job Title: Full Stack Data EngineerWe are seeking a highly skilled Full Stack Data Engineer to join our team at rockITdata. As a key member of our team, you will be responsible for designing, implementing, and maintaining scalable data solutions that meet the needs of our clients.Key Responsibilities:Data Ingestion and Integration:Design and implement data...

  • Data Engineer

    4 weeks ago


    Philadelphia, Pennsylvania, United States Constellation Search Group Full time

    Data Engineer Job DescriptionWe are seeking a skilled Data Engineer to join our dynamic team at Constellation Search Group. As a Data Engineer, you will play a crucial role in building and maintaining scalable data pipelines and ensuring seamless data integration, storage, and processing using the latest cloud technologies.Key Responsibilities:Develop and...

  • Data Engineer

    2 weeks ago


    Philadelphia, United States Pomeroy Technologies, LLC. Full time

    Job DescriptionJob DescriptionPomeroy is looking for a Data Engineer to join our client's team for a 7-month contract opportunity. This is a hybrid/remote opportunity with eligible candidates residing in EITHER Philadelphia or Washington D.C.The Data Engineer will take a pivotal role in building, optimizing, and maintaining our data platform. As a...


  • Philadelphia, United States KlearLink Full time

    Data & Integration EngineerCompany OverviewOur client provides innovative student engagement and success software that helps educational institutions foster student participation, track experiential learning, and empower students with unique storytelling capabilities through co-curricular transcripts. The solutions are designed to modernize and personalize...


  • Philadelphia, United States KlearLink Full time

    Data & Integration EngineerCompany OverviewOur client provides innovative student engagement and success software that helps educational institutions foster student participation, track experiential learning, and empower students with unique storytelling capabilities through co-curricular transcripts. The solutions are designed to modernize and personalize...


  • philadelphia, United States KlearLink Full time

    Data & Integration EngineerCompany OverviewOur client provides innovative student engagement and success software that helps educational institutions foster student participation, track experiential learning, and empower students with unique storytelling capabilities through co-curricular transcripts. The solutions are designed to modernize and personalize...

  • Project Manager

    3 days ago


    Philadelphia, Pennsylvania, United States Data Intelligence, LLC Full time

    Job Title: Project ManagerData Intelligence, LLC is seeking a highly skilled Project Manager to support our government clients in Philadelphia, PA. This is a full-time position that may be remote.Job Responsibilities:Develop and execute project plans to ensure timely completion of tasksAssess resource requirements and estimate costsCreate project schedules...


  • Philadelphia, Pennsylvania, United States Data Intelligence Full time

    Job Title: Senior Cloud Software EngineerData Intelligence, LLC is seeking a highly skilled Senior Cloud Software Engineer to join our team in Philadelphia, PA. As a key member of our software development team, you will be responsible for designing, developing, and deploying cloud-based software systems.Job Responsibilities:Provide full software development...