Principal Data Engineer with Azure Data Lake

3 weeks ago

Somerville, United States Mastech Digital Full time

TITLE: Azure Data Lake Principal Architect/Principal Data Engineer

LOCATION: Somerville MA (HYBRID): 1-2x a week onsite - Hours: 8:30am-5pm

Type: PERM

Salary range: $150K (Depending on Exp)

*** Candidate needs to have experience doing solution architecture on Azure. The customer has a mature data platform currently and this person will be responsible for all new designs moving forward.***

DESCRIPTION

Seeking a highly skilled and experienced Principal Data Engineer that will play a crucial role in designing, developing, and maintaining data infrastructure and platform integration solutions. The Principal Data Engineer will be responsible for advancing data engineering capabilities, ensuring data quality, and contributing to the success of data-driven initiatives.

RESPONSIBILITIES:

Infrastructure, Architecture and Design:

  • Design end-to-end data solutions on the Azure data platform, considering Scalability, Security, Compliance and Performance.
  • Contribute to technical foundation of the Data Lake platform, expanding and optimizing the data ecosystem and pipeline architecture.
  • Provide expertise in selecting and implementing appropriate data platform services (e.g. Azure, Databricks, Snowflake) for various data processing and storage needs.
  • Collaborate with cross-functional teams to integrate data solutions with existing platform systems and applications, integrate new data management technologies aligned with the latest vendor products & capabilities (Microsoft Azure and Fabric, Databricks, Snowflake, Collibra, etc.) and software engineering tools into existing structures.
  • Design, develop, construct, test, and maintain Data Lake architectures and large-scale data processing systems.
  • Support big data, data lake, lakehouse ecosystem related tool selection and POC analysis.


Data Pipeline Development:

  • Gather and process raw data at scale including large complex data sets, meeting functional/non-functional business requirements (using ADF, Databricks, Python, Pyspark, scripts, REST API calls, SQL Queries, etc.).
  • Develop data set processes for data modeling, mining, and production.
  • Create and maintain optimal data pipeline architecture on cloud-based platforms (e.g., Azure) and relational data systems (SQL Server, SSIS, Snowflakes).


Cross-Functional Collaboration:

  • Work on cross-functional teams delivering enterprise solutions for internal and external clients.
  • Collaborate with Software Developers, Database Architects, Data Analysts, and Data Scientists on data initiatives.
  • Support stakeholders, including the Management team, Product owners, and Architecture teams, in addressing data-related technical issues and fulfilling data infrastructure needs.
  • Partner with integrated platform & data science teams to define integrated solutions to meet the evolving needs of data analysts, developers & consumers (e.g. data ingestion framework, AI/ML capabilities, scalable compute), contributing to the innovation and leadership of the organization.


Data Optimization and Automation:

  • Identify, design, and implement internal process improvements, automation of manual processes, optimization of data delivery, etc.
  • Build the data infrastructure required for optimal extraction, transformation, and loading of data from traditional/legacy sources.


Subject Matter Expertise and Leadership:

  • Act as a subject matter expert for internal or external data products.
  • Contribute to solution architecture design and advise on engineering solution best practices.
  • Mentor and guide junior data engineers by leading code reviews and documenting best practices.


QUALIFICATIONS:

  • Bachelor's or Master's degree in Computer Science, Information Technology, or related fields; or comparable work experience
  • 8+ years of related professional experience including 5+ years in data lake development in large reporting environment(s)
  • Expert in Azure cloud computing, specializing in Azure data engineering stacks like ADF (Azure Data Factory), ADLS, Event Hubs, Snowflake, Databricks, streaming, Azure PowerShell, and Log Analytics.
  • Hands-on development experience with Design and Architecture of big data frameworks/tools: Azure Data Lake, Snowflake, Azure Data Bricks.
  • Expert experience with Hadoop based technology (e.g. Spark) and programming (Python, SQL, PySpark)
  • Knowledgeable about cloud computing costs and performance and capable of providing ongoing suggestions for cost & performance optimization.
  • Solid understanding of Snowflake computing, including its integration with Azure Data Lake, utilizing ADLS as a source for data processing.
  • Experience with Design and Architecture of relational SQL and NoSQL databases, including MS SQL Server, Snowflake.
  • Experience with Azure DevOps, familiar with CI/CD (Continuous Integration/Continuous Deployment) processes and capable of scaling them across engineering teams, leveraging Azure native resources for data ingestion.
  • Experience leading and working with cross-functional teams in a dynamic environment, demonstrated track record of team leadership, technical acumen, innovation, tactical and strategic.
  • Proven verbal, communication, and presentation skills, ability to clearly and concisely communicate complex technical concepts to both technical and non-technical audiences.
  • Proven ability to work independently.


SKILLS

  • Advanced hands-on SQL, Spark, Python, PySpark knowledge and experience working with relational databases for data querying and retrieval on multiple platform.
  • Proficiency in Data Modeling tools (e.g. Erwin, Visio).
  • Strong interpersonal and communication skills, both written and verbal.
  • Strong Scrum/Agile development experience.
  • Excellent organizational skills and attention to detail, manage multiple tasks and projects, meet deadlines, follow through, and manage to schedule.
  • Strong innovation capabilities and the ability to think creatively.
  • Strong collaboration and team building skills within, across and outside of an organization. Maintain and promote a positive team environment.
  • Maintains stable performance under pressure, demonstrating sensitivity to diverse organizational culture.
  • Ability to effectively cope with change, remain flexible and adaptable within a fast-paced environment with rapidly changing requirements, and ability to negotiate situations when the big picture is not clearly defined.


PREFERRED SKILLS/ABILITIES/COMPETENCIES

  • Strong root cause analysis and problem-solving skills
  • Ability to juggle multiple projects in a high volume, fast-paced Production environment
  • Ability to identify process improve opportunities and areas of potential conflict
  • Ability to quickly adapt to new technologies, concepts, and approaches
  • Familiar with current healthcare and data management trends and industry practices
  • Demonstrated ability to manage multiple priorities
  • Microsoft Certified: Azure Solutions Architect Expert ( Nice to Have)
  • Microsoft Certified: Azure Data Engineer ( Nice to Have)

  • Principal Data Engineer with Azure Data Lake

    3 weeks ago

    Somerville, United States Mastech Digital Full time

    TITLE: Azure Data Lake Principal Architect/Principal Data EngineerLOCATION: Somerville MA (HYBRID): 1-2x a week onsite - Hours: 8:30am-5pmType: PERMSalary range: $150K (Depending on Exp) *** Candidate needs to have experience doing solution architecture on Azure. The customer has a mature data platform currently and this person will be responsible for all...

  • Principal Data Engineer with Azure Data Lake

    3 weeks ago

    Somerville, United States Mastech Digital Full time

    TITLE: Azure Data Lake Principal Architect/Principal Data EngineerLOCATION: Somerville MA (HYBRID): 1-2x a week onsite - Hours: 8:30am-5pmType: PERMSalary range: $150K (Depending on Exp) *** Candidate needs to have experience doing solution architecture on Azure. The customer has a mature data platform currently and this person will be responsible for all...

  • Azure Data Lake Principal Architect

    4 days ago

    Somerville, Massachusetts, United States Randstad USA Full time

    job summary: Team Overview The mission of the Data Lake Team is to establish and maintain a centralized and scalable data platform - Data Lake, that consolidates diverse datasets from various sources across the healthcare system. This initiative aims to break down data silos, foster collaboration, and empower stakeholders with timely and accurate...

  • Azure Data Lake Principal Architect

    5 days ago

    Somerville, United States Cat America Full time

    Job Description Job Description The candidate needs to have experience doing solution architecture on Azure. The Architect will play a crucial role in designing, developing, and maintaining our data infrastructure and platform integration solutions. Responsible for advancing our data engineering capabilities, ensuring data quality, and contributing to the...

  • Azure Data Lake Principal Architect

    2 weeks ago

    Somerville, United States Cat America Full time

    Job DescriptionJob DescriptionThe candidate needs to have experience doing solution architecture on Azure.The Architect will play a crucial role in designing, developing, and maintaining our data infrastructure and platform integration solutions. Responsible for advancing our data engineering capabilities, ensuring data quality, and contributing to the...

  • Azure Data Lake Principal Architect

    4 days ago

    Somerville, Massachusetts, United States Meduvi LLC Full time

    The Architect will play a crucial role in designing, developing, and maintaining our data infrastructure and platform integration solutions. Responsible for advancing our data engineering capabilities, ensuring data quality, and contributing to the success of data-driven initiatives. Candidate needs to have experience doing solution architecture on Azure....

  • Principal Data Architect with Azure Data Factory

    1 week ago

    Somerville, United States Mastech Digital Full time

    TITLE: Azure Data Lake Principal Architect/Principal Data EngineerLOCATION: Somerville MA (HYBRID): 1-2x a week onsite - Hours: 8:30am-5pmType: PERMSalary range: $170K (Depending on Exp)*** Candidate needs to have experience doing solution architecture on Azure. The customer has a mature data platform currently and this person will be responsible for all new...

  • Principal Data Architect with Azure Data Factory

    1 week ago

    Somerville, United States Mastech Digital Full time

    TITLE: Azure Data Lake Principal Architect/Principal Data EngineerLOCATION: Somerville MA (HYBRID): 1-2x a week onsite - Hours: 8:30am-5pmType: PERMSalary range: $170K (Depending on Exp)*** Candidate needs to have experience doing solution architecture on Azure. The customer has a mature data platform currently and this person will be responsible for all new...

  • AzureData Lake Principal Architect

    1 week ago

    Somerville, United States Stellent IT LLC Full time

    Job Title : Azure Data Lake Principal Architect Location : Somerville, Massachusetts (Hybrid) Duration: Full Time / FTE Only VISA: OR EAD The Architect will play a crucial role in designing, developing, and maintaining our data infrastructure and platform integration solutions. Responsible for advancing our data engineering capabilities, ensuring data...

  • Looking for Solution Architect on Azure

    2 weeks ago

    Somerville, United States 4-Serv Solutions Inc. Full time

    Principal / Solution Architect on AzurHybrid Position 1-2x a week onsite in Somerville, MassachusettsClient will not accept candidates requiring sponsorship.Hours: 8:30am-5pm Azure Data Lake Principal Architect. Candidate needs to have experience doing solution architecture on Azure. The Architect will play a crucial role in designing, developing, and...

  • Azure Datalake Administrator

    2 weeks ago

    Somerville, United States The Cranberry Country Chamber of Commerce Full time

    Bachelor's or Master's degree in Computer Science, Information Technology, or related fields; or comparable work experience 8+ years of related professional experience including 5+ years in data lake development in large reporting environment(s) Expert in Azure cloud computing, specializing in Azure data engineering stacks like ADF (Azure Data Factory),...

  • Azure Datalake Administrator

    2 weeks ago

    Somerville, United States Mastech Digital Full time

    Bachelor's or Master's degree in Computer Science, Information Technology, or related fields; or comparable work experience 8+ years of related professional experience including 5+ years in data lake development in large reporting environment(s) Expert in Azure cloud computing, specializing in Azure data engineering stacks like ADF (Azure Data Factory),...

  • Azure Datalake Administrator

    3 weeks ago

    Somerville, United States Mastech Digital Full time

    Bachelor's or Master's degree in Computer Science, Information Technology, or related fields; or comparable work experience8+ years of related professional experience including 5+ years in data lake development in large reporting environment(s)Expert in Azure cloud computing, specializing in Azure data engineering stacks like ADF (Azure Data Factory), ADLS,...

  • Azure Datalake Administrator

    3 weeks ago

    Somerville, United States Mastech Digital Full time

    Bachelor's or Master's degree in Computer Science, Information Technology, or related fields; or comparable work experience8+ years of related professional experience including 5+ years in data lake development in large reporting environment(s)Expert in Azure cloud computing, specializing in Azure data engineering stacks like ADF (Azure Data Factory), ADLS,...

  • Senior Cloud DevOps Engineer and Administrator

    4 days ago

    Somerville, United States Mass General Brigham Full time

    About Us: As a not-for-profit organization, Mass General Brigham is committed to supporting patient care, research, teaching, and service to the community by leading innovation across our system. Founded by Brigham and Women's Hospital and Massachusetts General Hospital, Mass General Brigham supports a complete continuum of care including community and...

  • Cloud Infrastructure Architect

    2 days ago

    Somerville, United States Mastech Digital Full time

    RESPONSIBILITIES Infrastructure, Architecture and Design: Design end-to-end data solutions on the Azure data platform , considering Scalability, Security, Compliance and Performance. Contribute to technical foundation of the Data Lake platform, expanding and optimizing the data ecosystem and pipeline architecture. Provide expertise in selecting and...

  • Cloud Infrastructure Architect

    5 days ago

    Somerville, United States Mastech Digital Full time

    RESPONSIBILITIES Infrastructure, Architecture and Design:Design end-to-end data solutions on the Azure data platform, considering Scalability, Security, Compliance and Performance. Contribute to technical foundation of the Data Lake platform, expanding and optimizing the data ecosystem and pipeline architecture. Provide expertise in selecting and...

  • Data Engineering Intern

    2 weeks ago

    Somerville, United States Formlabs Full time

    To reinvent an industry, you have to build the best team. Join Formlabs if you want to bring groundbreaking professional 3D printers to the desktop of every designer, engineer, researcher, and artist in the world. Formlabs is growing fast, and that means a lot of data to crunch, software to build, and web applications to scale. Our Data Engineering team...

  • Cloud Infrastructure Architect

    7 days ago

    Somerville, United States Mastech Digital Full time

    RESPONSIBILITIESInfrastructure, Architecture and Design:Design end-to-end data solutions on the Azure data platform, considering Scalability, Security, Compliance and Performance.Contribute to technical foundation of the Data Lake platform, expanding and optimizing the data ecosystem and pipeline architecture.Provide expertise in selecting and implementing...

  • Cloud Infrastructure Architect

    1 week ago

    Somerville, United States Mastech Digital Full time

    RESPONSIBILITIESInfrastructure, Architecture and Design:Design end-to-end data solutions on the Azure data platform, considering Scalability, Security, Compliance and Performance.Contribute to technical foundation of the Data Lake platform, expanding and optimizing the data ecosystem and pipeline architecture.Provide expertise in selecting and implementing...