Principal Data Engineer

2 weeks ago


Boston, United States iTech Solutions Full time

Azure Data Lake Principal Architect

Hybrid Position 1-2x a week onsite at client
Direct Hire/Perm
Hours: 8:30am-5pm

Candidate needs to have experience doing solution architecture on Azure. The customer has a mature data platform currently and this person will be responsible for all new designs moving forward.

Team Overview
The our client's Data Lake Team plays a pivotal role in harnessing the power of data to drive innovation, enhance patient care, and advance research initiatives within the our clienthealthcare system. As a leading integrated health system, our clientrelies on a robust data infrastructure to make informed decisions and optimize healthcare delivery.
The mission of the Data Lake Team is to establish and maintain a centralized and scalable data platform - Data Lake, that consolidates diverse datasets from various sources across the healthcare system. This initiative aims to break down data silos, foster collaboration, and empower stakeholders with timely and accurate information.
The Data Lake Team collaborates closely with various stakeholders, including clinicians, researchers, administrators, and IT professionals. Cross-functional partnerships ensure that the Data Lake aligns with the evolving needs of the our client community.
Embracing a culture of innovation, the team is committed to exploring emerging technologies and methodologies to enhance the capabilities of the Data Lake, Lake house and Enterprise Data Warehouse. Continuous improvement initiatives are undertaken to refine processes, optimize performance, and stay at the forefront of healthcare data management.
The our client's Data Lake Team is at the forefront of revolutionizing healthcare through effective data management. By fostering collaboration, ensuring data integrity, and facilitating advanced analytics, the team contributes significantly to Mass General Brigham's mission of providing high-quality patient care and advancing medical knowledge.

Position Overview

We are seeking a highly skilled and experienced Principal Data Engineer to join our dynamic team. As a key member of our data engineering team, you will play a crucial role in designing, developing, and maintaining our data infrastructure and platform integration solutions. The Principal Data Engineer will be responsible for advancing our data engineering capabilities, ensuring data quality, and contributing to the success of data-driven initiatives.
Principal Duties and Responsibilities: Indicate key areas of responsibility, major job duties, special projects and key objectives for this position. These items should be evaluated throughout the year and included in the written annual evaluation.

Infrastructure, Architecture and Design:

  • Design end-to-end data solutions on the Azure data platform, considering Scalability, Security, Compliance and Performance.
  • Contribute to technical foundation of the Data Lake platform, expanding and optimizing the data ecosystem and pipeline architecture.
  • Provide expertise in selecting and implementing appropriate data platform services (e.g. Azure, Databricks, Snowflake) for various data processing and storage needs.
  • Collaborate with cross-functional teams to integrate data solutions with existing MGB data platform systems and applications, integrate new data management technologies aligned with the latest vendor products & capabilities (Microsoft Azure and Fabric, Databricks, Snowflake, Collibra, etc.) and software engineering tools into existing structures.
  • Design, develop, construct, test, and maintain Data Lake architectures and large-scale data processing systems.
  • Support big data, data lake, lakehouse ecosystem related tool selection and POC analysis.

Data Pipeline Development:

  • Gather and process raw data at scale including large complex data sets, meeting functional/non-functional business requirements (using ADF, Databricks, Python, Pyspark, scripts, REST API calls, SQL Queries, etc.).
  • Develop data set processes for data modeling, mining, and production.
  • Create and maintain optimal data pipeline architecture on cloud-based platforms (e.g., Azure) and relational data systems (SQL Server, SSIS, Snowflakes).

Cross-Functional Collaboration:
  • Work on cross-functional teams delivering enterprise solutions for internal and external clients.
  • Collaborate with Software Developers, Database Architects, Data Analysts, and Data Scientists on data initiatives.
  • Support stakeholders, including the Management team, Product owners, and Architecture teams, in addressing data-related technical issues and fulfilling data infrastructure needs.
  • Partner with integrated platform & data science teams to define integrated solutions to meet the evolving needs of data analysts, developers & consumers (e.g. data ingestion framework, AI/Client capabilities, scalable compute), contributing to the innovation and leadership of the organization.

Data Optimization and Automation:
  • Identify, design, and implement internal process improvements, automation of manual processes, optimization of data delivery, etc.
  • Build the data infrastructure required for optimal extraction, transformation, and loading of data from traditional/legacy sources.

Subject Matter Expertise and Leadership:
  • Act as a subject matter expert for internal or external data products.
  • Contribute to solution architecture design and advise on engineering solution best practices.
  • Mentor and guide junior data engineers by leading code reviews and documenting best practices.
  • Use our clientvalues to govern decisions, actions, and behaviors.
  • Perform other duties and responsibilities as assigned.

Qualifications:
(Must be realistic, neither overstated nor understated, and related to the essential functions of the job.)
  • Bachelor's or Master's degree in Computer Science, Information Technology, or related fields; or comparable work experience
  • 8+ years of related professional experience including 5+ years in data lake development in large reporting environment(s)
  • Expert in Azure cloud computing, specializing in Azure data engineering stacks like ADF (Azure Data Factory), ADLS, Event Hubs, Snowflake, Databricks, streaming, Azure PowerShell, and Log Analytics.
  • Hands-on development experience with Design and Architecture of big data frameworks/tools: Azure Data Lake, Snowflake, Azure Data Bricks.
  • Expert experience with Hadoop based technology (e.g. Spark) and programming (Python, SQL, PySpark)
  • Knowledgeable about cloud computing costs and performance and capable of providing ongoing suggestions for cost & performance optimization.
  • Solid understanding of Snowflake computing, including its integration with Azure Data Lake, utilizing ADLS as a source for data processing.
  • Experience with Design and Architecture of relational SQL and NoSQL databases, including MS SQL Server, Snowflake.
  • Experience with Azure DevOps, familiar with CI/CD (Continuous Integration/Continuous Deployment) processes and capable of scaling them across engineering teams, leveraging Azure native resources for data ingestion.
  • Experience leading and working with cross-functional teams in a dynamic environment, demonstrated track record of team leadership, technical acumen, innovation, tactical and strategic.
  • Proven verbal, communication, and presentation skills, ability to clearly and concisely communicate complex technical concepts to both technical and non-technical audiences.
  • Proven ability to work independently.

Skills/Abilities/Competencies: (Must be realistic, neither overstated nor understated, and related to the essential functions of the job.)

  • Advanced hands-on SQL, Spark, Python, PySpark knowledge and experience working with relational databases for data querying and retrieval on multiple platform.
  • Proficiency in Data Modeling tools (e.g. Erwin, Visio).
  • Strong interpersonal and communication skills, both written and verbal.
  • Strong Scrum/Agile development experience.
  • Excellent organizational skills and attention to detail, manage multiple tasks and projects, meet deadlines, follow through, and manage to schedule.
  • Strong innovation capabilities and the ability to think creatively.
  • Strong collaboration and team building skills within, across and outside of an organization.
  • Maintain and promote a positive team environment.
  • Maintains stable performance under pressure, demonstrating sensitivity to diverse organizational culture.
  • Ability to effectively cope with change, remain flexible and adaptable within a fast-paced environment with rapidly changing requirements, and ability to negotiate situations when the big picture is not clearly defined.

PREFERRED SKILLS/ABILITIES/COMPETENCIES
  • Strong root cause analysis and problem-solving skills
  • Ability to juggle multiple projects in a high volume, fast-paced Production environment
  • Ability to identify process improve opportunities and areas of potential conflict
  • Working knowledge of MGB data products and platforms a plus
  • Ability to quickly adapt to new technologies, concepts, and approaches
  • Familiar with current healthcare and data management trends and industry practices
  • Demonstrated ability to manage multiple priorities
  • Microsoft Certified: Azure Solutions Architect Expert ( Nice to Have)
  • Microsoft Certified: Azure Data Engineer ( Nice to Have)
For more information please contact Vijay Karney at or

Since 1995, iTech Solutions Inc., has been providing IT Consulting and Direct Hire Services to the Insurance, Financial, Communications, Manufacturing and Government sectors with local offices in Connecticut, Minnesota, Colorado, Massachusetts, Tennessee, North Carolina, and New Jersey / Pennsylvania area.

Our recruiting strategy is simple, if you want to find qualified IT professionals then use IT professionals to find them. So at iTech Solutions, our personnel are all career IT professionals with a wide range of IT experience. We can honestly say our staff understands the technologies, the complexities of finding and selecting the appropriate personnel and the pressures of running successful IT projects.

Employer will not sponsor applicants for any employment visas, at hiring or in the future, including but not limited to H-1B visas. Corp-to-Corp or subcontract personnel will not be considered for this position.



  • Boston, United States Quantexa Full time

    Job DescriptionJob DescriptionFounded in 2016 with only a handful of individuals, Quantexa was built with a purpose that through a greater understanding of context, better decisions can be made. 8 years, 15 locations and 700+ employees later we still believe that today. We connect the dots within our customers data using dynamic entity resolution and...


  • Boston, Massachusetts, United States Geode Capital Management Full time

    Geode Capital Management is actively searching for a full-time Principal Data Engineer to become an integral part of our Data team. The Principal Data Engineer will play a very critical role within our data team by providing technical leadership, implementation expertise, design best practices and guiding other highly skilled and motivated data engineers. In...


  • Boston, MA, United States Commonwealth of Massachusetts Full time

    The Executive Office of Technology Services and Security (EOTSS) places our customers and constituents at the heart of everything we do. We offer responsive digital services and productivity tools to more than 40,000 state employees, who provide essential information and services to the citizens of the Commonwealth. We also serve constituents, providing...


  • Boston, MA, United States Quantexa Full time

    Job Description Job Description Founded in 2016 with only a handful of individuals, Quantexa was built with a purpose that through a greater understanding of context, better decisions can be made. 8 years, 15 locations and 700+ employees later we still believe that today. We connect the dots within our customers data using dynamic entity resolution and...


  • Boston, MA, United States Dice Full time

    Dice is the leading career destination for tech experts at every stage of their careers. Our client, iTech Solutions, is seeking the following. Apply via Dice today! Azure Data Lake Principal Architect Hybrid Position 1-2x a week onsite at client Direct Hire/Perm Hours: 8:30am-5pm Candidate needs to have experience doing solution architecture on Azure....


  • Boston, United States Ccube Full time

    Job DescriptionJob DescriptionLead Principal Data EngineerPrimary Location: Boston, MA (Hybrid 3 Days a  Week)    Additional Locations; Merrimack, NH; Raleigh, NC; Jersey City, NJLong TermLooking for someone with hands-on experience and architect level person who can design, evangelize, and mentor junior engineers The Expertise and Skills You...


  • Boston, United States Motion Recruitment Full time

    Job Title: Principal Data Engineer Location: Downtown Boston, MA - You bus already live in Massachusetts no relocation is available  Company Overview: Join a prestigious asset management firm in the heart of Boston, boasting over $1 trillion in assets under management. Our firm is renowned for its innovative investment strategies and commitment to...


  • Boston, MA, United States Motion Recruitment Full time

    Job Title : Principal Data Engineer Location : Downtown Boston, MA - You bus already live in Massachusetts no relocation is available Company Overview : Join a prestigious asset management firm in the heart of Boston, boasting over $1 trillion in assets under management. Our firm is renowned for its innovative investment strategies and commitment...


  • Boston, United States Nasuni Full time

    Description The PositionNasuni is expanding its Platform development team with a Principal Software Engineer. This Principal Software Engineer should have strong Linux and CentOS experience. The ideal candidate will be looking to take ownership of the development of the many Python libraries and services that run on the Nasuni Appliance that enables our...


  • Boston, United States Snowflake Computing Full time

    Build the future of data. Join the Snowflake team.Snowflake started with a clear vision: develop a cloud data platform that is effective, affordable, and accessible to all data users. Snowflake developed an innovative new product with a built-for-the-cloud architecture that combines the power of data warehousing, the flexibility of big data platforms, and...


  • Boston, United States Nasuni Full time

    Description The Position Nasuni is expanding its Platform development team with a Principal Software Engineer. This Principal Software Engineer should have strong Linux and CentOS experience. The ideal candidate will be looking to take ownership of the development of the many Python libraries and services that run on the Nasuni Appliance that enables our...


  • Boston, United States The Panther Group Full time

    Our client, a financial institution, is seeking a Principal Infrastructure Engineer to join their team to help execute a new mission-critical program. Responsibilities: Identify and address the root causes of issues, engaging early and comprehensivel Infrastructure Engineer, Principal, Infrastructure, Engineer, Construction, Staffing


  • Boston, Massachusetts, United States Nasuni Full time

    Principal Software Automation Engineer Nasuni is hiring a Principal Software Automation Engineer to expand the internal tool, libraries, frameworks and infrastructure for test management and test automation. Build test automation infrastructure using Python. This role requires an understanding of enterprise software product development, product testing, and...


  • Boston, United States Motion Recruitment Full time

    We're currently working with a company that specializes in systematic asset management providing core beta exposures across a wide range of niche asset classes and other equity sources. Their focus is to offer the scale of a large asset firm with the benefits of a smaller organization.  The ideal candidate will play a critical role on the Data Team,...


  • Boston, United States Nasuni Full time

    Description Principal Software Engineer - Control Path Principal Python Software Engineer. Nasuni is expanding scrum teams for the Nasuni Management Console (NMC). The NMC provides a single pane of glass for our global enterprise customers to manage Nasuni’s Hybrid Cloud Storage services. As a Principal Software Engineer, you would be a leader and mentor...


  • Boston, United States Arrowstreet Capital Full time

    Principal, Software Engineering Research Systems page is loaded Principal, Software Engineering Research Systems Apply locations Boston time type Full time posted on Posted 3 Days Ago job requisition id R994 Job Overview The Research Systems Principal Engineer will work across multiple development teams that builds data and compute platforms and solutions to...


  • Boston, United States Apollo Solutions Full time

    Principal DevOps Engineer/SREApollo Solutions have partnered with a disruptive early stage AI/ML start-up backed by top tier venture capital.In this role, you will be working closely with their founders and founding engineers to ensure fast, secure and reliable features can be delivered as well as building their infrastructure to feature massive...


  • Boston, United States Apollo Solutions Full time

    Principal DevOps Engineer/SREApollo Solutions have partnered with a disruptive early stage AI/ML start-up backed by top tier venture capital.In this role, you will be working closely with their founders and founding engineers to ensure fast, secure and reliable features can be delivered as well as building their infrastructure to feature massive...


  • Boston, United States Apollo Solutions Full time

    Principal DevOps Engineer/SREApollo Solutions have partnered with a disruptive early stage AI/ML start-up backed by top tier venture capital.In this role, you will be working closely with their founders and founding engineers to ensure fast, secure and reliable features can be delivered as well as building their infrastructure to feature massive...

  • Principal Engineer

    5 days ago


    Boston, United States Westinghouse Electric Company, LLC Full time

    We are searching for a Principal Engineer - Substation to join the Engineering & Construction Business Unit team in Juno Beach, FL. The person will be all the assigned substation projects from conception (Early Stage) through construction and commissioning (Execution). In the conceptual phase, the person will work with the Early-Stage Project Manager to...