Senior Data Engineer

Senior Data Engineer Mayo Clinic is top-ranked in more specialties than any other care provider according to U.S. News & World Report. As we work together to put the needs of the patient first, we are also dedicated to our employees, investing in competitive compensation and comprehensive benefit plans to take care of you and your family, now and in the future. And with continuing education and advancement opportunities at every turn, you can build a long, successful career with Mayo Clinic. We are seeking a talented Senior Data Engineer to join our Advanced Data Lake (ADL) team. This is an infrastructure-heavy, hybrid cloud role with Google Cloud Platform (GCP) as a core requirement. You will build and operate enterprise data Lakehouse platforms that support large-scale analytics and digital transformation. Your responsibilities will include architecting and maintaining automated data pipelines for ingesting, transforming, and integrating complex datasets. You will use DataStream for real-time data movement and Dataflow for processing at scale. Composer/Airflow will be leveraged for seamless scheduling, monitoring, and automation of pipeline operations. Infrastructure provisioning and workflow management will be handled with Terraform and Dataform to ensure reproducibility and adherence to best practices. All code and pipeline assets will be managed through git repositories, with CI/CD automation and streamlined releases enabled by Azure DevOps (ADO). Changes will be governed by ServiceNow processes to ensure traceability, auditability, and operational compliance. Core duties involve working with cross-functional teams to translate business needs into pipeline specifications, building and optimizing data models for advanced analytics, and maintaining data quality and security throughout all processes. You will automate workflow monitoring and proactively resolve data issues, applying strong technical and problem-solving skills. The ideal candidate will have proficiency in Python and SQL, with significant experience in Google Cloud Platform (especially Dataflow and DataStream), Terraform, Dataform, and orchestration with Composer/Airflow. Experience managing code in git repositories, working with Azure DevOps workflows, and following ServiceNow change management processes is required. Strong communication skills and the ability to manage multiple priorities in a remote, team-oriented environment are also necessary. Develops and deploys data pipelines, integrations and transformations to support analytics and machine learning applications and solutions as part of an assigned product team using various open-source programming languages and vended software to meet the desired design functionality for products and programs. The position requires maintaining an understanding of the organization's current solutions, coding languages, tools, and regularly requires the application of independent judgment. May provide consultative services to departments/divisions and leadership committees. Demonstrated experience in designing, building, and installing data systems and how they are applied to the Department of Data & Analytics technology framework is required. Candidate will partner with product owners and Analytics and Machine Learning delivery teams to identify and retrieve data, conduct exploratory analysis, pipeline and transform data to help identify and visualize trends, build and validate analytical models, and translate qualitative and quantitative assessments into actionable insights. This is a full time remote position within the United States. Mayo Clinic will not sponsor or transfer visas for this position including F1 OPT STEM. A Bachelor's degree in a relevant field such as engineering, mathematics, computer science, information technology, health science, or other analytical/quantitative field and a minimum of five years of professional or research experience in data visualization, data engineering, analytical modeling techniques; OR an Associate's degree in a relevant field such as engineering, mathematics, computer science, information technology, health science, or other analytical/quantitative field and a minimum of seven years of professional or research experience in data visualization, data engineering, analytical modeling techniques. In-depth business or practice knowledge will also be considered. Incumbent must have the ability to manage a varied workload of projects with multiple priorities and stay current on healthcare trends and enterprise changes. Interpersonal skills, time management skills, and demonstrated experience working on cross functional teams are required. Requires strong analytical skills and the ability to identify and recommend solutions and a commitment to customer service. The position requires excellent verbal and written communication skills, attention to detail, and a high capacity for learning and problem resolution. Advanced experience in SQL is required. Strong experience in scripting languages such as Python, JavaScript, PHP, C++ or Java & API integration is required. Experience in hybrid data processing methods (batch and streaming) such as Apache Spark, Hive, Pig, Kafka is required. Experience with big data, statistics, and machine learning is required. The ability to navigate linux and windows operating systems is required. Knowledge of workflow scheduling (Apache Airflow Google Composer), Infrastructure as code (Kubernetes, Docker) CI/CD (Jenkins, Github Actions) is preferred. Experience in DataOps/DevOps and agile methodologies is preferred. Experience with hybrid data virtualization such as Denodo is preferred. Working knowledge of Tableau, Power BI, SAS, ThoughtSpot, DASH, d3, React, Snowflake, SSIS, and Google Big Query is preferred. Google Cloud Platform (GCP) certification is preferred. Preferred qualifications include hybrid or multi-cloud experience, familiarity with enterprise data governance, metadata, and lineage tools, and experience working in large, regulated environments. A GCP Professional Data Engineer certification is required. Exempt $138,257.60 - $200,512.00 / year Yes Full Time 80 Monday - Friday, 8:00 a.m. - 5:00 p.m. May be required to provide 24/7 on-call support. May be required to provide 24/7 on-call support. No Each Mayo Clinic location is a special place where our employees thrive in both their work and personal lives. Learn more about what each unique Mayo Clinic campus has to offer, and where your best fit is. All qualified applicants will receive consideration for employment without regard to race, color, religion, sex, gender identity, sexual orientation, national origin, protected veteran status or disability status. Mayo Clinic participates in E-Verify and may provide the Social Security Administration and, if necessary, the Department of Homeland Security with information from each new employee's Form I-9 to confirm work authorization. As an Affirmative Action and Equal Opportunity Employer Mayo Clinic is committed to creating an inclusive environment that values the diversity of its employees and does not discriminate against any employee or candidate. Women, minorities, veterans, people from the LGBTQ communities and people with disabilities are strongly encouraged to apply to join our teams. Reasonable accommodations to access job openings or to apply for a job are available.