Senior Databricks Engineer

2 days ago

Charlotte, NC, United States Tata Consultancy Services Full time

Must Have Technical/Functional Skills
• 10+ years overall experience in data engineering or related fields.
• 35 years hands-on experience with Databricks and Spark.
• Strong proficiency in SQL and data analysis techniques.
• Experience with ETL processes, data modeling, and performance tuning.
• Familiarity with Python or Scala for data engineering tasks.
• Excellent problem-solving and communication skills. Roles & Responsibilities We are seeking a hands-on Sr. Databricks Data Engineer to design, develop, and optimize data pipelines and analytics solutions. The ideal candidate will have strong experience in data engineering, ETL development, and production support, ensuring reliable, scalable, and high-performing data operations within Azure environment and can work in a fast-paced environment. Knowledge of insurance domain and Power BI is a plus but not mandatory. Development
• Design, develop, and deploy scalable ETL/ELT data pipelines using Apache Spark, PySpark, and Databricks.
• Develop and optimize SQL queries for data transformation and analysis.
• Collaborate with product owners, data architects and analysts to build data models, delta lake structures, and data workflows.
• Collaborate with data analysts and business teams to deliver actionable insights.
• Build job orchestration and monitoring solutions
• Ensure data quality, performance, and reliability across workflows.
• Develop and maintain CI/CD pipelines for Databricks notebooks, jobs, and workflows.
• Work with cloud-based data platforms (Azure preferred).

Production Support
• Provide L2/L3 support for Databricks production jobs, resolving incidents and performance issues.
• Monitor and troubleshoot Spark jobs, cluster performance, job failures, and resource utilization.
• Ensure data pipelines meet SLAs and data accuracy standards.
• Coordinate with infrastructure and DevOps teams to optimize cluster management and costs.
• Perform root cause analysis (RCA) and implement long-term fixes for recurring issues.
• Maintain documentation for production jobs, data flows, and incident management procedures.

Required Skills & Experience:
• 10+ years overall experience in data engineering or related fields.
• 35 years hands-on experience with Databricks and Spark.
• Strong proficiency in SQL and data analysis techniques.
• Experience with ETL processes, data modeling, and performance tuning.
• Familiarity with Python or Scala for data engineering tasks.
• Excellent problem-solving and communication skills.

Nice-to-Have:
• Knowledge of insurance industry data.
• Experience with Power BI or other BI tools.

Salary Range: $110,000 to $140,000 per year

  • Senior Data Engineer

    8 hours ago

    Charlotte, NC, United States Sumitomo Mitsui Financial Group, Inc. Full time

    SMBC Group is a top-tier global financial group. Headquartered in Tokyo and with a 400-year history, SMBC Group offers a diverse range of financial services, including banking, leasing, securities, credit cards, and consumer finance. The Group has more than 130 offices and 80,000 employees worldwide in nearly 40 countries. Sumitomo Mitsui Financial Group,...

  • AWS Databricks Admin

    2 weeks ago

    Charlotte, NC, United States Kaav Inc. Full time

    Job Description: Bachelor's degree in computer science, Information Technology, or a related field. Proven experience in cloud computing, specifically AWS, with a focus on GovCloud. Strong understanding of Databricks and its integration with AWS services Responsibilities: 1. Platform Management: Administer and maintain the Databricks environment within AWS...

  • AWS Databricks Admin

    5 days ago

    Charlotte, NC, United States Kaav Inc. Full time

    Job Description: Bachelor's degree in computer science, Information Technology, or a related field. Proven experience in cloud computing, specifically AWS, with a focus on GovCloud. Strong understanding of Databricks and its integration with AWS services Responsibilities: 1. Platform Management: Administer and maintain the Databricks environment within AWS...

  • AWS Databricks Admin

    13 hours ago

    Charlotte, NC, United States Kaav Inc. Full time

    Job Description: Bachelor's degree in computer science, Information Technology, or a related field. Proven experience in cloud computing, specifically AWS, with a focus on GovCloud. Strong understanding of Databricks and its integration with AWS services Responsibilities: 1. Platform Management: Administer and maintain the Databricks environment within AWS...

  • Senior Data Engineer

    8 hours ago

    Charlotte, NC, United States Synechron Full time

    We areAt Synechron, we believe in the power of digital to transform businesses for the better. Our global consulting firm combines creativity and innovative technology to deliver industry-leading digital solutions. Synechron s progressive technologies and optimization strategies span end-to-end Artificial Intelligence, Consulting, Digital, Cloud & DevOps,...

  • Senior Data Architect

    6 days ago

    Charlotte, NC, United States CRC Group Full time

    The position is described below. If you want to apply, click the Apply button at the top or bottom of this page. You'll be required to create an account or sign in to an existing one. If you have a disability and need assistance with the application, you can request a reasonable accommodation. Send an email to Accessibility (accommodation requests only;...

  • Senior Data Engineer

    18 hours ago

    Charlotte, NC, United States The Hartford Full time

    Sr Data Engineer - GE07BESr Developer - IE07GE We're determined to make a difference and are proud to be an insurance company that goes well beyond coverages and policies. Working here means having every opportunity to achieve your goals - and to help others accomplish theirs, too. Join our team as we help shape the future. Senior Data Engineer The...

  • Senior Data Engineer

    7 hours ago

    Charlotte, NC, United States The Hartford Full time

    Sr Data Engineer - GE07BESr Developer - IE07GE We're determined to make a difference and are proud to be an insurance company that goes well beyond coverages and policies. Working here means having every opportunity to achieve your goals - and to help others accomplish theirs, too. Join our team as we help shape the future. Senior Data Engineer The...

  • Senior Data Architect

    2 days ago

    Charlotte, NC, United States CliftonLarsonAllen Full time

    CLA is a top 10 national professional services firm where our purpose is to create opportunities every day, for our clients, our people, and our communities through industry-focused wealth advisory, digital, audit, tax, consulting, and outsourcing services. Even with more than 8,500 people, 130 U.S. locations, and a global reach, we promise to know you and...

  • Senior Data Architect

    1 week ago

    Charlotte, NC, United States CliftonLarsonAllen Full time

    CLA is a top 10 national professional services firm where our purpose is to create opportunities every day, for our clients, our people, and our communities through industry-focused wealth advisory, digital, audit, tax, consulting, and outsourcing services. Even with more than 8,500 people, 130 U.S. locations, and a global reach, we promise to know you and...