Senior Databricks Data Engineer
Save this job and keep your search organized
Create a free account to save jobs, create alerts and return to this listing from your dashboard.
By continuing, you agree to our Terms & Privacy Policy.
- Remote
We are seeking an experienced Senior Databricks Data Engineer with strong expertise in designing, developing, and optimizing large-scale data engineering solutions on the Databricks platform. The ideal candidate will have extensive hands-on experience in building enterprise-grade data pipelines, processing high-volume datasets, and delivering scalable analytics solutions using PySpark, Spark SQL, and AWS cloud technologies.
Key Responsibilities
Design, develop, and maintain scalable data engineering solutions using Databricks, PySpark, Spark SQL, and Apache Spark. Build and optimize end-to-end ETL/ELT pipelines integrating Databricks with AWS S3, Snowflake, Airflow, and AWS Glue. Process and transform large-scale structured and semi-structured datasets to support enterprise reporting, analytics, and regulatory requirements. Develop reusable frameworks and components for: Data ingestion Data cleansing and transformation Data validation and reconciliation Data enrichment Business rule implementation Implement robust data quality checks, monitoring, and governance controls to ensure data accuracy and reliability. Perform performance tuning and optimization of Spark applications and Databricks workloads to improve scalability and processing efficiency. Collaborate with business stakeholders, architects, and analytics teams to understand data requirements and deliver scalable solutions. Troubleshoot and resolve production data pipeline issues while ensuring high availability and reliability. Support advanced analytics, customer segmentation, campaign analytics, AML/regulatory reporting, and data validation initiatives. Required
Skills:
Technical Skills 7+ years of hands-on experience with Databricks and modern data engineering platforms. Strong expertise in: PySpark Spark SQL Apache Spark Databricks Lakehouse Architecture. Experience building and managing ETL/ELT pipelines on cloud platforms. Strong experience with: AWS S3 Snowflake Apache Airflow AWS Glue Knowledge of data modeling, data warehousing, and big data processing techniques. Experience working with structured and semi-structured data formats such as JSON, Parquet, Avro, and CSV. Hands-on experience implementing data quality frameworks and monitoring solutions. Strong SQL and data analysis skills. Preferred Skills Experience with regulatory reporting, AML, compliance, or financial data domains. Exposure to CI/CD and DevOps practices for data engineering. Experience with Delta Lake, Databricks Workflows, and Unity Catalog. Knowledge of Agile/Scrum delivery methodologies. Key Experience Areas Candidates should have demonstrated experience in one or more of the following: Customer Segmentation Analytics Campaign Analytics AML and Regulatory Reporting Enterprise Data Platforms Production Data Pipeline Support Data Validation and Reconciliation Large-Scale Data Migration and Modernization Programs
-
Senior Databricks Data Engineer
3 days ago
New York, NY, United States Intone Inc Full-timeIntone Inc. is seeking a Senior Data Engineer in New York City with deep Databricks expertise to design and own data architecture for complex financial systems. The role emphasizes end-to-end ownership of data models, ingest, transformation, and publishing layers, and requires leadership in data architecture decisions. You will build full-pipeline solutions,...
-
Senior Databricks Data Engineer: Delta Lake
3 days ago
New York, NY, United States Talentify.io Full-timeTalentify is seeking a Databricks Data Engineer to design, develop, and implement data engineering solutions on the Databricks platform to support ingestion, transformation, reporting, analytics, and SIC processing. You will collaborate with finance, IT, and BI teams to ensure seamless integration and high-quality data, while staying up-to-date with Unity...
-
Senior Databricks
5 days ago
New York, NY, United States Franklin Fitch Full-time $80,000 - $110,000A consulting firm is seeking a Senior Consultant specializing in data engineering to join a hybrid team in Manhattan, NY. The role involves designing and implementing scalable data solutions using a modern tech stack including Databricks and Snowflake. Ideal candidates will have strong SQL and Python skills, alongside experience with data integration and...
-
Senior Databricks Data Engineer — SIC
6 days ago
New York, NY, United States Voktio Full-time $85,000 - $121,000Voktio in New York, NY is seeking a senior data engineer to design and manage Databricks data solutions, including pipelines, data models, analytics, and SIC processing. You will build and optimize Databricks pipelines, Delta Lake tables, and workflows; develop data solutions for quotas, commissions, attainment, and payout processing; translate SIC rules...
-
Databricks Senior Data Engineer
20 hours ago
New York, NY, United States SmartIT Frame Full-timeHiring: Senior Data Engineer – Databricks/PySpark Location: New York, NY – Onsite Requirements: 7+ years of Data Engineering experience 4+ years hands-on with Databricks & Spark/PySpark Strong Python, SQL, Delta Lake Experience with AWS (S3, Glue) and Google Cloud Platform (BigQuery, GCS) Strong ETL/ELT, Spark optimization & data...
-
Senior Databricks Python Engineer
5 days ago
New York, NY, United States Envision Technology Solutions Full-time $120,000 - $180,000Envision Technology Solutions in New York City, NY / Jersey City, NJ is seeking a senior Data Engineer to design and optimize large-scale data pipelines and data lakehouse architectures. You will work with Databricks, Python, PySpark, Spark SQL and Delta Lake to enable robust analytics across diverse data sources.The role requires strong SQL, data modelling,...
-
Senior Databricks Data Engineer
1 week ago
New York, NY, United States Ethereum Technologies Full-time $140,000 - $190,000ETHEREUM TECHNOLOGIES LLC is seeking a senior data engineer to manage and support batch processes, data pipelines in Azure Databricks and Azure Data Factory, and Bloomberg market data feeds within trading platforms.You will monitor, troubleshoot, and optimize ETL pipelines using Python, Spark, and SQL, and collaborate with operations, support, and...
-
Senior Data Engineer
3 days ago
New York, NY, United States Inizio Partners Full-timeAbout the job Senior Data Engineer - Databricks Job Overview: We are seeking a skilled Data Engineer to join our team. The successful candidate will be responsible for development and optimization of data pipelines, implementing robust data checks, and ensuring the accuracy and integrity of data flows. This role is critical in supporting data-driven...
-
Data Engineer
3 days ago
New York, NY, United States Motion Recruitment Full-timeData Engineer (Databricks)Dallas, Texas 100% Remote Full Time $115k - $150k Seeking a Data Engineer for a full-time opportunity supporting a leading research, analytics, and technology organization within the utility and energy industry. This role will focus on designing, building, and optimizing modern data solutions leveraging Databricks, Spark, Python,...
-
Senior Data Engineer
3 days ago
New York, NY, United States Optiveum Full-timeSenior Data Engineer (Databricks & PySpark) Location: 100% Remote (Warsaw, PL) Rate: Up to 38 EUR / hour Mode of work: Full-time (40 hours/week) Contracting party: Optiveum (B2B Cooperation Agreement) About the role: We are seeking an experienced Senior Data Engineer on behalf of our client to design, build, and operate their Sustainability Data...