Senior Data Engineer

2 days ago

Beverly Hills, California, United States Aloola.io Full-time
Senior Data Engineer

Location:
Beverly Hills, CA (in person)

ABOUT THE ROLE
We are hiring a Senior Data Engineer with 5+ years of experience to design, build, and operate a production data platform end to end. You will own ingestion, transformation, orchestration, and data quality on a modern cloud warehouse and lakehouse stack: Amazon Redshift or Databricks, Apache Airflow, dbt, and Apache Spark. This is a hands-on role for someone who has run these systems in production and is accountable for their performance, cost, and reliability.

WHAT YOU'LL DO
• Own data models and pipelines on Amazon Redshift or Databricks, including performance tuning, cost management, and workload isolation.
• Author and operate Apache Airflow DAGs at scale, managing dependencies, retries, SLAs, and observability.
• Build and maintain dbt transformations: model design, testing, macros, documentation, and CI/CD.
• Develop and run ingestion pipelines using tools such as Fivetran, Airbyte, AWS DMS, Kafka/Kinesis, or custom CDC and API ingestion frameworks.
• Manage lakehouse tables in open table formats (Apache Iceberg, Delta Lake, or Hudi), covering partitioning, schema evolution, compaction, and time travel.
• Write large-scale batch and streaming transformations in PySpark or Spark SQL.
• Write advanced, well-optimized SQL and production-grade Python across the platform.
• Put data governance, quality checks, and observability in place so issues are caught before they reach data consumers. REQUIRED QUALIFICATIONS Candidates should demonstrate 5+ years of data engineering experience, with depth in:
• Cloud data warehouse / lakehouse: Amazon Redshift or Databricks, with real production ownership of modeling, performance tuning, cost management, and workload isolation.
• Python: strong, production-level proficiency.
• Apache Airflow: authoring and operating DAGs at scale, including dependency management, retries, SLAs, and observability.
• dbt: model design, testing, macros, documentation, and CI/CD for transformations.
• Data movement / ingestion tooling: for example Fivetran, Airbyte, AWS DMS, Kafka/Kinesis, or custom CDC and API ingestion frameworks.
• Open table formats: Apache Iceberg, Delta Lake, or Hudi, including partitioning, schema evolution, compaction, and time travel.
• Apache Spark: PySpark or Spark SQL for large-scale batch and streaming transformation.
• SQL: advanced, including window functions and query optimization.
• Data governance, quality, and observability: hands-on experience with tooling such as Great Expectations, Monte Carlo, or Elementary. NICE TO HAVE
• AWS ecosystem depth: S3, Glue, Lambda, IAM, and EMR.
• Experience with streaming or near-real-time pipelines.
• Infrastructure-as-code (Terraform), Docker, and CI/CD (GitHub Actions).