Senior Data Engineer in Oaks PA

4 days ago

Phoenixville, PA, United States Amtex Systems Inc. Full-time

Amtex Systems Inc is an information technology and talent solutions company offering talent and BI consulting to the companies in US for over 25 years.

Our solutions are designed to fill resource gaps, by providing the right candidates who deliver value to the organization. Our propensity to nurture and build strong relationships with our clients helps us better understand their business demands and gives us the ability to provide services that are on time and rise above the rest.


Senior Data Engineer  

Location: Oaks PA | Onsite

Duration: 6+ months

Locals preferred for onsite

 

Job Description: 

We are seeking a Senior Data Engineer with 10-15 years of experience, including deep hands-on expertise in dbt and Apache Airflow, to architect, build, and own scalable, production-grade data transformation and orchestration platforms. The successful candidate will set technical direction for dbt project design and Airflow orchestration patterns, establish engineering standards and data-quality practices, mentor other engineers, and partner with architects and business stakeholders to deliver reliable, well-governed data products. Strong SQL, solid Oracle database knowledge, and proficient Python skills are expected. Hands-on dbt expertise on Snowflake is mandatory, and the candidate must have practical, hands-on Apache Airflow experience in both creating new DAGs and executing, monitoring, and managing DAG runs.
 
Key Responsibilities
  • Lead the design and architecture of dbt projects, including project structure, layering conventions (staging, intermediate, marts), naming standards, and reusable package strategy.
  • Design and develop advanced dbt models using complex SQL, choosing optimal materializations including views, tables, ephemeral, and incremental models with appropriate unique keys, merge strategies, and on_schema_change handling.
  • Build, deploy, and optimize dbt models on Snowflake, applying Snowflake-specific configurations (transient tables, clustering keys, query tags, grants, and warehouse selection) and tuning dbt workloads for performance and credit consumption.
  • Apply advanced dbt capabilities including Jinja macros, custom generic tests, snapshots (SCD Type 2), seeds, hooks, exposures, model contracts, versions, groups, access controls, and environment-specific configuration via profiles and vars.
  • Build and maintain shared dbt macros and packages (e.g., dbt_utils, dbt_expectations) and promote reuse and consistency across teams.
  • Own dbt documentation, lineage, and metadata, and use state-based selection (state:modified, deferral) and slim CI to speed up development and deployment.
  • Create new Apache Airflow DAGs from scratch and architect Airflow orchestration, designing scalable, idempotent DAGs with clear dependency management, retries, SLAs, alerting, and failure-handling patterns.
  • Execute and operate DAGs hands-on: trigger scheduled, manual, and parameterized runs through the Airflow UI and CLI, monitor task execution, review logs, clear and re-run failed tasks, and perform backfills.
  • Apply advanced Airflow concepts including TaskFlow API, dynamic task mapping, task groups, sensors and deferrable operators, datasets/data-aware scheduling, XComs, pools, priority weights, custom operators and hooks, and connections and variables management.
  • Integrate dbt with Airflow using proven patterns (e.g., Astronomer Cosmos, BashOperator/KubernetesPodOperator, or dbt Cloud operators), enabling model-level visibility, selective runs, and safe re-runs and backfills.
  • Define and enforce orchestration standards for scheduling, catchup and backfill strategy, parameterization, logging, observability, and environment promotion.
  • Write and tune high-performance SQL using joins, CTEs, aggregations, window functions, and set-based logic; analyze execution plans and optimize for performance and cost at scale.
  • Work with Oracle databases to understand and optimize schemas, partitioning, indexing, data types, constraints, query behavior, and connectiv