Senior Data Engineer in Oaks PA
Save this job and keep your search organized
Create a free account to save jobs, create alerts and return to this listing from your dashboard.
By continuing, you agree to our Terms & Privacy Policy.
Amtex Systems Inc is an information technology and talent solutions company offering talent and BI consulting to the companies in US for over 25 years.
Our solutions are designed to fill resource gaps, by providing the right candidates who deliver value to the organization. Our propensity to nurture and build strong relationships with our clients helps us better understand their business demands and gives us the ability to provide services that are on time and rise above the rest.
Senior Data Engineer
Location: Oaks PA | Onsite
Duration: 6+ months
Locals preferred for onsite
Job Description:
- Lead the design and architecture of dbt projects, including project structure, layering conventions (staging, intermediate, marts), naming standards, and reusable package strategy.
- Design and develop advanced dbt models using complex SQL, choosing optimal materializations including views, tables, ephemeral, and incremental models with appropriate unique keys, merge strategies, and on_schema_change handling.
- Build, deploy, and optimize dbt models on Snowflake, applying Snowflake-specific configurations (transient tables, clustering keys, query tags, grants, and warehouse selection) and tuning dbt workloads for performance and credit consumption.
- Apply advanced dbt capabilities including Jinja macros, custom generic tests, snapshots (SCD Type 2), seeds, hooks, exposures, model contracts, versions, groups, access controls, and environment-specific configuration via profiles and vars.
- Build and maintain shared dbt macros and packages (e.g., dbt_utils, dbt_expectations) and promote reuse and consistency across teams.
- Own dbt documentation, lineage, and metadata, and use state-based selection (state:modified, deferral) and slim CI to speed up development and deployment.
- Create new Apache Airflow DAGs from scratch and architect Airflow orchestration, designing scalable, idempotent DAGs with clear dependency management, retries, SLAs, alerting, and failure-handling patterns.
- Execute and operate DAGs hands-on: trigger scheduled, manual, and parameterized runs through the Airflow UI and CLI, monitor task execution, review logs, clear and re-run failed tasks, and perform backfills.
- Apply advanced Airflow concepts including TaskFlow API, dynamic task mapping, task groups, sensors and deferrable operators, datasets/data-aware scheduling, XComs, pools, priority weights, custom operators and hooks, and connections and variables management.
- Integrate dbt with Airflow using proven patterns (e.g., Astronomer Cosmos, BashOperator/KubernetesPodOperator, or dbt Cloud operators), enabling model-level visibility, selective runs, and safe re-runs and backfills.
- Define and enforce orchestration standards for scheduling, catchup and backfill strategy, parameterization, logging, observability, and environment promotion.
- Write and tune high-performance SQL using joins, CTEs, aggregations, window functions, and set-based logic; analyze execution plans and optimize for performance and cost at scale.
- Work with Oracle databases to understand and optimize schemas, partitioning, indexing, data types, constraints, query behavior, and connectiv