Lead Data Engineer + AI

2 weeks ago

Boston, MA, United States DCM INFOTECH LIMITED Full time
Position - Lead Data Engineer
Location - Boston or, Remote
Experience - 10 years to 15years
Note: Please submit only genuine candidate with LinkedIn.

Need minimum 3 years of experience as Lead.
About the role
We're looking for a Senior Data Engineer to build and scale our lakehouse and AI data pipelines on Databricks. You'll design robust ETL/ELT, enable feature engineering for ML/LLM use cases, and drive best practices for reliability, performance, and cost.
What you'll do
  • Design, build, and maintain batch/streaming pipelines in Python + PySpark on Databricks (Delta Lake, Autoloader, Structured Streaming).
  • Implement data models (Bronze/Silver/Gold), optimize with partitioning, Z-ORDER, and indexing, and manage reliability (DLT/Jobs, monitoring, alerting).
  • Enable ML/AI: feature engineering, MLflow experiment tracking, model registries, and model/feature serving; support RAG pipelines (embeddings, vector stores).
  • Establish data quality checks (e.g., Great Expectations), lineage, and governance (Unity Catalog, RBAC).
  • Collaborate with Data Science/ML and Product to productionize models and AI workflows; champion CI/CD and IaC.
  • Troubleshoot performance and cost issues; mentor engineers and set coding standards.
Must-have qualifications
  • 10+ years in data engineering with a track record of production pipelines.
  • Expert in Python and PySpark (UDFs, Window functions, Spark SQL, Catalyst basics).
  • Deep hands-on Databricks: Delta Lake, Jobs/Workflows, Structured Streaming, SQL Warehouses; practical tuning and cost optimization.
  • Strong SQL and data modeling (dimensional, medallion, CDC).
  • ML/AI enablement experience: MLflow, feature stores, model deployment/monitoring; familiarity with LLM workflows (embeddings, vectorization, prompt/response logging).
  • Cloud proficiency on AWS/Azure/GCP (object storage, IAM, networking).
  • CI/CD (GitHub/GitLab/Azure DevOps), testing (pytest), and observability (logs/metrics).
Nice to have
  • Databricks Delta Live Tables, Unity Catalog automation, Model Serving.
  • Orchestration (Airflow/Databricks Workflows), messaging (Kafka/Kinesis/Event Hubs).
  • Data quality & lineage tools (Great Expectations, OpenLineage).
  • Vector DBs (FAISS, pgvector, Pinecone), RAG frameworks (LangChain/LlamaIndex).
  • IaC (Terraform), security/compliance (PII handling, data masking).
  • Experience interfacing with BI tools (Power BI, Tableau, Databricks SQL).

  • AI and Gen. AI Data Scientist and Data Engineers

    5 days ago

    Boston, MA, United States Saviance Full time

    Job Title: AI and Gen. AI Data Scientist and Data Engineers Location: India, Remote Part time/Consulting About BigRio: BigRio is a remote-based, technology consulting firm with headquarters in Boston, MA. We deliver software solutions ranging from: custom development, software implementation, data analytics, and machine learning/AI integrations. We...

  • AI and Gen. AI Data Scientist and Data Engineers

    1 week ago

    Boston, MA, United States Saviance Full time

    Job Title: AI and Gen. AI Data Scientist and Data Engineers Location: India, Remote Part time/Consulting About BigRio: BigRio is a remote-based, technology consulting firm with headquarters in Boston, MA. We deliver software solutions ranging from: custom development, software implementation, data analytics, and machine learning/AI integrations. We...

  • AI and Gen. AI Data Scientist and Data Engineers

    2 weeks ago

    Boston, MA, United States Saviance Full time

    Job Title: AI and Gen. AI Data Scientist and Data Engineers Location: India, Remote Part time/Consulting About BigRio: BigRio is a remote-based, technology consulting firm with headquarters in Boston, MA. We deliver software solutions ranging from: custom development, software implementation, data analytics, and machine learning/AI integrations. We...

  • Data Engineer + AI

    2 weeks ago

    Boston, MA, United States DCM INFOTECH LIMITED Full time

    Position - Data Engineer + AI Location - Boston or, Remote Experience - 5 years to 10 years Note: Please submit only genuine candidate with LinkedIn. Local will be preferred. About the role We're looking for a Senior Data Engineer to build and scale our lakehouse and AI data pipelines on Databricks. You'll design robust ETL/ELT, enable feature engineering...

  • Data Engineer + AI

    1 week ago

    Boston, MA, United States DCM INFOTECH LIMITED Full time

    Position - Data Engineer + AI Location - Boston or, Remote Experience - 5 years to 10 years Note: Please submit only genuine candidate with LinkedIn. Local will be preferred. About the role We're looking for a Senior Data Engineer to build and scale our lakehouse and AI data pipelines on Databricks. You'll design robust ETL/ELT, enable feature engineering...

  • Data Engineer + AI

    1 week ago

    Boston, MA, United States DCM INFOTECH LIMITED Full time

    Position - Data Engineer + AI Location - Boston or, Remote Experience - 5 years to 10 years Note: Please submit only genuine candidate with LinkedIn. Local will be preferred. About the role We're looking for a Senior Data Engineer to build and scale our lakehouse and AI data pipelines on Databricks. You'll design robust ETL/ELT, enable feature engineering...

  • AI Data Engineer

    2 weeks ago

    Boston, MA, United States C the Signs Full time

    Position Summary The Data Engineer will play a crucial role in developing and fine-tuning data specifically for our LLMs and machine learning models. This individual will be responsible for the entire data lifecycle, including gathering, cleaning, structuring, and optimizing large, diverse healthcare datasets. The ideal candidate will have a strong...

  • AI Data Engineer

    2 weeks ago

    Boston, MA, United States C the Signs Full time

    Position Summary The Data Engineer will play a crucial role in developing and fine-tuning data specifically for our LLMs and machine learning models. This individual will be responsible for the entire data lifecycle, including gathering, cleaning, structuring, and optimizing large, diverse healthcare datasets. The ideal candidate will have a strong...

  • Senior Software Engineer

    1 week ago

    Boston, MA, United States Saviance Full time

    Senior Software Engineer - AI & Data Engineering Location: Boston, MA (Onsite every Thursday) About BigRio BigRio is a Boston-based technology consulting firm specializing in AI/ML, data engineering, custom software development, and digital transformation, with a strong focus on the healthcare and life sciences domains. We partner with leading...

  • Salesforce Tech Lead

    2 weeks ago

    Boston, MA, United States Zelis Full time

    At Zelis, we Get Stuff Done. So, let's get to it! A Little About Us Zelis is modernizing the healthcare financial experience across payers, providers, and healthcare consumers. We serve more than 750 payers, including the top five national health plans, regional health plans, TPAs and millions of healthcare providers and consumers across our platform of...