Data Engineer + AI

2 weeks ago

Boston, MA, United States DCM INFOTECH LIMITED Full time
Position - Data Engineer + AI
Location - Boston or, Remote
Experience - 5 years to 10 years
Note: Please submit only genuine candidate with LinkedIn.
Local will be preferred.
About the role
We're looking for a Senior Data Engineer to build and scale our lakehouse and AI data pipelines on Databricks. You'll design robust ETL/ELT, enable feature engineering for ML/LLM use cases, and drive best practices for reliability, performance, and cost.
What you'll do
  • Design, build, and maintain batch/streaming pipelines in Python + PySpark on Databricks (Delta Lake, Autoloader, Structured Streaming).
  • Implement data models (Bronze/Silver/Gold), optimize with partitioning, Z-ORDER, and indexing, and manage reliability (DLT/Jobs, monitoring, alerting).
  • Enable ML/AI: feature engineering, MLflow experiment tracking, model registries, and model/feature serving; support RAG pipelines (embeddings, vector stores).
  • Establish data quality checks (e.g., Great Expectations), lineage, and governance (Unity Catalog, RBAC).
  • Collaborate with Data Science/ML and Product to productionize models and AI workflows; champion CI/CD and IaC.
  • Troubleshoot performance and cost issues; mentor engineers and set coding standards.
Must-have qualifications
  • 6-10+ years in data engineering with a track record of production pipelines.
  • Expert in Python and PySpark (UDFs, Window functions, Spark SQL, Catalyst basics).
  • Deep hands-on Databricks: Delta Lake, Jobs/Workflows, Structured Streaming, SQL Warehouses; practical tuning and cost optimization.
  • Strong SQL and data modeling (dimensional, medallion, CDC).
  • ML/AI enablement experience: MLflow, feature stores, model deployment/monitoring; familiarity with LLM workflows (embeddings, vectorization, prompt/response logging).
  • Cloud proficiency on AWS/Azure/GCP (object storage, IAM, networking).
  • CI/CD (GitHub/GitLab/Azure DevOps), testing (pytest), and observability (logs/metrics).
Nice to have
  • Databricks Delta Live Tables, Unity Catalog automation, Model Serving.
  • Orchestration (Airflow/Databricks Workflows), messaging (Kafka/Kinesis/Event Hubs).
  • Data quality & lineage tools (Great Expectations, OpenLineage).
  • Vector DBs (FAISS, pgvector, Pinecone), RAG frameworks (LangChain/LlamaIndex).
  • IaC (Terraform), security/compliance (PII handling, data masking).
  • Experience interfacing with BI tools (Power BI, Tableau, Databricks SQL).

  • AI and Gen. AI Data Scientist and Data Engineers

    1 week ago

    Boston, MA, United States Saviance Full time

    Job Title: AI and Gen. AI Data Scientist and Data Engineers Location: India, Remote Part time/Consulting About BigRio: BigRio is a remote-based, technology consulting firm with headquarters in Boston, MA. We deliver software solutions ranging from: custom development, software implementation, data analytics, and machine learning/AI integrations. We...

  • AI and Gen. AI Data Scientist and Data Engineers

    2 weeks ago

    Boston, MA, United States Saviance Full time

    Job Title: AI and Gen. AI Data Scientist and Data Engineers Location: India, Remote Part time/Consulting About BigRio: BigRio is a remote-based, technology consulting firm with headquarters in Boston, MA. We deliver software solutions ranging from: custom development, software implementation, data analytics, and machine learning/AI integrations. We...

  • AI and Gen. AI Data Scientist and Data Engineers

    5 days ago

    Boston, MA, United States Saviance Full time

    Job Title: AI and Gen. AI Data Scientist and Data Engineers Location: India, Remote Part time/Consulting About BigRio: BigRio is a remote-based, technology consulting firm with headquarters in Boston, MA. We deliver software solutions ranging from: custom development, software implementation, data analytics, and machine learning/AI integrations. We...

  • AI Data Engineer

    2 weeks ago

    Boston, MA, United States C the Signs Full time

    Position Summary The Data Engineer will play a crucial role in developing and fine-tuning data specifically for our LLMs and machine learning models. This individual will be responsible for the entire data lifecycle, including gathering, cleaning, structuring, and optimizing large, diverse healthcare datasets. The ideal candidate will have a strong...

  • AI Data Engineer

    2 weeks ago

    Boston, MA, United States C the Signs Full time

    Position Summary The Data Engineer will play a crucial role in developing and fine-tuning data specifically for our LLMs and machine learning models. This individual will be responsible for the entire data lifecycle, including gathering, cleaning, structuring, and optimizing large, diverse healthcare datasets. The ideal candidate will have a strong...

  • Data Engineer with AI

    2 weeks ago

    Boston, MA, United States Lorven Technologies Full time

    I hope you are doing well, Please share your updated profile if you are interested in the below role. Our client seeks an Data Engineer + AI for a 12 Months project in Boston, MA. Below is the detailed requirement Job Title: Data Engineer + AI Work location : Boston, MA Duration: 12 Months Job Summary: We're looking for a Senior Data Engineer to build...

  • Lead Data Engineer + AI

    1 week ago

    Boston, MA, United States DCM INFOTECH LIMITED Full time

    Position - Lead Data Engineer Location - Boston or, Remote Experience - 10 years to 15years Note: Please submit only genuine candidate with LinkedIn.Need minimum 3 years of experience as Lead. About the role We're looking for a Senior Data Engineer to build and scale our lakehouse and AI data pipelines on Databricks. You'll design robust ETL/ELT, enable...

  • Lead Data Engineer + AI

    1 week ago

    Boston, MA, United States DCM INFOTECH LIMITED Full time

    Position - Lead Data Engineer Location - Boston or, Remote Experience - 10 years to 15years Note: Please submit only genuine candidate with LinkedIn.Need minimum 3 years of experience as Lead. About the role We're looking for a Senior Data Engineer to build and scale our lakehouse and AI data pipelines on Databricks. You'll design robust ETL/ELT, enable...

  • Lead Data Engineer + AI

    2 weeks ago

    Boston, MA, United States DCM INFOTECH LIMITED Full time

    Position - Lead Data Engineer Location - Boston or, Remote Experience - 10 years to 15years Note: Please submit only genuine candidate with LinkedIn.Need minimum 3 years of experience as Lead. About the role We're looking for a Senior Data Engineer to build and scale our lakehouse and AI data pipelines on Databricks. You'll design robust ETL/ELT, enable...

  • Senior Software Engineer

    1 week ago

    Boston, MA, United States Saviance Full time

    Senior Software Engineer - AI & Data Engineering Location: Boston, MA (Onsite every Thursday) About BigRio BigRio is a Boston-based technology consulting firm specializing in AI/ML, data engineering, custom software development, and digital transformation, with a strong focus on the healthcare and life sciences domains. We partner with leading...