Principal Data Engineer

11 hours ago

Fort Mill, SC, United States InterSources Full time
Role - Principal Data Engineer
Location - Fort Mill SC or Austin (Hybrid)

We are looking for a skilled Data Engineer to join our team and help build robust, scalable, and efficient data pipelines. The ideal candidate will have strong expertise in AWS, Python, Spark, ETL Pipelines, SQL, and Pytest. This role involves designing, implementing, and optimizing data pipelines to support analytics, business intelligence, and machine learning initiatives.

Key Responsibilities:
• Design, develop, and maintain ETL pipelines using AWS services, Python, and Spark.
• Optimize data ingestion, transformation, and storage processes for high-performance data processing.
• Work with structured and unstructured data, ensuring data integrity, quality, and governance.
• Develop SQL queries to extract and manipulate data efficiently from relational databases.
• Implement data validation and testing frameworks using Pytest to ensure data accuracy and reliability.
• Collaborate with data scientists, analysts, and software engineers to build scalable data solutions.
• Monitor and troubleshoot data pipelines to ensure smooth operation and minimal downtime.
• Stay up-to-date with industry trends, tools, and best practices for data engineering and cloud technologies.

Required Skills & Qualifications:
• Experience in Data Engineering or a related field.
• Strong proficiency in AWS (S3, Glue, Lambda, EMR, Redshift, etc.) for cloud-based data processing.
• Hands-on experience with Python for data processing and automation.
• Expertise in Apache Spark for distributed data processing.
• Solid understanding of ETL pipeline design and data warehousing concepts.
• Proficiency in SQL for querying and managing relational databases.
• Experience writing unit and integration tests using Pytest.
• Familiarity with CI/CD pipelines and version control systems (e.g., Git).
• Strong problem-solving skills and ability to work in a fast-paced environment.

Preferred Qualifications:
• Experience with Terraform, Docker, or Kubernetes.
• Knowledge of big data tools such as Apache Kafka or Airflow.
• Exposure to data governance and security best practices.

  • Principal Data Engineer

    4 days ago

    Fort Mill, SC, United States Shutterfly Career Site Full time

    At Shutterfly, we make life's experiences unforgettable. We believe there is extraordinary power in the self-expression. That's why our family of brands helps customers create products and capture moments that reflect who they uniquely are. We're seeking a Principal Data Engineer/Systems Architect with 7-10 years of industry experience working in...

  • Principal Data Engineer

    1 week ago

    Fort Mill, SC, United States Shutterfly Career Site Full time

    At Shutterfly, we make life's experiences unforgettable. We believe there is extraordinary power in the self-expression. That's why our family of brands helps customers create products and capture moments that reflect who they uniquely are. We're seeking a Principal Data Engineer/Systems Architect with 7-10 years of industry experience working in...

  • Principal Data Engineer

    4 days ago

    Fort Mill, SC, United States Shutterfly Career Site Full time

    At Shutterfly, we make life's experiences unforgettable. We believe there is extraordinary power in the self-expression. That's why our family of brands helps customers create products and capture moments that reflect who they uniquely are. We're seeking a Principal Data Engineer/Systems Architect with 7-10 years of industry experience working in...

  • Principal AI Engineer

    1 week ago

    Fort Mill, SC, United States Shutterfly Career Site Full time

    At Shutterfly, we make life's experiences unforgettable. We believe there is extraordinary power in the self-expression. That's why our family of brands helps customers create products and capture moments that reflect who they uniquely are. We are seeking a Principal AI Engineer with deep expertise in computer vision and machine learning to lead the...

  • Principal AI Engineer

    15 hours ago

    Fort Mill, SC, United States Shutterfly Career Site Full time

    At Shutterfly, we make life's experiences unforgettable. We believe there is extraordinary power in the self-expression. That's why our family of brands helps customers create products and capture moments that reflect who they uniquely are. We are seeking a Principal AI Engineer with deep expertise in computer vision and machine learning to lead the...

  • Principal Angular Developer

    2 weeks ago

    Fort Mill, SC, United States Insight Global Full time

    A client is looking to identify a Principal Angular Developer. As a Principal Developer you will have strong UI engineering and architecture experience. Potential candidate will be responsible to work closely with business to gain deep understanding of complex user journeys, workflows and data interactions involved in Index business lifecycle so to come up...

  • AWS Data Engineer

    6 days ago

    Fort Mill, SC, United States Experis Full time

    Our client, a leading organization in the technology and data solutions sector, is seeking a Cloud Engineer to join their dynamic team. As a Cloud Engineer, you will be an integral part of the Data Development Department supporting innovative cloud-based data platforms. The ideal candidate will demonstrate strong problem-solving skills, adaptability, and...

  • Sr Engineer, Data Reporting

    6 days ago

    Fort Mill, SC, United States LPL Financial Full time

    What if you could build a career where ambition meets innovation? At LPL Financial, we empower professionals to shape their success while helping clients pursue their financial goals with confidence. What if you could have access to cutting-edge resources, a collaborative environment, and the freedom to make an impact? If you're ready to take the next step,...

  • Sr Engineer, Data Reporting

    5 days ago

    Fort Mill, SC, United States LPL Financial Holdings, Inc. Full time

    What if you could build a career where ambition meets innovation? At LPL Financial, we empower professionals to shape their success while helping clients pursue their financial goals with confidence. What if you could have access to cutting-edge resources, a collaborative environment, and the freedom to make an impact? If you're ready to take the next step,...

  • Sr Engineer, Data Reporting

    1 week ago

    Fort Mill, SC, United States LPL Financial Holdings, Inc. Full time

    What if you could build a career where ambition meets innovation? At LPL Financial, we empower professionals to shape their success while helping clients pursue their financial goals with confidence. What if you could have access to cutting-edge resources, a collaborative environment, and the freedom to make an impact? If you're ready to take the next step,...