PySpark Developer with Kubernetes Experience

3 days ago


New York, United States Mindlance Full time

PySpark Developer with Kubernetes Experience Duration: 12 months plus (possible extension or conversion) Location: HYBRID W/ ONSITE REQUIREMENT - 3 days onsite New York City, NY 10013 Job Summary: We are looking for an experienced PySpark Developer with strong expertise in Kubernetes to join our dynamic team. The ideal candidate will be responsible for designing, developing, and maintaining big data solutions using Apache Spark (PySpark), deploying scalable applications on Kubernetes, and ensuring smooth data pipeline operations . This role requires hands-on experience with data engineering, container orchestration, and cloud technologies . Responsibilities : - Develop and maintain data pipelines using PySpark for data transformation, aggregation, and loading . - Design, implement, and manage containerized applications on Kubernetes to ensure scalability, reliability, and efficiency. - Collaborate with data engineering and analytics teams to optimize PySpark jobs for performance and resource utilization. - Automate the deployment and scaling of PySpark jobs using Kubernetes tools and best practices. - Implement and monitor data workflows, ensuring high availability and resilience . - Troubleshoot and resolve issues related to data processing, Kubernetes deployments, and cloud infrastructure . - Work closely with DevOps and infrastructure teams to manage Kubernetes clusters and improve CI/CD pipelines for big data applications. - Create detailed documentation for data workflows, processes, and Kubernetes configuration . - Stay updated on the latest industry trends, particularly within the big data and container orchestration spaces. Requirements : - Bachelor's degree in Computer Science, Information Technology, or related field. - Proven experience as a PySpark Developer , with a solid understanding of Spark architecture and data processing concepts . - Hands-on experience with Kubernetes for container orchestration and deployment . - Strong knowledge of big data technologies, including Hadoop, HDFS, Spark, and Kafka (preferred). - Experience with cloud platforms (AWS, Google Cloud Platform, or Azure) and their Kubernetes services. - Familiarity with CI/CD tools and automated deployment processes. - Proficient in scripting languages (Python, Shell scripting) and experience with version control (Git). - Strong analytical and problem-solving skills, with an emphasis on optimizing data workflows. - Excellent communication and collaboration abilities, especially within cross-functional teams. Preferred Skills: - Experience with Docker and containerization . - Familiarity with monitoring and logging tools for Kubernetes, such as Prometheus and Grafana. - Knowledge of other big data tools and frameworks (Hive, HBase, Flink, etc.). - Certifications in Apache Spark, Kubernetes, or relevant cloud platforms. "Mindlance is an Equal Opportunity Employer and does not discriminate in employment on the basis of - Minority/Gender/Disability/Religion/LGBTQI/Age/Veterans."



  • New York, United States Mindlance Full time

    PySpark Developer with Kubernetes Experience Duration: 12 months plus (possible extension or conversion) Location: HYBRID W/ ONSITE REQUIREMENT - 3 days onsite New York City, NY 10013 Job Summary: We are looking for an experienced PySpark Developer with strong expertise in Kubernetes to join our dynamic team. The ideal candidate will be responsible for...


  • New York, New York, United States Diverse Lynx Full time

    Job Title: Databricks and PySpark DeveloperWe are seeking a highly skilled Databricks and PySpark Developer to join our team at Diverse Lynx LLC. The ideal candidate will have experience in developing scalable and efficient data processing pipelines using Databricks and PySpark.Key Responsibilities:Design and implement data processing workflows using...


  • New York, United States Diverse Lynx Full time

    • Minimum 4 years + working exp as Databricks Developer• Minimum 4 years + working exp on Python, PySpark and AWSNice to Have :• Preferable Software engineering background• Preferable AWS experience• Experience with Typescript Digital : Python, Digital : Databricks, Digital : PySpark Diverse Lynx LLC is an Equal Employment Opportunity employer....


  • New York, United States Diverse Lynx Full time

    •Primary Skills:o Pysparko AWS Cloudo Airflowo SQLo Databricks•Secondary Skills:o Snowflakeo GCP Cloud Digital : Databricks, Digital : PySpark 6-8 •Minimum 5 years + working exp as Databricks Developer•Minimum 3 years + working exp on PySpark and AWS •Minimum 5 years + working exp as Databricks Developer•Minimum 3 years + working exp on PySpark...


  • New York, United States Diverse Lynx Full time

    •Primary Skills:o Pysparko AWS Cloudo Airflowo SQLo Databricks•Secondary Skills:o Snowflakeo GCP Cloud Digital : Databricks, Digital : PySpark 6-8 •Minimum 5 years + working exp as Databricks Developer•Minimum 3 years + working exp on PySpark and AWS •Minimum 5 years + working exp as Databricks Developer•Minimum 3 years + working exp on PySpark...


  • New York, United States Diverse Lynx Full time

    • Primary Skills:o Pysparko AWS Cloudo Airflowo SQLo Databricks• Secondary Skills:o Snowflakeo GCP Cloud Digital : Databricks, Digital : PySpark 6-8 • Minimum 5 years + working exp as Databricks Developer• Minimum 3 years + working exp on PySpark and AWS • Minimum 5 years + working exp as Databricks Developer• Minimum 3 years + working exp on...


  • New York, United States Tech Tammina Full time

    Senior Fullstack Pyspark / Python Engineers Location: NYC, NY Duration: 1+ YearThe client is looking for Senior PySpark/Python Engineers who can: Improve performance and user experience of new applications. Work with data lakes, data warehouses, and ETL processes. Contribute to digital transformation projects. Utilize Agile software development frameworks....

  • Data Engineer

    7 days ago


    New York, New York, United States Diverse Lynx Full time

    Job Description:We are seeking a skilled Data Engineer with expertise in Databricks and PySpark to join our team at Diverse Lynx LLC.Key Responsibilities:Design, develop, and maintain large-scale data processing pipelines using Databricks and PySpark.Collaborate with cross-functional teams to integrate data solutions into business operations.Analyze complex...


  • New York, United States Diverse Lynx Full time

    Must Have Skills? Proficiency in containerization technologies such as Docker and Kubernetes.? Hands-on experience with front-end framework like React, Vue.js, Next.js or Angular.js? Experience in infrastructure as code (IaC) using Terraform, particularly with AWS provider? Strong Coding skills in Python, Java or Scala for data processing task.? Proficient...


  • New York, United States Diverse Lynx Full time

    Must Have Skills? Proficiency in containerization technologies such as Docker and Kubernetes.? Hands-on experience with front-end framework like React, Vue.js, Next.js or Angular.js? Experience in infrastructure as code (IaC) using Terraform, particularly with AWS provider? Strong Coding skills in Python, Java or Scala for data processing task.? Proficient...


  • New York, New York, United States Diverse Lynx Full time

    About the RoleDiverse Lynx LLC is seeking a highly skilled Senior Data Engineering Specialist to join our team as a Databricks/PySpark Developer. As a key member of our data engineering team, you will be responsible for designing, developing, and maintaining large-scale data processing systems using Databricks and PySpark.Key ResponsibilitiesDesign and...

  • Kubernetes Engineer

    1 month ago


    New York, United States Meta Resources Group Full time

    Our Client a large Pharmaceutical company is seeking a Kubernetes Engineer who will assist in the migration/integration of multiple companies into the clients current environment. The role will initially be a 6 month contract with the likely conversion to a fulltime hire. This is a remote position but may require to be onsite occasionally in the NYCSuburb...

  • Cloud Engineer

    6 days ago


    New York, New York, United States Capital One Full time

    Job Title: Cloud Engineer - Kubernetes Fleet ManagementAbout the Role:    We are seeking a skilled Cloud Engineer to join our team in New York City. As a DevOps Software Engineer, you will play a critical role in building and maintaining our self-service Kubernetes provisioning platform.    Your primary focus will be on developing Kubernetes...

  • Full Stack Developer

    3 weeks ago


    New London, United States Perfict Global, Inc. Full time

    About Us: Perfict Global is a leading IT consulting services provider focused on providing innovative and successful business workforce solutions to Fortune 500 companies. Our trained and experienced professionals constantly strive to bring together the best technologies available to manage client's complex business and technology, participate in...


  • New York, United States CoreWeave Full time

    Job DescriptionJob DescriptionCoreWeave is the AI Hyperscaler™, delivering a cloud platform of cutting edge services powering the next wave of AI. The company's technology provides enterprises and leading AI labs with the most performant, efficient and resilient solutions for accelerated computing. Since 2017, CoreWeave has operated a growing footprint...


  • New York, New York, United States Capital One Full time

    Enterprise Platforms Technology (EPTech) is a vital component of Capital One, playing a pivotal role in establishing best practices for building technology solutions across the company. We are seeking a Senior Software Engineer, DevOps to join our OneKube team.The successful candidate will be responsible for developing Kubernetes operators and other tooling...


  • New York, New York, United States Diverse Lynx Full time

    Company OverviewDiverse Lynx LLC is an Equal Employment Opportunity employer. We promote and support a diverse workforce across all levels in the company.SalaryThe estimated salary for this position is $120,000 - $180,000 per year, depending on experience.Job DescriptionWe are seeking a highly skilled Databricks / PySpark Developer to join our team. As a key...


  • West New York, United States Zenith services Full time

    Job Summary: We are seeking a highly skilled Full Stack Developer to join our team. The successful candidate will have experience in developing scalable applications using Java React Spring and SQL. The ideal candidate should have a strong understanding of software development principles be able to write clean and efficient code and have experience with...

  • Sr. Spark Engineer

    20 hours ago


    New York, United States TEKsystems Full time

    *Description* Description The role of the Sr. Data Engineer is responsible for building and maintaining optimized and highly available data pipelines that facilitate deeper analysis and reporting. This engineer's duty is to monitor the existing metrics, analyze data, and lead partnerships with other Data and Analytics teams in an effort to identify and...


  • York, United States Capital One Full time

    Center 1 (19052), United States of America, McLean, VirginiaSenior Software Engineer (Python, AWS, Pyspark )Do you love building and pioneering in the technology space? Do you enjoy solving complex business problems in a fast-paced, collaborative, inclusive, and iterative delivery environment? At Capital One, you'll be part of a big group of makers,...