Big Data Specialist
1 week ago
Job Description
Big Data (PySpark) Tech Lead
- 10 + Years Overall Experience in Data Management, Data Lake and Data Warehouse
- 6+ Years Hadoop, Hive, Sqoop, SQL, Teradata
- 6+ Years PySpark(Python and Spark), Unix
- Good to have Industry leading ETL experience
- Banking Domain experience
Key Responsibilities
- Ability to design, build and unit test applications on Spark framework on Python.
- Build PySpark based applications for both batch and streaming requirements, which will require in-depth knowledge on majority of Hadoop and NoSQL databases as well.
- Develop and execute data pipeline testing processes and validate business rules and policies
- Optimize performance of the built Spark applications in Hadoop using configurations around Spark Context, Spark-SQL, Data Frame, and Pair RDD's.
- Optimize performance for data access requirements by choosing the appropriate native Hadoop file formats (Avro, Parquet, ORC etc) and compression codec respectively.
- Ability to design & build real-time applications using Apache Kafka & Spark Streaming
- Build integrated solutions leveraging Unix shell scripting, RDBMS, Hive, HDFS File System, HDFS File Types, HDFS compression codec.
- Build data tokenization libraries and integrate with Hive & Spark for column-level obfuscation
- Experience in processing large amounts of structured and unstructured data, including integrating data from multiple sources.
- Create and maintain integration and regression testing framework on Jenkins integrated with BitBucket and/or GIT repositories
- Participate in the agile development process, and document and communicate issues and bugs relative to data standards in scrum meetings
- Work collaboratively with onsite and offshore team.
- Develop & review technical documentation for artifacts delivered.
- Ability to solve complex data-driven scenarios and triage towards defects and production issues
- Ability to learn-unlearn-relearn concepts with an open and analytical mindset
- Participate in code release and production deployment.
- Challenge and inspire team members to achieve business results in a fast paced and quickly changing environment
-
Big Data Analytics Specialist
6 days ago
Irving, Texas, United States Diverse Lynx Full timeJob SummaryDiverse Lynx LLC seeks a highly skilled Big Data Analytics Specialist to leverage Apache Spark and PySpark expertise in our digital transformation initiatives. Key responsibilities include:Designing and implementing scalable data pipelines using Apache SparkDeveloping and maintaining complex data processing workflows with PySparkCollaborating with...
-
Big Data Specialist
4 weeks ago
Irving, United States Tata Consultancy Services Full timeJob DescriptionBig Data (PySpark) Tech Lead10 + Years Overall Experience in Data Management, Data Lake and Data Warehouse6+ Years Hadoop, Hive, Sqoop, SQL, Teradata6+ Years PySpark(Python and Spark), UnixGood to have Industry leading ETL experienceBanking Domain experience Key ResponsibilitiesAbility to design, build and unit test applications on Spark...
-
Big Data Developer
2 weeks ago
Irving, United States Resource Informatics Group Full timeNote: It is a W2 Opportunity, Client is accepting candidates from (New Jersey and New York) other location candidates will be straight away rejected and application candidate must have prior banking domain work experience. Role: Big Data Developer Location: Iselin, NJ In office: 3 days a week onsite Contract: 6-24 months to perm Interview process: screening...
-
Big Data Engineer
6 days ago
Irving, Texas, United States Diverse Lynx Full timeJob DescriptionWe are seeking a highly skilled Big Data Engineer to join our team at Diverse Lynx LLC.About UsDiverse Lynx LLC is an Equal Employment Opportunity employer committed to promoting diversity and inclusion in the workplace. We believe that a diverse workforce brings unique perspectives and ideas, leading to innovation and success.SalaryThe...
-
Senior Big Data Engineer
2 months ago
Irving, United States Anblicks Full timeDescription: The Data Engineer is responsible for building Data Engineering Solutions using next generation data techniques. The individual will be working directly with product owners, customers and technologists to deliver data products/solutions in a collaborative and agile environment.Responsibilities:Responsible for design and development of big data...
-
Senior Big Data Engineer
4 weeks ago
Irving, United States Anblicks Full timeDescription: The Data Engineer is responsible for building Data Engineering Solutions using next generation data techniques. The individual will be working directly with product owners, customers and technologists to deliver data products/solutions in a collaborative and agile environment.Responsibilities:Responsible for design and development of big data...
-
Lead/Sr. Big Data Engineer
1 month ago
Irving, United States America Technology Professionals LLC Full timeJob Title: Lead / Sr. Big Data Engineer Location: Irving, TX (Hybrid – 2-3 days/week) Duration: 12+ Months USC/GC/H1B only C2C/1099 only This role can play the hand-on dev role along with tech lead skills Required Skills: Big data expert with 9+ years experience in Hadoop Big data ecosystem Spark - Batch Streaming (Python, Scala) Experience in cloud...
-
Lead / Sr. Big Data Engineer
4 weeks ago
Irving, United States Infovision Full timeJob Title: Lead / Sr. Big Data EngineerLocation: Irving, TX (Hybrid 2-3 days/week)Duration: 12+ MonthsRequired Skills: Big data expert with 9+ years experience in Hadoop Big data ecosystemSpark - Batch Streaming (Python, Scala)Experience in cloud environment specially Google Cloud PlatformExperience in developing both batch and real-time streaming data...
-
Lead / Sr. Big Data Engineer
2 months ago
Irving, United States InfoVision Inc. Full timeJob Title: Lead / Sr. Big Data EngineerLocation: Irving, TX (Hybrid 2-3 days/week)Duration: 12+ Months Required Skills: Big data expert with 9+ years experience in Hadoop Big data ecosystemSpark - Batch Streaming (Python, Scala)Experience in cloud environment specially Google Cloud PlatformExperience in developing both batch and real-time streaming data...
-
Lead Big Data Engineer
7 days ago
Irving, United States Infovision Full timeJob title: Lead Big Data Engineer (GCP Cloud) Location: Irving, TX Duration: Long-term Skills Needed: 10+ years of experience in designing and building data pipelines in large-scale distributed systems. Lead the design, development, and maintenance of scalable batch and real-time data processing pipelines. Proficiency with Google Cloud Platform (GCP) and...
-
Lead Big Data Engineer
4 weeks ago
Irving, United States Infovision Full timeJob title: Lead Big Data Engineer (GCP Cloud)Location: Irving, TXDuration: Long-termSkills Needed:10+ years of experience in designing and building data pipelines in large-scale distributed systems.Lead the design, development, and maintenance of scalable batch and real-time data processing pipelines.Proficiency with Google Cloud Platform (GCP) and tools...
-
Cloud Engineer
1 week ago
Irving, United States Resource Informatics Group Full timeJob Title: Cloud Engineer - Senior / Big Data Location: McLean, VA / Dallas, TX (hybrid role) Duration: 6+ Months (possibility of extension or conversion) Experience: - 5 - 8 years Job Description 5+ Years of exp needed. Will be working in AWS environment. Product - Data Lake Platform in AWS This Developer will be working on Glue, DynamoDB, S3, Lambda...
-
Lead Big Data Engineer
2 months ago
Irving, United States InfoVision Inc. Full timeJob title: Lead Big Data Engineer (GCP Cloud)Location: Irving, TXDuration: Long-term Skills Needed:10+ years of experience in designing and building data pipelines in large-scale distributed systems.Lead the design, development, and maintenance of scalable batch and real-time data processing pipelines.Proficiency with Google Cloud Platform (GCP) and tools...
-
Senior Big Data Engineer
4 weeks ago
Irving, TX, United States Anblicks Full timeDescription: The Data Engineer is responsible for building Data Engineering Solutions using next generation data techniques. The individual will be working directly with product owners, customers and technologists to deliver data products/solutions in a collaborative and agile environment.Responsibilities:Responsible for design and development of big data...
-
Big Data Professional
6 days ago
Irving, Texas, United States Forward Air Full timeJob Title: Data EngineerJob Summary:Forward Air is seeking a skilled Data Engineer to work in the IT department. As a Data Engineer, you will be responsible for designing and implementing scalable cloud-based streaming data solutions that enable real-time analytics and mission-critical data visibility to our organization. You will work across departments...
-
Senior Enterprise Data Management Specialist
2 days ago
Irving, Texas, United States NTT DATA Full timeAt NTT DATA, we are seeking a skilled Technical Project Manager to join our team in Irving, Texas. This is a hybrid position requiring 2-3 days on site and remote work.Job SummaryWe are currently looking for an experienced Technical Project Manager who can effectively manage projects from inception to closure. The ideal candidate will have a solid...
-
Senior Data Solutions Specialist
7 days ago
Irving, Texas, United States Verizon Full timeJob DescriptionAt Verizon, we're looking for a highly skilled Senior Data Solutions Specialist to join our team. In this role, you'll be responsible for designing, building, and maintaining end-to-end data products that meet our business needs.Key Responsibilities:Develop high-quality code using cutting-edge technologies such as GCP, AWS, and Hadoop...
-
Big Data Cluster Manager
6 days ago
Irving, Texas, United States Donato Technologies Inc Full timeJob DescriptionWe are seeking experienced professionals to manage and optimize Cassandra database clusters for our client in Irving, TX or Miami, FL. The ideal candidate will have a strong background in NoSQL database administration and performance tuning.Key Responsibilities:Install, configure, and manage Cassandra clusters to ensure high-performance data...
-
Senior Data Engineer
2 weeks ago
Irving, United States Newt Global Full timeGreeting from Newt Global LLC!!!We are currently looking for Big Data Engineer to join our team for long term project with our Banking client. Interested with the below job description, please share your resume with expected rate/hr.Big Data EngineerIrving, TX (Hybrid – 2 Days a Week)Contract Type: W2Long TermExp: 6+ yrsBanking Exp RequiredBig Data...
-
Data Engineer
1 month ago
Irving, United States Newt Global Full timeBig Data EngineerIrving, TX - HybridW2 ContractDirect ClientQualifications:8+ years of experience in hadoop/big data technologies.3+ years of experience in spark.2+ years experience in Snowflake2+ year of experience working on Google or AWS cloud developing data solutions. Certifications preferred.Hands-on experience with Python/Pyspark/Scala and basic...