DataStage Administrator

Role: DataStage Administrator for On-premises to AWS Migration

Location: Miami, FL (Hybrid)

Contract Role

***Need only USC, GC & EADs on W2***

*** No H1B***

Project Overview:

Objective: To migrate an existing on-premises IBM InfoSphere DataStage environment to the cloud (AWS), ensuring seamless data integration, transformation, and ETL (Extract, Transform, Load) processes for telecom-related data.

Roles & Responsibilities of a Datastage Administrator in this Migration Project:

Pre-Migration Tasks:

• Conduct a comprehensive audit of the current Datastage setup.
• Identify and document all data flows and dependencies, particularly those related to the telecom domain (e.g., call data records, billing data, network performance metrics).
• Work with other stakeholders (e.g., network engineers, telecom domain experts) to ensure all use cases are understood.
• Choose appropriate AWS services for Datastage components, such as:
• EC2 for computing
• S3 for data storage
• RDS/Aurora or Redshift for databases and analytics
• Lambda and Glue for serverless processing

Migration Planning:

• Create a migration strategy that includes timelines, resource allocation, and risk mitigation steps.
• Develop a cutover plan to switch from on-premises to AWS with minimal service disruption.
• Identify the migration method: lift-and-shift vs. re-architecture.
• Coordinate with telecom domain architects for alignment with business requirements (e.g., billing cycles, data consistency, SLAs).

Infrastructure Setup on AWS:

• Provision AWS resources (e.g., EC2 instances with suitable specifications, storage in S3, RDS for databases).
• Configure IAM roles for access control and security.
• Set up AWS Direct Connect or VPN if private network connectivity is required between on-prem and cloud.
• Implement data security and compliance measures in line with telecom industry regulations (e.g., GDPR, HIPAA).

Datastage Migration Execution:

• Migrate Datastage ETL jobs, sequences, and configurations to the cloud.
• Ensure that existing custom scripts and connectors work in the cloud environment.
• Adjust job parallelism and resource allocation to leverage AWS scalability features like Auto Scaling.
• Address any issues with file transfer protocols (e.g., FTP, SFTP) between on-prem and AWS.

Testing & Validation:

• Unit testing and integration testing of Datastage jobs post-migration.
• Verify data accuracy and completeness for telecom datasets.
• Test data transformation rules and ensure that they function correctly in the AWS environment.
• Conduct performance testing to verify that the new AWS infrastructure can handle the data loads and workloads typical of the telecom domain.

Post-Migration Support:

• Provide ongoing monitoring of Datastage jobs and AWS resources using CloudWatch, Datadog, or similar.
• Troubleshoot performance issues (e.g., memory, CPU usage) and optimize configurations.
• Fine-tune resource allocation to ensure cost efficiency without sacrificing performance.
• Work with the security team to ensure data protection and compliance.

Documentation:

• Document the new cloud-based Datastage environment and any customizations made.
• Update disaster recovery plans to account for cloud-based resources.

Training & Knowledge Transfer:

• Conduct training sessions for the internal team on the new AWS environment, tools, and best practices.
• Ensure that knowledge is transferred for long-term management of the Datastage system in the cloud.

Technologies/Tools You Might Use:

• AWS Services: EC2, S3, RDS, Glue, Lambda, Redshift, Kinesis
• Datastage (installed on AWS EC2 or via managed services)
• Networking: AWS Direct Connect, VPN, VPC Peering
• Monitoring: AWS CloudWatch, CloudTrail, Datadog
• Security: AWS IAM, KMS, encryption, audit logging