Azure Data Engineering Architect

1 month ago

Brentwood, United States Indiglobe IT Solutions Pvt. Ltd. Full time
Job DescriptionJob DescriptionDescriptionLead Azure Data Engineer Architect role is to strategically design, develop and implement Azure Cloud based Data Warehouse, Data Lake, Data Lakehouse, Data Marts BI solution using modern data engineering techniques and advanced analytics methods to provide information insights to end users. The role requires modeling, collecting, harmonizing, aggregating, storing and reconciling data from variety of data sources for our petroleum refineries IT/OT technology platform including time series Database, SCADA data, batch, real time, enterprise applications, Rest API’s , streaming IOT data.& The role will help design and build data pipelines, streams, data warehouse, data lake, reporting application, data generators and a whole range of tools to provide information and insight.The ideal candidate will be responsible for designing, developing, and maintaining scalable data processing pipelines in a cloud-based environment. The candidate should have a deep understanding of Azure ADF, Databricks, streaming analytics, data engineering best practices, and a proven track record of delivering high-quality real time solutions.
ResponsibilitiesStrategy Planning
  • Assess and cultivate long-term strategic goals for data warehouse, data lake, BI& advance analytics development in conjunction with end users, managers, clients, and other stakeholders.
  • Coordinate and work with other technical staff to develop Azure Data BI architecture, coding standards, and quality assurance policies and procedures.
  • Promote a culture of self-serve data analytics by minimizing technical barriers to data access and understanding.
  • Is responsible for data collaboration and communication with various stakeholders.
& Operational Management
  • Lead the integration efforts for Azure data warehouse, data lakehouse - sourcing data from disparate enterprise systems including applications both on premises cloud, IOT devices, Data Service (Rest API’s), time-series database, semi-structure, unstructured data external data sources.
  • Organize and lead projects in the implementation and use of BI software tools and systems.=
  • Design, code, test, and document all new or modified BI systems, applications, and programs.
  • Develop the semantic layer, metadata, reports, and report definitions.
  • Assist in the design of databases to ensure interoperability with DW BI solutions.
  • Produce ELT/ETL design guidelines to ensure a manageable ELT/ETL infrastructure for the analytics system.
  • Participate in design discussions about new features and ensure real-time technology integrates seamlessly with other pieces of the platform
  • Applying technical knowledge to architect solutions that meet business needs, AI/ML roadmaps, and contributing to architectures that enable the ability to scale to support additional modelling use cases (e.g. Azure ML, Azure Databricks, Azure EventHub/IoT Hub, etc)
  • Build processes supporting data transformation, data structures, metadata, dependency and workload management.
  • Work with project managers to ensure that data entry, retrieval, change, and delete functions meet business requirements for project completion.
  • Act as the primary owner and maintainer of new and existing Lakehouse/DW/BI data models
  • Design, implement, and maintain CI/CD pipelines using Azure DevOps to integrate analytical solutions into cloud based custom developed applications
  • Support Cloud strategy team to integrate analytical capabilities into an overall cloud architecture and business case development
  • Facilitate Master/ Reference Data Management and overall Data Governance process
  • Conduct job duties and responsibilities according to the organization’s business systems development methodology and its Systems Development Life Cycle (SDLC) methodology.
Design and deliver end-user training and training materials; provide technical support as necessary.
  • Troubleshoot Data Integration BI tools, systems, and software; performance-tune these applications as necessary.
  • Defines and lead proof of concept activity associated with cloud data analytics technology assessment.
  • Act as evangelist for Azure Cloud data analytics benefits across the organization; increase usage to relevant departments.
  • Act as a subject matter expert on all things cloud data architecture and pipeline development
Position RequirementsKnowledge Experience&
  • Minimum of 12 to 15 years of Data Engineering, Data Warehouse/ Business intelligence with hands-on experience leading and delivering multiple full life cycle implementations.
  • Comprehensive data warehouse analysis and design experience, with full knowledge of data warehouse methodologies and data modeling.
  • Hands-on experience deep understanding in designing, development and implementation of Azure Data Analytics architecture, data modeling, transformations, and development.
  • In-depth knowledge and minimum 5 to 7 years’ experience developing implementing data engineering solutions using Azure services for& Data Warehouse, Data Lakehouse, Advance analytics- Primarily- Azure Data Factory, ADLS, Azure Databricks, Azure Functions, Logic Apps, IoT Hub, Event Hub Azure SQL
  • Experience with Data pipeline and workflow management tools Azure DevOps processes.
  • Strong proficiency in programming languages such as Python, PySpark, or Scala
  • End to End Implementations expertise in& Azure Data Warehouse Data Lakehouse
  • Solid understanding of database structure principles, Master and Reference Data management
  • Experience expertise in optimizing the ETL/ELT data pipeline technology options to minimize the on-going operational cost of Azure data engineering storage solutions.
  • 8+ years experience& in agile Data warehouse development methodology.
  • 8+ years’ experience designing and developing dimensional models for data warehousing
  • 3+ years’ experience in MSBI suite primarily SSIS
  • 3+ years of implementing operational data model is plus not essential
  • 3+ years of experience with big data batch/streaming processing.
  • 5+ years’ experience using Pyspark, Python or Scala
  • Experience delivering reporting solution using visualization tools like PowerBI
  • Experience in implementing SAP data analytics solution in Azure is plus not essential
  • Experience using data modeling tool like ERWin
  • Ability to lead and work with remote offshore teams.
  • Experience integrating analyzing data from time-series databases, SCADA data, IOT, voice to text, images videos for creating Industrial Intelligence solution.
  • Knowledge/expertise on Azure ML/AI predictive modeling capabilities is a plus
  • Experience using proven methods to solve business problems using Azure Data and Analytics services in combination with building data pipelines, data streams and system integration
  • Hands-on experience with analytics and big data technologies within Microsoft Azure Cloud platform in tools such as Azure Machine Learning Azure Cognitive Services is plus not essential
  • Experience with data profiling, cataloguing, and data mapping for technical design using a use case-based approach that drives the construction of technical data flows
  • Experience with Azure Administration Security (Connection setup, User setup, Object security) is plus
  • Understanding of statistical and machine learning modeling with experience applying these modeling techniques to business problems is a plus not essential.
  • Ability to set up data and experimental platforms using Azure cloud service
  • Ability to work independently with business stakeholders to collect and understand requirements for reporting analytics.
  • Full project management and software development life cycle experience.
  • Experience with data process flowcharting techniques.
  • Oil Gas industry experience preferred but not essential.
  • Experience knowledge of SAP S4 and SAP BW4HANA is a plus not essential.
  • Excellent quantitative, analytical mathematical skills is a plus
  • Strong understanding of the organization’s goals and objectives
& Formal Education Certification
  • Bachelor’s degree in computer science, information systems, or a related field, or equivalent work experience.
  • Certifications in Azure Data Engineering and Analytics space is a plus
& Personal Attributes
  • Ability to work independently and collaboratively in a fast-paced environment
  • Strong written and oral communication skills.
  • Strong presentation and interpersonal skills.
  • Exceptional analytical, conceptual, and problem-solving abilities.
  • Able to prioritize and execute tasks in a high-pressure environment.
Experience working in a team-oriented, collaborative environment.

  • Telecom Engineer

    2 days ago

    Brentwood, United States Colliers Engineering & Design Full time

    Overview Colliers Engineering & Design is currently seeking Engineers to join our Telecom team in Nashville, TN! Wireless communication is an ever-changing technology embedded in our culture that seamlessly links us to the world around us. Our progressive telecommunication professionals work tirelessly to provide leading-edge engineering solutions for the...

  • System Engineer

    14 hours ago

    brentwood, United States Lifepoint Health Support Center Full time

    p>System Engineer (Data Protection) – Remote Who we are:At Lifepoint Health, we provide quality healthcare to rural communities. As a valued member of our team, you will be an integral part of a group working together to elevate Lifepoint's healthcare delivery network. Our network includes 60+ community hospitals, 60+ rehabilitation/behavioral health...

  • IAM Engineer I

    1 day ago

    Brentwood, United States Ardent Health Full time

    Overview: Ardent Health Services is a leading provider of healthcare in communities across the country. Through its subsidiaries, Ardent owns and operates nearly 200 sites of care and 30 hospitals across six states. Together, our 24,000 employees and 1,200 employed providers touch more than 10,000 lives each day. With six Ardent entities earning recognition...

  • Project Engineer

    3 weeks ago

    Brentwood, United States W.E. O'Neil Construction Full time

    Job DescriptionJob DescriptionProject Engineer is an entry-level position in operations, and is the starting point for the typical career path of engineer-superintendent-project manager. O’Neil engineers are typically assigned to one project, are located at the jobsites, and are managed by the project manager with a significant amount of oversight. Project...

  • Cyber Security Engineer III

    2 days ago

    Brentwood, United States CoreCivic Full time

    At CoreCivic, our employees are driven by a deep sense of service, high standards of professionalism and a responsibility to better the public good. CoreCivic is currently seeking a **Cyber Security Engineer III** located at our corporate office in Brentwood, TN. Come join a team that is dedicated to making an impact for the people and communities we serve....

  • Sr. .NET Software Engineer

    2 days ago

    Brentwood, United States Virtual Full time

    Job Summary: Develop and maintain quality applications using Angular, interfacing with Rest APIs. Participate in the complete software development lifecycle, contributing to defining solutions, estimating efforts, and delivering high-quality work. Engage in Docker and Azure pipelines for efficient application deployment. Support and upgrade existing MVC and...

  • Data Analyst/Production Support

    2 weeks ago

    Brentwood, United States Apex Systems Full time

    Data Analyst working on a production support team. Looking for somebody strong with SQL (Creating queries, stored procedures, etc). Ideally have worked within/have knowledge of a Data warehouse. Standard M-F day shift. Hybrid position being 3 days onsite and 2 days remote a week. Office is located near the Brentwood, Missouri area. Contract to hire position....

  • Data Analyst/Production Support

    2 weeks ago

    Brentwood, United States Apex Systems Full time

    Data Analyst working on a production support team. Looking for somebody strong with SQL (Creating queries, stored procedures, etc). Ideally have worked within/have knowledge of a Data warehouse. Standard M-F day shift. Hybrid position being 3 days onsite and 2 days remote a week. Office is located near the Brentwood, Missouri area. Contract to hire position....

  • Data Analyst/Production Support

    2 weeks ago

    Brentwood, United States Apex Systems Full time

    Data Analyst working on a production support team. Looking for somebody strong with SQL (Creating queries, stored procedures, etc). Ideally have worked within/have knowledge of a Data warehouse. Standard M-F day shift. Hybrid position being 3 days onsite and 2 days remote a week. Office is located near the Brentwood, Missouri area. Contract to hire position....

  • Solr Search Engine

    4 weeks ago

    Brentwood, United States Resource Informatics Group Inc Full time

    Job DescriptionJob DescriptionRole:  Solr Search Engine – Lead/ Developer  Location: Brentwood, TNDuration; 6+ monthsRate: $market All inclusiveResponsibilitiesArchitect, design and develop solutions within a multi-functional Agile team to support key business needs for the platform search applicationDesign, implement and test front-end solutions for...

  • Storage Engineer

    6 days ago

    Brentwood, United States IDR Healthcare Full time

    IDR has a new opportunity available for a Storage Engineer in the Nashville area. Our client is a highly respected organization with plans for continued growth. If you are looking to join a stable team with a fast paced, innovative environment, please apply today! Storage Systems Engineer Responsibilities: Administer, build, troubleshoot, and maintain data...

  • Senior Frontend Engineer

    4 weeks ago

    Brentwood, United States Trilliant Health Full time

    Senior Frontend Engineer - Level IV Role Overview:This is an atypical frontend position. The small frontend team (currently 4 developers) housed within the Trilliant Health Engineering team (20+ developers) has nearly full control over our tech stack and deployment method.At this time, we create new sub-applications, or add new features to existing...

  • Senior Frontend Engineer

    4 weeks ago

    Brentwood, United States Trilliant Health Full time

    Senior Frontend Engineer - Level IV Role Overview:This is an atypical frontend position. The small frontend team (currently 4 developers) housed within the Trilliant Health Engineering team (20+ developers) has nearly full control over our tech stack and deployment method.At this time, we create new sub-applications, or add new features to existing...

  • Senior Frontend Engineer

    4 weeks ago

    Brentwood, United States Trilliant Health Full time

    Senior Frontend Engineer - Level IV Role Overview:This is an atypical frontend position. The small frontend team (currently 4 developers) housed within the Trilliant Health Engineering team (20+ developers) has nearly full control over our tech stack and deployment method.At this time, we create new sub-applications, or add new features to existing...

  • Implementation Engineer

    4 days ago

    Brentwood, United States CYBERMAXX LLC Full time

    Job DescriptionJob DescriptionDescription:POSITION SUMMARY:As an Implementation Manager, you will be responsible for overseeing and managing the successful implementation of Cybermaxx products and services for our clients. You will work closely with cross-functional teams, including Sales and Customer Success, to ensure a smooth transition from sales to...

  • Sr. Lead Geotechnical Engineer

    3 weeks ago

    Brentwood, United States WSP Global Inc. Full time

    Our Business We are a global leader in environmental consulting - made stronger with the recent acquisition of Wood’s Environment & Infrastructure business and Golder - with world-class expertise in environmental, social and governance (ESG), climate resiliency & sustainability (CRS), and earth sciences. Combined we have over 20,000 multidisciplinary...

  • Remote Site Reliability Engineer

    2 weeks ago

    Brentwood, United States Insight Global Full time

    A client of Insight Global is looking for a Remote Site Reliability Engingeer to join their team and be apart of their DevOps movement as a full time employee. This person will be responsible for miantaining cloud services and hosted infastructure including IaC configurations, Triaging incidents affecting application performance, uptime, and infrastructure...

  • Lead Project Engineer

    2 days ago

    Brentwood, United States CESO Full time

    Job Description Are you a Civil Lead Project Engineer seeking purpose, challenge, and talented colleagues? With a diverse portfolio of residential, energy, commercial, light industrial and public programs, CESO has the opportunity for you to develop your career. We develop leaders and empower our associates to use their skills and talents to positively...

  • Project Engineer

    3 weeks ago

    Brentwood, United States exp Federal Full time

    Job DescriptionJob DescriptionProject Engineer Arnold Airforce Base, TNExp Federal is an established full-service architectural and engineering firm that provides a full range of exceptional design solutions and construction support to the federal government. Our team of professionals operates in some of the most challenging environments to provide support...

  • Estimator

    4 weeks ago

    Brentwood, United States W.E. O'Neil Construction Full time

    Job DescriptionJob DescriptionThe Estimator handles budgeting, scheduling, and cost analysis during the preconstruction period. Estimators interact regularly with clients and architects and are an important part of securing new work. The position has a moderate amount of supervision. Major Duties and Responsibilities• Generating schematic, conceptual, and...