Training Type

Select faculty

Select Date

Dur: 60 days
Course fee : 14000 /-

Azure Data Engineer

Course Overview

The Azure Data Engineer course is designed to equip individuals with the skills and knowledge required to effectively work with data within the Azure ecosystem. This comprehensive course covers a wide range of topics related to data engineering, including data ingestion, transformation, storage, and analysis using various Azure services and tools.

Learn software skills with real experts, either in live classes with videos or without videos, whichever suits you best.

Description

The course delves into the core concepts and technologies essential for modern data engineering on the Azure platform. Participants will learn how to leverage Azure services such as Azure Data Factory, Azure Databricks, Azure Synapse Analytics, Azure HDInsight, Azure Cosmos DB, and others to build scalable, reliable, and efficient data pipelines and analytics solutions. Hands-on exercises and real-world projects are integrated into the curriculum to provide practical experience and reinforce learning.

Course Objectives

  • Understand the fundamentals of data engineering and its role in the Azure ecosystem.
  • Gain proficiency in designing and implementing data ingestion pipelines using Azure Data Factory.
  • Learn how to process and transform data at scale using Azure Databricks.
  • Explore various storage options in Azure for both structured and unstructured data.
  • Master the techniques for building and optimizing big data solutions with Azure HDInsight.
  • Discover how to design and implement data warehousing solutions using Azure Synapse Analytics.
  • Learn best practices for data governance, security, and compliance in Azure.
  • Develop skills in data visualization and reporting using Power BI and other Azure services.
  • Gain practical experience through hands-on labs and real-world projects.

Prerequisites
Course Curriculum

  • Understanding different Cloud Models
  • Advantages of Cloud Computing
  • Different Cloud Services
  • Different Cloud vendors in the market

  • Introduction to Azure
  • Azure cloud computing features
  • Azure Services for Data Engineering
  • Introduction of Azure Resources/Services with examples
  • Azure management portal
  • Advantage of Azure Cloud Computing
  • Managing Azure resources with the Azure portal
  • Overview of Azure Resource Manager
  • Azure management services
  • What is Azure Resource Groups
  • Configuration and management of Azure Resource groups for hosting Azure services

  • Completed walkthrough of the Azure Portal with all the features
  • What is Resource Groups and why we need RG’s in Azure cloud computing platform to host resources?
  • Different types of Storage Accounts provisioning in Cloud computing with different storage services
  • Details explanation & understanding of different Blob/container storage services
  • Creating and managing the data in container storage services with Public and Private accesses as per the need of a project
  • Implementation of Snapshots for Blob storage services and File share storage service
  • Generating SAS for different storage services to make the storage content browseable across all the globe or Publicly
  • What is Standard Storage Account and Premium Storage account and which to use accordingly as per the real time scenarios
  • Detail explanation and implementation of Data Lake storage Gen2 Storage Account to store the unstructured data in cloud storage services
  • All the features/properties(Overview, activity log, Tags, Access control(IAM), Storage browser…etc) of Azure Storage Accounts
  • Maintenance and management of Storage keys and connection string for Azure Storage services
  • Implementing different levels of access(Reader, contributor, owners…etc) to the Azure Storage accounts

  • Moving the storage account with storage content across different Resources Groups based on real time scenarios
  • Migrating the data from On-prem(Private cloud) to Azure Storage account (Public cloud) using Az copy(forward migration)
  • Migrating the data from public cloud to Private cloud(revers migration)
  • Implementing the Az copy commands to migrate the data
  • Moving the SA & its content from one Resource Group to another

  • Azure Storage explorer for creating, managing, and maintaining the Azure storage services data
  • Installation of Azure Storage Explorer and what is the purpose of this tool for Azure Storage accounts(its Purpose & benefits with real time scenarios)
  • Generate Shared Access Signature(SAS) in Azure Storage Explorer(ASE) for security implementation of Storage account content
  • Managing of Access keys & connection strings of SA with Azure Storage Explorer
  • Configuration of Authentication and Authorization for Storage Account via Azure Active Directory
  • Hosting File share Storage services to On prem servers or Cloud Servers as shared drive for File share servers

  • Introduction to SQL DB’s
  • Creation of new SQL DB’s & Sample SQL DB’s both in On-prem and Cloud computing
  • Planning and deploying Azure SQL Database
  • Implementing and managing Azure SQL Database
  • Managing Azure SQL Database security
  • Planning and deployment of SQL DB’s in Azure cloud computing with real time scenarios
  • Different DB’s Deployment options
  • Databases purchasing models.(VCore & DTU’s)
  • Visualization of cloud DB server, Database, and validation of data from on-prem(private cloud)
  • Implementation of Firewall security rules on Azure DB servers to access and connect from on-prem SSMS
  • Creation of Database in on-premises and synch with azure cloud

  • Migrating SQL DB’s from On-premises to Azure cloud computing using Microsoft Data migration assistant
  • Restoring SQL DB’s from On-prem to cloud computing
  • Migration of Specific DB objects from on-prem to cloud based upon base upon project requirements
  • Implementation of RSV and scheduling the backups of SQL DB’s and Azure Storage Account file share services on schedule, on demand based upon real time scenarios

  • Introduction to SQL DB Queries
  • SQL queries detail explanations, syntax & execution based upon real time scenarios

  • Deep understanding and implementation of concepts/Components of ADF
  • Building blocks of Azure Data Factory
  • Complete features and walk through of Azure Data factory studio
  • Different triggers and their implementation in ADF
  • What is integration run time and different types of integration run time in ADF
  • When to use ADF
  • Why to use ADF
  • Different types of ADF pipelines
  • Pipelines in ADF
  • Different types of Activities in ADF
  • Datasets in Azure Data factory
  • Linked services in ADF

  • Copying the data from Blb Storage account to ADL’s Gen2 Storage account
  • Copying of zip files(.csv) from Blob SA to ADL’s Gen2 SA using ADF
  • Implementation and explanation of Metadata control in ADF to find the structure before copying the data
  • Implementation and explanation of Validation and If Condition
  • Implementation of Get Metadata control, filter control & For Each Control or activities in ADF
  • Implementation & execution to copy the data from GitHub platform to Azure Storage services with variables and parameters
  • Implementation of Foreach control, copy data control and Set variable to dynamically load the data from source to target using ADF
  • Creating Dynamic pipelines with lookup activity to copy multiple .csv files data picking form Json format data in Azure Storage services
  • Copying the files from GitHub Dynamically with the use of Dynamic parameters allocation-AUTOMATION PROCESS
  • Copying the data from different files formats(.csv, .xlsx, .txt, .Parquet, .Json, .SQL…etc) using suitable ADF controls/activities
  • Implementation and execution of Loading the data from Blb SA to SQL DB single table & multiple tables using copy data activity, ForEach activity
  • Executing multiple pipelines in parallel with Execute pipeline activity

  • Implementation of Schedule based triggers for different ADF pipeline containing different activities.
  • Implementation of Event based triggers for different ADF pipeline containing different activities.
  • Implementation of Thumbling window-based triggers for different ADF pipeline containing different activities.
  • Implementation and execution of storage and Event based triggers.

  • Detail explanation & implementation of Azure Keyvaults
  • Making the SQL DB connection string to store in Keyvault to enhance the security for SA content and SQL DB
  • Generating the secrets inside the Azure keyvault and granting access by implementing the access policies for different users.

  • Detail walk through of GitHub portal
  • Creating an account, repo’s, in GitHub portal
  • Integrating Azure Data Factory with GitHub Portal as per project requirements.
  • Placing, maintaining and executing the source code via GitHub portal for Azure Data Factory.
  • Creating master branch, practice branches in GitHub portal to merge the newly created code via Pull Requests.
  • Setting up the Repo for ADF pipelines and converting to live mode from GitHub portal covering with real time scenarios.

  • Designing new Data flows
  • Designing and implementing transformations
  • Inline Datasets in data flow source control
  • Designing and implementing of Data flow with Source transformations, Filter transformations & Sink transformations in ADF with inline Datasets
  • Implementation of Select transformations with Data flows for various source controls
  • Implementation of Dataflows using Aggregate & Sink transformation
  • Implementation of Dataflow with conditional split & Sink transformation with copy data activity
  • Implementation of Dataflow with Exists & Sink transformation
  • Implementation of Azure Dataflows for Derived column transformation with Source & Sink transformation
  • Implementation of Azure Dataflows to connect to SQL DB with Source & Sink transformation
  • Union & Union flow transformation implementation with ADF Data flows
  • Implementation of Azure Dataflows to connect to SQL DB with Source & Sink transformation
  • Implementation of windows functions…like Rank() function, Dense_Rank() function, Row_Number() function…etc.

  • What is Apache Spark, details explanation and implementation of Apache Spark
  • Illustration and Elaboration of Apache Spark Architecture
  • Explanation of RDD & DAG
  • Understanding of different Apache Spark components
  • What are worker nodes and slaves nodes in Azure Data Bricks clusters
  • Implementation of Azure Databricks cluster by considering different worker nodes and slave nodes
  • Different features and properties of Azure Data Bricks clusters

  • Creating single node and multi nodes clusters
  • Creation of Pyspark notebooks in Databricks cluster to fulfil different business requirements

  • What is Azure Synapse Analytics
  • Implementation of Linked Services/Datasets in Synapse Analytics
  • Implementation of dedicated SQL Pool inside Synapse Analytics
  • Implementation of serverless SQL Pool inside Synapse Analytics
  • Creation of Apache spark pool in Azure Synapse Analytics
  • Writing SQL Script in Azure Synapse analytics to get the result set in tabular and chart formats
  • Visualizing the data in Synapse analytics in variety of different charts (like pie charts, line charts, bar charts…. etc)
  • Designing of Synapse Analytics pipelines by considering various activities as per the business requirements
  • Creation of Datasets, Linked services for Synapse Analytics pipelines
  • Data analysis with serverless spark pools in Azure Synapse Analytics
  • What is Apache spark in azure synapse analytics
  • Designing and development of Apache spark pool in Azure synapse
  • Creating Spark Databases and tables to load the data from source system and analysing the data in Synapse analytics

  • What is Azure Stream Analytics
  • Purposes and usage of Stream Analytics in Azure cloud computing
  • Benefits and advantages of stream analytics
  • Architecture diagram of data flow in Azure stream analytics with other cloud services
  • Understanding & usage of browser-based Raspberry Pi simulator
  • Deployment of IoT Hub services as an input for Stream analytics jobs
  • Implementation & execution of stream analytics jobs and designing inputs and outputs for IoT Hub and Datalake Gen2
  • Writing SQL scripts to generate live streaming data and loading it in destination
Who can learn this course

The Azure Data Engineer course is suitable for a wide range of individuals interested in working with data in the Azure cloud environment. This includes:

  • Data engineers looking to enhance their skills and knowledge in Azure data technologies.
  • Database administrators seeking to migrate or manage data solutions in Azure.
  • Data analysts and business intelligence professionals aiming to leverage Azure for advanced analytics and reporting.
  • Software developers interested in building data-intensive applications on the Azure platform.
  • IT professionals and system architects involved in designing and implementing data solutions in Azure.
  • Anyone looking to pursue a career in data engineering or cloud computing with a focus on Azure technologies.

Average package of course (Azure Data Engineer)

50% Avg
salary hike
50L Avg
Package
Upcoming Batches
Live Training Batches Timetable
Course Name Faculty Date Time Mode of Training Batch Type Meeting Link
Azure Data Engineer Mr. Gareth 22 May 8:30 AM (IST) online Online Training
Training Features
Comprehensive Course Curriculum

Elevate your career with essential soft skills training for effective communication, leadership, and professional success.

Experienced Industry Professionals

Learn from trainers with extensive experience in the industry, offering real-world insights.

24/7 Learning Access

Enjoy round-the-clock access to course materials and resources for flexible learning.

Comprehensive Placement Programs

Benefit from specialized programs focused on securing job opportunities post-training.

Hands-on Practice

Learn by doing with hands-on practice, mastering skills through real-world projects

Lab Facility with Expert Mentors

State-of-the-art lab facility, guided by experienced mentors, ensures hands-on learning excellence in every session

Our Trainees are Working with
...
...
...
...
...
...
...
...
...
...
...
...
...
...
...
...
...
...
...

Top 5 Technologies to learn Register for the Course !

By Providing your contact details, you agree to our Terms of use & Privacy Policy