001 Introduction to Data Engineering using AWS Analytics Services (32.14 MB)
002 Video Lectures and Reference Material (21.23 MB)
003 Taking the Udemy Course for new Udemy Users (23.44 MB)
004 Additional Costs for AWS Infrastructure for Hands-on Practice (9.48 MB)
005 Signup for AWS Account (8.51 MB)
006 Logging in into AWS Account (10.22 MB)
007 Overview of AWS Billing Dashboard - Cost Explorer and Budgets (19.64 MB)
001 Setup Local Environment on Windows for AWS (19.33 MB)
002 Overview of Powershell on Windows 10 or Windows 11 (10.75 MB)
003 Setup Ubuntu VM on Windows 10 or 11 using wsl (14.92 MB)
004 Setup Ubuntu VM on Windows 10 or 11 using wsl - Contd (41.91 MB)
005 Setup Python venv and pip on Ubuntu (51.76 MB)
006 Setup AWS CLI on Windows and Ubuntu using Pip (6.51 MB)
007 Create AWS IAM User and Download Credentials (9.52 MB)
008 Configure AWS CLI on Windows (28.49 MB)
009 Create Python Virtual Environment for AWS Projects (10 MB)
010 Setup Boto3 as part of Python Virtual Environment (17.2 MB)
011 Setup Jupyter Lab and Validate boto3 (39.24 MB)
001 Setup Local Environment for AWS on Mac (5.88 MB)
002 Setup AWS CLI on Mac (12.27 MB)
003 Setup AWS IAM User to configure AWS CLI (5.66 MB)
004 Configure AWS CLI using IAM User Credentials (14.81 MB)
005 Setup Python Virtual Environment on Mac using Python 3 (10.93 MB)
006 Setup Boto3 as part of Python Virtual Environment (17.21 MB)
007 Setup Jupyter Lab and Validate boto3 (39.23 MB)
001 Introduction to Cloud9 (1.94 MB)
002 Setup Cloud9 (28.98 MB)
003 Overview of Cloud9 IDE (16.89 MB)
004 Docker and AWS CLI on Cloud9 (22.23 MB)
005 Cloud9 and EC2 (12.48 MB)
006 Accessing Web Applications (14.89 MB)
007 Allocate and Assign Static IP (13.72 MB)
008 Changing Permissions using IAM Policies (18.13 MB)
009 Increasing Size of EBS Volume (14.26 MB)
010 Opening ports for Cloud9 Instance (10.56 MB)
011 Setup Jupyter lab on Cloud9 Instance (39.08 MB)
012 Open SSH Port for Cloud9 EC2 Instance (13.31 MB)
013 Connect to Cloud9 EC2 Instance using SSH (16.14 MB)
001 Introduction - AWS Getting Started (13.16 MB)
003 Create AWS s3 Bucket using AWS Web Console (14.26 MB)
005 Create AWS IAM Group and User using AWS Web Console (15.81 MB)
007 Overview of AWS IAM Roles to grant permissions between AWS Services (6.12 MB)
009 Create and Attach AWS IAM Custom Policy using AWS Web Console (27.67 MB)
011 Configure and Validate AWS Command Line Interface to run AWS Commands (26.65 MB)
001 Getting Started with AWS Simple Storage aka S3 (19.18 MB)
003 Setup Data Set locally to upload into AWS s3 (11.52 MB)
005 Adding AWS S3 Buckets and Objects using AWS Web Console (23.27 MB)
007 Version Control of AWS S3 Objects or Files (32.29 MB)
009 AWS S3 Cross-Region Replication for fault tolerance (59.78 MB)
011 Overview of AWS S3 Storage Classes or Storage Tiers (35.41 MB)
013 Overview of Glacier in AWS s3 (25.65 MB)
015 Managing AWS S3 buckets and objects using AWS CLI (33.36 MB)
017 Managing Objects in AWS S3 using AWS CLI - Lab (92.7 MB)
001 Creating AWS IAM Users with Programmatic and Web Console Access (33.02 MB)
003 Logging into AWS Management Console using AWS IAM User (7.34 MB)
005 Validate Programmatic Access to AWS IAM User via AWS CLI (12.11 MB)
007 Getting Started with AWS IAM Identity-based Policies (61.48 MB)
009 Managing AWS IAM User Groups (24.86 MB)
011 Managing AWS IAM Roles for Service Level Access (62.43 MB)
013 Overview of AWS Custom Policies to grant permissions to Users, Groups, and Roles (64.56 MB)
015 Managing AWS IAM Groups, Users, and Roles using AWS CLI (44.08 MB)
001 Getting Started with AWS Elastic Cloud Compute aka EC2 (9.29 MB)
003 Create AWS EC2 Key Pair for SSH Access (35.84 MB)
005 Launch AWS EC2 Instance or Virtual Machine (72.83 MB)
007 Connecting to AWS EC2 Instance or Virtual Machine using SSH (12.8 MB)
009 Overview of AWS Security Groups for firewall security of AWS EC2 Instance (28.62 MB)
011 Overview of Public and Private IP Addresses of AWS EC2 Instance (48.92 MB)
013 Understanding AWS EC2 Instance or Virtual Machine Life Cycle (14.8 MB)
015 Allocating and Assigning AWS Elastic IP or Static IP address to AWS EC2 Instance (16.98 MB)
017 Managing AWS EC2 Instances or Virtual Machines Using AWS CLI (59.78 MB)
019 Upgrade or Downgrade of AWS EC2 Instances or Virtual Machines (48.61 MB)
001 Understanding AWS EC2 Instance or Virtual Machine Metadata (29.38 MB)
003 Querying on AWS EC2 Instance or Virtual Machine Metadata (27.9 MB)
005 Fitering on AWS EC2 Instance or Virtual Machine Metadata (37.21 MB)
007 Using Bootstrapping Scripts on AWS EC2 Instance or Virtual Machine (63.59 MB)
009 Create an Amazon Machine Image aka AMI using AWS EC2 Instance (19.67 MB)
011 Validate Amazon Machine Image aka AMI - Lab (34.77 MB)
001 Hello World using AWS Lambda (14.22 MB)
003 Setup Project for local development (44.92 MB)
005 Deploy Project to AWS Lambda console (15.73 MB)
007 Develop download functionality using requests (39.51 MB)
009 Using 3rd party libraries in AWS Lambda (41.97 MB)
011 Validating s3 access for local development (56.87 MB)
013 Develop upload functionality to s3 (57.16 MB)
015 Validating using AWS Lambda Console (12.28 MB)
017 Run using AWS Lambda Console (15.4 MB)
019 Validating files incrementally (33.72 MB)
021 Reading and Writing Bookmark using s3 (33.22 MB)
023 Maintaining Bookmark using s3 (44.7 MB)
025 Review the incremental upload logic (42.97 MB)
026 Deploying lambda function (94.76 MB)
027 ghactivity-downloader-src (1.99 KB)
028 Schedule Lambda Function using AWS Event Bridge (34.49 MB)
001 Setup Virtual Environment and Install Pyspark (25.45 MB)
003 Getting Started with Pycharm (25.92 MB)
005 Passing Run Time Arguments (16.7 MB)
006 Accessing OS Environment Variables (10.83 MB)
007 Getting Started with Spark (14.81 MB)
008 Create Function for Spark Session (24.03 MB)
010 Setup Sample Data (13.62 MB)
011 Read data from files (36.03 MB)
013 Process data using Spark APIs (41 MB)
015 Write data to files (20.22 MB)
017 Validating Writing Data to Files (24.75 MB)
018 Productionizing the Code (12.02 MB)
001 Introduction - Overview of Glue Components (8.97 MB)
003 Create Crawler and Catalog Table (30.88 MB)
005 Analyze Data using Athena (14.43 MB)
007 Creating S3 Bucket and Role (25.37 MB)
009 Create and Run the Glue Job (63.69 MB)
011 Validate using Glue CatalogTable and Athena (26.74 MB)
013 Create and Run Glue Trigger (22.14 MB)
015 Create Glue Workflow (23.05 MB)
017 Run Glue Workflow and Validate (33.07 MB)
001 Introduction - Spark History Server for Glue (5.81 MB)
002 Setup Spark History Server on AWS (47.08 MB)
003 Clone AWS Glue Samples repository (13.82 MB)
005 Build Glue Spark UI Container (3.61 MB)
007 Update IAM Policy Permissions (22.08 MB)
008 Start Glue Spark UI Container (39.44 MB)
001 Prerequisites for Glue Catalog Tables (2.32 MB)
003 Steps for Creating Catalog Tables (4.05 MB)
005 Download Data Set (26.24 MB)
007 Upload data to s3 (57.3 MB)
009 Create Glue Catalog Database - itvghlandingdb (4.14 MB)
011 Create Glue Catalog Table - ghactivity (16.38 MB)
013 Running Queries using Athena - ghactivity (18.23 MB)
015 Crawling Multiple Folders (42.04 MB)
017 Managing Glue Catalog using AWS CLI (57.71 MB)
019 Managing Glue Catalog using Python Boto3 (41.18 MB)
001 Update IAM Role for Glue Job (6.94 MB)
003 Generate baseline Glue Job (14.26 MB)
005 Running baseline Glue Job (66.99 MB)
007 Glue Script for Partitioning Data (14.89 MB)
009 Validating using Athena (43.75 MB)
001 Introduction to Glue Job Boomarks (3.58 MB)
002 Cleaning up the data (8.39 MB)
004 Overview of AWS Glue CLI (18.81 MB)
006 Run Job using Bookmark (16.46 MB)
008 Validate Bookmark using AWS CLI (28.78 MB)
010 Add new data to landing (19.18 MB)
012 Rerun Glue Job using Bookmark (24.1 MB)
014 Validate Job Bookmark and Files for Incremental run (13.92 MB)
016 Recrawl the Glue Catalog Table using CLI (36.12 MB)
018 Run Athena Queries for Data Validation (25.79 MB)
001 Planning of EMR Cluster (3.23 MB)
002 Create EC2 Key Pair (15.17 MB)
003 Setup EMR Cluster with Spark (18.65 MB)
004 Understanding Summary of AWS EMR Cluster (11.41 MB)
005 Review EMR Cluster Application User Interfaces (14.81 MB)
006 Review EMR Cluster Monitoring07 Review EMR Cluster Monitoring (5.66 MB)
007 Review EMR Cluster Hardware and Cluster Scaling Policy (3.5 MB)
008 Review EMR Cluster Configurations (7.08 MB)
009 Review EMR Cluster Events (8.83 MB)
010 Review EMR Cluster Steps (4.94 MB)
011 Review EMR Cluster Bootstrap Actions (11.83 MB)
012 Connecting to EMR Master Node using SSH (14.95 MB)
013 Disabling Termination Protection and Terminating the Cluster (4.49 MB)
014 Clone and Create New Cluster (18.73 MB)
015 Listing AWS S3 Buckets and Objects using AWS CLI on EMR Cluster (13.48 MB)
016 Listing AWS S3 Buckets and Objects using HDFS CLI on EMR Cluster (27.49 MB)
017 Managing Files in AWS s3 using HDFS CLI on EMR Cluster (45.47 MB)
001 Deploying Applications using AWS EMR - Introduction (1.39 MB)
002 Setup EMR Cluster to deploy applications (45.2 MB)
003 Validate SSH Connectivity to Master node of AWS EMR Cluster (16.91 MB)
004 Setup Jupyter Notebook Environment on EMR Cluster (14.36 MB)
005 Create required AWS s3 Bucket (5.02 MB)
006 Upload GHActivity Data to s3 (32.15 MB)
007 Validate Application using AWS EMR Compatible Versions (21.08 MB)
008 Deploy Application to AWS EMR Master Node (22 MB)
009 Create user space for ec2-user on AWS EMR Cluster (36.75 MB)
010 Run Spark Application using spark-submit on AWS EMR Master Node (69.51 MB)
011 Validate Data using Jupyter Notebooks on AWS EMR Cluster (44.58 MB)
012 Clone and Start Auto Terminated AWS EMR Cluster (51.35 MB)
013 Delete Data Populated by GHAcitivity Application using AWS EMR Cluster (6.01 MB)
014 Differences between Spark Client and Cluster Deployment Modes (76.47 MB)
015 Running Spark Application using Cluster Mode on AWS EMR Cluster (36.78 MB)
016 Overview of Adding Pyspark Application as Step to AWS EMR Cluster (17.89 MB)
017 Deploy Spark Application to AWS S3 (19.02 MB)
018 Running Spark Applications as AWS EMR Steps in client mode (11.21 MB)
019 Running Spark Applications as AWS EMR Steps in cluster mode (25.13 MB)
020 Validate AWS EMR Step Execution of Spark Application (25.45 MB)
001 Building Streaming Pipeline using Kinesis (14.53 MB)
002 Rotating Logs (73.16 MB)
003 Setup Kinesis Firehose Agent (51 MB)
004 Create Kinesis Firehose Delivery Stream (26.96 MB)
005 Planning the Pipeline (21.54 MB)
006 Create IAM Group and User (28.09 MB)
007 Granting Permissions to IAM User using Policy (32.24 MB)
008 Configure Kinesis Firehose Agent (34.1 MB)
009 Start and Validate Agent (95.67 MB)
010 Conclusion - Building Simple Steaming Pipeline (8.42 MB)
001 Customizing s3 folder using Kinesis Delivery Stream (30.82 MB)
002 Create Policy to read from s3 Bucket (31.36 MB)
003 Validate s3 access using AWS CLI (27.9 MB)
004 Setup Python Virtual Environment to explore boto3 (12.52 MB)
005 Validating access to s3 using Python boto3 (25.74 MB)
006 Read Content from s3 object (57.61 MB)
007 Read multiple s3 Objects (32.86 MB)
008 Get number of s3 Objects using Marker (24.2 MB)
009 Get size of s3 Objects using Marker (14.29 MB)
001 Install required libraries (19.99 MB)
002 Understanding GitHub APIs (30.52 MB)
003 Setting up GitHub API Token (28.46 MB)
004 Understanding GitHub Rate Limit (6.44 MB)
005 Create New Repository for since (13.28 MB)
006 Extracting Required Information (32.4 MB)
007 Processing Data (65.02 MB)
008 Grant Permissions to create dynamodb tables using boto3 (11.03 MB)
009 Create Dynamodb Tables (33.28 MB)
010 Dynamodb CRUD Operations (50.66 MB)
011 Populate Dynamodb Table (44.26 MB)
012 Dynamodb Batch Operations (33.48 MB)
001 Getting Started with Amazon Athena (21.87 MB)
002 Quick Recap of Glue Catalog Databases and Tables (14.42 MB)
003 Access Glue Catalog Databases and Tables using Athena Query Editor (10.2 MB)
004 Create Database and Table using Athena (28.15 MB)
005 Populate Data into Table using Athena (14.16 MB)
006 Using CTAS to create tables using Athena (49.48 MB)
007 Overview of Amazon Athena Architecture (28.29 MB)
008 Amazon Athena Resources and relationship with Hive (11.86 MB)
009 Create Partitioned Table using Athena (15.52 MB)
010 Develop Query for Partitioned Column (21.82 MB)
011 Insert into Partitioned Tables using Athena (7.63 MB)
012 Validate Data Partitioning using Athena (15.4 MB)
013 Drop Athena Tables and Delete Data Files (28.25 MB)
014 Drop Partitioned Table using Athena (10.22 MB)
015 Data Partitioning in Athena using CTAS (36.58 MB)
001 Amazon Athena using AWS CLI - Introduction (1.86 MB)
002 Get help and list Athena databases using AWS CLI (6.04 MB)
004 Managing Athena Workgroups using AWS CLI (30.2 MB)
006 Run Athena Queries using AWS CLI (14.97 MB)
008 Get Athena Table Metadata using AWS CLI (10.32 MB)
010 Run Athena Queries with custom location using AWS CLI (34.88 MB)
012 Drop Athena table using AWS CLI (13.81 MB)
014 Run CTAS under Athena using AWS CLI (17.04 MB)
001 Amazon Athena using Python boto3 - Introduction (4.32 MB)
002 Getting Started with Managing Athena using Python boto3 (43.59 MB)
004 List Amazon Athena Databases using Python boto3 (18.21 MB)
006 List Amazon Athena Tables using Python boto3 (55.87 MB)
008 Run Amazon Athena Queries using Python boto3 (32.77 MB)
010 Review Athena Query Results using boto3 (48.76 MB)
001 Getting Started with Amazon Redshift - Introduction (2.33 MB)
002 Create Redshift Cluster using Free Trial (12.74 MB)
003 Connecting to Database using Redshift Query Editor (13.72 MB)
004 Get list of tables querying information schema (16.22 MB)
006 Run Queries against Redshift Tables using Query Editor (10.49 MB)
008 Create Redshift Table using Primary Key (17.28 MB)
011 Insert Data into Redshift Tables (18.42 MB)
012 Update Data in Redshift Tables (12.28 MB)
013 Delete data from Redshift tables (9.63 MB)
014 Redshift Saved Queries using Query Editor (17.48 MB)
015 Deleting Redshift Cluster (11.33 MB)
016 Restore Redshift Cluster from Snapshot (21.2 MB)
001 Copy Data from s3 to Redshift - Introduction (3.64 MB)
002 Setup Data in s3 for Redshift Copy (42.48 MB)
003 Copy Database and Table for Redshift Copy Command (10.14 MB)
004 Create IAM User with full access on s3 for Redshift Copy (13.12 MB)
005 Run Copy Command to copy data from s3 to Reshift Table (11.77 MB)
006 Troubleshoot Errors related to Redshift Copy Command (6.69 MB)
007 Run Copy Command to copy from s3 to Redshift table (7.11 MB)
008 Validate using queries against Redshift Table (6.9 MB)
009 Overview of Redshift Copy Command (18.5 MB)
010 Create IAM Role for Redshift to access s3 (22.5 MB)
011 Copy Data from s3 to Redshift table using IAM Role (33.26 MB)
012 Setup JSON Dataset in s3 for Redshift Copy Command (25.23 MB)
013 Copy JSON Data from s3 to Redshift table using IAM Role (24.48 MB)
001 Develop application using Redshift Cluster - Introduction (2.39 MB)
002 Allocate Elastic Ip for Redshift Cluster (19.83 MB)
003 Enable Public Accessibility for Redshift Cluster (26.4 MB)
004 Update Inbound Rules in Security Group to access Redshift Cluster (33.6 MB)
005 Create Database and User in Redshift Cluster (14.5 MB)
006 Connect to database in Redshift using psql (19.56 MB)
007 Change Owner on Redshift Tables (12.22 MB)
008 Download Redshift JDBC Jar file (5.78 MB)
009 Connect to Redshift Databases using IDEs such as SQL Workbench (22.06 MB)
010 Setup Python Virtual Environment for Redshift (34.6 MB)
011 Run Simple Query against Redshift Database Table using Python (33.07 MB)
012 Truncate Redshift Table using Python (13.54 MB)
013 Create IAM User to copy from s3 to Redshift Tables (7.22 MB)
014 Validate Access of IAM User using Boto3 (15.25 MB)
015 Run Redshift Copy Command using Python (22.09 MB)
001 Redshift Tables with Distkeys and Sortkeys - Introduction (13.7 MB)
002 Quick Review of Redshift Architecture (18.29 MB)
003 Create multi-node Redshift Cluster (14.17 MB)
004 Connect to Redshift Cluster using Query Editor (10.76 MB)
005 Create Redshift Database (3.75 MB)
006 Create Redshift Database User (16.25 MB)
007 Create Redshift Database Schema (16.72 MB)
008 Default Distribution Style of Redshift Table (12.94 MB)
009 Grant Select Permissions on Catalog to Redshift Database User (15.86 MB)
010 Update Search Path to query Redshift system tables (39.37 MB)
011 Validate table with DISTSTYLE AUTO (35.98 MB)
012 Create Cluster from Snapshot to the original state (30.8 MB)
013 Overview of Node Slices in Redshift Cluster (15.1 MB)
014 Overview of Distribution Styles (29.8 MB)
015 Distribution Strategies for retail tables in Redshift (6.45 MB)
016 Create Redshift tables with distribution style all (29.78 MB)
017 Troubleshoot and Fix Load or Copy Errors (24.52 MB)
018 Create Redshift Table with Distribution Style Auto (11.51 MB)
019 Create Redshift Tables using Distribution Style Key (43.07 MB)
020 Delete Cluster with manual snapshot (3.74 MB)
001 Redshift Federated Queries and Spectrum - Introduction (3.66 MB)
002 Overview of integrating RDS and Redshift for Federated Queries (15 MB)
003 Create IAM Role for Redshift Cluster (7.75 MB)
004 Setup Postgres Database Server for Redshift Federated Queries (23.06 MB)
005 Create tables in Postgres Database for Redshift Federated Queries (29.16 MB)
006 Creating Secret using Secrets Manager for Postgres Database (21.52 MB)
007 Accessing Secret Details using Python Boto3 (38.28 MB)
008 Reading Json Data to Dataframe using Pandas (54.08 MB)
009 Write JSON Data to Database Tables using Pandas (72.09 MB)
010 Create IAM Policy for Secret and associate with Redshift Role (33.21 MB)
011 Create Redshift Cluster using IAM Role with permissions on secret (19.77 MB)
012 Create Redshift External Schema to Postgres Database (37.68 MB)
013 Update Redshift Cluster Network Settings for Federated Queries (55.72 MB)
014 Performing ETL using Redshift Federated Queries (27.73 MB)
015 Clean up resources added for Redshift Federated Queries (16.89 MB)
016 Grant Access on Glue Data Catalog to Redshift Cluster for Spectrum (16.2 MB)
017 Setup Redshift Clusters to run queries using Spectrum (9.87 MB)
018 Quick Recap of Glue Catalog Database and Tables for Redshift Spectrum (14.58 MB)
019 Create External Schema using Redshift Spectrum (17.18 MB)
020 Run Queries using Redshift Spectrum (20.26 MB)
021 Cleanup the Redshift Cluster (3.75 MB)