Skill Path

Python Data Engineering - Learn Data Engineering with Python: visualize, manage and manipulate data at scale

Intermediate18h 44min7 chapters1 simulationsLast updated 10/2025

Embark on the Python Data Engineering skill path, tailored for software engineers aiming to master data management. This course guides you from the fundamentals of Python to advanced techniques in data manipulation and management. You'll learn to efficiently process and analyze large datasets using pandas and NumPy, and visualize your insights with Matplotlib. The curriculum covers all phases of data treatment—from initial data wrangling to the final presentation of your findings. Ideal for software engineers seeking to enhance their data engineering skills and work effectively with data at scale. 

By Chris Koch · Instructional designer, Instructor and Sr. Software Engineer

Skill objectives

Chapter 01

What is Data Engineering

In this chapter we are going to learn more about the definition of data engineering and what that really takes into account. You will learn the tasks and core competencies of data engineers and we will introduce you to a few key concepts like ETL

Skills: Data engineering skills

Link10min

What is data engineering?

To get started with this path we need to talk about what data engineering is in the first place: As a data engineer you will need to extract data from a variety of sources, organize that data, and finally export it so other people/applications can use it. 

Link20min

Intro to ETL tools

ETL is the foundation of data engineering, it stands for  extract, transform and load and describes fairly well what you will be doing as a data engineer. 

Video11min

How I would learn Data Engineering

How an experienced Data Engineer would learn the field. 

Link25min

Data engineer vs Data scientist

It is very easy to confuse data engineering with analyzing data, and the fields are connected and have some overlaps. However, in the most basic sense, a data engineer extracts and prepares the data while a data analyst uses that data to get deeper insights. 

Link10min

Best programming language for Data Engineers

As a data engineer you will work with a broad range of sources and will thus be required to learn various tools. The primary one is usually Python and SQL but, given the project, you might have to learn other languages as well. 

Link5min

Data Engineer wiki

Data engineers have created a wiki that contains a lot of useful learning tools and a glossary of technical terms; it also contains events and communities. Definitely a page you want to bookmark. 

Chapter 02

Python Data Tools: how to manage data at scale with Python

This chapter is fully focused on the tools you can use with Python to manage large datasets of data. You will learn how to use Jupyter notebooks, Numpy and Pandas.

Skills: Pandas, Python, NumPy, Jupyter Notebook

Video25min

Jupyter notebooks

When working with data you usually don't use a traditional code editor like VS code but rather Jupyter notebooks: These allow you to separate your code into chunks and run each of those  separately, they also include nice visual features to display graphs and tables. 

Video14min

Learn Numpy in 5 minutes

The single most important tool in Python for data analysis is an array, which you get from Numpy. An Array is basically a list that is much faster and more powerful. 

Link1h

Numpy documentation and tutorial

While Numpy arrays are fundamentally simple, you can do a lot with them. Reading through the documentation/manual will give you a good idea what you can do with it. 

Video39min

A gentle introduction to Pandas

Numpy arrays are only for single sets of data but usually you are working with lots of data that you need to organize. For that you have Pandas, which gives you a DataFrame. This combines multiple Numpy arrays into a larger DataFrame that lets you organize things efficiently. 

Link1h

Pandas documentation and tutorial

Like with Numpy, Pandas' DataFrames are fundamentally not too complicated but you can do a lot with them. Hence, it is really important to understand the documentation and read through the manual inside of it. 

Link40min

101 Pandas exercises for Data Analysts

To really get good at Pandas you need to practice. The attached website contains 101 exercises and if you get through all of them you should have a good understanding of the library. 

Chapter 03

Extracting data with Python and other tools

In this chapter, we dive deep into data extraction using various techniques. We will use Python and SQL in a variety of environments and use cases.

Skills: Data Mapping and Extraction, Microsoft Excel, Selenium, SQL

Link20min

What is Data Extraction

The first step to working with data is to extract it. This is not a  standardized process as the data can be very diverse. Ideally, you simply import an SQL database or some kind of Excel file but there can be lots of other, more unconventional, datasources that you might have to work with. 

Video21min

Web scraping with Python

Web scraping is a very common source of data. All you do for that is download an HTML file and then extract specific bits from it. Python has a few libraries for that. 

Video1h

BeautifulSoup

To expand on web scraping, the main library in Python for that is BeautifulSoup4, or BS4, which allows you to download and analyze HTML files very efficiently. You can find here the link to the full playlist: https://www.youtube.com/playlist?list=PLzMcBGfZo4-lSq2IDrA6vpZEV92AmQfJK 

Link25min

Getting started with Selenium

With tools like BeautifulSoup you can download HTML files but quite often that isn't enough: Occasionally you will need to create a bot that can itself click through websites and store important data. Selenium is the Python library best for this purpose. 

Video24min

SQL

SQL, or Structured Query Language, is that standard language to work with databases and extract data from them. Python has a number of modules to work with SQL and import parts of a database. 

Link5min

Reading SQL data with Python

While Python has dedicated SQL modules a common shortcut is to use Pandas for that. This might not be as powerful as a separate module but very often you just want a basic import. Also, Pandas can do basic web scraping as well! 

Video20min

Import Excel data

Excel is another source of data that you need to work with quite often. Once again, Python can import this data well, either via a dedicated module like OpenPyXL or via Pandas. 

Video16min

Python CSV tutorial

CSV, or comma separated values, are text values that contain values separated by a comma. For homogenous datasets these are quite common as they are easy to create and very small. You can import them either via Pandas, Numpy or even with standard Python. 

Chapter 04

Exploratory Data Analysis: understand your data and their characteristics

Exploratory Data Analysis is an approach to analyzing datasets that involves summarizing their main characteristics, often using visual methods. It’s a critical first step in data analysis, allowing analysts to discover patterns and more

Skills: Data Mapping, Exploratory Analysis

Link20min

What is exploratory data analysis

An introduction to exploratory data analysis to get started. 

Video32min

Exploratory Data Analysis in Pandas

Just having a basic understanding of data exploration isn't too helpful, you actually need to do practical work in a DataFrame. This video is a really good start to go through an example and see how Pandas can help you. 

Link30min

Intro to Exploratory Data Analysis in Python

To get into more detail on data exploration you can read through this article to get more ideas like how to manipulate columns in a DataFrame. 

Link25min

Exploratory Data Analysis is mostly just vibes, man

A really important thing to understand is that Numpy and Pandas are just tools that make your life easier. they do not prevent you from making statistical errors. This article is very good at discussing various ideas how to approach these kinds of problems. 

Chapter 05

Data organization and cleaning

In this chapter, you will learn how to clean up data before working on it. This is usually related to the removal of columns, information, and other data that is not needed.

Skills: Data Cleaning and Preprocessing

Video39min

Data cleaning in Python

A really important task for data organization is data cleaning, meaning that you remove unnecessary data and errors, hide columns that aren't required, ensure that you have the right datatype and that columns are named appropriately etc... 

Link30min

Top 20 Pandas functions which are commonly used for Exploratory Data Analysis

Reorganizing the data can be a step in exploratory data analysis or in the data organization part. To go into depth, this article  outlines key methods to reorganize your DataFrames to suit your data. 

Link10min

How to organize your Python Data Science project

Just organizing the data itself isn't enough, you also need to make sure that the whole project is organized well. This includes the folder setup, naming convention, file types etc... 

Link30min

Pandas Data Science by example

With the basics covered you need to practice reorganizing data. The linked article contains a lot of examples and recipes that should help you to master handling basically any kind of data. 

Chapter 06

Data visualization MatPlotLib: a complete Python library for data visualization

Now that you have cleaned your data it's time to visualize them and it's chapter is all about learning how to use a few Python libraries to easily visualize data and build any type of chart.

Skills: Data Visualization, Python

Link20min

Pyplot tutorial

An introduction to creating graphs in Python using Pyplot, a submodule of MatPlotLib. 

Video22min

Intro to Data Analysis and visualization with Python, MatPlotLib and Pandas

An introduction to creating graphs using Pandas and Pyplot together. Python's data tools have been developed together, meaning that Numpy and Pandas can be very easily integrated with MatPlotLib. 

Link20min

MatPlotLib examples

Matplotlib can create basically any kind of graphic and on this page you will find an example for all the available options. Don't read all of it in one go instead look through it when you need a specific graph and then adapt the example. 

Video12min

MatPlotLib animations

MatPlotLib is also able to animate graphs, which can give a graph a very professional look. 

Chapter 07

Building a data pipeline

A data pipeline automates the flow of data from sources to storage and analysis. It is the final result of everything we have seen in this path: It involves extraction, transformation, and loading (ETL) processes to prepare and move data efficiently.

Skills: Data Pipelines, ETL Pipeline Development, ETL (Extract, Transform, Load), Data engineering skills, Dataset Validation, Data Manipulation, Data Mapping, Data Modeling, Data Science Fundamentals

Video8min

Data pipelines explained

An introduction to what data pipelines are and how to approach them. A data pipeline is simply the whole path from gathering data to transforming it and then loading it into other tools. 

Link25min

More details on how data pipelines work

An article that goes into a lot of detail on how data pipelines  can look like. It talks about the various possible sources of data, if the pipeline generates real world or "static" data; what tools are going to be used and where the data can end up. 

Link45min

How to create data pipelines in Python

A very detailed and practical guide how a data pipeline can be established in Python. It nicely brings together the concepts we have covered so far in the path. While the created data pipeline is simple it outlines the basic approaches to understand the process. 

Video10min

Building a data engineering project

This is a practical project to get started with extracting json data and working with AWS. This would be an excellent start to get data imported for a data pipeline. 

Video11min

How to build an ETL pipeline with Python

Another example to get started with a data pipeline. This video imports data from an SQL server and organizes it using a DataFrame. 

Link45min

Data pipeline recipes in Python

To finish, it is important to look at examples and references what other people are doing. This article contains a few ideas that should be handy. 

AI Simulation3hTraining

Test your Data Engineering skills with our AI Simulation

There is no better way to test your new skills that try our final AI Simulation for Data Engineers: you will be presented with a real life scenario where you need to take the lead and help your company analyze a large amount of maritime data. 