An Interest In:
Web News this Week
- April 23, 2024
- April 22, 2024
- April 21, 2024
- April 20, 2024
- April 19, 2024
- April 18, 2024
- April 17, 2024
The Data Engineer Roadmap
Introduction
This has been inspired by the Full Stack Developer's Roadmap post written by @ender_minyard
With the ever growing data volumes and demands, the data engineering career has been one of the fastest growing jobs for the past few years.
According to the 2021 Stack Overflow survey, data engineers are one of the top 5 highest paid professionals right after SREs and DevOps engineers:
If you are looking to become a data engineer, here are some resources for data engineering that you can save for later.
Table Of Contents
- Fundamentals
- Programming basics
- Testing
- Database Fundamentals
- Data warehouses
- Object storage
- Data processing
- Messaging
- Cluster computing
- Workflow Scheduling
- Monitoring data pipelines
- Infrastructure as Code
- CI/CD
Fundamentals
Having a solid understanding of the Linux operating system could be a must in many IT related roles. You are going to benefit a lot if you know the basics of the following:
- Basic Terminal Usagehttps://devdojo.com/course/linux-command-line-basics
- Shell Scripting
- Git and GitHub
- Networking
Programming basics
As with any IT related role it is essential to have fundamental knowledge of programming in general. The programming language itself does not matter that much, but you need to have good understanding of programming paradigms and best practices.
Testing
- Unit Testing
- Functional testing
Database Fundamentals
Having a solid understanding of SQL, data normalization and ACID transactions is a must for all data engineers.
Relational Databases
Non-relational databases
- Document: MongoDB, Elasticsearch
- Wide column: Apache Cassandra, Apache HBase
- Graph: Neo4j
- Key-value: Redis, Memcached
Data warehouses
Object storage
Data processing
Batch
Hybrid
Streaming
- Materialize - The Streaming Database for Real-time Analytics
- Apache Kafka
- Apache Storm
Messaging
Cluster computing
Workflow Scheduling
Monitoring data pipelines
Infrastructure as Code
- Containers: Docker
- Orchestration: Kubernetes, Docker Swarm
- Provisioning: Terraform
- Automation: Ansible
CI/CD
Conclusion
This was inspired by the Data Engineer Roadmap open source repository here:
https://github.com/datastacktv/data-engineer-roadmap
I wanted to build upon the roadmap and provide a list of resources for each topic.
Let me know if I've missed anything! Hope you find this useful and make sure to keep learning
You can follow me on Twitter at: @bobbyiliev_
Original Link: https://dev.to/bobbyiliev/the-data-engineer-roadmap-4o30
Dev To
An online community for sharing and discovering great ideas, having debates, and making friendsMore About this Source Visit Dev To