Data Engineering topics to be covered

  • Introduction of data engineering
  • Stages of data engineering
  • Overview of Spark
  • Overview of Kafka
  • Best data engineering practices and database concepts
    • Handling and logging errors
    • Building human-fault-tolerant pipelines
    • System monitoring
    • Understanding what is necessary to scale up
    • Addressing continuous integration
    • Knowledge of database administration
    • Maintaining data cleaning
    • Ensuring a deterministic pipeline

What software or tools do you need?

  • TBC (to be confirmed)