This document is designed to be read in parallel with the code in the pyspark-template-project repository. Together, these constitute what we consider to be a 'best practices' approach to writing ETL ...

Big data adoption has been growing by leaps and bounds over the past few years, which has necessitated new technologies to analyze that data holistically. Individual big data solutions provide their ...

When Spark submits a job on Kubernetes it creates a driver pod that hosts the Spark UI on port 4040. Those pods are short-lived and their addresses change between runs, making them hard to bookmark or ...