Azure DataBricks Architecture Part-2 (DataBricks Cluster)
Introduction: In this Blog post we are going to discuss about DataBricks cluster type and creation options that need to be selected when creating DataBricks cluster. Type of DataBricks Cluster: Mainly we can divide it into two types. · All-purpose Cluster · Job Cluster Difference between Them: All-purpose Cluster Job Cluster Created Manually Created By JOB Persisted (Can Terminated Manually) Non Persisted (Terminated when Job Ends) Suitable for Interactive Workload Suitable for Automated Work Load Shared among many users Isolated Just for JOB Expensive to run Cheaper to run Note: We cannot Create Job cluster. It is automatically created when Job runs. Cluster Configuration option Details: We need to select ...