Optimizing Spark can dramatically improve performance and reduce resource consumption. Typically, optimization in Spark can be approached from three distinct levels: cluster level, code level, and CPU/memory level. Each level addresses different aspects of Spark's operation and contributes uniquely to overall efficiency in distributed processing with Spark.

Cluster Level Optimization

At the cluster level, optimization involves configuring the Spark environment to efficiently manage and utilize the hardware resources available across the cluster. This includes tuning resource allocation settings such as executor memory, core counts, and maximizing data locality to reduce network overhead. Effective cluster management ensures that Spark jobs are allocated the right amount of resources, balancing between under-utilization and over-subscription.

Code Level Optimization

Code level optimization refers to the improvement of the actual Spark code. It involves refining the data processing operations to make better use of Spark’s execution engine. This includes optimizing the way data transformations and actions are structured, minimizing shuffling of data across the cluster, and employing Spark's Catalyst optimizer to enhance SQL query performance. Writing efficient Spark code not only speeds up processing times but also reduces the computational overhead on the cluster.

CPU/Memory Level Optimization

At the CPU/memory level, the focus shifts to the internals of how Spark jobs are executed on each node. This entails tuning garbage collection processes, managing serialization and deserialization of data, and optimizing memory usage patterns to prevent spilling and excessive garbage collection. These optimizations, utilizing the Tungsten optimizer, enhance the utilization of available CPU and memory resources, which is essential for efficiently processing large-scale data.

In this article, we will mostly be diving into code level optimisation based on the code we worked with in the previous articles.

Narrow and Wide Transformations

Narrow Transformations

Narrow transformations involve operations where data required to compute the elements of the resulting RDD can be derived from a single partition of the parent RDD. This means there is no need for data to be shuffled across partitions or across nodes in the cluster. Each partition of the parent RDD contributes to only one partition of the child RDD.

Examples of Narrow Transformations:

• map(): Applies a function to each element in the RDD.

• filter(): Returns a new RDD containing only the elements that satisfy a predicate.

• flatMap(): Similar to map, but each input item can be mapped to zero or more output items.

Because data does not need to move between partitions, narrow transformations are generally more efficient and faster.

Wide Transformations

Wide transformations, also known as shuffle transformations, involve operations where data needs to be shuffled across partitions or even across different nodes. This happens because the data required to compute the elements in a single partition of the resulting RDD may reside in many partitions of the parent RDD.

Examples of Wide Transformations:

• groupBy(): Groups data according to a specified key.

• reduceByKey(): Merges the values for each key using an associative reduce function.

• join(): Joins different RDDs based on their keys.

Wide transformations are costlier than narrow transformations because they involve shuffling data, which is an expensive operation in terms of network and disk I/O.

Spark Computation

When you write and run Spark code, how does it execute from start to finish to produce the output?

• Number of Jobs: In Spark, a job is triggered by an action. Each action triggers the creation of a job, which orchestrates the execution of stages needed to return the results of the action.

  Number of jobs = Number of actions

• Number of Stages: Stages in Spark are sets of tasks that are executed together on the same cluster node and can run in parallel. Stages are divided by wide transformations.

  Number of stages = Number of wide transformations + 1

• This +1 accounts for the final stage that gathers results following any necessary shuffles.

• Number of Tasks: Tasks are the smallest units of work in Spark and correspond to individual operations that can be performed on data partitions in parallel. The number of tasks is directly proportional to the number of partitions of the RDD (Resilient Distributed Dataset). A single task is spawned for each partition for every stage. Therefore, adjusting the number of partitions can significantly affect parallelism and performance.

  Number of tasks = Number of partitions

Catalyst Optimiser - The Internals

Unresolved Logical Plan

This is the initial plan generated directly from the user's query, representing the operations required without detailed knowledge of the actual data types and structure. If there are syntax-level issues such as missing brackets, incorrect function names, or syntactical errors, this plan will generate errors because it can't correctly interpret the query.

Analyzed Logical Plan

After the initial parsing, the query's logical plan is analyzed. This step involves resolving column and table names against the existing schema catalog. It checks whether the columns, tables, and functions referenced in the query actually exist and are accessible. If there are errors in the names or if referenced tables or columns don't exist, the query will fail at this stage.

Optimized Logical Plan - Rule-Based Optimization

Once the logical plan is analyzed and validated, it undergoes rule-based optimizations. These optimizations aim to simplify the query and improve execution efficiency:

• Predicate Pushdown

  Filters are applied as early as possible to reduce the amount of data processed in subsequent steps.

• Projection Pushdown

  Only necessary columns are retrieved, skipping over unneeded data, which minimizes I/O.

• Partition Pruning

  Reduces the number of partitions to scan by excluding those that do not meet the query's filter criteria.

Physical Plan

The optimized logical plan is then converted into one or more physical plans. These plans outline specific strategies for executing the query, including how joins, sorts, and aggregations are to be physically implemented on the cluster. The Catalyst Optimizer evaluates various strategies to find the most efficient one based on the actual data layout and distribution.

Cost Model Evaluation

From the possible physical plans, the Catalyst Optimizer uses a cost model to determine the best execution strategy. This model considers:

• Data distribution

• Disk I/O

• Network latency

• CPU usage

• Memory consumption

The optimizer assesses each plan's cost based on these factors and selects the one with the lowest estimated cost for execution.

Code Generation

This final step involves generating optimized bytecode or SQL code that can be executed by Spark's engine. The code is specifically tailored to be efficient based on the physical plan and the underlying execution environment (like JVM bytecode for operations on RDDs). The SQL code is further optimised by the SQL engine.

Execution of Optimized RDDs

The generated code leads to the execution of the plan on Spark's distributed infrastructure, creating optimized RDDs that carry out the intended data transformations and actions as defined by the query.

Caching

Cache and persist are mechanisms for optimizing data processing by storing intermediate results that can be reused in subsequent stages of data transformations. Here's a concise explanation of both:

Cache

• cache() in Spark stores the RDD, DataFrame, or Dataset in memory (default storage level), making it quickly accessible, speeding up future actions that reuse this data.

• The data is stored as deserialized objects in the JVM memory. This means the objects are ready for computation but occupy more space compared to serialized data.

Persist

• persist() method allows you to specify different storage levels, offering more flexibility compared to cache().

• You can choose how Spark stores the data, whether in memory only, disk only, memory and disk, and more. The choice affects the performance and efficiency of data retrieval and processing.

  • MEMORY_ONLY

    Default for persist(), similar to cache(), stores data as deserialized objects in memory.

  • DISK_ONLY

    Useful for very large datasets that do not fit into memory. It avoids the overhead of recomputing the data by storing it directly on the disk.

  • MEMORY_AND_DISK

    Stores partitions in memory, but if the memory is insufficient, it spills them to disk. This can help handle larger datasets than memory can accommodate without re-computation.

  • OFF_HEAP

    Available only in Java and Scala, this option stores data outside of the JVM heap using serialized objects. This is useful for managing memory more efficiently, especially with large heaps. We’ll discuss this further when looking at the tungsten optimizer in the future articles in this series.

• PySpark Specifics: In PySpark, data is always stored as serialized objects even when using MEMORY_ONLY. This helps reduce the JVM overhead but might increase the computation cost because objects need to be deserialized before processing.

By using caching or persistence strategically in Spark applications, you can significantly reduce the time spent on recomputation of intermediate results, especially for iterative algorithms or complex transformations. This leads to performance improvements in processing workflows where the same dataset is accessed multiple times.

Using Spark UI for Optimizations

Spark UI offers invaluable tools like lineage graphs and stage/task analysis to enhance your application's performance. Here’s how to effectively use these features:

Lineage Graphs for Debugging

The lineage graph in Spark UI visually maps out the RDD or DataFrame transformations, providing a clear and detailed trace of the data flow from start to finish. This visual representation is crucial for understanding how your data is being transformed across various stages of your Spark application, and it plays a key role in identifying inefficiencies:

• Identify Narrow and Wide Transformations

  By examining the lineage graph, you can see whether certain stages involve wide transformations, which can cause substantial shuffling and are likely performance bottlenecks.

• Trace Data Processing Paths

  The lineage allows you to trace the path taken by your data through transformations. This can help in understanding unexpected results or errors in data processing.

• Optimization Opportunities

  The graph can highlight redundant or unnecessary operations. For example, multiple filters might be consolidated, or cache operations could be strategically placed to optimize data retrieval in subsequent actions.

Stage and Task Analysis

The Stage and Task tabs in Spark UI provide a deeper dive into the execution of individual stages and tasks within those stages. They offer metrics such as scheduler delay, task duration, and shuffle read/write stats, which are critical for diagnosing performance issues:

• Task Skew Identification

  By analyzing task durations and shuffle read/write statistics, you can identify data skew, where certain tasks take much longer than others, suggesting an imbalance in data partitioning.

• Memory Allocation Metrics

  By keeping an eye on these metrics, you can fine-tune the memory settings in your Spark application. This helps minimize pauses caused by garbage collection—a process where Spark clears out unused data from memory—and ensures memory is used more efficiently. We will discuss more about this in the future articles.

• Performance Tuning

  Adjustments in your code can be made based on these insights. For example, increasing or decreasing the level of parallelism, modifying join types, or altering the way data is partitioned and distributed across the cluster.

Adjusting Code Based on Insights

Based on the information from the lineage graph and stage/task analysis, you can return to your code to make specific adjustments:

• Refactor Code for Efficient Data Processing:

  Simplify transformation chains where possible, merge operations logically to minimize the creation of intermediate data, and ensure transformations are placed optimally to reduce shuffling.

• Tune Data Shuffling (Shuffle Partitions)

  Adjust configurations related to data shuffling, such as spark.sql.shuffle.partitions, to optimize the distribution of data across tasks and nodes.

• Leverage Caching

  Decide more strategically where to cache data based on the understanding of which datasets are reused extensively across different stages of the application.

Here is a simple example of how to understand and utilize the Spark UI.

Optimisation Cheat Sheet

Dos

1. Focus on per node computation strategy over optimising node to node communication strategy i.e minimize data shuffling: Design transformations to minimize the need for shuffling data across the network. Use narrow transformations instead of wide transformations wherever you can. Example - When reducing the number of partitions, use coalesce, (a narrow transformation) over repartition (a wide transformation).

   Coalesce and Repartition

   These are used to adjust the number of partitions in an RDD or DataFrame. coalesce is used to decrease the number of partitions, while you can both increase and decrease the number of partitions using repartition.

    # Example DataFrame
    df = spark.range(1000)
   
    # Using coalesce to reduce the number of partitions
    df_coalesced = df.coalesce(5)
    print(f"Number of partitions after coalescing: {df_coalesced.rdd.getNumPartitions()}")
   
    # Using repartition to change the number of partitions
    df_repartitioned = df.repartition(10)
    print(f"Number of partitions after repartitioning: {df_repartitioned.rdd.getNumPartitions()}")
   

2. Use DataFrames and Datasets: Instead of using RDDs, use DataFrames and Datasets wherever possible. These abstractions allow the Catalyst Optimizer to apply advanced optimizations like predicate pushdown, projection pushdown, and partition pruning.

3. Filter Early: Apply filters as early as possible in your transformations. This reduces the amount of data processed in later stages, enhancing performance.

4. Use Explicit Partitioning: When you know your data distribution, use explicit partitioning to optimize query performance. This helps in operations like join and groupBy, which can benefit significantly from data being partitioned appropriately.

5. Use the best joining strategy according to use case:

   Joining Strategies Recap

   Here is a recap of the join strategies sorted by best to worst in terms of performance:

   • Broadcast Join (Map-Side Join):

     • Best performance for smaller datasets.

     • Efficient when one dataset is small enough (typically less than 10MB) to be broadcast to all nodes, avoiding the need for shuffling the larger dataset.

   • Sort Merge Join:

     • Good for larger datasets that are sorted or can be sorted efficiently.

     • Involves sorting both datasets and then merging, which is less resource-intensive than shuffling but more than a broadcast join.

   • Shuffle Hash Join:

     • Suitable for medium-sized datasets.

     • Both datasets are partitioned and shuffled by the join key, then joined using a hash table. More expensive than sort merge due to the cost of building hash tables but less than Cartesian join.

   • Cartesian Join (Cross Join):

     • Worst performance; should be avoided if possible.

     • Every row of one dataset is joined with every row of the other, leading to a massive number of combinations and significant resource consumption.

6. Leverage Broadcast variables for Small Data: When joining a small DataFrame with a large DataFrame, use broadcast variables and joins to minimize data shuffling by broadcasting the smaller DataFrame to all nodes.

7. Cache Strategically: Cache DataFrames that are used multiple times in your application. Caching can help avoid recomputation, but it should be used judiciously to avoid excessive memory consumption.

8. Use Explain Plans and Lineage graphs in Spark UI: Regularly use the EXPLAIN command to understand the physical and logical plans that Catalyst generates. Additionally, using the Spark UI to examine the lineage of RDDs and DataFrames can further enhance your understanding of how Spark executes your queries, enabling more targeted optimizations.

Don'ts

1. Avoid Unnecessary Columns: Do not select more columns than you need. Use projection pushdown to limit the columns read from disk, especially when dealing with large datasets.

2. Don't Neglect Data Skew: Be cautious of operations on skewed data, which can lead to inefficient resource utilization and long processing times. Try to understand and manage data skew, possibly by redistributing the data more evenly and using skew join optimisations.

3. Avoid Frequent Schema Changes: Frequent changes to the data schema can lead to plan recomputation and reduce the benefits of query plan caching.

4. Don't Overuse UDFs: While User Defined Functions (UDFs) are powerful, they can hinder Catalyst's ability to fully optimize query plans because the optimizer cannot always infer what the UDF does. Whenever possible, use built-in Spark SQL functions.

5. Avoid Large Broadcasts: Broadcasting very large datasets can consume a lot of memory and network bandwidth.

In the realm of data storage and processing, file formats play a pivotal role in defining how information is organized, stored, and accessed.

These formats, ranging from simple text files to complex structured formats, serve as the blueprint for data encapsulation, enabling efficient data retrieval, analysis, and interchange. Whether it's the widely used CSV and JSON for straightforward data transactions, or more specialized formats like Avro, ORC, and Parquet designed for high-performance computing environments, each file format brings unique advantages tailored to specific data storage needs and processing workflows.

Comprehensive Snapshot of File Formats

Here's a table summarizing the characteristics, advantages, limitations, and use cases of the mentioned file formats:

(1) Landing zone suitability refers to how well a data storage format or system accommodates the initial collection, storage, and basic processing of raw data in a preparatory or intermediate stage before further analytics or transformation.

(2) Predicate pushdown is an optimization technique that filters data as close to the source as possible, reducing the amount of data processed by pushing down the filter conditions to the storage layer. (ORC, Avro, Parquet)

Comma-Separated Values (CSV)

• Despite its simplicity, CSV does not enforce any schema, leading to potential issues with data consistency and integrity. It’s extremely flexible but requires careful handling to avoid errors, such as field misalignment or varying data formats.

• CSV files can be used as a lowest-common-denominator data exchange format due to their wide support across programming languages, databases, and spreadsheet applications.

XML/JSON

XML:

• XML is highly extensible, allowing developers to create their tags and data structures, but this flexibility can lead to increased file size and complexity.

• XML is often used in SOAP-based web services and as a configuration language in many software applications due to its structured and hierarchical nature.

JSON:

• JSON's lightweight and text-based structure make it ideal for mobile and web applications where bandwidth may be limited.

• JSON has become the preferred choice for RESTful APIs due to its simplicity and ease of use with JavaScript, allowing for seamless data interchange between clients and servers.

Avro: JSON on Steroids

• Avro's support for direct mapping to and from JSON makes it easy to understand and use, especially for developers familiar with JSON. This feature enables efficient data serialization without compromising human readability during schema design.

• Avro is particularly favored in streaming data architectures, such as Apache Kafka, where schema evolution and efficient data serialization are crucial for handling large volumes of real-time data.

Overview of Avro File Structure

The file structure can be broken down into two main parts: the header and the data blocks.

Header

The header is the first part of the Avro file and contains metadata about the file, including:

• Magic Bytes: A sequence of bytes (specifically, "Obj" followed by a null byte) used to identify the file as Avro format.

• Metadata: A map (key-value pairs) containing information such as the schema (in JSON format), codec (compression method used, if any), and any other user-defined metadata. The schema in the metadata describes the structure of the data stored in the file, defining fields, data types, and other relevant details.

• Sync Marker: A 16-byte, randomly-generated marker used to separate blocks in the file. The same sync marker is used throughout a single Avro file, ensuring that data blocks can be efficiently separated and processed in parallel, if necessary.

Data Blocks

Following the header, the file contains one or more data blocks, each holding a chunk of the actual data:

• Block Size and Count: Each block starts with two integers indicating the number of serialized records in the block and the size of the serialized data (in bytes), respectively.

• Serialized Data: The actual data, serialized according to the Avro schema defined in the header. The serialization is binary, making it compact and efficient for storage and transmission.

• Sync Marker: Each block ends with the same sync marker found in the header, serving as a delimiter to mark the end of a block and facilitate efficient data processing and integrity checks.

Schema Evolution

Schema evolution in Avro allows for backward and forward compatibility, making it possible to modify schemas over time without disrupting existing applications. This feature is critical for long-lived data storage systems and for applications that evolve independently of the data they process.

1. Backward Compatibility: Older data can be read with a newer schema, enabling applications to evolve without requiring simultaneous updates to all data.

2. Forward Compatibility: Newer data can be read with an older schema, allowing legacy systems to access updated data formats.

Optimized Row Columnar (ORC)

• ORC files include lightweight indexes that store statistics (such as min, max, sum, and count) about the data in each column, allowing for more efficient data retrieval operations. This indexing significantly enhances performance for data warehousing queries.

• The format is designed to optimize the performance of Hive queries. By storing data in a columnar format, ORC minimizes the amount of data read from disk, improving query performance, especially for analytical queries that aggregate large volumes of data.

File Structure

An ORC file consists of multiple components, each serving a specific function to enhance storage and query capabilities:

1. File Header: This section marks the ORC file's beginning, detailing the file format version and other overarching properties.

2. Stripes: The core data storage units within an ORC file are its stripes, substantial blocks of data usually set between 64MB and 256MB. Stripes contain a subset of the data, compressed and encoded independently, which supports both parallel processing and efficient data access. For example, in a 500MB ORC file, you might find two stripes, each carrying 250MB and consisting of 1 lakh (100,000) rows.

3. Stripe Header: Holds metadata about the stripe, including the row count, stripe size, and specifics on the stripe's indexes and data streams.

4. Row Groups: Data within each stripe is organized into row groups, with each grouping a set of rows to be processed collectively. Suppose each stripe is divided into 10 row groups, with each group containing 10,000 records. This structuring facilitates efficient processing by batching the columnar data of several rows.

5. Column Storage: Within the row groups, data is stored column-wise. Different data types may leverage distinct compression and encoding strategies, optimizing storage and read operations.

6. Indexes: Integral to ORC's efficiency are the lightweight indexes within each stripe, detailing the contained data. These indexes, including metrics like minimum and maximum values and row counts, enable precise data filtering without necessitating a full data read.

7. Stripe Footer: Contains detailed metadata about the stripe, such as encoding types per column and stream directories.

8. File Footer: Summarizes the file by listing stripes and indexes for all levels, detailing the data schema, and aggregating statistics like column row counts and min/max values.

9. Postscript: Holds file metadata, including encoding and compression details, file footer length, and serialization library versions.

Indexes and Storage Optimization

Each stripe possesses a set of indexes that hold vital data statistics:

• File Level: Offers global statistics for the entire file.

• Stripe Level: Provides specific statistics for each stripe, facilitating efficient data querying by enabling the system to skip irrelevant stripes.

• Row Group Index: Within each stripe, enabling finer control over data access by detailing row group contents.

Parquet

• Parquet supports efficient compression and encoding schemes such as dictionary encoding, which can significantly reduce the size of the data stored. This feature makes it an excellent choice for cost-effective storage in cloud environments.

• Parquet’s columnar storage model is not just beneficial for analytical querying; it also supports complex nested data structures, making it suitable for semi-structured data like JSON. This adaptability makes it a popular choice for big data ecosystems and data lake architectures, where data comes in various shapes and sizes.

Columnar Storage

Unlike traditional row-based storage, where data is stored as a sequence of rows, Parquet stores data in a columnar format. This means each column is stored separately, enabling more efficient data retrieval and compression, especially for analytical workloads where only a subset of columns may be needed for a query.

Row Groups

Similar to row groups in stripes in the ORC format, a row group in Parquet is a vertical partitioning of data into columns within a certain chunk of rows. This organization allows for both columnar storage benefits and efficient data scanning across rows.

Let's make this a little clearer- The data is organized into columns that are further divided into row groups. Each row group contains a segment of the dataset's rows, but stores this data in a column-oriented manner.

Example Scenario

Let's take an example where data about products and countries is stored in Parquet format. In a traditional row store, a query filtering for specific product types and countries would need to scan every row. However, with Parquet's structure, the engine can directly access the columns for products and countries, and within those, only the row groups relevant to the query criteria, significantly reducing the amount of data processed. (Reference for a better understanding- Data Mozart's article on Parquet)

Efficiency in Analytical Queries

Projection and Predicate Pushdown

Parquet's file structure is particularly advantageous for analytical queries that involve projection (selecting specific columns) and predicates (applying filters on rows). The columnar storage allows for projection by accessing only the needed columns, while the row groups facilitate predicate pushdown by enabling selective scanning of rows based on the filters applied.

Metadata

Metadata in Parquet files includes details about the data's structure, such as schema information, and statistics about the data stored within the file.

Facilitating Data Skips

With metadata, query engines can perform optimizations such as predicate pushdown. By providing detailed information about the data's organization and content, the metadata allows query engines to quickly determine which parts of the data need to be read and which can be skipped.

Reducing I/O Operations

By minimizing the need to read irrelevant data from disk, metadata significantly reduces I/O operations, a common bottleneck in data processing. This efficiency is crucial for handling large datasets where reading the entire dataset from disk would be prohibitively time-consuming and resource-intensive.

Metadata Structure

Every Parquet file contains a footer section, which holds the metadata. This includes format version, schema information and column metadata like statistics for individual columns within each row group.

Encoding

Run Length Encoding (RLE)

Run-Length Encoding is a straightforward yet powerful compression technique. Its primary objective is to reduce the size of data that has numerous repeatitive occurrences of the same value (runs). Instead of storing each value individually, RLE compresses a run of data into two elements: the value itself and the count of how many times the value repeats consecutively.

1. Identification of Runs: RLE scans the data to identify runs, which are sequences where the same value occurs consecutively.

2. Compression: Each run is then compressed into a pair: the value and its count. For example, the sequence AAAABBBCC is compressed to A4B3C2.

3. Efficiency: The efficiency of RLE is highly dependent on the data's nature. It performs exceptionally well on data with many long runs but might not be effective for data with high variability.

Run Length Encoding with Bit Packing

Combining RLE with bit-packing enhances its compression capability, especially for numerical data with a limited range of values. Bit-packing is a technique that minimizes the number of bits needed to represent a value, packing multiple numbers into a single byte or word in memory.

Let me expand on bit-packing a little more to make it easier.

A byte consists of 8 bits, and traditionally, a whole byte might be used to store a single piece of information. But if your information can be represented with fewer bits, you can store multiple pieces of information in the same byte.

For example, for integers ranging from 0 to 15, you only need 4 bits to represent all possible values. Why? Because 4 bits can create 16 different combinations (2^4). Here’s how the bit representation looks for these numbers:

• The number 0 is 0000 in binary.

• The number 1 is 0001.

• ...

• The number 15 is 1111.

Each of these can fit in just half a byte (or 4 bits).

Now, instead of using a whole byte (8 bits) for each number, you decide to pack two numbers into one byte.

Let's consider you have the numbers 3 (0011) and 14 (1110) to store:

1. Convert each number to its 4-bit binary representation: 0011 for 3 and 1110 for 14.

2. Pack these two sets of 4 bits into one byte: 00111110.

If you are considering actual values from bit packing beyond the scope of the example above (repeats greater than 15), then you either split the runs or use a flag to identify only those run lengths and then display the exact number of repeats after that.

Now, when RLE is combined with bit-packing, it efficiently compresses runs of integers by first using RLE to encode the run lengths and then applying bit-packing to compress the actual values. This dual approach is particularly effective for numerical data with small ranges or repetitive sequences.

1. RLE on Runs: Identifies and encodes the length of runs.

2. Bit-Packing on Values: Applies bit-packing to the values themselves, further reducing the storage space required.

3. Use Case Example: A sequence of integers with many repeats and a small range, such as [0, 0, 0, 1, 1, 2, 2, 2, 2], would first have RLE applied to compress the repeats, then bit-packing could compress the actual numbers since they require very few bits to represent.

Dictionary Encoding

Dictionary encoding is a compression technique where a unique dictionary is created for a dataset, mapping each distinct value to a unique integer identifier. This approach is highly effective in datasets with many unique records, but they repeat a few specific values often.

1. Dictionary Creation: Scan the dataset to identify all unique values. Each unique value is assigned a unique integer identifier. This mapping of values to identifiers forms the dictionary.

2. Data Transformation: Replace each value in the dataset with its corresponding identifier from the dictionary. This results in a transformed dataset where repetitive values are replaced with their integer identifiers, often much smaller in size than the original values.

3. Storage: The dictionary and the transformed dataset are stored together. To reconstruct the original data, a lookup in the dictionary is performed using the integer identifiers.

Example: Consider a column with the values ["Red", "Blue", "Red", "Green", "Blue"]. A dictionary might map these colors to integers as follows: {"Red": 1, "Blue": 2, "Green": 3}. The encoded column becomes [1, 2, 1, 3, 2].

Dictionary Encoding with Bit Packing

If the range of unique integers is small, you can further compress this data with bit-packing.

1. Apply Dictionary Encoding: First, create a dictionary for the column's unique values, assigning each an integer identifier. Replace each original value in the column with its corresponding identifier.

2. Assess the Dictionary Size: Determine the maximum identifier value used in your dictionary. This tells you the range of values you need to encode.

3. Determine Bit Requirements: Calculate the minimum number of bits required to represent the highest identifier. For example, if your highest identifier is 15, you need 4 bits.

4. Bit-Pack the Identifiers: Instead of storing these identifiers using standard byte or integer formats, you pack them into the minimum bit spaces determined. This might mean putting two 4-bit identifiers into a single 8-bit byte, effectively doubling the storage efficiency for these values. (Similar to what we went over in the 3 (0011) and 14 (1110) example above)

Data Serialization

Having explored file formats, our focus shifts to the critical aspect of how we mobilize this data. This is where serialization comes in- it acts as the conduit or channel, transforming data structures and objects into a streamlined sequence of bytes that can be easily stored, transmitted, and subsequently reassembled.

Spark uses serialization to:

• Transfer data across the network between different nodes in a cluster.

• Save data to disk when necessary, such as persisting RDDs (which will be discussed in the future articles) or when spilling data from memory to disk during shuffles.

Serialization Libraries

Spark supports two main serialization libraries:

Java Serialization Library

• Default Method: The default serialization method used by Spark is the native Java serialization facilitated by implementing the java.io.Serializable interface.

• Ease of Use: This method is straightforward to use since it requires minimal code changes, but it can be relatively slow and produce larger serialized objects, which may not be ideal for network transmission or disk storage.

Kryo Serialization Library

• Performance-Optimized: Kryo is a faster and more compact serialization framework compared to Java serialization. It can significantly reduce the amount of data that needs to be transferred over the network or stored.

• Configuration Required: To use Kryo serialization, it must be explicitly enabled and configured in Spark. This involves registering the classes you'll serialize with Kryo, which can be a bit more work but is often worth the effort for the performance gains.

Implementation and Use Cases

• Configuring Spark for Serialization: In Spark, you can configure the serialization library at the start of your application through the Spark configuration settings, using spark.serializer property. For Kryo, you would set this to org.apache.spark.serializer.KryoSerializer.

• Serialization for Performance: Choosing the right serialization library and tuning your configuration can have a significant impact on the performance of your Spark applications, especially in data-intensive operations. Efficient serialization reduces the overhead of shuffling data between Spark workers and nodes, and when persisting data.

Best Practices

• Use Kryo for Large Datasets: For applications that process large volumes of data or require extensive network communication, using Kryo serialization is recommended.

• Register Classes with Kryo: When using Kryo, explicitly registering your classes can improve performance further, as it allows Kryo to optimize serialization and deserialization operations.

• Tune Spark’s Memory Management: Properly configuring memory management in Spark, alongside efficient serialization, can reduce the occurrence of spills to disk and out-of-memory errors, leading to smoother and faster execution of Spark jobs. We'll discuss more on memory management in the upcoming articles.

Whether encoding data in JSON for web transport, using Avro for its compact binary efficiency, or adopting Parquet's columnar approach for analytic processing, the method of serialization chosen is pivotal. It not only dictates the ease and efficiency of data movement but also shapes how data is preserved, accessed, and utilized across various platforms and applications.

Deserialization: Putting the puzzle back together

After detailing the serialization process, it's essential to understand its counterpart: deserialization. Deserialization reverses serialization's actions, converting the sequence of bytes back into the original data structure or object. This process is crucial for retrieving serialized data from storage or after transmission, allowing it to be utilized effectively within applications.

Spark's Approach to Deserialization

In Apache Spark, deserialization plays a vital role in:

• Reading Data from Disk: When data persisted in storage (like RDDs or datasets) needs to be accessed for computation, Spark deserializes it into the original format for processing.

• Receiving Data Across the Network: Data sent between nodes in a cluster is deserialized upon receipt to be used in computations or stored efficiently.

Deserialization Libraries in Spark

As with serialization, Spark leverages the same libraries for deserialization:

• Java Deserialization: By default, Spark uses Java’s built-in deserialization mechanism. This method is automatically applied to data serialized with Java serialization, ensuring compatibility and ease of use but may carry performance and size overheads.

• Kryo Deserialization: For data serialized with Kryo, Spark employs Kryo’s deserialization methods, which are designed for high performance and efficiency. Similar to serialization, proper configuration and class registration are essential for leveraging Kryo's full potential in deserialization processes.

Implementing Deserialization in Spark Applications

• Configuration for Deserialization: Similar to serialization, deserialization settings in Spark are configured at the application's outset via Spark configuration. Ensuring that the deserialization library matches the serialization method used is crucial for correct data retrieval.

• Deserialization for Data Recovery and Processing: Effective deserialization is key for the rapid recovery of persisted data and the efficient execution of distributed computations, impacting overall application performance.

Best Practices for Deserialization

• Match Serialization and Deserialization Libraries: Ensure consistency between serialization and deserialization methods to avoid errors and data loss.

• Optimize Data Structures for Serialization/Deserialization: Design data structures with serialization efficiency in mind, particularly for applications with heavy data exchange or persistence requirements.

• Monitor Performance Impacts: Regularly assess the impact of serialization and deserialization on application performance, adjusting configurations as necessary to optimize speed and resource usage.

Serialization between Memory and Disk

1. Serialization: Before writing data to disk, Spark serializes the data into a binary format. This is done to minimize the memory footprint and optimize disk I/O.

2. Spilling: When the memory usage exceeds the available memory threshold, Spark spills data to disk to free up memory. This spilled data, in its serialized form, is written to disk partitions, typically in temporary spill files.

3. Deserialization: When the spilled data needs to be read back into memory for further processing, Spark deserializes the data from its binary format back into objects.

The interplay between serialization and deserialization is a foundational aspect of data processing in distributed systems like Apache Spark- while serialization ensures that data is efficiently prepared for transfer or storage, deserialization makes that data usable again, completing the cycle of data mobility. As we continue exploring Spark's advanced capabilities and optimizations in future articles, the importance of efficient serialization and deserialization practices in enhancing performance and enabling complex computations over distributed data will become increasingly apparent.

In the first article, I explained Dataframes in detail, and in the second, I talked about Resilient Distributed Datasets. But what exactly sets them apart?

Here's a table that summarizes the key differences between RDDs and DataFrames in Apache Spark, along with examples for each:

Note:

* Will be covered in future articles

Examples in Context

• RDD Example: Suppose you have a list of sales transactions and you want to filter out transactions with a value greater than 100.

    transactions_rdd = sc.parallelize([(1, 150), (2, 75), (3, 200)])
    filtered_rdd = transactions_rdd.filter(lambda x: x[1] > 100)
  

• DataFrame Example: Achieving the same with a DataFrame:

    transactions_df = spark.createDataFrame([(1, 150), (2, 75), (3, 200)], ["id", "amount"])
    filtered_df = transactions_df.filter("amount > 100")
  

The choice between RDDs and DataFrames depends on the specific needs of your application, such as the need for custom, complex transformations (favouring RDDs) or for optimized, structured data processing (favoring DataFrames).

Understanding Joins in Apache Spark

Let's break down the concept of joins in Apache Spark, starting with the basics and moving towards how to optimize these operations for efficiency.

Introduction to Joins

If you've spent some time in the data domain, you might already be familiar with Joins. These allow us to merge datasets based on common attributes.

Spark supports several types of joins to cater to various needs:

• Inner Join: Retains only the row pairs with matching IDs from both tables.

• Left Join: Retains all rows from the left table, along with their matches from the right table.

• Full Outer Join: Merges the outcomes of both left and right joins, keeping all rows from both tables.

• Cross Join: Generates a Cartesian product, pairing every row from the first table with every row from the second table.

I won't be spending too much time on this topic (SQL basics) and instead will move into how Spark deals with Joins.

How Joins Work in Spark

To perform these joins, Spark may need to shuffle data. This shuffling, a mix of network and sort operations, aligns matching rows for the join but can be resource-intensive. I will go into detail on Shuffling in further articles on Spark Optimisation. But, for now, I want you to keep in mind that:

• Shuffle Process: Spark redistributes data across its distributed system to align matching rows, ensuring all data with the same key (ID) are in the same partition for joining.

• Impact on Performance: Shuffling involves moving data across the network, which can be a bottleneck. Hence, optimizing this process is crucial for performance.

What if, instead of optimizing data transfer between nodes (Node-to-node communication strategy), we focus on maximizing the efficiency of data processing within each individual node by bringing the data into each node (per node computation strategy)?

Broadcast Joins

A broadcast join in Apache Spark is an optimization technique for join operations that can significantly improve performance, especially when one of the datasets involved in the join is much smaller than the other. To understand broadcast joins deeply, let’s explore their mechanism, advantages, and when to use them, using an analogy and then delving into the technical details.

The Broadcast Join Explained

Imagine you’re a teacher in a large school, and you have a small list of special announcements that you need to share with every classroom. Instead of asking every classroom to come to you (which would be chaotic and inefficient), you make copies of the announcements and deliver them to each classroom. Now, every classroom has direct access to the announcements, making the whole process smoother and faster.

In Apache Spark, a similar approach is taken with broadcast joins. Instead of moving large amounts of data across the network to perform a join, the smaller dataset is sent (broadcast) to every node in the cluster where the larger dataset resides. This eliminates the need for shuffling the larger dataset across the network, significantly reducing the amount of data transfer and, consequently, the time taken to perform the join.

In-depth Mechanics of Broadcast Joins

1. Identification: Spark automatically identifies when one side of the join operation is small enough to be broadcasted based on the spark.sql.autoBroadcastJoinThreshold configuration setting. This setting specifies the maximum size (in bytes) of a table that can be broadcast.

2. Broadcasting: The smaller dataset is broadcasted to all executor nodes in the Spark cluster. This involves serializing the dataset and sending it over the network to each node.

3. Local Join Operation: Once the smaller dataset is on every node, Spark performs the join operation locally, combining it with the relevant partition of the larger dataset that resides on the node. Since the data is located together, this operation is very efficient and avoids the costly shuffle operation that would otherwise be necessary.

To illustrate the implementation of a broadcast join in Apache Spark using PySpark (the Python API for Spark), let's consider a simple example. Suppose we have two datasets: a large datasetemployees with employee details, and a smaller datasetdepartments that contains department names and their IDs. Our goal is to use a broadcast join to efficiently join these two datasets on the department ID, ensuring that each employee record is enriched with the department name.

from pyspark.sql import SparkSession
from pyspark.sql.functions import broadcast

# Initialize a SparkSession
spark = SparkSession.builder \
    .appName("Broadcast Join Example") \
    .getOrCreate()

# Example datasets
employees_data = [
    (1, "John Doe", 2),
    (2, "Jane Doe", 1),
    (3, "Mike Brown", 2),
    # .... Assume more records
]

departments_data = [
    (1, "Human Resources"),
    (2, "Technology"),
    # .... Assume more departments
]

# Creating DataFrames
employees_df = spark.createDataFrame(employees_data, ["emp_id", "name", "dept_id"])
departments_df = spark.createDataFrame(departments_data, ["dept_id", "dept_name"])

# Performing a broadcast join (inner)
# Broadcasting the smaller DataFrame (departments_df)
joined_df = employees_df.join(broadcast(departments_df), employees_df.dept_id == departments_df.dept_id, "inner")

# Show the result of the join
joined_df.show()

# Stop the SparkSession
spark.stop()

Advantages of Broadcast Joins

• Performance: The primary advantage is the significant reduction in network traffic, which can greatly improve the performance of join operations.

• Efficiency: By avoiding shuffling, broadcast joins reduce the load on the network and the cluster, making the join process more efficient.

• Scalability: For joins with significantly disparate dataset sizes, broadcast joins allow Spark to scale more effectively, handling large datasets more efficiently by leveraging the parallel processing capabilities of the cluster.

Use Cases

Broadcast joins are particularly effective in scenarios where:

• One dataset is much smaller than the other and can fit into the memory of each node.

• The smaller dataset is used repeatedly in join operations with different larger datasets, making the cost of broadcasting it justified over several operations.

Guidelines for Use

• Size Consideration: Be mindful of the spark.sql.autoBroadcastJoinThreshold setting. If the smaller dataset is close to this size limit, consider manually broadcasting it if it's beneficial.

• Memory Management: Ensure that there is sufficient memory on each node to accommodate the broadcasted data without impacting the execution of other tasks.

In summary, broadcast joins are a powerful optimization in Apache Spark that can lead to significant performance improvements in distributed data processing tasks, particularly when the size disparity between joining datasets is large. Understanding when and how to use broadcast joins can greatly enhance the efficiency of Spark applications.

Other Advanced Joins

Advanced join techniques in Apache Spark are designed to optimize the performance of join operations beyond the basic and broadcast join strategies. These techniques can significantly reduce the computational and memory overhead associated with joins, especially when dealing with large datasets. Let’s dive into some of these advanced techniques:

Sort-Merge Join

A sort-merge join is a method that Spark may choose when both sides of the join have already been sorted on the join key. This technique involves two main steps:

• Sorting: If the datasets are not already sorted, they are sorted by the join key.

• Merging: The sorted datasets are then merged together. Because the data is sorted, Spark can efficiently merge the two datasets by sequentially scanning through them, significantly reducing the need for shuffling data across the network.

Use Cases: Sort-merge joins are particularly effective when dealing with large datasets that are already sorted or when the cost of sorting the data is offset by the reduced need for shuffling during the join.

# Assuming df1 and df2 are the DataFrames to join
df1_sorted = df1.sort("joinKey")
df2_sorted = df2.sort("joinKey")

# Perform the join
joined_df = df1_sorted.join(df2_sorted, df1_sorted.joinKey == df2_sorted.joinKey)

Shuffle Hash Join

The shuffle hash join is another technique that Spark might use, especially when one dataset is much larger than the other but not small enough for a broadcast join. This method works by:

• Shuffling: The smaller dataset is shuffled (partitioned) across the cluster based on the join key.

• Hashing: Spark then builds an in-memory hash table of the smaller dataset on each partition.

• Joining: The larger dataset is scanned partition-wise, and for each row, Spark uses the hash table to quickly find matching rows from the smaller dataset.

Use Cases: This method is efficient when one dataset is moderately small, and its hash table can fit into the memory of each node, allowing for fast lookups.

# Setting these parametersis is more about influencing Spark's 
# choice rather than directly enforcing a Shuffle Hash Join.
spark.conf.set("spark.sql.join.preferSortMergeJoin", "false")
spark.conf.set("spark.sql.autoBroadcastJoinThreshold", "-1")

# Perform the join, Spark might use Shuffle Hash Join depending 
# on data size and other conditions
joined_df = df1.join(df2, ["joinKey"])

Bucketed Joins

Bucketed joins in Spark leverage bucketing, a technique where data is divided based on a join key:

• Bucketing: Data is split into buckets based on a hash of the join key, with each bucket containing a subset of data. This is done during data ingestion. To understand this better, refer to my previous article where I discuss bucketing in detail.

• Joining: When performing a join on bucketed tables, Spark can avoid shuffling by directly joining buckets with matching keys.

Use Cases: Bucketed joins are most effective when dealing with frequent joins on large datasets where the join keys are known ahead of time. By avoiding shuffling, they can significantly improve join performance.

# Bucketed Joins require your data to be bucketed and saved 
# to disk beforehand
df1.write.bucketBy(42, "joinKey").sortBy("joinKey").saveAsTable("df1_bucketed")
df2.write.bucketBy(42, "joinKey").sortBy("joinKey").saveAsTable("df2_bucketed")

# Read bucketed data
df1_bucketed = spark.table("df1_bucketed")
df2_bucketed = spark.table("df2_bucketed")

# Perform the bucketed join without shuffling
joined_bucketed_df = df1_bucketed.join(df2_bucketed, "joinKey")

Here, 42 specifies the number of buckets to create, and "joinKey" is the column used for bucketing. This means that rows with the same value in the joinKey column will end up in the same bucket.

Skew Join Optimization

Data skew occurs when the distribution of values within a dataset is uneven, leading to some tasks taking much longer than others. Spark offers optimizations for skew joins:

• Detecting and Handling Skew: Spark can detect skewed keys and replicate the skewed partition's data to all tasks, allowing each task to process a portion of the skewed data in parallel.

• Salting: A common technique to manually handle skew involves "salting" the join keys by adding a random value, thus distributing the skewed keys across multiple partitions to balance the load. This process creates multiple variations of each key, distributing the data more evenly across the cluster.

Use Cases: Skew join optimizations are crucial for performance when joins involve skewed datasets, helping to distribute the load more evenly across the cluster.

The following steps describe how you can perform the Skew Join:

1. Adding Salt to Join Keys

df1 = df1.withColumn("salt", (rand()*5).cast("int"))
df2 = df2.withColumn("salt", monotonically_increasing_id() % 5)

Fordf1: A new column named salt is added. This column is populated with random integers between 0 and 4, inclusive. The rand() function generates a random floating-point number between 0 and 1, which is then multiplied by 5 to scale it. Casting it to an integer effectively creates a random salt value within the desired range.

Fordf2: Similarly, a salt column is added. Here, monotonically_increasing_id() generates a unique, monotonically increasing integer for each row. Taking this value modulo 5 ensures that the salt values are distributed between 0 and 4, much like in df1, but in a sequential and deterministic manner rather than randomly.

2. Replicating Skewed Data indf2

from pyspark.sql.functions import concat, col
from pyspark.sql.functions import lit

# Create a DataFrame with Salt Values
salt_values = spark.range(5).withColumnRenamed("id", "salt")

# Cross-Join df2 with the Salt Values DataFrame
df1_cross_joined = df1.crossJoin(salt_values)
df2_cross_joined = df2.crossJoin(salt_values)

# Modify the Join Key in Both DataFrames to Include the Salt
df1_cross_joined = df1_cross_joined.withColumn("saltedJoinKey", concat(col("joinKey"), lit("_"), col("salt")))
df2_cross_joined = df2_cross_joined.withColumn("saltedJoinKey", concat(col("joinKey"), lit("_"), col("salt")))

This operation is intended to replicate each row of df2 five times, with each replication having a different salt value from 0 to 4. The flatMap function is used to expand each original row into multiple rows, one for each salt value. This approach ensures that when joining df1 and df2, the join operation is not bottlenecked by any single value of joinKey since the data corresponding to each joinKey in df2 is now evenly distributed across all salt values.

3. Performing the Join with the Salted Column

joined_skewed_df = df1_cross_joined.join(df2_cross_joined, df1_cross_joined.saltedJoinKey == df2_cross_joined.saltedJoinKey)

The join operation now includes a condition which ensures that rows are matched by their salt value, effectively distributing the join operation more evenly across the cluster. But, make a note to always test and monitor the performance impact when using this technique since Cross-joins can significantly increase the volume of data being processed, and hence, should be used judiciously, especially with large datasets.

Implementing these advanced join techniques requires a good understanding of the data and the specific computational challenges it presents. Spark attempts to automatically choose the most efficient join strategy based on the data characteristics and the available cluster resources. However, for optimal performance, developers might need to manually adjust configurations, such as enabling or tuning specific join optimizations based on their understanding of the data and the nature of the join operation.

Leveraging Shared Variables in Apache Spark

Apache Spark provides two types of shared variables to support different use cases when tasks across multiple nodes need to share data: broadcast variables and accumulators. Let's explore each type in detail.

Introduction to Shared Variables

In distributed computing, operations are executed across many nodes, and sometimes these operations need to share some state or data among them. However, Spark's default behavior is to execute tasks on cluster nodes in an isolated manner. Shared variables come into play to allow these tasks to share data efficiently.

Use Cases: Shared variables are used when you need to:

• Share a read-only variable with tasks across multiple nodes efficiently.

• Accumulate results from tasks across multiple nodes in a central location.

Broadcast Variables

Broadcast variables allow the program to efficiently send a large, read-only variable to all worker nodes for use in one or more Spark operations.

Advantages

• Efficiency: Significantly reduces the amount of data that needs to be shipped to each node.

• Performance: Can cache the data in memory on each node, avoiding the need to re-send the data for each action.

Example Use Case

Suppose you have a large lookup table that tasks across multiple nodes need to reference. Instead of sending this table with every task, you can broadcast it so that each node has a local copy.

from pyspark import SparkContext
sc = SparkContext()

lookup_table = {"id1": "value1", "id2": "value2"}  # Large lookup table
broadcastVar = sc.broadcast(lookup_table)

data = sc.parallelize(["id1", "id2", "id1"])
result = data.map(lambda x: (x, broadcastVar.value.get(x))).collect()
# Output: [("id1", "value1"), ("id2", "value2"), ("id1", "value1")]

Accumulators

Accumulators are variables that are only "added" to through an associative and commutative operation and can therefore be efficiently supported in parallel. They can be used to implement counters or sums.

Advantages

• Central Aggregation: Provide a simple syntax for aggregating values across multiple tasks.

• Fault Tolerance: If a task fails and is re-executed, its updates to the accumulator are not reapplied, thus maintaining accuracy.

Example Use Case

You want to count

• how many times each key appears across all partitions of a dataset, or

• debug by counting how many records were processed or

• how many had missing information.

from pyspark import SparkContext
sc = SparkContext()

data = sc.parallelize([1, 2, 3, 4, 5])
counter = sc.accumulator(0)

def count_function(x):
    global counter
    counter += x

data.foreach(count_function)
print(counter.value)  
# Output will be the sum of the numbers in the dataset: 15

Both broadcast variables and accumulators provide essential capabilities for optimizing and managing state in distributed computations with Apache Spark, facilitating efficient data sharing and aggregation across the cluster.

Datasets

We've covered the various abstractions in PySpark, including DataFrames and RDDs. But what about those not native to PySpark?

Datasets are a strongly-typed collection of distributed data. They extend the functionality of DataFrames by adding compile-time type safety. This is particularly useful in Scala due to its static type system, where type information is known and checked at compile time.

Note - To simplify the comparison, imagine Python as similar to JavaScript, and Scala as akin to TypeScript, especially if you're familiar with application development (Only in terms of type strictness).

Key Features

• Type Safety: Datasets provide compile-time type safety. This means errors like accessing a non-existent column name or performing operations with incompatible data types can be caught during compilation rather than at runtime.

• Lambda Functions: Similar to RDDs, Datasets allow you to manipulate data using functional programming constructs. However, unlike RDDs, Datasets benefit from Spark SQL's optimization strategies (Catalyst optimizer).

• Interoperability with DataFrames: A Dataset can be considered a typed DataFrame. In fact, DataFrame is just a type alias for Dataset[Row], where Row is a generic untyped JVM object. This allows seamless conversion and operation between typed Datasets and untyped DataFrames.

Example Usage in Scala

Don't worry too much about Scala or the syntax. Just focus on the comments to understand the code's purpose.

import spark.implicits._

// Define a case class that represents the schema of your data
case class Person(name: String, age: Long)

// Creating a Dataset from a Seq collection
val peopleDS = Seq(Person("Alice", 30), Person("Bob", 25)).toDS()

// Perform transformations using lambda functions
val adultsDS = peopleDS.filter(_.age >= 18)

// Show results
adultsDS.show()

// Is 25 really adulting though?

Encoders

Encoders are what Spark uses under the hood to convert between Java Virtual Machine (JVM) objects and Spark SQL's internal tabular format. This conversion is necessary for Spark to perform operations on data within Datasets efficiently. I'll talk more on JVM in the future articles.

Key Features

• Efficiency: Encoders are responsible for Spark's ability to handle serialization and deserialization cost-effectively. They allow Spark to compress data into a binary format and operate directly on this binary representation, reducing memory usage and improving processing speed.

• Custom Objects: For custom objects, like the Person case class in the example above, Spark requires an implicit Encoder to be available. Scala's implicits and the Datasets API make working with custom types seamless.

Example Usage in Scala

Implicitly, Encoders are used in the Dataset operations shown above. However, when creating Datasets from RDDs or dealing with more complex types, you might need to provide an encoder explicitly:

import org.apache.spark.sql.Encoders

// Explicitly specifying an encoder for a Dataset of type String
val stringEncoder = Encoders.STRING

val namesDS = spark.sparkContext
  .parallelize(Seq("Alice", "Bob"))
  .toDS()(stringEncoder) // Applying the encoder explicitly

namesDS.show()

For a PySpark user, understanding Datasets and Encoders can offer insights into how Spark manages type safety, serialization, and optimization at a deeper level. While PySpark leverages dynamic typing and does not directly expose Datasets, the principles of efficient data representation, processing, and the ability to perform strongly-typed operations are universally beneficial across Spark's API ecosystem.

SparkContext

The entry point to programming Spark with the RDD interface. It is responsible for managing and distributing data across the Spark cluster and creating RDDs. It can load data from various sources like HDFS (Hadoop Distributed File System), local file systems, and external databases. Through SparkContext, you can set various Spark properties and configurations that control aspects like memory usage, core utilization, and the behavior of Spark’s scheduler.

from pyspark import SparkContext
sc = SparkContext(master="local", appName="My App")

Info

Retrieves the SparkContext version, the Python version, and the master URL, respectively.

print(sc.version, sc.pythonVer, sc.master)

SparkContext vs SparkSession

While SparkContext is used for accessing Spark features through RDDs, SparkSession provides a single point of entry for DataFrames, streaming, or Hive features including HiveContext, SQLContext or Streaming Context. Also, with the introduction of Spark 2.0, SparkSession was introduced as a new entry point for Spark applications.

from pyspark.sql import SparkSession
spark = SparkSession.builder.appName("My App").getOrCreate()

Resilient Distributed Datasets - Under the Hood

In the last article, I briefly explained about the lineage of transformations that promotes the fault tolerant nature of Resilient Distributed Datasets. Let's look at that in detail now:

Lazy Evaluation

When you apply a transformation like sort to your data in Spark, it's crucial to understand that Spark operates under a principle known as Lazy Evaluation. This means that when you call a transformation, Spark doesn't immediately manipulate your data. Instead, what happens is that Spark queues up a series of transformations, building a plan for how it will eventually execute these operations across the cluster when necessary.

Lazy evaluation in Spark ensures that transformations like sort don't trigger any immediate action on the data. The transformation is registered as a part of the execution plan, but Spark waits to execute any operations until an action (such as collect, count, or saveAsTextFile) is called.

This delay in execution allows Spark to map out a comprehensive strategy for distributing the data processing workload, enhancing the system's ability to manage resources and execute tasks in an optimized manner.

Explain Plan

To inspect the execution plan that Spark has prepared, including the sequence of transformations and how they will be applied, you can use the explain method on any DataFrame object. This method reveals the DataFrame's lineage, illustrating how Spark intends to execute the given query. This insight into the execution plan is particularly valuable for optimizing performance and understanding the sequence of operations Spark will undertake to process your data.

Lineage Graph

Now, the question arises- What actually are these "lineages"?

RDD lineage is essentially a record of what operations were performed on the data from the time it was loaded into memory up to the present. It’s a directed acyclic graph (DAG) of the entire parent RDDs of an RDD. This graph consists of edges and nodes, where the nodes represent the RDDs and the edges represent the transformations that lead from one RDD to the next.

Creating and Using RDDs

Parallelized collection

Distributes a local Python collection to form an RDD. Useful for parallelizing existing Python collections into Spark RDDs.

parallelized_data = sc.parallelize([1, 2, 3, 4, 5])

From external tables

Used for creating distributed collections of unstructured data from a text file.

rdd_from_text_file = sc.textFile("path/to/textfile")

Partitioning

Partitioning in RDDs refers to the division of the dataset into smaller, logical portions that can be distributed across multiple nodes in a Spark cluster. This concept is fundamental to distributed computing in Spark, enabling parallel processing, which significantly enhances performance for large datasets. More on this topic will be covered in an upcoming article.

rdd_with_partitions = rdd_from_file.repartition(10)
print(rdd_with_partitions.getNumPartitions())

Alters the number of partitions in an RDD to distribute data more evenly across the cluster.

Converting an RDD to a DataFrame

The conversion of an RDD to a DataFrame in PySpark can be accomplished through multiple methods, involving the definition of the schema in different ways.

• StructType

from pyspark.sql.types import StructType, StructField, StringType, IntegerType
schema = StructType([
    StructField("name", StringType(), True),
    StructField("age", IntegerType(), True),
    StructField("city", StringType(), True)
])
rdd = sc.parallelize([("Ankit", 30, "Bangalore"), ("Anitha", 25, "Chennai"), ("Aditya", 35, "Mumbai")])
df = spark.createDataFrame(rdd, schema)

• pyspark.sql.Row

from pyspark.sql import Row
rdd = sc.parallelize([Row(name="Ankit", age=30, city="Bangalore"), Row(name="Anitha", age=25, city="Chennai"), Row(name="Aditya", age=35, city="Mumbai")])
df = spark.createDataFrame(rdd)

Converting CSV to DataFrame

df_from_csv = spark.read.csv("path/to/file.csv", header=True, inferSchema=True)

Directly reads a CSV file into a DataFrame, inferring the schema.

Inspecting Data in DataFrame

df_from_csv.printSchema()
df_from_csv.describe().show()

Displays the schema of the DataFrame and shows summary statistics.

DataFrame Visualization

pandas_df = df_from_csv.toPandas()

Converts a Spark DataFrame to a Pandas DataFrame for visualization.

SQL queries

df_from_csv.createOrReplaceTempView("people")
spark.sql("SELECT * FROM people WHERE age > 30").show()

Executes SQL queries directly on DataFrames that have been registered as a temporary view.

Types of RDDs

RDDs are categorized based on their characteristics, which can include their partitioning strategy or the data types they contain. Understanding these categories is essential for optimizing the performance of Spark applications, as each type has its implications for how data is processed and distributed across the cluster. Here's a deeper look into these categorizations:

Based on Partitioning

ParallelCollection RDDs

These RDDs are created from existing Scala or Python collections. When you parallelize a collection to create an RDD using the parallelize() method, Spark distributes the elements of the collection across the cluster. The distribution is determined by the number of partitions you specify or by the default parallelism level of the cluster. ParallelCollection RDDs are useful for parallelizing small datasets or for testing and prototyping with Spark.

MapPartitions RDDs

MapPartitions RDDs result from transformations that apply a function to every partition of the parent RDD, rather than to individual elements. Transformations like mapPartitions() and mapPartitionsWithIndex() create MapPartitions RDDs. This approach can be more efficient than applying a function to each element, especially when initialising an expensive operation (like opening a database connection) that is better done once per partition rather than once per element.

Shuffled RDDs

Shuffled RDDs are the result of transformations that cause data to be redistributed across the cluster, such as repartition() or by key-based transformations like reduceByKey(), groupBy(), and join(). These operations involve shuffling data across the partitions to regroup it according to the transformation's requirements. Shuffles are expensive operations in terms of network and disk I/O, so understanding when your operations result in Shuffled RDDs can help in optimizing your Spark jobs.

Based on Data Types

RDDs can also be distinguished by the types of data they contain. This classification is more about the content of the RDD rather than its structural characteristics.

Generic RDDs (like Python RDD)

These are RDDs that contain any type of object, such as integers, strings, or custom objects. They are the most flexible type of RDD, as Spark does not impose any constraints on the data type. However, operations on these RDDs may not be as optimized as those on more specific types of RDDs.

Key-Value Pair RDDs (Pair RDDs)

Pair RDDs contain tuples as elements, where each tuple consists of a key and a value. This structure is especially useful for operations that need to process data based on keys, such as aggregating or grouping data by key. I will also go into detail about this specific type of RDD later.

Understanding the different types of RDDs and their characteristics is crucial for writing efficient Spark applications. By choosing the appropriate type of RDD and transformations, developers can optimize data distribution and processing strategies, improving the performance and scalability of their Spark applications.

Transformations on RDDs

Transformations create a new RDD from an existing one. Here are some key transformations:

map()

Applies a function to each element of the RDD, returning a new RDD with the results.

rdd = sc.parallelize([1, 2, 3, 4])
mapped_rdd = rdd.map(lambda x: x * 2)
# Resulting RDD: [2, 4, 6, 8]

filter()

Returns a new RDD containing only the elements that satisfy a condition.

filtered_rdd = rdd.filter(lambda x: x % 2 == 0)
# Resulting RDD: [2, 4]

This operation does not modify the original RDD but creates a new one with the filtered results.

flatMap()

Similar to map, but each input item can be mapped to 0 or more output items.

sentences_rdd = sc.parallelize(["hello world", "how are you"])
flattened_rdd = sentences_rdd.flatMap(lambda x: x.split(" "))
# Resulting RDD: ["hello", "world", "how", "are", "you"]

Actions on RDDs

Actions trigger the execution of the transformations and return results.

collect()

Gathers the entire RDD's data to the driver program. Suitable for small datasets.

all_elements = mapped_rdd.collect()
# Result: [2, 4, 6, 8]

take(N)

Returns an array with the first N elements of the RDD.

first_two = mapped_rdd.take(2)
# Result: [2, 4]

first()

Retrieves the first element of the RDD.

first_element = rdd.first()
# Result: 1

count()

Counts the number of elements in the RDD.

total = rdd.count()
# Result: 4

reduce()

Aggregates the elements of the RDD using a function, bringing the result to the driver.

sum = rdd.reduce(lambda x, y: x + y)
# Result: 10
sum_mapped = mapped_rdd.reduce(lambda x, y: x + y)
# Result: 20
product = rdd.reduce(lambda x, y: x * y)
# Result: 24

saveAsTextFile()

Saves the RDD to the filesystem as a text file.

rdd.saveAsTextFile("path/to/output")

coalesce()

Reduces the number of partitions in the RDD, useful for reducing the dataset size before collecting it on the driver. This method is particularly useful for increasing the efficiency of operations that may benefit from fewer partitions, such as reducing the number of files generated by actions like saveAsTextFile() or optimizing data for export to a single file.

# This operation prepares the RDD for efficient saving or collecting 
# but does not alter its content.

coalesced_rdd = rdd.coalesce(1)

#Result - reduces the number of partitions in a dataset to 1

Each of these transformations and actions plays a vital role in manipulating and retrieving data from RDDs in Spark, enabling efficient distributed data processing across a cluster.

Key-Value Pair RDDs in Detail

Spark includes dedicated transformations and actions designed for use with Pair RDDs, and I'll cover these in the following discussion.

Creating Pair RDDs

Tuples

You can create Pair RDDs by mapping an existing RDD to a format where each element is a tuple representing a key-value pair.

rdd = sc.parallelize(['apple', 'banana', 'apple', 'orange', 'banana', 'apple'])
pair_rdd = rdd.map(lambda fruit: (fruit, 1))

Regular RDDs

Regular RDDs can be transformed into Pair RDDs using transformations that produce key-value pairs, demonstrating the flexibility in handling structured data.

pair_rdd_from_regular = sc.parallelize([("apple", 1), ("banana", 2), ("apple", 3)])

Transformations

Transformations on Pair RDDs allow for sophisticated aggregation and sorting based on keys.

reduceByKey()

Aggregates values for each key using a specified reduce function.

reduced_rdd = pair_rdd.reduceByKey(lambda a, b: a + b)
# Output example: [('banana', 2), ('apple', 3), ('orange', 1)]

sortByKey()

Sorts the RDD by keys.

sorted_rdd = reduced_rdd.sortByKey()
# Output example: [('apple', 3), ('banana', 2), ('orange', 1)]

groupByKey()

Groups values with the same key.

grouped_rdd = pair_rdd.groupByKey().mapValues(list)
# Output example: [('banana', [1, 1]), ('apple', [1, 1, 1]), ('orange', [1])]

This operation groups all the values under each key into a list. It demonstrates the distribution of individual occurrences before aggregation.

join()

Joins two RDDs based on their keys.

price_rdd = sc.parallelize([("apple", 0.5), ("banana", 0.75), ("orange", 0.8)])
joined_rdd = reduced_rdd.join(price_rdd)

Actions

Actions on Pair RDDs trigger computation and return results to the driver program or store them in external storage.

countByKey()

Counts the number of elements for each key.

counts_by_key = pair_rdd.countByKey()
# Output example: defaultdict(<class 'int'>, {'apple': 3, 'banana': 2, 'orange': 1})

This result is a Python dictionary-like object (specifically a defaultdict of type <class 'int'>) that shows how many times each fruit appears in the original dataset.

reduceByKey(): Performs a shuffle across the nodes to aggregate values by key. It's more flexible, allowing for complex aggregation functions beyond simple counting. It's suitable when your RDD contains complex data manipulations or when the dataset is not initially in the form of (key, 1) pairs.

countByKey(): Provides a more direct approach for counting keys when the RDD is already structured as (key, 1) pairs, making it efficient for simple counting tasks. However, it collects the results to the driver, which could be problematic for very large datasets.

Now you understand which option to use and when.

collectAsMap()

Collects the result as a dictionary to bring back to the driver program.

result_map = reduced_rdd.collectAsMap()
# Output example: {'banana': 2, 'apple': 3, 'orange': 1}

These operations on Pair RDDs illustrate Spark's powerful capabilities for distributed data processing, especially when working with data that naturally fits into key-value pairs.

Conclusion

In conclusion, this article has delved deeply into the foundational aspects of Apache Spark, highlighting the pivotal role of SparkContext as the gateway to leveraging Spark's powerful capabilities. Additionally, we've explored the intricacies of Resilient Distributed Datasets (RDDs), uncovering how they serve as the backbone for distributed data processing within Spark. As we move forward, anticipate further exploration into the realms of Spark optimization and memory management. I'm excited to have you join me on this learning journey!

Core Concepts

At its heart, Spark is designed to efficiently process large volumes of data, spreading its tasks across many computers for rapid computation. It achieves this through several core concepts:

• Resilient Distributed Datasets (RDDs)

  Resilient Distributed Datasets (RDDs) are a fundamental concept in Spark. They are immutable distributed collections of objects. RDDs can be created through deterministic operations on either data in storage or other RDDs. They are fault-tolerant, as they track the lineage of transformations applied to them, allowing for re-computation in case of node failures. (Analogous to Git version tracking)

• Directed Acyclic Graph (DAG)

  Spark uses DAG to optimize workflows. Unlike traditional MapReduce, which executes tasks in a linear sequence, Spark processes tasks in a graph structure, allowing for more efficient execution of complex computations.

• DataFrame and Dataset

  Introduced in later versions, these abstractions are similar to RDDs but offer richer optimizations. They allow you to work with structured data more naturally and leverage Spark SQL for querying.

Key Features

Apache Spark is known for its remarkable speed, versatile nature, and ease of use. Here's why:

• Speed

  Spark's in-memory data processing is vastly faster than traditional disk-based processing for certain types of applications. It can run up to 100x faster than Hadoop MapReduce for in-memory processing and 10x faster for disk-based processing.

• Ease of Use

  With high-level APIs in Java, Scala, Python, and R, Spark allows developers to quickly write applications. Spark's concise and expressive syntax significantly reduces the amount of code required.

• Advanced Analytics

  Beyond simple map and reduce operations, Spark supports SQL queries, streaming data, machine learning (ML), and graph processing. These capabilities make it a comprehensive platform for complex analytics on big data.

Architecture

Apache Spark runs on a cluster, the basic structure of which includes a master node and worker nodes. The master node runs the Spark Context, which orchestrates the activities of the cluster. Worker nodes execute the tasks assigned to them. This design allows Spark to process data in parallel, significantly speeding up tasks.

Deployment

Spark can run standalone, on EC2, on Hadoop YARN, on Mesos, or in Kubernetes. This flexibility means it can be integrated into existing infrastructure with minimal hassle.

Use Cases

• Real-Time Stream Processing: Spark's streaming capabilities allow for processing live data streams from sources like Kafka and Flume.

• Machine Learning: Spark MLlib provides a suite of machine learning algorithms for classification, regression, clustering, and more.

• Interactive SQL Queries: Spark SQL lets you run SQL/HQL queries on big data.

Getting Started

The map above lists the functions and methods used to perform basic operations in Pyspark. The code for each is listed in the headings below.

Understanding DataFrames and Catalogs

The image in the map above demonstrates the lifecycle and scope of data structures and components in a typical Spark application, focusing on how the user's code creates and interacts with DataFrames and how those relate to the Spark Cluster and the Catalog within a session.

The arrows show the relationships or interactions between these components:

• The user creates and interacts with DataFrames locally in their Spark application through SparkContext.

• The user can register DataFrames as temporary views in the catalog, making it possible to query them using SQL syntax.

• The user can also directly query tables registered in the catalog using the .table() method through the SparkSession.

• While the Catalog is accessed via SparkSession, the actual DataFrames are managed within the user's application and they interact with the cluster to distribute the computation.

Spark DataFrames are immutable, meaning once created, they cannot be altered. Any transformation creates a new DataFrame.

SparkContext and SparkSession

The SparkContext is the entry point to any Spark application, providing a connection to the Spark cluster, while the SparkSession offers an interface to that connection, enabling higher-level operations and easier interaction with Spark functionalities.

from pyspark.sql import SparkSession

# Creating SparkSession (which also provides SparkContext)
spark = SparkSession.builder \
    .appName("Spark Example") \
    .getOrCreate()

# SparkContext example, accessing through SparkSession
sc = spark.sparkContext

SparkSession.builder.getOrCreate() method

# This returns an existing SparkSession if there's already one in 
# the environment, or creates a new one if necessary.
spark = SparkSession.builder \
    .appName("Spark Example") \
    .getOrCreate()

Catalog

This is a metadata repository within Spark SQL. It keeps track of all the data and metadata (like databases, tables, functions, etc.) in a structured format. The catalog is available via the SparkSession, which is the unified entry point for reading, writing, and configuring data in Spark.

# Lists all the tables inside the cluster
tables_list = spark.catalog.listTables()
print(tables_list)
print([table.name for table in tables_list])

SparkSession.sql() and .show() methods

# This method takes a string containing the query and returns a 
# DataFrame with the results
result_df = spark.sql("SELECT * FROM tableName WHERE someColumn > 10")
result_df.show()

.toPandas() method

# Convert Spark DataFrame to pandas DataFrame
pandas_df = result_df.toPandas()

createDataFrame() and .createOrReplaceTempView()

import pandas as pd

# Assuming pandas_df is a pandas DataFrame
spark_df = spark.createDataFrame(pandas_df)

# The DataFrame is now in Spark but not in the SparkSession catalog 
spark_df.createOrReplaceTempView("temp_table")

createOrReplaceTempView(): This method is a part of the DataFrame API. When you call this method on a DataFrame, it creates (or replaces if it already exists) a temporary view in the Spark catalog. This view is temporary in that it will disappear if the session that created it terminates. The temporary view allows the data to be queried as a table in Spark SQL.

.read.csv() method

# Convert a CSV file to a Spark DataFrame
df = spark.read.csv("path/to/your/file.csv", header=True)

filter method

# Equivalent to WHERE clause in SQL
filtered_df = df.filter(df["some_column"] > 10)

select method

# SELECT clause
selected_df = df.select("column1", "column2")

selectExpr method

# SELECT which takes SQL expressions as input
expr_df = df.selectExpr("column1 as new_column_name", "abs(column2)")

groupBy() and aggregation functions

from pyspark.sql import functions as F

grouped_df = df.groupBy("someColumn").agg(
    F.min("anotherColumn"),
    F.max("anotherColumn"),
    F.count("anotherColumn"),
    F.stddev("anotherColumn")
)

SELECT some_column,
       MIN(another_column) AS min_another_column,
       MAX(another_column) AS max_another_column,
       COUNT(another_column) AS count_another_column,
       STDDEV(another_column) AS stddev_another_column
FROM df
GROUP BY some_column

.withColumn() and .withColumnRenamed() methods

# Adding a new column or replacing an existing one
new_df = df.withColumn("new_column", df["some_column"] * 2)

# Renaming a column
renamed_df = df.withColumnRenamed("old_name", "new_name")

join method

# Joining two tables
joinedDF = table1.join(table2, "common_column", "leftouter")

.cast() method

# Casting a column to a different type
df = df.withColumn("some_column", df["some_column"].cast("integer"))

Conclusion

In wrapping up our exploration of the basics of Apache Spark, it's clear that this powerful framework stands out for its ability to process large volumes of data efficiently across distributed systems. Spark’s versatile ecosystem, which encompasses Spark SQL, DataFrames, and advanced analytics libraries, empowers data engineers and scientists to tackle a wide array of data processing and analysis tasks. Keep an eye out for future articles in this Spark series, where I'll delve deeper into Spark's architecture, execution models, and optimization strategies, providing you with a comprehensive understanding to leverage its full capabilities.

Structure Level Optimization

Optimizing Hive for maximum performance requires not only writing better queries but paying attention to schema and structure. By carefully designing data models, balancing normalization and denormalization, and considering factors like bucketing, partitioning, indexing, and compression, you can significantly improve the performance of your Hive environment. Let's dive into bucketing and partitioning.

Partitioning

Partitioning involves dividing a table into smaller, more manageable parts based on a particular column or set of columns. This can help with data organization, querying, and filtering. I will go over the two types of partitioning - Static and Dynamic Partitioning. I will also dive into a common issue with partitioning - Over Partitioning.

Static Partitioning

In static partitioning, the partition values are explicitly specified manually during the data insertion process. Static partitions require prior knowledge of the partition values, and separate INSERT statements are needed for each partition. It provides better control and predictability over partitioning as the partitions are pre-defined and known in advance. Static partitions are typically suitable for scenarios where the partitioning criteria are well-defined and stable, such as partitioning data by year or region.

Dynamic Partitioning

Dynamic partitioning, also known as automatic partitioning, allows Hive to automatically determine the partitions based on the values present in the data being inserted. With dynamic partitions, a single INSERT statement can insert data into multiple partitions without explicitly specifying partition values. Dynamic partitions are flexible and can adapt to changes in data without requiring manual intervention. This type of partitioning is useful when the partitioning criteria are not known in advance or when new partitions need to be created frequently, such as partitioning data by date or customer ID.

Dealing With Over Partitioning

When working with Hive and HDFS, it's important to consider the drawbacks of having too many partitions. The creation of numerous Hadoop files and directories for each partition can quickly overwhelm the NameNode's capacity to manage the filesystem metadata. As the number of partitions grows, the strain on memory and the potential for exhausting metadata capacity increases.

Additionally, MapReduce processing introduces overhead in the form of JVM start-up and tear-down, which can be significant when dealing with small files.

When implementing time-range partitioning in Hive, it is recommended to estimate the data size across different time granularities. Begin with a granularity that ensures controlled partition growth over time, while ensuring that each partition's file size is at least equal to the filesystem block size or its multiples. This approach optimizes query performance for general queries. However, it's crucial to consider when the next level of granularity becomes appropriate, particularly if query WHERE clauses often select ranges of smaller granularities.

Another approach is to employ two levels of partitions based on different dimensions. For example, you can partition by day at the first level and by geographic region (e.g., state) at the second level. This allows for more targeted querying, such as retrieving data for a specific day within a particular state.

However, it is worth noting that processing imbalances may occur when handling larger states that have significantly more data, resulting in longer processing times compared to smaller states with lesser data volumes. This is where bucketing comes into the picture.

Bucketing

Bucketing involves grouping data based on a hash function applied to one or more columns. This can improve query performance by reducing the amount of data that needs to be scanned.

Use Case

• Partitioning is a useful technique when the cardinality or number of unique values in a column is low. For example, if you have a table of sales data with a "year" column, partitioning the data based on this column can make it easier to filter and analyze sales for specific years.

  Bucketing, on the other hand, is ideal when the cardinality of a column is high.

See the image below on how Hive stores partitions and buckets. There are two partitions based on product ID (P1 and P2) and 3 buckets (bucket 0, bucket 1 and bucket 2).

• Use the function -

SHOW PARTITIONS sales_table

to see all the partitions in your table.

• Use the following query to see the bucket

SELECT * FROM sales_table TABLESAMPLE (bucket 1 out of 3);

Change the bucket number to 1, 2 and 3 to see the data in each bucket.

Query Level Optimization

Join optimization techniques in Hive aim to improve the performance and efficiency of join operations, which are essential for querying and analyzing large datasets.

Join optimization techniques in Hive involve the use of hashtables and leveraging distributed processing to improve performance. The process generally follows these steps:

1. A hashtable is created from the smaller table involved in the join operation.

2. The hashtable is moved to HDFS for storage and accessibility.

3. From HDFS, the hashtable is broadcasted to all nodes in the cluster, ensuring it is available locally.

4. The hashtable is then stored in the distributed cache, residing on the local disk of each node.

5. Each node loads the hashtable into memory from its local disk, allowing for efficient in-memory access.

6. Finally, the MapReduce job is invoked to perform the join operation, utilizing the loaded hashtables to optimize the process.

Here is a summary of common join optimization techniques used in Hive:

Map Side Join Optimization

In certain cases, if one of the tables being joined is small enough to fit in memory, Hive can perform a map-side join. This technique avoids costly shuffling and reduces the need for network transfers by distributing the smaller table to all mapper tasks.

Settings

SET hive.auto.convert.join=true; -- default true after v0.11.0

SET hive.mapjoin.smalltable.filesize=600000000; -- default 25m

SET hive.auto.convert.join.noconditionaltask=true; -- default value above is true so map join hint is not needed

SET hive.auto.convert.join.noconditionaltask.size=10000000; -- default value above controls the size of table to fit in memory

With join auto-convert enabled in Hive, the system will automatically assess if the file size of the smaller table exceeds the specified threshold defined by hive.mapjoin.smalltable.filesize. If the file size is smaller than the threshold, Hive will attempt to convert the common join into a map join. This automatic conversion eliminates the need to explicitly provide map join hints in the query, simplifying the query writing process and reducing manual optimization efforts. By leveraging this feature, users can optimize join operations without the need for manual intervention.

Code

-- Perform a map join on two tables 
SELECT  /*+ MAPJOIN(table2) */ *
FROM table1 
JOIN table2 ON table1.key = table2.key;

Conditions

• Except for one big table, all other tables in the join operation should be small.

Here is a quick guide for the different joins and which table should be the smaller one.

Bucket Map Join Optimization

When both tables are bucketed on the join key, Hive can leverage bucketed map joins. This technique allows data with the same bucket ID to be processed locally, minimizing data movement and reducing the need for a full shuffle.

It's important to note that unlike map side join,

• it can be done on two big tables.

• Only one bucket is loaded in memory essentially.

Settings

SET hive.auto.convert.join=true;

SET hive.optimize.bucketmapjoin=true; -- default false

Code

-- Perform a bucketed map join on two tables
SELECT /*+ MAPJOIN(table2) */ *
FROM table1
JOIN table2 ON table1.key = table2.key
CLUSTERED BY (key) INTO <num_of_buckets>;

Conditions

• Both tables should be bucketed on the join column.

• Join tables must have integral multiple of buckets. Eg - If table 1 has 4 buckets, table 2 can have 4,8,12, etc buckets.

Sort Merge Bucket Join Optimization

When both tables are bucketed and sorted on the join key, Hive can use sort merge bucketed joins. This technique avoids unnecessary data shuffling and performs an efficient merge operation on the sorted buckets, enhancing join performance. This join can be done on two big tables.

Settings

SET hive.input.format= org.apache.hadoop.hive.ql.io.BucketizedHiveInputFormat;

SET hive.auto.convert.sortmerge.join=true;

SET hive.optimize.bucketmapjoin=true;

SET hive.optimize.bucketmapjoin.sortedmerge=true;

SET hive.auto.convert.sortmerge.join.noconditionaltask=true;

Code

-- Perform a sort merge bucketed join on two tables
SELECT /*+ SMBJOIN(table2) */ *
FROM table1
JOIN table2 ON table1.key = table2.key
CLUSTERED BY (key) SORTED BY (key) INTO <num_of_buckets>;

Conditions

• Both tables should be bucketed on the join column.

• The number of buckets in one table must be equal to the number of buckets in the other table.

• Both tables should be sorted based on the join column.

Sort Merge Bucket Map Join Optimization

An SMBM (Sort-Merge-Bucket-Merge) join is a specific type of bucket join that exclusively triggers a map-side join. Unlike a traditional map join that requires caching all rows in memory, an SMBM join avoids this overhead.

Settings

SET hive.auto.convert.sortmerge.join=true

SET hive.optimize.bucketmapjoin=true;

SET hive.optimize.bucketmapjoin.sortedmerge=true;

SET hive.auto.convert.sortmerge.join.noconditionaltask=true;

SET hive.auto.convert.sortmerge.join.bigtable.selection.policy= org.apache.hadoop.hive.ql.optimizer.TableSizeBasedBigTableSelectorForAutoSM J;

Conditions

• Both tables should be bucketed on the join column.

• The number of buckets in one table must be equal to the number of buckets in the other table.

• Both tables should be sorted based on the join column.

Skew Join Optimization

When dealing with data that exhibits a significant imbalance in distribution, data skew can occur, leading to a situation where a few compute nodes bear the brunt of the computation workload.

Settings

SET hive.optimize.skewjoin=true; --If there is data skew in join, set it to true. Default is false.

SET hive.skewjoin.key=100000; --This is the default value. If the number of key is bigger than --this, the new keys will send to the other unused reducers.

SET hive.groupby.skewindata=true;

Data skew can also occur when performing GROUP BY operations, leading to uneven distribution and performance bottlenecks. To address this, Hive offers the setting above, enabling skew data optimization specifically for GROUP BY results. Once enabled, Hive initiates an additional MapReduce job that strategically redistributes the map output across reducers in a random manner, effectively mitigating data skew and improving overall query performance. By configuring this setting, Hive ensures a more balanced workload distribution and optimized processing for GROUP BY operations affected by skewed data.

Code

-- Perform a join with skew join optimization
SELECT /*+ SKEWJOIN(table1) */ *
FROM table1
JOIN table2 ON table1.key = table2.key;

• If you are performing the same calculation within a function for different input arguments, use nested functions.

This nested function can take one input argument and return the value after the calculation is performed on that argument. In the main (outer) function, return this nested function with the different input arguments passed to it.

So your initial function definition:

def square_three_numbers(n1,n2,n3):
    sq_n1 = n1**2
    sq_n2 = n2**2
    sq_n3 = n3**2
    return (sq_n1,sq_n2,sq_n3)

Becomes:

def square_three_numbers(n1,n2,n3):
    def square_number(number):
        return (number**2)
    return (square_number(n1),square_number(n2),square_number(n3))

The second snippet is a way more efficient way of modularizing your code.

• Use flexible arguments - '*args' and '**kwargs' when you don't know how many arguments you will be passing.

Use '*args' when you are just plainly passing the arguments. This gets passed into the function body as a tuple and then you can iterate through it to get your arguments.

def sum_numbers(*args):
    total=0
    for number in args:
        total=total+number
    return(total)

#Function call
print(sum_numbers(1,2,3))

#Output:
#6

Use '**kwargs' when you pass an identifier before your argument value. This gets passed into the function body as a dictionary and then you can iterate through it to get your arguments (with keys being the identifiers and values being the argument value).

#Function definition
def my_favourites(**kwargs):
    for key,value in kwargs.items():
        print("My favourite {0} is \'{1}\'!".format(key,value))



#Function call
my_favourites(series="Handmaid's Tale", movie="Rogue One: A Star Wars Story",song="Less I Know The Better")


#Output:
#My favourite series is 'Handmaid's Tale'!
#My favourite movie is 'Rogue One: A Star Wars Story'!
#My favourite song is 'Less I Know The Better'!

Is it weird that I took more time to decide my favourite movie than to write this code?

Notice the format() method for string. This is way easier than trying to concatenate strings using +.

• Lambda functions (anonymous function) - Used usually when you want to pass a function as an argument to higher-order functions.

def exponent_function(exp):
    return lambda num : num ** exp

raised_to_10 = exponent_function(10)
raised_to_5 = exponent_function(5)

print(raised_to_10(2))
print(raised_to_5(2))

#Output:
#1024
#32

Lambda functions are used with:

map(function,sequence) - Applies the lambda function to all the elements in the sequence. Returns a map object.

numbers = [1,2,3,4]


#Use map() to apply the lambda function for all elements in your list
squares = map(lambda num:num**2, numbers)


#Remember, squares is a map object. So, convert it into a list to see what the values are!
print(list(squares))

#Output:
#[1,4,9,16]

filter(function,sequence) - Applies the lambda function with a condition to the elements in the sequence. Returns a filter object.

numbers = [1,2,3,4,5,6,7]


#Use the lambda function in filter to print even numbers from the list
even_numbers = filter(lambda nums:nums%2==0,numbers)


#Remember, even_numbers is a filter object. So, convert it into a list to see what the values are!
print(list(even_numbers))

#Output
#[2, 4, 6]

reduce(function,sequence) - Applies the lambda function to all the elements in the sequence. Returns a single value.

numbers = [1,2,3,4]


# Use reduce() to apply a lambda function over numbers
product = reduce(lambda item1,item2:item1*item2, numbers)


# Print the result
print(product)

#Output:
#24

The function is first applied to the first two elements. The result of this computation and the third value are again passed into the function and so on.

• Iterators and Iterables - For efficient and flexible processing of data, especially when working with large datasets.

Iterables

fruits = ["apple", "banana", "orange"]
for fruit in fruits:
    print(fruit)

Output:

apple
banana
orange

In this example, the fruits list is an iterable object, and we can loop over its elements using a for loop.

Iterators

fruits = ["apple", "banana", "orange"]
fruit_iterator = iter(fruits)
print(next(fruit_iterator))
print(next(fruit_iterator))
print(next(fruit_iterator))

Output:

apple
banana
orange

In this example, we create an iterator object fruit_iterator using the iter() function. We then use the next() function to iterate over the elements of the fruits list. Each call to next() returns the next element in the list until there are no more elements to iterate over.

• List Comprehensions - For loop reimagined.

When this code:

my_list=[]
for i in range(10):
    for j in range(5):
        my_list.append((j,i))
print(my_list)

becomes this:

my_list_comp=[(j,i) for i in range(10) for j in range(5) ]
print(my_list_comp)

Your program will send you a pop-up saying 'Thank you!'

Seriously. Try it.

However, to ensure that you stay pythonic, it is advisable to have a clear understanding of when to use list comprehensions and when not to.