Databricks Certified Associate-Developer for Apache Spark 3.5 サンプル問題プレビュー（抜粋）
収録内容から抜粋したサンプル問題です。「解答・解説を表示」をクリックすると正解を確認できます。
サンプル問題 1： A Spark DataFrame df is cached using the MEMORY_AND_DISK storage level, but the DataFrame is too large to fit entirely in memory. What is the likely behavior when Spark runs out of memory to store the DataFrame?
A: Spark duplicates the DataFrame in both memory and disk. If it doesn't fit in memory, the DataFrame is stored and retrieved from the disk entirely.
B: Spark splits the DataFrame evenly between memory and disk, ensuring balanced storage utilization.
C: Spark will store as much data as possible in memory and spill the rest to disk when memory is full, continuing processing with performance overhead.
D: Spark stores the frequently accessed rows in memory and less frequently accessed rows on disk, utilizing both resources to offer balanced performance.
正解：C
Explanation: When using the MEMORY_AND_DISK storage level, Spark attempts to cache as much of the DataFrame in memory as possible. If the DataFrame does not fit entirely in memory, Spark will store the remaining partitions on disk. This allows processing to continue, albeit with a performance overhead due to disk I/O. As per the Spark documentation: "MEMORY_AND_DISK: It stores partitions that do not fit in memory on disk and keeps the rest in memory. This can be useful when working with datasets that are larger than the available memory." ― Perficient Blogs: Spark - StorageLevel This behavior ensures that Spark can handle datasets larger than the available memory by spilling excess data to disk, thus preventing job failures due to memory constraints.
サンプル問題 2： A data engineer is building a Structured Streaming pipeline and wants the pipeline to recover from failures or intentional shutdowns by continuing where the pipeline left off. How can this be achieved?
A: By configuring the option checkpointLocation during readStream
B: By configuring the option recoveryLocation during the SparkSession initialization
C: By configuring the option recoveryLocation during writeStream
D: By configuring the option checkpointLocation during writeStream
正解：D
Explanation: To enable a Structured Streaming query to recover from failures or intentional shutdowns, it is essential to specify the checkpointLocation option during the writeStream operation. This checkpoint location stores the progress information of the streaming query, allowing it to resume from where it left off. According to the Databricks documentation: "You must specify the checkpointLocation option before you run a streaming query, as in the following example: .option("checkpointLocation", "/path/to/checkpoint/dir") .toTable("catalog.schema.table") ― Databricks Documentation: Structured Streaming checkpoints By setting the checkpointLocation during writeStream, Spark can maintain state information and ensure exactly-once processing semantics, which are crucial for reliable streaming applications.
サンプル問題 3： A data scientist is analyzing a large dataset and has written a PySpark script that includes several transformations and actions on a DataFrame. The script ends with a collect() action to retrieve the results. How does Apache Spark™'s execution hierarchy process the operations when the data scientist runs this script?
A: The script is first divided into multiple applications, then each application is split into jobs, stages, and finally tasks.
B: The entire script is treated as a single job, which is then divided into multiple stages, and each stage is further divided into tasks based on data partitions.
C: The collect() action triggers a job, which is divided into stages at shuffle boundaries, and each stage is split into tasks that operate on individual data partitions.
D: Spark creates a single task for each transformation and action in the script, and these tasks are grouped into stages and jobs based on their dependencies.
正解：C
Explanation: In Apache Spark, the execution hierarchy is structured as follows: Application: The highest-level unit, representing the user program built on Spark. Job: Triggered by an action (e.g., collect(), count()). Each action corresponds to a job. Stage: A job is divided into stages based on shuffle boundaries. Each stage contains tasks that can be executed in parallel. Task: The smallest unit of work, representing a single operation applied to a partition of the data. When the collect() action is invoked, Spark initiates a job. This job is then divided into stages at points where data shuffling is required (i.e., wide transformations). Each stage comprises tasks that are distributed across the cluster's executors, operating on individual data partitions. This hierarchical execution model allows Spark to efficiently process large-scale data by parallelizing tasks and optimizing resource utilization.
Databricks Certified Associate-Developer for Apache Spark 3.5試験の準備に向けた復習資料としてDatabricks Certified Associate-Developer for Apache Spark 3.5問題集を提供しています。Databricks Certified Associate Developer for Apache Spark 3.5 - Python試験の主要な知識ポイントをカバーしています。ページ上部の「Databricks Certified Associate-Developer for Apache Spark 3.5 試験を開始する」ボタンからオンライン無料サンプル問題を確認できます。
Databricks Certified Associate-Developer for Apache Spark 3.5問題集の収録内容と更新状況はどうなっていますか？
答え：現在提供されているDatabricks Certified Associate-Developer for Apache Spark 3.5問題集には、Databricks Certified Associate-Developer for Apache Spark 3.5試験対策に向けた全135問の練習問題と解答が収録されています。Databricks Certified Associate Developer for Apache Spark 3.5 - Python試験に向けた復習に利用できます。
