Databricks Generative AI Engineer Associate サンプル問題プレビュー（抜粋）
収録内容から抜粋したサンプル問題です。「解答・解説を表示」をクリックすると正解を確認できます。
サンプル問題 1： A Generative Al Engineer has created a RAG application to look up answers to questions about a series of fantasy novels that are being asked on the author’s web forum. The fantasy novel texts are chunked and embedded into a vector store with metadata (page number, chapter number, book title), retrieved with the user’s query, and provided to an LLM for response generation. The Generative AI Engineer used their intuition to pick the chunking strategy and associated configurations but now wants to more methodically choose the best values. Which TWO strategies should the Generative AI Engineer take to optimize their chunking strategy and parameters? (Choose two.)
A: Change embedding models and compare performance.
B: Add a classifier for user queries that predicts which book will best contain the answer. Use this to filter retrieval.
C: Choose an appropriate evaluation metric (such as recall or NDCG) and experiment with changes in the chunking strategy, such as splitting chunks by paragraphs or chapters. Choose the strategy that gives the best performance metric.
D: Pass known questions and best answers to an LLM and instruct the LLM to provide the best token count. Use a summary statistic (mean, median, etc.) of the best token counts to choose chunk size.
E: Create an LLM-as-a-judge metric to evaluate how well previous questions are answered by the most appropriate chunk. Optimize the chunking parameters based upon the values of the metric.
正解：C, E
Explanation: To optimize a chunking strategy for a Retrieval-Augmented Generation (RAG) application, the Generative AI Engineer needs a structured approach to evaluating the chunking strategy, ensuring that the chosen configuration retrieves the most relevant information and leads to accurate and coherent LLM responses. Here's why C and E are the correct strategies: Strategy C: Evaluation Metrics (Recall, NDCG) Define an evaluation metric: Common evaluation metrics such as recall, precision, or NDCG (Normalized Discounted Cumulative Gain) measure how well the retrieved chunks match the user's query and the expected response. Recall measures the proportion of relevant information retrieved. NDCG is often used when you want to account for both the relevance of retrieved chunks and the ranking or order in which they are retrieved. Experiment with chunking strategies: Adjusting chunking strategies based on text structure (e.g., splitting by paragraph, chapter, or a fixed number of tokens) allows the engineer to experiment with various ways of slicing the text. Some chunks may better align with the user's query than others. Evaluate performance: By using recall or NDCG, the engineer can methodically test various chunking strategies to identify which one yields the highest performance. This ensures that the chunking method provides the most relevant information when embedding and retrieving data from the vector store. Strategy E: LLM-as-a-Judge Metric Use the LLM as an evaluator: After retrieving chunks, the LLM can be used to evaluate the quality of answers based on the chunks provided. This could be framed as a "judge" function, where the LLM compares how well a given chunk answers previous user queries. Optimize based on the LLM's judgment: By having the LLM assess previous answers and rate their relevance and accuracy, the engineer can collect feedback on how well different chunking configurations perform in real-world scenarios. This metric could be a qualitative judgment on how closely the retrieved information matches the user's intent. Tune chunking parameters: Based on the LLM's judgment, the engineer can adjust the chunk size or structure to better align with the LLM's responses, optimizing retrieval for future queries. By combining these two approaches, the engineer ensures that the chunking strategy is systematically evaluated using both quantitative (recall/NDCG) and qualitative (LLM judgment) methods. This balanced optimization process results in improved retrieval relevance and, consequently, better response generation by the LLM
サンプル問題 2： A Generative AI Engineer is designing a RAG application for answering user questions on technical regulations as they learn a new sport. What are the steps needed to build this RAG application and deploy it?
A: Ingest documents from a source C> Index the documents and saves to Vector Search C> User submits queries against an LLM C> LLM retrieves relevant documents C> Evaluate model C> LLM generates a response C> Deploy it using Model Serving
B: Ingest documents from a source C> Index the documents and save to Vector Search C> User submits queries against an LLM C> LLM retrieves relevant documents C> LLM generates a response -> Evaluate model C> Deploy it using Model Serving
C: Ingest documents from a source C> Index the documents and save to Vector Search C> Evaluate model C> Deploy it using Model Serving
D: User submits queries against an LLM C> Ingest documents from a source C> Index the documents and save to Vector Search C> LLM retrieves relevant documents C> LLM generates a response C> Evaluate model C> Deploy it using Model Serving
正解：B
Explanation: The Generative AI Engineer needs to follow a methodical pipeline to build and deploy a Retrieval-Augmented Generation (RAG) application. The steps outlined in option B accurately reflect this process: Ingest documents from a source: This is the first step, where the engineer collects documents (e.g., technical regulations) that will be used for retrieval when the application answers user questions. Index the documents and save to Vector Search: Once the documents are ingested, they need to be embedded using a technique like embeddings (e.g., with a pre-trained model like BERT) and stored in a vector database (such as Pinecone or FAISS). This enables fast retrieval based on user queries. User submits queries against an LLM: Users interact with the application by submitting their queries. These queries will be passed to the LLM. LLM retrieves relevant documents: The LLM works with the vector store to retrieve the most relevant documents based on their vector representations. LLM generates a response: Using the retrieved documents, the LLM generates a response that is tailored to the user's question. Evaluate model: After generating responses, the system must be evaluated to ensure the retrieved documents are relevant and the generated response is accurate. Metrics such as accuracy, relevance, and user satisfaction can be used for evaluation. Deploy it using Model Serving: Once the RAG pipeline is ready and evaluated, it is deployed using a model-serving platform such as Databricks Model Serving. This enables real-time inference and response generation for users. By following these steps, the Generative AI Engineer ensures that the RAG application is both efficient and effective for the task of answering technical regulation questions.
サンプル問題 3： A Generative AI Engineer just deployed an LLM application at a digital marketing company that assists with answering customer service inquiries. Which metric should they monitor for their customer service LLM application in production?
A: Number of customer inquiries processed per unit of time
B: Energy usage per query
C: Final perplexity scores for the training of the model
D: HuggingFace Leaderboard values for the base LLM
正解：A
Explanation: When deploying an LLM application for customer service inquiries, the primary focus is on measuring the operational efficiency and quality of the responses. Here's why A is the correct metric: Number of customer inquiries processed per unit of time: This metric tracks the throughput of the customer service system, reflecting how many customer inquiries the LLM application can handle in a given time period (e.g., per minute or hour). High throughput is crucial in customer service applications where quick response times are essential to user satisfaction and business efficiency. Real-time performance monitoring: Monitoring the number of queries processed is an important part of ensuring that the model is performing well under load, especially during peak traffic times. It also helps ensure the system scales properly to meet demand. Why other options are not ideal: B. Energy usage per query: While energy efficiency is a consideration, it is not the primary concern for a customer-facing application where user experience (i.e., fast and accurate responses) is critical. C. Final perplexity scores for the training of the model: Perplexity is a metric for model training, but it doesn't reflect the real-time operational performance of an LLM in production. D. HuggingFace Leaderboard values for the base LLM: The HuggingFace Leaderboard is more relevant during model selection and benchmarking. However, it is not a direct measure of the model's performance in a specific customer service application in production. Focusing on throughput (inquiries processed per unit time) ensures that the LLM application is meeting business needs for fast and efficient customer service responses.
Databricks Generative AI Engineer Associate試験の準備に向けた復習資料としてDatabricks Generative AI Engineer Associate問題集を提供しています。Databricks Certified Generative AI Engineer Associate試験の主要な知識ポイントをカバーしています。ページ上部の「Databricks Generative AI Engineer Associate 試験を開始する」ボタンからオンライン無料サンプル問題を確認できます。
Databricks Generative AI Engineer Associate問題集の収録内容と更新状況はどうなっていますか？
答え：現在提供されているDatabricks Generative AI Engineer Associate問題集には、Databricks Generative AI Engineer Associate試験対策に向けた全45問の練習問題と解答が収録されています。Databricks Certified Generative AI Engineer Associate試験に向けた復習に利用できます。
どうやってDatabricks Generative AI Engineer Associate問題集を使って復習しますか？
答え：時間の許す限り、内容を理解しながら計画的に復習することをお勧めします。Databricks Generative AI Engineer Associate問題集の多くの設問と知識ポイントを一度の学習ですべて定着させるのは難しいため、3回に分けて反復する方法が効果的です。1回目は全体を通して解説とあわせて理解し、2回目は分野ごとに自分の言葉で説明できるか確認し、3回目は試験直前に間違えやすい設問を中心に総復習します。
学習時間が限られている場合は、Databricks Generative AI Engineer Associate問題集の選択問題を中心に重要キーワードと正答の根拠を優先して整理してください。また、ドラッグ＆ドロップ問題やシミュレーション形式の設問については、手順や前提条件を含めて仕組みを理解しておくことが重要です。
