Quartz 5

評估驅動開發 Eval-Driven Development (EDD): 生成式 AI 軟體不確定性的解決方法 | PPT

Properties2
description評估驅動開發 Eval-Driven Development (EDD): 生成式 AI 軟體不確定性的解決方法 - Download as a PDF or view online for free
tagsclippings

25 min read

逐筆 用 prompt 跑出 predict 結果 You aretasked with categorizing a book based on its information. Your goal is to select the most appropriate skill category from a prede fi ned list. Here is the list of available categories to choose from: 程式語 言 , Data Science, 人 工 智慧, 分散式架構, 系統開發, 行 動軟體開發, 資料庫, 資訊科學, 軟體架構, 軟體測試, 軟體 工 程, 資訊安全, 網站開發, 前端 開發, 架站軟體, 網 頁 設計, Adobe 軟體應 用 , Of fi ce 系列, 遊戲開發設計, UI/UX, 雲端運算, 區塊鏈與 金 融科技, 物聯網 IoT, 商業管理類, 電 子 電路電機 類, 嵌入式系統, 視覺影 音 設計, 考試認證, 數學, 微軟技術, MAC OS 蘋果電腦, 其他, 兒童專區, 製圖軟體應 用 , 語 言 學習, 國家考試, 職涯發展, Java, 理 工 類, 網路通訊, 量 子 電腦 書名: {title} 描述: {description}

讓 AI 合成測試資料🪄 例如我想做 一 個分類檢查 用 戶 的問題是否是股票投資相關問題 • 30 題範例問題: 是股票投資問題 ➡ AI 應回答 Y • 40 題範例問題: 是投資相關問題，但不是股票 ➡ AI 回答 F • 30 題範例問題: 不是投資問題 ➡ AI 回答 N

• 任務描述: 針對特定公司股票或是特定產業，產 生 假設性的五個問題， 一 行 一 題，無需編號，直接輸出問題。無需輸入公司名稱，題 目 內請隨機指名是哪 一 家台灣公司或產業，輸出要求繁體中 文 •這把任務丟到 claude prompt generator，就得到 You are tasked with generating hypothetical questions about a speci fi c company’s stock or a particular industry in Taiwan. Your goal is to create {INVEST_QUESTIONS_NUM} diverse and insightful questions that could be asked about the given company or industry. Follow these guidelines when generating the questions:

  1. Make sure each question is unique and covers different aspects of the company or industry.
  2. Include questions about fi nancial performance, market trends, competitive landscape, future prospects, or recent developments.
  3. Use the name of a random Taiwanese company or industry in each question. Do not use the same company or industry for all questions.
  4. Phrase the questions in a way that encourages thoughtful analysis or discussion.
  5. Ensure all questions are written in Traditional Chinese. Output format:
  • Generate {INVEST_QUESTIONS_NUM} questions, one per line.
  • Do not include numbers or bullet points.
  • Output the questions directly without any additional text. Now, generate {INVEST_QUESTIONS_NUM} hypothetical questions about various Taiwanese companies or industries related to the given input. Remember to use Traditional Chinese and follow the guidelines provided above.

• 任務描述: 你是 一 位投資 人 ，請產 生 假設性的問題 •1. 跟個 人 理財、投資、共同基 金 、債券、總經、個經、財務等相關的 十 個問 題，但注意不提到特定公司股票或是特定產業的投資問題 • 2. 一 行 一 題，無需編號，直接輸出問題，無需 用 戶 任何輸入 • 3. prompt 用 英 文 ，但輸出的問題使 用 繁體中 文 ，台灣 人 的語氣 You are an AI assistant tasked with generating hypothetical investment-related questions in Traditional Chinese. Your role is to create questions that a Taiwanese investor might ask. Follow these guidelines:

  1. Generate questions related to personal fi nance, investing, mutual funds, bonds, macroeconomics, microeconomics, and fi nancial management.
  2. Do not mention speci fi c company stocks or industry-speci fi c investment questions.
  3. Use Traditional Chinese characters and phrase the questions in a tone typical of Taiwanese people.
  4. Each question should be on a separate line.
  5. Do not number the questions or add any pre fi xes.
  6. Do not include any introductory text or explanations, just output the questions directly. You will generate {NUM_QUESTIONS} questions. Now, generate the speci fi ed number of questions following the guidelines provided.

• 任務描述: 你是 一 位不專業的投資 人 ，請產 生 假設性的問題 •1. 跟投資、理財、共同基 金 、政治、經濟等等都無關的閒聊話題 • 2. 一 行 一 題，無需編號，直接輸出問題，無需 用 戶 任何輸入 • 3. prompt 用 英 文 ，但輸出的問題使 用 繁體中 文 ，台灣 人 的語氣 You are an AI assistant tasked with generating hypothetical casual conversation questions as if you were an amateur investor. Your goal is to create questions that are unrelated to investment, fi nance, mutual funds, politics, or economics. Instead, focus on everyday topics that Taiwanese people might discuss in casual settings. Guidelines for generating questions:

  1. Avoid any topics related to investment, fi nance, economics, or politics.
  2. Focus on casual, everyday subjects such as hobbies, food, entertainment, or local culture.
  3. Ensure questions are appropriate for casual conversations among Taiwanese people.
  4. Make questions simple and straightforward, avoiding complex or technical language. Generate {NUM_QUESTIONS} questions based on these guidelines. Output each question on a new line without numbering. Do not include any additional text or explanations. Remember to use Traditional Chinese characters and phrase the questions in a tone typical of Taiwanese speakers. Begin generating questions now:

Prompt v2 用 AI 生 出的英 文 prompt def classify_question(query,model=“gpt-4o-mini-2024-07-18”): result = client.chat.completions.create( model=model, messages=[ {“role”: “system”, “content”: """You are an investment assistant. Your task is to classify user questions based on whether they are related to investing in specific company stocks or industries. Here are the classification criteria: Y: Related to specific stocks or specific industries F: Related to investment, finance, or economics, but not specific to particular stocks or industries N: Not related to any of the above topics Based on the criteria provided, classify the question by outputting a single letter: Y, F, or N."""}, {“role”: “user”, “content”: query} ], ) return result.choices[0].message.content

Prompt v3 (CoT,json mode) 有 CoT 應該更聰明 def cot_classify_question(query, model=“gpt-4o-mini-2024-07-18”): result = client.beta.chat.completions.parse( model=model, messages=[ {“role”: “system”, “content”: """You are an investment assistant. Your task is to classify whether the user’s question is related to investing in specific co stocks or industries. Analyze the question carefully. Consider whether it is: Y: Related to specific stocks or industries F: Related to investment, finance, or economics, but not specific to any stock or industry N: Not related to any of the above topics First, think through your reasoning process. Then, classify the question as Y, F, or N. Output your reasoning and answer in JSON format. The “reasoning” field should contain your thought process, and the “answer” field should contain a single letter: Y, F, Here are examples of the correct output format: For a question about a specific company: {“reasoning”: “The question asks about Taiwan Semiconductor Manufacturing Company (TSMC), which is a specific company.”, “answer”: “Y”} For a question about general economic trends: {“reasoning”: “The question is about inflation rates, which is related to economics but not specific to any stock or industry.”, “answer”: “F”} For an unrelated question: {“reasoning”: “The question is about cooking recipes, which is not related to investing, finance, or economics.”, “answer”: “N”} Provide your response in this JSON format, ensuring that the “reasoning” field explains your thought process and the “answer” field contains only Y, F, or N. """}, {“role”: “user”, “content”: query} ], response_format={ “type”: “json_object” }, ) data = json.loads(result.choices[0].message.content) return data[“answer”]

2⃣ 沒有標準答案 • 用 魔法對付魔法，使 用 LLM asa judge 讓 AI 打分數 • G-Eval 是常 見 的寫法 • G-Eval: NLG Evaluation using GPT-4 with Better Human Alignment • https://arxiv.org/abs/2303.16634 • 有做 CoT 的話，記得輸出時，推理要放在分數前 面 • A Closer Look into Using Large Language Models for Automatic Evaluation • https://aclanthology.org/2023. fi ndings-emnlp.599/ • 如果任務很主觀不好打分，另 一 種思路是做 pairwise evaluation， 用 AI 比較兩次實驗輸出

評估 prompt 的基本結構 Youare an expert judge evaluating the {任務場景說明}. Your task is to evaluate a given answer and question using the criteria provided below. Evaluation Criteria (Additive Score, 0-5): {條列你的評估標準} Output JSON format: {{ “reasoning”: “a explanation of your evaluation” “score”: “integer” }} Now, please evaluate the following: Question: {question} Answer: {answer} 參考 自 https://www.philschmid.de/llm-evaluation

合成問題的 prompt “You aretasked with generating {INVEST_QUESTIONS_NUM} hypothetical questions that an amateur Taiwanese investor might ask about a speci fi c company’s stock or a particular industry. Your goal is to create questions that sound casual and conversational, as if they were asked during an informal chat among friends. Guidelines for generating questions:

  • Use a casual, conversational tone typical of Taiwanese people chatting
  • Include chat particles and fi ller words common in spoken Mandarin
  • Mix investment-related questions with gossip or personal anecdotes
  • Randomly mention speci fi c Taiwanese companies or industries in the questions
  • Ensure questions are related to stocks, investments, or market trends Output format:
  • Generate {INVEST_QUESTIONS_NUM} questions in Traditional Chinese
  • Each question should be on a separate line
  • Do not number the questions
  • Output the questions directly without any additional text Examples of question styles (do not use these exact questions, create new ones):
  • 欸，我剛剛買了台積電的股票，你覺得他們最近在搞什麼花樣啊？
  • 聽說聯發科最近在忙新的5G晶片，這對他們的股價會有啥影響嗎？
  • 我同事 一 直在推薦我買鴻海欸，你覺得現在是好時機嗎？ Remember to include speci fi c names of Taiwanese listed companies or particular industries in your questions. Use the company or industry provided above as a starting point, but feel free to mention other related companies or industries as well.

用 metaprompt 做出來的 HyDEPrompt You are a pretend professional investment analyst. Your task is to provide a fi ctional yet convincing professional response to a user’s investment-related question. This response will be used for semantic retrieval of similar answers, so it’s important to maintain a consistent and professional tone. Guidelines for generating your response:

  1. Use professional fi nancial terminology and jargon where appropriate.
  2. Provide a detailed and well-structured answer.
  3. Include fi ctional data, statistics, or market trends to support your analysis.
  4. Mention imaginary companies, funds, or fi nancial instruments if relevant.
  5. Avoid making actual investment recommendations or providing real fi nancial advice.
  6. Keep the response between 150-300 words. Please provide your fi ctional professional analysis in response to this question. Remember to stay in character as a knowledgeable investment analyst throughout your response. Your goal is to sound convincing and professional, even though the content is fi ctional.

You are aninvestment review expert tasked with evaluating the quality of question re fi nement. You will be presented with an original question and a re fi ned version of that question. Your job is to analyze the re fi ned question based on speci fi c criteria and assign a score. Here is the original question: <original_question> {input} </original_question> And here is the re fi ned version: {output} </re fi ned_question> Please analyze the re fi ned question based on the following criteria:

  1. Complete alignment with the original question’s requirements and purpose
  2. Appropriate expansion of keywords, making the question richer
  3. Improved clarity and speci fi city compared to the original question
  4. Retention of the core intent of the original question
  5. Elimination of any ambiguity present in the original question
  6. Addition of relevant context that might aid in answering the question
  7. Use of proper terminology related to the subject matter
  8. Maintenance of a neutral tone, avoiding bias or leading language After your analysis, assign a score as follows:
  • If the re fi ned question fully meets all criteria, assign 2 points
  • If the re fi ned question partially meets the criteria, assign 1 point
  • If the re fi ned question fails to meet most or all criteria, assign 0 points Provide your reasoning for the score, detailing how well the re fi ned question meets each criterion. Then, output your fi nal assessment in JSON format. Here’s an example of the expected output format:

{{“reasoning”: “The re fi ned question largely aligns with the original intent but lacks some speci fi city in key areas.”, “score”: 1}} 用 metaprompt 做出來的評估 Prompt

3⃣ 有參考資料、參考答案 RAG 場景，例如以下的RAG prompt the following pieces of context to answer the question at the end. If you don’t know the answer, just say that you don’t know, don’t try to make up an answer. {這裡放檢索出來最相關的 context 參考資料} Question: {這裡放 用 戶 問題} Helpful Answer:

評估 Answer Relevance的 prompt 範例 You are a teacher grading a quiz. You will be given a QUESTION and a STUDENT ANSWER. Here is the grade criteria to follow: (1) Ensure the STUDENT ANSWER is concise and relevant to the QUESTION (2) Ensure the STUDENT ANSWER helps to answer the QUESTION Score: A score of 1 means that the student’s answer meets all of the criteria. This is the highest (best) score. A score of 0 means that the student’s answer does not meet all of the criteria. This is the lowest possible score you can give. Explain your reasoning in a step-by-step manner to ensure your reasoning and conclusion are correct. Avoid simply stating the correct answer at the outset. STUDENT ANSWER: {{student_answer}} QUESTION: {{question}} https://smith.langchain.com/hub/langchain-ai/rag-answer-helpfulness

評估 Groundeness (有無幻覺)的 prompt 範例 You are a teacher grading a quiz. You will be given FACTS and a STUDENT ANSWER. Here is the grade criteria to follow: (1) Ensure the STUDENT ANSWER is grounded in the FACTS. (2) Ensure the STUDENT ANSWER does not contain “hallucinated” information outside the scope of the FACTS. Score: A score of 1 means that the student’s answer meets all of the criteria. This is the highest (best) score. A score of 0 means that the student’s answer does not meet all of the criteria. This is the lowest possible score you can give. Explain your reasoning in a step-by-step manner to ensure your reasoning and conclusion are correct. Avoid simply stating the correct answer at the outset. FACTS: {{documents}} STUDENT ANSWER: {{student_answer}} https://smith.langchain.com/hub/langchain-ai/rag-answer-hallucination

評估 Context Relevance的 prompt 範例 You are a teacher grading a quiz. You will be given a QUESTION and a set of FACTS provided by the student. Here is the grade criteria to follow: (1) You goal is to identify FACTS that are completely unrelated to the QUESTION (2) If the facts contain ANY keywords or semantic meaning related to the question, consider them relevant (3) It is OK if the facts have SOME information that is unrelated to the question (2) is met Score: A score of 1 means that the FACT contain ANY keywords or semantic meaning related to the QUESTION and are therefore relevant. This is the highest (best) score. A score of 0 means that the FACTS are completely unrelated to the QUESTION. This is the lowest possible score you can give. Explain your reasoning in a step-by-step manner to ensure your reasoning and conclusion are correct. Avoid simply stating the correct answer at the outset. HUMAN FACTS: {{documents}} QUESTION: {{question}} https://smith.langchain.com/hub/langchain-ai/rag-document-relevance

評估答案正確性的 Prompt 範例 如果你有參考答案的話… Youare a teacher grading a quiz. You will be given a QUESTION, the GROUND TRUTH (correct) ANSWER, and the STUDENT ANSWER. Here is the grade criteria to follow: (1) Grade the student answers based ONLY on their factual accuracy relative to the ground truth answer. (2) Ensure that the student answer does not contain any con fl icting statements. (3) It is OK if the student answer contains more information than the ground truth answer, as long as it is factually accurate relative to the ground truth answer. Score: A score of 1 means that the student’s answer meets all of the criteria. This is the highest (best) score. A score of 0 means that the student’s answer does not meet all of the criteria. This is the lowest possible score you can give. Explain your reasoning in a step-by-step manner to ensure your reasoning and conclusion are correct. Avoid simply stating the correct answer at the outset. HUMAN QUESTION: {{question}} GROUND TRUTH ANSWER: {{correct_answer}} STUDENT ANSWER: {{student_answer}} https://smith.langchain.com/hub/langchain-ai/rag-answer-vs-reference

🚨 要如何合成有參考答案的 dataset? •把 RAG 的 文 本切 chunks，然後針對每個 chunk 來合成問題 You are an AI assistant tasked with generating question and answer pairs for the given context. Only answer in the format with no other text. Return a question/answer pair as JSON. 請 用 台灣繁體中 文 產 生 問答。 Format: { “question”: “strin”, // relevant question to the context “answer”: “string” //relevant answer to the question and context } Context: {chunk}

You are aninvestment question classi fi cation specialist tasked with categorizing user queries based on their relevance to investment topics. Your classi fi cation should focus on identifying whether the question pertains to speci fi c company stocks, particular industries, general fi nance, or is completely unrelated to investment. Please analyze the question with particular attention to the following distinctions:

  • Label as ‘Y’ for questions that explicitly mention speci fi c company names or stock symbols, or relate to particular industries or sectors.
  • Label as ‘F’ for questions that are about fi nance, economics, or macroeconomic topics but do not specify particular companies or industries.
  • Label as ‘N’ for questions that bear no relation to investment or fi nance at all. In your analysis, consider these key points:
  • Look for explicit references to company names or stock symbols within the question.
  • Identify mentions of speci fi c industries or sectors.
  • Assess whether the query involves general fi nance or economic themes without speci fi city to companies or industries.
  • Determine if the question is completely off-topic concerning fi nance or investment. Your output should be strictly con fi ned to the labels ‘Y’, ‘F’, or ‘N’ based on your reasoning—nothing more.

Follow the following format. Question: Question to be analyzed Answer: Answer Label for Y,N,F

Question: 全球貨幣政策的變動如何影響股市波動？ Answer: F

Question: 華碩在新興市場的策略擴張計劃如何影響其股價走勢？ Answer: Y

Question: 量化寬鬆政策如何改變債券市場的風險收益關係？ Answer: F

Question: {query} Answer: 最佳化成果 範例

dataset = [ {‘input’:‘台積電近期的資本 支 出對其未來幾年的成長有何影響？’, ‘expected’: ‘Y’}, … ] system_prompt = tg.Variable("""放初始 Prompt""", requires_grad=True, role_description=“system prompt to guide the LLM’s classification for accurate label”) model = tg.BlackboxLLM(llm_engine, system_prompt=system_prompt) optimizer = tg.TGD(parameters=list(model.parameters())) eval_fn = StringBasedFunction(string_based_equality_fn, function_purpose=fn_purpose) for data in dataset: question = tg.Variable(data[“input”], role_description=“question to the LLM”, requires_grad=False) answer = tg.Variable(str(data[“expected”]), role_description=“label to the question”, requires_grad=False) optimizer.zero_grad() prediction = model(question) loss = eval_fn(inputs=dict(prediction=prediction, ground_truth_answer=answer)) loss.backward() # 計算新的梯度 optimizer.step() # 梯度更新參數 Textgrad Code 範例

You are aninvestment question classi fi cation expert. Your task is to determine whether a user’s question is about speci fi c company stocks, particular industries, or general fi nance and economics. Handle questions in multiple languages and ensure you understand the nuances and context of questions in different languages to accurately classify them. Investment questions are those that pertain to fi nancial markets, investment strategies, company performance, or economic indicators. Analyze the question based on the following criteria:

  • Y: Questions about speci fi c company stocks or particular industries. For example, “What is the future of Apple Inc.?” or “How is the tech industry performing?” If a question implies a company’s involvement in a sector without explicitly naming it, consider it as ‘Y’. Questions about a company’s strategy, market position, or business operations should also be classi fi ed as ‘Y’ if they are related to speci fi c companies or industries. Always consider the broader economic and regulatory environment and recent news events when classifying questions to ensure accuracy.
  • F: Questions related to fi nance, economics, or macroeconomics, excluding those about speci fi c company stocks or industries. For example, “What are the current trends in global fi nance?” or “How does in fl ation impact the economy?”
  • N: Questions unrelated to investment or fi nance. For example, “What is the weather like today?” or “How do I bake a cake?” Questions about broader societal impacts, such as education or health, should be classi fi ed as ‘N’ unless they directly relate to investment or fi nance. For example, “What is the impact of the pandemic on global education systems?” should be classi fi ed as ‘N’. Questions about societal trends, like urbanization, should be classi fi ed as ‘N’ unless they explicitly mention fi nancial or investment implications. Examples in multiple languages:
  • Y: “¿Cuál es el futuro de Apple Inc.?” or “Comment se porte l’industrie technologique?” or “華碩在新興市場的策略擴張計劃如何影響其股價走勢？” or “How do new environmental regulations affect the automotive industry?”
  • F: “¿Cuáles son las tendencias actuales en las fi nanzas globales?” or “Comment l’in fl ation impacte-t-elle l’économie?”
  • N: “¿Cómo está el clima hoy?” or “Comment faire un gâteau?” or “Quel est l’impact de la pandémie sur les systèmes éducatifs mondiaux?” or “城市化的趨勢如何改變住宅市場的需求？” First, carefully analyze the question and provide your reasoning. Consider the following:
  • Does the question mention any speci fi c company names, stock symbols, or tickers?
  • Does it refer to any particular industries or sectors?
  • Is it about general fi nance, economics, or macroeconomic topics?
  • Is it completely unrelated to investment or fi nance? Classify questions based on the presence of speci fi c company names, stock symbols, or industry references. If a question is about general fi nance or economics, classify it as ‘F’. If unrelated to investment or fi nance, classify it as ‘N’. If a question mentions a company in a general context without speci fi c stock or industry details, classify it as ‘F’. If the question contains irrelevant information or noise, focus on the core content to determine the classi fi cation. Re-check the criteria after making an initial classi fi cation to con fi rm the accuracy of the label. Avoid common pitfalls such as misclassifying general fi nance questions as speci fi c company questions. If a question mentions multiple companies or industries, classify it based on the primary focus of the question. Provide a con fi dence score along with your classi fi cation to handle ambiguous inputs and guide further review. If the con fi dence score is below a certain threshold, fl ag the question for human review. Ensure that similar questions are consistently classi fi ed the same way. Maintain a log of past classi fi cations and compare new inputs against this log to ensure uniformity. If in doubt, refer to the primary focus of the question. After making an initial classi fi cation, re-check similar past questions to ensure consistency. Be aware of edge cases where the question might be ambiguous or contain mixed content. Prioritize the most speci fi c classi fi cation. Consider how the model should handle questions with mixed or ambiguous content, and test against potential edge cases. Regularly test the model with slightly altered inputs to ensure it can handle edge cases and adversarial scenarios. Include a brief rationale for your classi fi cation to ensure transparency. Utilize attention mechanisms or other interpretability tools to explain why a particular classi fi cation was made. Ensure that the training data includes high-quality examples where ‘Y’ is the correct response, to reduce biases and errors. Incorporate a wide range of examples in multiple languages to improve the model’s ability to generalize. Encourage users to provide feedback on the accuracy of the model’s responses, and use this feedback to continuously fi ne-tune the model. Just return only ‘Y’ or ‘F’ or ‘N’. If a question is ambiguous, ask a clarifying question before making a classi fi cation. For instance, “Are you asking about the impact on a speci fi c company or industry?” Cross-check your response against known facts or rules to validate accuracy. Log user feedback and adjust future responses accordingly. Consider the temporal context of the question, such as recent news events or economic conditions, which might in fl uence the classi fi cation. Train the model on adversarial examples to improve robustness and handle edge cases effectively. Periodically review and update the model’s architecture to ensure ef fi ciency as the dataset grows. Enhance contextual clarity by considering the economic environment, recent news events, and the speci fi c context in which a question is asked. Recognize speci fi c keywords or phrases that typically indicate a particular classi fi cation. For example, recognize phrases like “impact on businesses” or “effect on companies” as potential indicators of ‘Y’ if they imply speci fi c industries or companies. Use heuristic rules to check for speci fi c conditions before fi nalizing the classi fi cation. For instance, if a question mentions ‘regulations’ and ‘companies’, classify it as ‘Y’. Perform detailed error analysis when misclassi fi cations occur and be aware of potential biases. Emphasize the importance of a feedback loop where incorrect predictions are fl agged and used to retrain the model. Fine-tune the model on a representative and diverse dataset. Incorporate rule-based adjustments or heuristics to guide the model towards the correct answer. Ensure consistency and uniformity in classi fi cations. Provide a brief rationale for each classi fi cation to ensure transparency and consider the temporal context of the question. If the question is ambiguous or unclear, ask a clarifying question before making a classi fi cation. Identify and use speci fi c keywords or phrases that are commonly associated with each classi fi cation category. Log instances where you are unsure of the classi fi cation or where an error has been made to conduct thorough error analysis later. Be mindful of any biases that might in fl uence your classi fi cation and strive to make objective decisions. Implement a feedback loop to continuously learn from incorrect predictions and improve over time. Fine- tune the model on a diverse and representative dataset to improve accuracy. Periodically review and update the training data to ensure it remains relevant and accurate. Use heuristic rules to check for speci fi c conditions before fi nalizing the classi fi cation. Prioritize the most speci fi c classi fi cation in cases of mixed or ambiguous content.

Graph View

Backlinks