Aristo 루브릭
버전 rubric-2.0
Aristo가 매기는 모든 점수는 이 루브릭으로 계산됩니다. 여러분이 이견을 제시할 수 있도록 공개하고, 기록을 비교할 수 있도록 버전을 관리합니다.
프롬프트 채점 방식
각 기준은 아래 앵커에 따라 0~4점으로 채점되며, 작업과 무관한 경우에는 해당 없음으로 표시됩니다. 종합 점수는 가중 평균이며 0~100점으로 표시됩니다. 독립적인 평가 샘플 세 개를 받아 중앙값을 채택하고, 샘플 간 판단이 엇갈리면 더 강력한 모델의 샘플 두 개를 추가하고 신뢰도 안내를 표시합니다.
목표와 성공 기준
원하는 결과가 명시되어 있고, 그 결과가 좋은지 어떻게 판단할 수 있습니까?
앵커
- 0
No discernible outcome.
예시: Tell me about marketing.
- 1
Topic or activity stated, outcome implicit.
예시: Write about our new product.
- 2
Outcome stated, no success criteria.
예시: Write a launch email for our product.
- 3
Outcome plus at least one testable criterion.
예시: …that gets existing customers to book a demo; under 150 words.
- 4
Outcome, testable criteria, and priority among them when they conflict.
예시: …Success = clear single call to action and a demo link above the fold; brevity matters more than covering every feature.
맥락과 입력
모델에 필요한 사실, 자료, 데이터, 상황이 명확히 구분되어 제공됩니까?
앵커
- 0
No context at all; the model must guess the situation, the subject and the material.
예시: Summarise this.
- 1
Situation named in passing; the material or facts the task needs are missing or only alluded to.
예시: Summarise the meeting we had about the budget.
- 2
The needed material is present but undelimited or incomplete, so the model cannot tell input from instruction.
예시: Summarise the meeting notes below for the finance team. Budget review 14 May: Ana raised…
- 3
Material is clearly delimited (tags, fences or headings) and the situation is explained; small gaps remain.
예시: <notes>…</notes> These are notes from the 14 May budget review; summarise them for the finance team.
- 4
Complete and delimited material, background the model could not infer, and a statement of what it does not have.
예시: <notes>…</notes> Background: this is the third review this quarter; last time we overspent on travel. Figures for June are not yet available, so do not estimate them.
대상과 활용 목적
결과물은 누구를 위한 것이며, 그들은 그것으로 무엇을 합니까?
앵커
- 0
No reader or purpose can be inferred.
예시: Explain photosynthesis.
- 1
Reader implied by the topic or the register, but never stated.
예시: Explain photosynthesis simply.
- 2
Reader named; what they will do with the output is not.
예시: Explain photosynthesis to a 12-year-old.
- 3
Reader and purpose of use both stated.
예시: Explain photosynthesis to a 12-year-old preparing for a science quiz next week.
- 4
Reader, purpose and what they already know or need, so the level, emphasis and register follow from it.
예시: …for a 12-year-old who has covered cells but not chemistry; she needs to answer short written questions, so favour precise terms over analogies.
제약과 범위
범위, 필수 사항과 금지 사항, 분량, 출처, 안전, 불확실할 때의 대처 방법.
앵커
- 0
No boundaries of any kind.
예시: Write a blog post about remote work.
- 1
One vague boundary, usually length or tone, nothing on scope or sources.
예시: Write a short blog post about remote work.
- 2
Several concrete boundaries (length, scope, sources) but silent on what to do when uncertain.
예시: …about 600 words, for managers, no product recommendations, cite at least two studies.
- 3
Concrete boundaries plus explicit must-nots and an instruction for the uncertain case.
예시: …if you cannot find a study published after 2023, say so rather than inventing one.
- 4
Boundaries are complete, consistent and prioritised, stated as what to do rather than only what to avoid.
예시: …Word count may stretch to 800 if needed to keep both studies; accuracy outranks length; mark any claim you are unsure of with “(verify)”.
작업 지시
명확하고 순서가 있으며 모호하지 않은 단계. 하나의 작업이거나 순서가 명시된 여러 작업.
앵커
- 0
No actionable instruction; a topic or a fragment.
예시: Customer churn Q3.
- 1
One ambiguous instruction, or several tasks tangled together with no order.
예시: Look at the churn data and also fix the dashboard and tell me what to do.
- 2
A clear single task, or multiple tasks listed but their order or dependencies unstated.
예시: Analyse the Q3 churn data and recommend three actions.
- 3
Clear, ordered steps or an explicit decomposition; each step has one verb and one object.
예시: 1. Compute churn by plan. 2. Identify the plan with the biggest rise. 3. Propose three actions for that plan.
- 4
Ordered steps with the reasoning behind them, what to do if a step cannot be completed, and where to stop.
예시: …Step 2 matters because we can only fund one intervention. If the data for a plan is missing, skip it and say so. Stop after the three actions; do not draft the rollout plan.
출력 명세
결과물의 형식, 구조, 스키마, 분량, 언어, 어조.
앵커
- 0
Nothing said about the form of the result.
예시: Compare the two proposals.
- 1
A single loose hint such as “briefly” or “in a table”.
예시: Compare the two proposals in a table.
- 2
Format and rough length stated; structure, language or tone left open.
예시: Compare the two proposals in a table of at most ten rows, then a one-paragraph recommendation.
- 3
Format, structure, length and language or tone all stated and consistent with the goal.
예시: …columns: criterion, proposal A, proposal B, winner. Write in Portuguese, neutral tone, no preamble.
- 4
A complete contract: schema or template, length, language, tone, and how edge cases appear in the output.
예시: …Return JSON {"rows":[{"criterion","a","b","winner"}],"recommendation"}; if a criterion cannot be judged, set winner to "tie" and explain in recommendation.
예시와 참고 자료
실제로 도움이 되는 곳의 예시나 참고 자료. 다양하게, 특정 예시에 과도하게 맞추지 않음.
앵커
- 0
Examples would clearly help (a specific format, style or classification) but none is given.
예시: Classify each support ticket by urgency.
- 1
A reference is mentioned but not provided, or one example that is incomplete.
예시: Classify each ticket by urgency, like we did last month.
- 2
One complete example, or several that all show the same case, so the model may over-fit it.
예시: <example>Ticket: “Site down for all users” → urgent</example>
- 3
Two to five varied examples in a consistent, delimited format.
예시: <examples><example>…→ urgent</example><example>…→ normal</example><example>…→ low</example></examples>
- 4
Varied examples that include an edge case or a counter-example, each with a short note on why.
예시: …<example>“Invoice typo” → low, because nothing is blocked</example><example>“Cannot log in (one user)” → normal, not urgent: single user</example>
역할과 어조
결과물을 바꾸는 관점이나 전문성. 어조에 대한 안내.
앵커
- 0
A perspective would change the output and none is given, or the role given contradicts the task.
예시: Review this contract clause.
- 1
A generic label with no bearing on the output (“you are a helpful assistant”).
예시: You are a helpful assistant. Review this contract clause.
- 2
A relevant role named, but its perspective is not used in the instructions.
예시: You are a commercial lawyer. Review this contract clause.
- 3
A role whose expertise shapes the task, plus guidance on voice or stance.
예시: As a commercial lawyer acting for the buyer, review this clause and flag risks to the buyer; be direct, no hedging.
- 4
Role, stance, voice and the standard to judge by, applied only where it changes the output.
예시: …Judge against standard Canadian commercial practice; where the clause is market-standard say so in one line and move on.
모델 적합성과 정돈
대상 모델 계열에 맞음. 모순이나 과도한 강조가 없음. 긴 자료는 질문 앞에 둠. 낡은 요령을 쓰지 않음.
앵커
- 0
Contradictory instructions, or shouting in capitals (“CRITICAL”, “MUST”), or a trick that harms current models (forced step-by-step, prefill-style openings).
예시: CRITICAL: you MUST think step by step and NEVER make mistakes. Also be brief but cover everything.
- 1
No contradictions, but obsolete tricks or heavy emphasis remain, or long material comes after the question.
예시: Think step by step. What are the risks? [20 pages of contract follow]
- 2
Plain, calm instructions; structure is weak (no delimiters) or the request ignores the target family’s conventions.
예시: Here is the contract and here are my questions, mixed together in one paragraph.
- 3
Calm, consistent instructions; long material first and the question last; delimiters or tags separate parts; explains the why where it matters.
예시: <contract>…</contract> Review the clause above. Keep the answer short because it goes into a client email.
- 4
Everything at 3, written for the chosen model family: structure and emphasis in the vendor’s recommended form, effort left to the model, nothing redundant.
예시: Same prompt with <contract> and <questions> tags, a one-line role, no “think carefully” padding, and success criteria the model can check itself.
작업 유형
가중치는 작업 유형에 따라 달라집니다. 예를 들어 코드 변환에는 대상이 필요 없고, 이미지 프롬프트에서는 출력 명세에 가장 큰 가중치가 주어집니다.
| 작업 유형 | 목표와 성공 기준 | 맥락과 입력 | 대상과 활용 목적 | 제약과 범위 | 작업 지시 | 출력 명세 | 예시와 참고 자료 | 역할과 어조 | 모델 적합성과 정돈 |
|---|---|---|---|---|---|---|---|---|---|
| 글쓰기 | 3 | 3 | 3 | 2 | 2 | 2 | 1 | 1 | 1 |
| 분석 | 3 | 3 | 2 | 2 | 2 | 2 | 해당 없음 | 1 | 1 |
| 코딩 | 3 | 3 | 해당 없음 | 3 | 2 | 2 | 1 | 해당 없음 | 1 |
| 리서치 | 3 | 3 | 2 | 3 | 2 | 2 | 해당 없음 | 1 | 1 |
| 데이터 추출 | 3 | 3 | 해당 없음 | 2 | 2 | 3 | 1 | 해당 없음 | 1 |
| 창작 | 2 | 3 | 2 | 2 | 2 | 2 | 1 | 2 | 1 |
| 튜터링 | 3 | 3 | 3 | 2 | 2 | 2 | 1 | 1 | 1 |
| 에이전트 시스템 프롬프트 | 3 | 3 | 2 | 3 | 3 | 2 | 1 | 2 | 2 |
| 이미지 생성 | 3 | 2 | 해당 없음 | 2 | 1 | 3 | 해당 없음 | 해당 없음 | 1 |
| 기타 | 3 | 3 | 2 | 2 | 2 | 2 | 1 | 1 | 1 |
이 루브릭은 프롬프트 교육에 관한 연구 근거(표현보다 요구 사항이 먼저)와 추론 모델에 대한 각 제공업체의 최신 지침을 따릅니다. 출처는 프로젝트 문서에 나와 있습니다.