# Evaluation A type that defines an evaluation. ``` protocol Evaluation : Sendable ``` ## Overview Implement this protocol to create custom evaluations. The evaluation runs your system under test against a dataset and applies evaluators to measure performance. ``` struct MyEvaluation: Evaluation { let metric = Metric("Match") let dataset = ArrayLoader(samples: [ ModelSample(prompt: "One plus one is...", expected: "Two.") ]) func subject(from sample: ModelSample) async throws -> ModelSubject { ModelSubject(value: "Two.") } var evaluators: Evaluators { Evaluator { sample, subject in let metric = Metric("Match") guard let expected = sample.expected else { return metric.ignore() } return subject.value == expected ? metric.passing() : metric.failing() } } func aggregateMetrics(using aggregator: inout MetricsAggregator) { aggregator.computeMean(of: metric) } } ``` ## Topics ### Providing data [`Sample`](/documentation/Evaluations/Evaluation/Sample) The type of input samples in the evaluation dataset. [`SampleLoader`](/documentation/Evaluations/Evaluation/SampleLoader) The type of the sample loader used to provide the evaluation dataset. [`dataset`](/documentation/Evaluations/Evaluation/dataset) The evaluation dataset. ### Testing an intelligent feature [`Subject`](/documentation/Evaluations/Evaluation/Subject) The type of subject the system under test produces. [`subject(from:)`](/documentation/Evaluations/Evaluation/subject(from:)) Produces the subject of evaluation from a given sample. [`EvaluationSubject`](/documentation/Evaluations/EvaluationSubject) A type that represents the output the system under test produces. [`ModelSubject`](/documentation/Evaluations/ModelSubject) The subject type for language model evaluations. [`name`](/documentation/Evaluations/Evaluation/name) The default name, taken from the type name. ### Scoring results [`evaluators`](/documentation/Evaluations/Evaluation/evaluators-swift.property) The evaluators to apply to each sample and its corresponding subject. [`Evaluators`](/documentation/Evaluations/Evaluation/Evaluators-swift.typealias) The evaluator array type for this conformance. [`EvaluatorProtocol`](/documentation/Evaluations/EvaluatorProtocol) A type that evaluates subjects and produces metrics. [`EvaluatorsBuilder`](/documentation/Evaluations/EvaluatorsBuilder) A result builder that enables declarative evaluator lists. [`aggregateMetrics(using:)`](/documentation/Evaluations/Evaluation/aggregateMetrics(using:)) Aggregates the collected metric results. ### Running an evaluation [`EvaluationTrait`](/documentation/Evaluations/EvaluationTrait) A test trait that runs an evaluation and records the result as attachments. [`EvaluationContext`](/documentation/Evaluations/EvaluationContext) A context that provides the evaluation result within a test scope. [`EvaluationResult`](/documentation/Evaluations/EvaluationResult) The results of running a model evaluation. [`run(info:)`](/documentation/Evaluations/Evaluation/run(info:)) Runs the evaluation against the dataset and computes metric results. ### Inspecting detailed results [`inputColumn`](/documentation/Evaluations/Evaluation/inputColumn) A typed column descriptor for the input samples in the detailed DataFrame. [`responseColumn`](/documentation/Evaluations/Evaluation/responseColumn) A typed column descriptor for the model responses in the detailed DataFrame. [`expectedColumn`](/documentation/Evaluations/Evaluation/expectedColumn) A typed column descriptor for the expected values in the detailed DataFrame. ### Errors [`EvaluationError`](/documentation/Evaluations/EvaluationError) Errors thrown during an evaluation run. [`EvaluatorError`](/documentation/Evaluations/EvaluatorError) A value that describes why an evaluator failed while scoring a produced subject. [`SubjectInferenceError`](/documentation/Evaluations/SubjectInferenceError) A value that describes a failure to produce a subject for a sample. [`EvaluationResultsError`](/documentation/Evaluations/EvaluationResultsError) Errors the framework throws when parsing evaluation results. ## Relationships ### Inherits From [`SendableMetatype`](/documentation/Swift/SendableMetatype) [`Sendable`](/documentation/Swift/Sendable) --- Copyright © 2026 Apple Inc. All rights reserved. | [Terms of Use](https://www.apple.com/legal/internet-services/terms/site.html) | [Privacy Policy](https://www.apple.com/privacy/privacy-policy)