Cortex

Appearance

Cortex Project Testing ​

This documentation provides in-depth information about the testing framework in Cortex projects.

Overview ​

Cortex projects include a powerful testing framework that allows you to validate your templates against various requirements. Tests are defined in YAML files with the suffix _test.yml and are associated with specific template files.

Test File Structure ​

A typical test file has the following structure:

yaml
name: [Test Suite Name]
model: [Default AI Model]
document: [Target Document]

tests:
  - name: [Test Name]
    assert: [Assertion Type]
    [Assertion-specific parameters]
    
  - name: [Another Test Name]
    assert: [Assertion Type]
    [Assertion-specific parameters]

Example from the NDA project:

yaml
name: NDA Compliance Test
model: cortex.law/claude-3-7
document: nda.md

tests:
  - name: Contains Confidentiality Definition
    assert: contains
    text: "Confidential Information"

Assertion Types ​

Text-Based Assertions ​

contains ​

Verifies that the document contains specific text.

yaml
- name: Has Survival Clause
  assert: contains
  text: "obligations of confidentiality and non-use set forth in this Agreement shall survive"

regex ​

Uses regular expressions to match patterns in the document.

yaml
- name: Has Proper Term Length
  assert: regex
  pattern: "period of \\{\\{term_years\\}\\} years"
  validate: "$term_years <= 5"

The validate parameter allows for additional validation of captured variables.

AI-Powered Assertions ​

ai ​

Leverages AI models to evaluate document content based on specific criteria.

yaml
- name: Evaluates Scope of Confidentiality
  assert: ai
  prompt: "Evaluate if the confidentiality clause is sufficiently comprehensive"
  criteria: "Must cover both disclosed information and derived information, and include both written and oral disclosures"

Parameters:

  • prompt: The instruction given to the AI
  • criteria: The evaluation criteria for the AI to apply
  • model: (Optional) Specifies a different AI model than the default

Example with custom model:

yaml
- name: Evaluates International Compliance
  assert: ai
  model: cortex.law/claude-3-5
  prompt: "Evaluate if this NDA complies with international data protection standards"
  criteria: "The agreement should address cross-border data transfers, comply with GDPR if applicable, and include provisions for data security measures"

Variable Validation ​

Tests can validate template variables to ensure they meet specific requirements:

yaml
- name: Has Proper Term Length
  assert: regex
  pattern: "period of \\{\\{term_years\\}\\} years"
  validate: "$term_years <= 5"

This example ensures that the term_years variable doesn't exceed 5 years, which might be important for compliance reasons.

Test Execution ​

Tests are typically executed using the Cortex CLI. Although the exact commands aren't shown in our examples, they would likely follow a format similar to:

cortex test path/to/project

Tests can be run individually or as part of a test suite, and results would indicate which assertions passed or failed.

Best Practices ​

  1. Comprehensive Testing: Create tests for all critical aspects of your templates
  2. Descriptive Names: Use clear, descriptive test names that explain what's being tested
  3. Appropriate Assertions: Choose the right assertion type for each requirement
  4. AI for Complex Validation: Use AI assertions for subjective or complex evaluations that can't be easily captured with simple patterns
  5. Variable Validation: Validate template variables to ensure they meet specific requirements
  6. Test Organization: Organize tests logically by topic or document section

Example Use Cases ​

  • Legal Compliance: Ensure templates comply with legal requirements
  • Style Consistency: Verify consistent formatting and terminology
  • Content Requirements: Check for mandatory clauses or sections
  • Variable Constraints: Validate that variables meet business rules or legal constraints
  • Comprehensive Coverage: Ensure all necessary topics are addressed in a document

Using Multiple Models ​

The testing framework allows using different AI models for different assertions, enabling you to:

  1. Use specialized models for specific domains
  2. Balance cost and performance for different types of evaluations
  3. Compare results across different models

For example:

yaml
- name: Basic Evaluation
  assert: ai
  model: cortex.law/claude-3-sonnet
  prompt: "Check basic legal requirements"
  
- name: Detailed Analysis
  assert: ai
  model: cortex.law/claude-3-7
  prompt: "Perform detailed legal analysis"

This flexibility allows for targeted, cost-effective testing strategies tailored to your project's needs.

© 2025 Cortex AB