ai_essay_evaluator.trainer package

ai_essay_evaluator.trainer.create_fine_tuning_job(file_id: str, api_key: str | None = None, model: str = 'gpt-4o-mini-2024-07-18') → str[source]

Creates a fine-tuning job with OpenAI using the uploaded JSONL file.

ai_essay_evaluator.trainer.generate_jsonl(story_folder: str | Path, question_path: str | Path, rubric_folder: str | Path, csv_path: str | Path, output_path: str | Path, output_format: str) → str | Path[source]

Generates a JSONL file for fine-tuning based on multiple stories, a question file, and a rubric.

  • Supports multiple stories by dynamically reading all .txt files from story_folder.

  • Includes detailed scoring feedback or simplified feedback based on output_format.

  • Considers grade level to ensure appropriate expectations for student responses.

  • Adapts feedback to the tested language (English or Spanish).

Returns:

str: Path to the generated JSONL file.

ai_essay_evaluator.trainer.merge_jsonl_files(input_folder: str | Path, output_file: str | Path) → str[source]

Merges all JSONL files in a folder into one JSONL file.

Args:

input_folder: Directory containing JSONL files to merge output_file: Path where the merged JSONL file will be saved

Returns:

str: Path to the merged output file

ai_essay_evaluator.trainer.upload_jsonl(jsonl_path: str, api_key: str | None = None) → str[source]

Upload JSONL file to OpenAI for fine-tuning.

Args:

jsonl_path: Path to the JSONL file to upload api_key: OpenAI API key

Returns:

str: The file ID of the uploaded file

ai_essay_evaluator.trainer.validate_jsonl(jsonl_path: str, scoring_format: str) → bool[source]

Validate JSONL file format for OpenAI fine-tuning.

Args:

jsonl_path: Path to the JSONL file to validate scoring_format: Scoring format for the JSONL file

Returns:

bool: True if file is valid, exits with code 1 otherwise

Submodules

ai_essay_evaluator.trainer.cli module

ai_essay_evaluator.trainer.cli.fine_tune(file: str | None = <typer.models.OptionInfo object>, file_id: str | None = <typer.models.OptionInfo object>, api_key: str | None = <typer.models.OptionInfo object>, scoring_format: str = <typer.models.OptionInfo object>) → None[source]

Start a fine-tuning job using OpenAI.

ai_essay_evaluator.trainer.cli.generate(story_folder: str = <typer.models.OptionInfo object>, question: str = <typer.models.OptionInfo object>, rubric: str = <typer.models.OptionInfo object>, csv: str = <typer.models.OptionInfo object>, output: str = <typer.models.OptionInfo object>, scoring_format: str = <typer.models.OptionInfo object>) → None[source]

Generate JSONL file from input files.

ai_essay_evaluator.trainer.cli.merge(folder: str = <typer.models.OptionInfo object>, output: str = <typer.models.OptionInfo object>) → None[source]

Merge all JSONL files in a folder into one.

ai_essay_evaluator.trainer.cli.upload(file: str = <typer.models.OptionInfo object>, api_key: str | None = <typer.models.OptionInfo object>) → None[source]

Upload a validated JSONL file to OpenAI.

ai_essay_evaluator.trainer.cli.validate(file: str = <typer.models.OptionInfo object>, scoring_format: str = <typer.models.OptionInfo object>) → None[source]

Validate a JSONL file.

ai_essay_evaluator.trainer.finetuner module

ai_essay_evaluator.trainer.finetuner.create_fine_tuning_job(file_id: str, api_key: str | None = None, model: str = 'gpt-4o-mini-2024-07-18') → str[source]

Creates a fine-tuning job with OpenAI using the uploaded JSONL file.

ai_essay_evaluator.trainer.generator module

class ai_essay_evaluator.trainer.generator.BaseGradingResponse[source]

Bases: BaseModel

Base class for grading responses to enable dynamic response models.

classmethod create_model(output_format: str) → type[source]

Returns the appropriate Pydantic model based on the output format.

Args:

output_format (str): The format type (e.g., “item-specific”, “short”, “extended”).

Returns:

Type[BaseModel]: A dynamically selected Pydantic model.

model_config = {}

Configuration for the model, should be a dictionary conforming to [ConfigDict][pydantic.config.ConfigDict].

ai_essay_evaluator.trainer.generator.generate_jsonl(story_folder: str | Path, question_path: str | Path, rubric_folder: str | Path, csv_path: str | Path, output_path: str | Path, output_format: str) → str | Path[source]

Generates a JSONL file for fine-tuning based on multiple stories, a question file, and a rubric.

  • Supports multiple stories by dynamically reading all .txt files from story_folder.

  • Includes detailed scoring feedback or simplified feedback based on output_format.

  • Considers grade level to ensure appropriate expectations for student responses.

  • Adapts feedback to the tested language (English or Spanish).

Returns:

str: Path to the generated JSONL file.

ai_essay_evaluator.trainer.generator.load_rubric_files(rubric_folder: str | Path, output_format: str) → dict[str, Any][source]

Loads multiple rubric files from a folder and organizes them into a dictionary.

  • If output_format == “extended”, the rubric is structured by categories.

  • Otherwise, the rubric is flattened to only contain score_3, score_2, etc.

Returns:

dict: A structured or flattened rubric dictionary.

ai_essay_evaluator.trainer.generator.load_story_files(story_folder: str | Path) → dict[str, str][source]

Loads multiple story files from a folder and organizes them into a dictionary.

Returns:

dict: A dictionary mapping “Story 1”, “Story 2”, etc., to story content.

ai_essay_evaluator.trainer.generator.load_text_file(file_path: str | Path) → str[source]

Load and normalize text file contents.

Args:

file_path: Path to the text file to load, as string or Path object

Returns:

str: File contents with normalized spaces

Raises:

FileNotFoundError: If the specified file does not exist

ai_essay_evaluator.trainer.generator.validate_response(response_content: str, output_format: str) → BaseModel | None[source]

Validates and parses a JSON response string into the correct grading model.

Args:

response_content (str): JSON string containing grading response. output_format (str): Determines which grading model to use.

Returns:

BaseModel | None: Validated response model if successful, None if validation fails.

ai_essay_evaluator.trainer.merge module

ai_essay_evaluator.trainer.merge.merge_jsonl_files(input_folder: str | Path, output_file: str | Path) → str[source]

Merges all JSONL files in a folder into one JSONL file.

Args:

input_folder: Directory containing JSONL files to merge output_file: Path where the merged JSONL file will be saved

Returns:

str: Path to the merged output file

ai_essay_evaluator.trainer.uploader module

ai_essay_evaluator.trainer.uploader.upload_jsonl(jsonl_path: str, api_key: str | None = None) → str[source]

Upload JSONL file to OpenAI for fine-tuning.

Args:

jsonl_path: Path to the JSONL file to upload api_key: OpenAI API key

Returns:

str: The file ID of the uploaded file

ai_essay_evaluator.trainer.validator module

ai_essay_evaluator.trainer.validator.validate_jsonl(jsonl_path: str, scoring_format: str) → bool[source]

Validate JSONL file format for OpenAI fine-tuning.

Args:

jsonl_path: Path to the JSONL file to validate scoring_format: Scoring format for the JSONL file

Returns:

bool: True if file is valid, exits with code 1 otherwise