Integrate suffixed encrypted storage across extraction, review, training, and inference. Add shared-password web authentication with two-hour sessions, private request logging, and an isolated profile-building pipeline.
Fix encrypted dataset discovery at training entrypoints and duplicate CSV imports. Add synthetic storage, authentication, and log checks; update documentation and frontend assets.
- Added default values for batch size, max tokens, and timeout in distill_profile.py.
- Refactored required configuration functions to support section-specific error messages.
- Introduced positive and optional positive integer validation functions for configuration.
- Updated resolve_llm_args to utilize new configuration validation functions.
- Modified organize_memories.py to load agent distill configuration and handle missing model gracefully.
- Improved stage 2 output file management in make_dataset_stage2.py, ensuring proper validation against manifest files.
- Added new web assets including HTML, CSS, and JS files for profile review interface.
- Set up default static directory for serving web assets in profile_review.py.
- Implemented a new authentication system in `auth.py` to manage user sessions and password security.
- Created an `AuthStore` class to handle password hashing, session management, and login/logout functionality.
- Added API endpoints for login, logout, and session validation in the FastAPI application.
- Developed a React context for authentication in `Auth.tsx`, including a login form and logout button.
- Introduced tests for private routes and session management in `test_web_auth.py`.
- Added CSS styles for the authentication page in `index-DOIy3AqG.css`.
- Included a new JavaScript file for the web application in `index-DwlWkAhO.js`.
- Implement comprehensive tests for profile review endpoints, including approval, editing, and exporting facts.
- Validate snapshot import behavior and ensure proper error handling for invalid data.
- Introduce tests for server CLI options and ensure correct application of configurations.
- Test unified routes with lifespan management, authentication, and streaming responses.
- Updated Python version requirement in `pyproject.toml` to 3.12.
- Adjusted dependency specifications for `llamafactory`, `torch`, `torchvision`, and `torchaudio` to include version constraints.
- Updated the `qa_generator.py` to include a new mechanism for managing chat member relationships, allowing the addition of contextual information about the relationship between users in conversations.
- Refactored the CSV loading function to support loading user relationship data from a `users.json` file, improving the context provided during QA generation.
- Added a new configuration option `add_relation` to the dataset settings, enabling users to toggle this feature.
- Updated the `.gitignore` to exclude additional data directories and cache files for better repository hygiene.
- Bumped version to 0.3.03 in `pyproject.toml` to reflect these changes.
Refactors dataset management to consistently append '-vl' for vision-language
datasets and dynamically name cleaned datasets (e.g., 'dataset-cleaned').
Enforces vLLM as a strict dependency for LLM-based data cleaning, exiting if
unavailable. Integrates 'enable_thinking' option for LLM cleaning and enables
cleaning by default in relevant test configurations.
Adds torchvision dependency for vision models and streamlines the cleaning call
in training to centralize decision-making.
Updates the Qwen3 model in test configurations from 4B to the smaller 0.6B version.
This change helps improve test performance and reduce resource requirements.
Ensures the data generation pipeline correctly includes the expected number of <image> tags for multimodal configurations, specifically for Qwen2.5-VL datasets.
Adds a blocked words filter to the dataset creation process.
This prevents undesirable words from appearing in the generated
datasets, enhancing data quality and safety.
A test case is added to verify the filter is working correctly.
Also, skips "text" type in qa_generator to avoid duplicated filter.
Refactors the test environment setup to use pytest fixtures
for session-level setup and teardown. This ensures a clean
and consistent environment before and after each test session.
It also improves the robustness by backing up and restoring
existing 'model_output' and 'dataset/csv' directories.
xming