Integrate suffixed encrypted storage across extraction, review, training, and inference. Add shared-password web authentication with two-hour sessions, private request logging, and an isolated profile-building pipeline.
Fix encrypted dataset discovery at training entrypoints and duplicate CSV imports. Add synthetic storage, authentication, and log checks; update documentation and frontend assets.
- Added default values for batch size, max tokens, and timeout in distill_profile.py.
- Refactored required configuration functions to support section-specific error messages.
- Introduced positive and optional positive integer validation functions for configuration.
- Updated resolve_llm_args to utilize new configuration validation functions.
- Modified organize_memories.py to load agent distill configuration and handle missing model gracefully.
- Improved stage 2 output file management in make_dataset_stage2.py, ensuring proper validation against manifest files.
- Added new web assets including HTML, CSS, and JS files for profile review interface.
- Set up default static directory for serving web assets in profile_review.py.
- Implemented `organize_memories.py` to manage and organize extracted profile/event memories by dimension.
- Introduced new schemas and rules for attribute hierarchy and preference attributes in `memory_organization.py`.
- Created `profile_hierarchy.py` for generating a hierarchical structure of attributes for better browsing.
- Enhanced classification and summarization processes to support new dimensions and attributes.
- Introduced `distill_event.py` for processing chat records and extracting events related to user interactions.
- Implemented functions for handling event payloads, rendering event chats, and managing state during processing.
- Enhanced `chat_distill.py` with a detailed prompt for extracting surface and inferred events from chat records, focusing on relevant information and event types.
- Updated logging and progress tracking for improved user feedback during event processing.
- Introduced `agent_distill_args` in `settings.template.jsonc` to specify LLM backend for agent distillation.
- Added `AgentDistillArgs` class in `config_models.py` to support new configuration parameters.
- Enhanced `CodexExecClient` in `llm_client.py` to manage request directories and improve command execution handling.
- Created `distill_state.py` for managing state extraction and processing logic related to agent distillation.
- Updated `distill.py` to enhance state extraction functions, introducing required configuration validation for model parameters.
- Refactored prompt definitions in `chat_distill.py`, replacing the previous state extraction prompt with a more structured template and adding a section for current state extraction.
- Improved memory filtering logic to better handle current state payloads and ensure relevant information is retained for future conversations.
- Updated subproject commit to reflect changes in the repository state.
- Introduced `__init__.py` for agent data construction utilities.
- Added `distill.py` containing functions for processing chat data, managing state, and saving JSON files.
- Created `chat_distill.py` with a detailed prompt for extracting memory states from chat records, focusing on various types of information such as stable facts, preferences, experiences, goals, and current states.
- Modified .gitignore to include all agent files in the dataset directory.
- Added a `sandbox` parameter to the `build_llm_client` function for improved client configuration.
- Introduced helper functions to generate chat IDs and assign group IDs to items in dataset processing, enhancing data organization.
- Introduced `chat_talker` and `partner_remarks` attributes in DataProcessor to improve chat context handling.
- Refactored `load_file` method to streamline user data loading and relation management.
- Replaced the previous batch processing logic with a more robust implementation that utilizes guided decoding for improved result parsing.
- Enhanced error logging for failed parsing attempts, providing clearer warnings for issues encountered during inference.
- Updated the score mapping to utilize the new parsed results structure, ensuring accurate score assignment to QA pairs.
- Added logic to handle the "<begin_chat>" marker in user messages, allowing for improved context in conversation flows.
- Updated the message processing to include the assistant's response when the user initiates a chat, ensuring more coherent QA pairs.
- Refactored the handling of conversation messages to maintain clarity and structure in the generated QA data.
- Updated the `qa_generator.py` to include a new mechanism for managing chat member relationships, allowing the addition of contextual information about the relationship between users in conversations.
- Refactored the CSV loading function to support loading user relationship data from a `users.json` file, improving the context provided during QA generation.
- Added a new configuration option `add_relation` to the dataset settings, enabling users to toggle this feature.
- Updated the `.gitignore` to exclude additional data directories and cache files for better repository hygiene.
- Bumped version to 0.3.03 in `pyproject.toml` to reflect these changes.
Expands the list of message types to be skipped or ignored by the application.
This ensures proper handling for user-uploaded GIF stickers and a new sticker
type identified as 'sticker2', preventing potential processing errors.
Introduce a ThreadPoolExecutor in the OnlineLLM class to enable
concurrent API calls, significantly boosting throughput for LLM operations.
Refactor the OlineLLMCleaningStrategy to leverage this new batching
capability, allowing multiple data points to be processed in parallel.
Increase the default `clean_batch_size` in settings and enhance
`n_process`/`batch_size` for PII detection to optimize for the new
concurrency. Simplify prompt management by removing a dedicated prompt
for online LLM cleaning. Add context manager support to OnlineLLM for
reliable resource cleanup. Ensure LLM chat responses explicitly request
JSON format.
Allow `vllm_infer` to return `None` for failed JSON parsing results,
clarifying the sequence index for such failures. This enables
`LLMCleaningStrategy` to robustly align parsed LLM scores with original data,
correctly handling examples where JSON parsing failed by assigning a default
score of 0.
Adds an option to disable data cleaning during training setup.
Refactors dataset management to consistently append '-vl' for vision-language
datasets and dynamically name cleaned datasets (e.g., 'dataset-cleaned').
Enforces vLLM as a strict dependency for LLM-based data cleaning, exiting if
unavailable. Integrates 'enable_thinking' option for LLM cleaning and enables
cleaning by default in relevant test configurations.
Adds torchvision dependency for vision models and streamlines the cleaning call
in training to centralize decision-making.
Ensures that files with a .gif extension are correctly identified and
categorized as animated expressions or stickers based on the platform.
This improves the accuracy of media type classification during data
processing.
Introduces a new 'enable_thinking' configuration for LLM interactions.
This allows dynamic control of thinking behavior in both offline inference
and the data cleaning process. When thinking is enabled for cleaning,
the maximum number of new tokens is increased to accommodate longer
reasoning outputs.
Also reduces the repetition penalty in the cleaning strategy for
improved response diversity and skips data entries containing images
during the LLM cleaning process to avoid irrelevant processing.
Updates pre-commit hooks.
Previously, the cleaning pipeline was entirely disabled for datasets
containing image messages.
Image conversations now receive a default score, bypassing text-focused
LLM cleaning. This enables partial cleaning for mixed datasets and
prevents pipeline failures.
Adds a blocked words filter to the dataset creation process.
This prevents undesirable words from appearing in the generated
datasets, enhancing data quality and safety.
A test case is added to verify the filter is working correctly.
Also, skips "text" type in qa_generator to avoid duplicated filter.
Implements batch processing for PII detection to improve
performance. Also, fixes a bug in the PII detection logic
and updates the installation instructions in the README.
Adds retry logic for OpenAI and image description API calls
to improve resilience against transient errors and rate
limiting. Implements exponential backoff with configurable
parameters.
Improves PII detection for Chinese language data by:
- Adding a Chinese-specific PII detector that inherits from
the base PII detector.
- Incorporating custom recognizers for common Chinese PII
patterns (e.g., phone numbers, ID cards).
- Filtering out irrelevant country-specific entities when
processing Chinese text.
- Adding unicode escape id recognizer.
These changes ensure more accurate and comprehensive
identification of sensitive information in Chinese datasets.
Adds a configuration option to include the current
datetime in the system prompt of the generated dataset.
This allows models to be aware of the temporal context
of the conversations.
This commit addresses an issue where newline characters
within text messages were causing problems in downstream
processing. It removes these characters to ensure that
the messages are processed correctly.
Consolidates the separator for combined content to a newline character, removing language-specific branching.
This simplifies the logic and ensures consistency across different languages.
Adds support for processing stickers in Telegram chat data.
- Includes 'sticker' as a valid DataModality.
- Allows the option to include sticker emojis as message content.
xming521