Integrate suffixed encrypted storage across extraction, review, training, and inference. Add shared-password web authentication with two-hour sessions, private request logging, and an isolated profile-building pipeline.
Fix encrypted dataset discovery at training entrypoints and duplicate CSV imports. Add synthetic storage, authentication, and log checks; update documentation and frontend assets.
- Added default values for batch size, max tokens, and timeout in distill_profile.py.
- Refactored required configuration functions to support section-specific error messages.
- Introduced positive and optional positive integer validation functions for configuration.
- Updated resolve_llm_args to utilize new configuration validation functions.
- Modified organize_memories.py to load agent distill configuration and handle missing model gracefully.
- Improved stage 2 output file management in make_dataset_stage2.py, ensuring proper validation against manifest files.
- Added new web assets including HTML, CSS, and JS files for profile review interface.
- Set up default static directory for serving web assets in profile_review.py.
- Deleted the obsolete tokenizer configuration file `tokenizer_config.json` from the deepseek_v3 tokenizer resources.
- Updated the `resolve_tokenizer_file` function in `token_counter.py` to raise a FileNotFoundError if the tokenizer file is not specified, improving error handling.
- Modified the help message for the `--tokenizer-file` argument to clarify that it can be set via the environment variable `WECLONE_TOKENIZER_FILE`.
- Introduced `agent_distill_args` in `settings.template.jsonc` to specify LLM backend for agent distillation.
- Added `AgentDistillArgs` class in `config_models.py` to support new configuration parameters.
- Enhanced `CodexExecClient` in `llm_client.py` to manage request directories and improve command execution handling.
- Created `distill_state.py` for managing state extraction and processing logic related to agent distillation.
- Removed `milvus-lite` and `pymilvus` from dependencies in `pyproject.toml`.
- Added `package-data` configuration for including tokenizer resources in the `weclone` package.
- Introduced new tokenizer configuration files in `weclone/resources/tokenizers/deepseek_v3`.
- Implemented a token counting utility in `token_counter.py` to facilitate text tokenization and counting.
- Added `milvus-lite` and `pymilvus` to project dependencies in `pyproject.toml`.
- Introduced `CodexExecArgs` and `EmbeddingServiceArgs` classes in `config_models.py` to support new execution and embedding service configurations.
- Updated `.gitignore` to exclude the newly added `AGENTS.md` file.
- Updated subproject commit to indicate a dirty state.
- Removed the `create_new_adapter` field from `TrainSftArgs` class in `config_models.py` to streamline training configuration.
- Added `train_pt` command to continue pre-training using prepared text datasets.
- Introduced `TrainPtArgs` class for pre-training parameters, extending `TrainSftArgs`.
Per review feedback (#212): the previous `Optional[str]` accepted any
string, deferring validation until runtime. Switch to
`Optional[Literal["auto", "float16", "bfloat16", "float32", "half", "bf16"]]`
so Pydantic rejects invalid values up-front (matching vLLM CLI's accepted
`--dtype` choices, including the `half` / `bf16` aliases).
Also expand the field description to document the supported values.
GPUs with compute capability < 8.0 (e.g., Tesla T4, V100) do not support
bfloat16, causing a ValueError when running vllm-based data cleaning.
Add an explicit dtype field to VllmArgs so users can set dtype: float16
in the vllm_args section of settings.jsonc. The existing engine_args.update()
path already propagates this to the vLLM engine.
Also document the option in settings.template.jsonc with a commented-out example.
- Updated the guided decoding parameter logic to support dynamic imports, improving compatibility with different configurations.
- Simplified the creation of guided decoding parameters by introducing a helper function.
- Modified the retry mechanism in the OpenAI API call to retry on all exceptions, enhancing robustness in error handling.
- Updated the `qa_generator.py` to include a new mechanism for managing chat member relationships, allowing the addition of contextual information about the relationship between users in conversations.
- Refactored the CSV loading function to support loading user relationship data from a `users.json` file, improving the context provided during QA generation.
- Added a new configuration option `add_relation` to the dataset settings, enabling users to toggle this feature.
- Updated the `.gitignore` to exclude additional data directories and cache files for better repository hygiene.
- Bumped version to 0.3.03 in `pyproject.toml` to reflect these changes.
Refactors dataset management to consistently append '-vl' for vision-language
datasets and dynamically name cleaned datasets (e.g., 'dataset-cleaned').
Enforces vLLM as a strict dependency for LLM-based data cleaning, exiting if
unavailable. Integrates 'enable_thinking' option for LLM cleaning and enables
cleaning by default in relevant test configurations.
Adds torchvision dependency for vision models and streamlines the cleaning call
in training to centralize decision-making.
Introduces a new 'enable_thinking' configuration for LLM interactions.
This allows dynamic control of thinking behavior in both offline inference
and the data cleaning process. When thinking is enabled for cleaning,
the maximum number of new tokens is increased to accommodate longer
reasoning outputs.
Also reduces the repetition penalty in the cleaning strategy for
improved response diversity and skips data entries containing images
during the LLM cleaning process to avoid irrelevant processing.
Adds retry logic for OpenAI and image description API calls
to improve resilience against transient errors and rate
limiting. Implements exponential backoff with configurable
parameters.
Adds a configuration option to include the current
datetime in the system prompt of the generated dataset.
This allows models to be aware of the temporal context
of the conversations.
Adds support for processing stickers in Telegram chat data.
- Includes 'sticker' as a valid DataModality.
- Allows the option to include sticker emojis as message content.
xming521