- Replaced the previous batch processing logic with a more robust implementation that utilizes guided decoding for improved result parsing.
- Enhanced error logging for failed parsing attempts, providing clearer warnings for issues encountered during inference.
- Updated the score mapping to utilize the new parsed results structure, ensuring accurate score assignment to QA pairs.
Introduce a ThreadPoolExecutor in the OnlineLLM class to enable
concurrent API calls, significantly boosting throughput for LLM operations.
Refactor the OlineLLMCleaningStrategy to leverage this new batching
capability, allowing multiple data points to be processed in parallel.
Increase the default `clean_batch_size` in settings and enhance
`n_process`/`batch_size` for PII detection to optimize for the new
concurrency. Simplify prompt management by removing a dedicated prompt
for online LLM cleaning. Add context manager support to OnlineLLM for
reliable resource cleanup. Ensure LLM chat responses explicitly request
JSON format.
Allow `vllm_infer` to return `None` for failed JSON parsing results,
clarifying the sequence index for such failures. This enables
`LLMCleaningStrategy` to robustly align parsed LLM scores with original data,
correctly handling examples where JSON parsing failed by assigning a default
score of 0.
Adds an option to disable data cleaning during training setup.
Refactors dataset management to consistently append '-vl' for vision-language
datasets and dynamically name cleaned datasets (e.g., 'dataset-cleaned').
Enforces vLLM as a strict dependency for LLM-based data cleaning, exiting if
unavailable. Integrates 'enable_thinking' option for LLM cleaning and enables
cleaning by default in relevant test configurations.
Adds torchvision dependency for vision models and streamlines the cleaning call
in training to centralize decision-making.
Introduces a new 'enable_thinking' configuration for LLM interactions.
This allows dynamic control of thinking behavior in both offline inference
and the data cleaning process. When thinking is enabled for cleaning,
the maximum number of new tokens is increased to accommodate longer
reasoning outputs.
Also reduces the repetition penalty in the cleaning strategy for
improved response diversity and skips data entries containing images
during the LLM cleaning process to avoid irrelevant processing.
Updates pre-commit hooks.
Previously, the cleaning pipeline was entirely disabled for datasets
containing image messages.
Image conversations now receive a default score, bypassing text-focused
LLM cleaning. This enables partial cleaning for mixed datasets and
prevents pipeline failures.
xming521