Introduce a ThreadPoolExecutor in the OnlineLLM class to enable
concurrent API calls, significantly boosting throughput for LLM operations.
Refactor the OlineLLMCleaningStrategy to leverage this new batching
capability, allowing multiple data points to be processed in parallel.
Increase the default `clean_batch_size` in settings and enhance
`n_process`/`batch_size` for PII detection to optimize for the new
concurrency. Simplify prompt management by removing a dedicated prompt
for online LLM cleaning. Add context manager support to OnlineLLM for
reliable resource cleanup. Ensure LLM chat responses explicitly request
JSON format.
Implements batch processing for PII detection to improve
performance. Also, fixes a bug in the PII detection logic
and updates the installation instructions in the README.
Initializes the analyzer engines within the
ChinesePIIDetector class, ensuring proper
setup and custom recognizer loading during object
instantiation. This enhances the PII detection
process for Chinese-specific contexts.
Improves PII detection for Chinese language data by:
- Adding a Chinese-specific PII detector that inherits from
the base PII detector.
- Incorporating custom recognizers for common Chinese PII
patterns (e.g., phone numbers, ID cards).
- Filtering out irrelevant country-specific entities when
processing Chinese text.
- Adding unicode escape id recognizer.
These changes ensure more accurate and comprehensive
identification of sensitive information in Chinese datasets.
xming521