189 Commits

Author SHA1 Message Date
xming521 ad288b7bd7 Refactor profile web UI into atlas/review modules and improve graph layout
- Split the frontend into atlas/ and review/ modules with a shared api client,
  and add Prettier, ESLint and node tests for graph layout and data helpers
- Lay out expanded dimensions as separate clusters around the root
- Let the force simulation cool slowly after a node drag instead of stopping abruptly
- Reopening the collapsed root resets the map to its initial layout
- Use one loading screen style for the page and canvas loading states
- Clear stale errors when retrying fact history
- Rebuild weclone/web/dist

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-16 20:09:16 +08:00
xming521 31ae4035fb feat: add test for long prompt handling in Codex and improve subprocess input handling 2026-10-07 12:46:02 +08:00
xming521 b43033f97d Update index.html to reference new JavaScript and CSS files; add new CSS styles for improved layout and design. 2026-10-03 18:03:09 +08:00
xming521 040b6f3af1 Fix password bounds, filtered profile scope and input hashing 2026-10-03 13:27:16 +08:00
xming521 4b95b885c7 Refine profile review navigation, controls and source chat display 2026-10-03 00:52:02 +08:00
xming521 58312f84ad feat: add SourceChatPanel component and backend support for source chat retrieval
- Implemented SourceChatPanel component for displaying chat records with UI enhancements.
- Added source_chat.py to handle chat sample extraction from persisted memory.
- Introduced new CSS styles for improved layout and responsiveness.
- Updated JavaScript bundle with new features and optimizations.
2026-10-02 23:46:15 +08:00
xming521 7daa270736 Add encrypted chat storage and shared-password workflows
Integrate suffixed encrypted storage across extraction, review, training, and inference. Add shared-password web authentication with two-hour sessions, private request logging, and an isolated profile-building pipeline.

Fix encrypted dataset discovery at training entrypoints and duplicate CSV imports. Add synthetic storage, authentication, and log checks; update documentation and frontend assets.
2026-10-02 23:18:07 +08:00
xming 86c51bca7a Merge branch 'master' into fix/retry-config-empty-lists 2026-09-28 19:58:35 +08:00
XiaoZheBrother ab15ebe50f fix: 保留重试配置中显式传入的空列表 2026-09-28 15:34:59 +08:00
Iams4kura 73c8bd0008 Merge upstream master and preserve configured dataset output 2026-09-27 19:50:41 +08:00
pre-commit-ci[bot] 8de3dc79d0 🎈 auto fixes by pre-commit hooks 2026-09-27 10:03:51 +00:00
xming521 27e7aefc85 feat: add LTP filtering logic and improve episode content validation 2026-09-27 18:03:35 +08:00
pre-commit-ci[bot] ca41978ee6 🎈 auto fixes by pre-commit hooks 2026-09-27 09:21:43 +00:00
xming 7b9f699d72 fix: avoid pathlib access in async config reader 2026-09-27 17:21:33 +08:00
pre-commit-ci[bot] ac033289c0 🎈 auto fixes by pre-commit hooks 2026-09-27 09:16:49 +00:00
xming521 3086a82934 feat: Enhance agent distillation configuration and improve memory organization
- Added default values for batch size, max tokens, and timeout in distill_profile.py.
- Refactored required configuration functions to support section-specific error messages.
- Introduced positive and optional positive integer validation functions for configuration.
- Updated resolve_llm_args to utilize new configuration validation functions.
- Modified organize_memories.py to load agent distill configuration and handle missing model gracefully.
- Improved stage 2 output file management in make_dataset_stage2.py, ensuring proper validation against manifest files.
- Added new web assets including HTML, CSS, and JS files for profile review interface.
- Set up default static directory for serving web assets in profile_review.py.
2026-09-27 17:16:37 +08:00
pre-commit-ci[bot] 86b7651f79 🎈 auto fixes by pre-commit hooks 2026-09-27 08:34:55 +00:00
xming eff8d2657d Merge branch 'master' into agent 2026-09-27 16:34:44 +08:00
xming521 e8d4b3f363 Remove deprecated tokenizer configuration and enhance tokenizer file resolution
- Deleted the obsolete tokenizer configuration file `tokenizer_config.json` from the deepseek_v3 tokenizer resources.
- Updated the `resolve_tokenizer_file` function in `token_counter.py` to raise a FileNotFoundError if the tokenizer file is not specified, improving error handling.
- Modified the help message for the `--tokenizer-file` argument to clarify that it can be set via the environment variable `WECLONE_TOKENIZER_FILE`.
2026-09-27 16:26:59 +08:00
xming521 583cba0b5b Add authentication and authorization features for the web UI
- Implemented a new authentication system in `auth.py` to manage user sessions and password security.
- Created an `AuthStore` class to handle password hashing, session management, and login/logout functionality.
- Added API endpoints for login, logout, and session validation in the FastAPI application.
- Developed a React context for authentication in `Auth.tsx`, including a login form and logout button.
- Introduced tests for private routes and session management in `test_web_auth.py`.
- Added CSS styles for the authentication page in `index-DOIy3AqG.css`.
- Included a new JavaScript file for the web application in `index-DwlWkAhO.js`.
2026-09-27 16:00:03 +08:00
xming521 373425dff2 feat: Update the command line startup instructions, add inference mode and port settings. 2026-09-27 14:37:13 +08:00
xming521 ad1720d4f5 feat: Add memory organization functionality with attribute hierarchy and classification
- Implemented `organize_memories.py` to manage and organize extracted profile/event memories by dimension.
- Introduced new schemas and rules for attribute hierarchy and preference attributes in `memory_organization.py`.
- Created `profile_hierarchy.py` for generating a hierarchical structure of attributes for better browsing.
- Enhanced classification and summarization processes to support new dimensions and attributes.
2026-09-27 14:36:37 +08:00
xming521 77c440e9df feat: enhance CLI with new commands for profile and event distillation, and update configuration settings 2026-09-27 14:35:50 +08:00
xming521 7a90f3ef1a feat: Add the distill_profile.py file, which contains user profile processing functionality and related configuration. 2026-09-27 10:52:22 +08:00
xming521 f015540734 feat: Add persona classification and summarization functionality, and define dimensions and processing rules. 2026-09-27 10:47:31 +08:00
xming 3e27d4bca5 Add description for quantization method in config 2026-09-17 12:39:34 +08:00
xming bf2de49085 Merge branch 'master' into fix/issue-85-support-float16-for-older-gpus 2026-09-17 12:35:22 +08:00
xming521 5c95829e5f feat: enhance CodexExecClient with MCP discovery and minimal context configuration 2026-09-15 16:32:02 +08:00
xming521 b0a4b18733 feat: enhance capacity handling and request pacing in Codex client 2026-09-15 15:59:44 +08:00
xming521 8e337d7eea refactor: unify inference kernel and migrate main callers 2026-09-13 13:58:39 +08:00
Iams4kura 5d66d32522 fix: validate configured dataset mapping 2026-08-26 12:40:05 +08:00
Iams4kura 23e47c63bd fix: save generated data to configured dataset 2026-08-26 12:35:21 +08:00
xming521 87679d6687 更新.gitignore以排除swanlog和.venv*目录;在online_infer.py中添加_as_chat_messages函数并修改消息处理逻辑 2026-08-18 21:47:04 +08:00
xming521 558249d4af Add event extraction functionality and update prompts
- Introduced `distill_event.py` for processing chat records and extracting events related to user interactions.
- Implemented functions for handling event payloads, rendering event chats, and managing state during processing.
- Enhanced `chat_distill.py` with a detailed prompt for extracting surface and inferred events from chat records, focusing on relevant information and event types.
- Updated logging and progress tracking for improved user feedback during event processing.
2026-06-24 15:51:15 +08:00
xming521 08a8ba9935 Add agent distillation configuration and enhance LLM client
- Introduced `agent_distill_args` in `settings.template.jsonc` to specify LLM backend for agent distillation.
- Added `AgentDistillArgs` class in `config_models.py` to support new configuration parameters.
- Enhanced `CodexExecClient` in `llm_client.py` to manage request directories and improve command execution handling.
- Created `distill_state.py` for managing state extraction and processing logic related to agent distillation.
2026-06-23 21:43:55 +08:00
xming521 a153110e73 Update tokenizer configuration and enhance token counting utility
- Removed `milvus-lite` and `pymilvus` from dependencies in `pyproject.toml`.
- Added `package-data` configuration for including tokenizer resources in the `weclone` package.
- Introduced new tokenizer configuration files in `weclone/resources/tokenizers/deepseek_v3`.
- Implemented a token counting utility in `token_counter.py` to facilitate text tokenization and counting.
2026-06-20 22:41:08 +08:00
xming521 d9db662c18 Update dependencies, enhance configuration models, and modify .gitignore
- Added `milvus-lite` and `pymilvus` to project dependencies in `pyproject.toml`.
- Introduced `CodexExecArgs` and `EmbeddingServiceArgs` classes in `config_models.py` to support new execution and embedding service configurations.
- Updated `.gitignore` to exclude the newly added `AGENTS.md` file.
2026-06-20 20:50:55 +08:00
xming521 2f842fb1be Refactor state extraction logic and update prompts
- Updated `distill.py` to enhance state extraction functions, introducing required configuration validation for model parameters.
- Refactored prompt definitions in `chat_distill.py`, replacing the previous state extraction prompt with a more structured template and adding a section for current state extraction.
- Improved memory filtering logic to better handle current state payloads and ensure relevant information is retained for future conversations.
- Updated subproject commit to reflect changes in the repository state.
2026-06-19 23:02:26 +08:00
xming521 e4a568c000 Add agent data utilities and state extraction prompt
- Introduced `__init__.py` for agent data construction utilities.
- Added `distill.py` containing functions for processing chat data, managing state, and saving JSON files.
- Created `chat_distill.py` with a detailed prompt for extracting memory states from chat records, focusing on various types of information such as stable facts, preferences, experiences, goals, and current states.
2026-06-17 23:03:41 +08:00
xming521 893a5c5e3b Update .gitignore, enhance LLM client with sandbox option, and add group ID assignment in dataset processing
- Modified .gitignore to include all agent files in the dataset directory.
- Added a `sandbox` parameter to the `build_llm_client` function for improved client configuration.
- Introduced helper functions to generate chat IDs and assign group IDs to items in dataset processing, enhancing data organization.
2026-06-17 23:02:35 +08:00
xming521 8ef753b28d Enhance online inference with proxy support and retry mechanism
- Added support for OpenRouter proxy configuration in the OnlineLLM class.
- Implemented a retry mechanism for OpenAI API calls with exponential backoff and jitter.
- Updated chat method to handle optional parameters for temperature and top_p.
- Improved error handling and logging for API call failures.
- Cleaned up unused code and comments for better readability.
2026-06-13 16:33:27 +08:00
xming521 0e2d15f2cd Update .gitignore, enhance DataProcessor with partner remarks and chat_talker, and refactor load_file method
- Introduced `chat_talker` and `partner_remarks` attributes in DataProcessor to improve chat context handling.
- Refactored `load_file` method to streamline user data loading and relation management.
2026-06-13 16:14:01 +08:00
xming521 04ba49e067 Update subproject commit and clean up training configuration
- Updated subproject commit to indicate a dirty state.
- Removed the `create_new_adapter` field from `TrainSftArgs` class in `config_models.py` to streamline training configuration.
2026-05-28 12:21:28 +08:00
xming521 49c7246a18 Enhance training configuration with pre-training support
- Added `train_pt` command to continue pre-training using prepared text datasets.
- Introduced `TrainPtArgs` class for pre-training parameters, extending `TrainSftArgs`.
2026-05-28 10:58:22 +08:00
xming 76e341feef Potential fix for pull request finding
Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>
2026-05-11 10:49:59 +08:00
xhrxgr b8747edf27 fix: 量化配置未传递给 LLaMA-Factory 导致退化为全精度加载
- 新增 QuantizationArgs 配置类,与 LLaMA-Factory QuantizationArguments 对齐
- 移除 load_in_4bit/load_in_8bit(由 quantization_bit 内部处理)
- 新增 quantization_method 支持多种量化后端(bnb, gptq, awq 等)
- 修复 dict_to_argv 跳过 None 值,避免 HfArgumentParser 报错
- 使用 exclude_none=True 防止空字段传递给 run_exp()
- 训练和推理时自动展平嵌套的量化配置
2026-05-10 18:26:09 +08:00
xming 9e892b9da7 Merge branch 'master' into fix/issue-115-check-api-server-before-test 2026-05-03 15:11:02 +08:00
Octopus e6325fd8b7 refactor: derive API server URL from client.base_url for error message
Address Sourcery review feedback - use client.base_url instead of
hardcoding http://127.0.0.1:8005 so the error message stays in sync
if the base URL configuration changes.
2026-04-24 13:02:10 +08:00
octo-patch d8d2c673f4 fix: check API server availability before running test-model (fixes #115)
Add _check_api_server() that probes GET /v1/models before the test loop
starts. When the server is not running, the user now gets a clear error
message with the exact command to start it instead of a cryptic
connection-refused traceback.

Co-Authored-By: Octopus <liyuan851277048@icloud.com>
2026-04-24 12:19:52 +08:00
octo-patch 990e002d10 fix: constrain VllmArgs.dtype to vLLM-supported Literal values
Per review feedback (#212): the previous `Optional[str]` accepted any
string, deferring validation until runtime. Switch to
`Optional[Literal["auto", "float16", "bfloat16", "float32", "half", "bf16"]]`
so Pydantic rejects invalid values up-front (matching vLLM CLI's accepted
`--dtype` choices, including the `half` / `bf16` aliases).

Also expand the field description to document the supported values.
2026-04-19 13:22:34 +08:00