61 Commits
Author SHA1 Message Date
xming521andClaude Opus 5.5 ab7f4c500e feat(web): privacy levels and tags in review, staggered domain rings
- facts and evidence show their privacy level, the overview shows the level spread in scope,
  and level toggles narrow the review list
- fact cards list their sources' extraction tags, the most shared first
- "全选" takes the whole filtered list and confirms when it includes facts not shown yet
- with eight or more domains the clusters alternate between an inner and an outer ring without
  overlapping; domain colours follow one OKLCH hue walk
- the synthetic demo profile gains domains, privacy levels and source tags

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-22 20:37:55 +08:00
xming521andClaude Opus 5.5 14ff633563 feat(memory): record which chat each peer id stands for
- prepare writes peers.json: the chat_with_id and chat_with behind each peer id (P1, P2, ...)
- read_memories returns the peers along with the records
- organized_memories.json carries the peers next to the sources

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-22 20:37:55 +08:00
xming521andClaude Opus 5.5 88d839439e feat(memory): time scope on summarized facts
- summarize gives every fact a time_scope: stable (never changes), current (latest known state),
  superseded (ended, replaced by a later statement, or a one-off claim contradicting well-supported ones)
  or dated (a plan, arrangement or event tied to a time)
- a state that ends takes its dependent states and goals with it; finished experiences and place facts are stable
- age, grade and tenure are written as birth year, entry year or start month and labelled stable
- identity (dimension 1) groups of every domain share one summarize request, so the model can see which
  candidate a later one replaced
- validation rejects missing or unknown time scopes; the rules sit in the fixed summarize prefix

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-22 20:31:41 +08:00
xming521andClaude Opus 5.5 6fe747294a feat(memory): privacy level on classified memories and summarized facts
- classify gives every memory a sensitivity (public/normal/private/sensitive), including memories with no fields
- summarize and reduce see each source's level and give every fact its own level, lower when the value drops private details
- the level rubric sits in the fixed prompt prefix of classify and summarize; validation rejects missing or unknown levels
- organized_memories sources keep the classify level

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-22 12:41:11 +08:00
xming521andClaude Opus 5.5 f8a1036234 feat(style): distill-style extraction and the /api/style view
- weclone-cli distill-style extracts speaking-style measurements per person and per contact group
- /api/style serves the newest complete style run to the web 说话风格 view
- distill-style passes --help through to its own option parser

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-21 20:48:38 +08:00
xming521andClaude Opus 5.5 5ff73c2df7 refactor(memory): organize memories into life domains and topics
- organize-memories induces life domains, classifies memories into them, groups attributes into topics and summarizes per topic
- llm-profile-hierarchy splits oversized topics, rewrites domain descriptions and requires --output-dir
- remove canonicalize_state_memories, group_state_memories and organize_identity_attributes
- rename prompts/chat_distill.py to prompts/distill_memory.py; drop merge_enabled from the settings template
- demo profile sources rewritten as casual fictional chats

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-21 20:46:51 +08:00
xming521andClaude Opus 5.5 d182434efc feat(server): night intro endpoint, dev command, newest-run review paths
- /api/profile/intro picks the atlas night-intro sentences from the profile sources
- review_paths reads only the newest profile_runs/*/profile_hierarchy run
- weclone-cli dev starts the backend and the web dev server together
- make-dataset joins merged messages with a newline
- llm_client sends service_tier=default; auth drops the unused DEFAULT_DATABASE

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-21 19:50:31 +08:00
xming521andClaude Opus 5.5 c267cf946f feat(harness): pick profile facts for a reply
- weclone/harness/profile: render the chat, route it to topics (jev
  two-stage choice or one LLM call), fill a fact budget and let an LLM
  keep the facts the reply needs, in must and helpful tiers. Any failure
  leaves the reply without profile facts and records the error.
- Unknown topic or fact ids in an LLM answer are dropped with a warning;
  only an answer with no valid id is retried.
- The profile tree applies the review store when its database exists:
  rejected facts are dropped and edited attributes and values are used.
- Review store: /api/profile returns the profile's domains (name,
  description, topic order).
- Config: harness_args.profile_context.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-21 19:45:56 +08:00
xming 40987742ed test: update profile hierarchy demo timeline data 2026-10-18 22:30:07 +08:00
xming521 31ae4035fb feat: add test for long prompt handling in Codex and improve subprocess input handling 2026-10-07 12:46:02 +08:00
xming521 7daa270736 Add encrypted chat storage and shared-password workflows
Integrate suffixed encrypted storage across extraction, review, training, and inference. Add shared-password web authentication with two-hour sessions, private request logging, and an isolated profile-building pipeline.

Fix encrypted dataset discovery at training entrypoints and duplicate CSV imports. Add synthetic storage, authentication, and log checks; update documentation and frontend assets.
2026-10-02 23:18:07 +08:00
xming521 54785561c7 Merge remote-tracking branch 'origin/master' into agent 2026-10-02 12:43:55 +08:00
xming521 0de8235b2f Add .gitignore file for dev-data directory to exclude SQLite and log files 2026-10-02 12:33:25 +08:00
Iams4kura 73c8bd0008 Merge upstream master and preserve configured dataset output 2026-09-27 19:50:41 +08:00
xming521 3086a82934 feat: Enhance agent distillation configuration and improve memory organization
- Added default values for batch size, max tokens, and timeout in distill_profile.py.
- Refactored required configuration functions to support section-specific error messages.
- Introduced positive and optional positive integer validation functions for configuration.
- Updated resolve_llm_args to utilize new configuration validation functions.
- Modified organize_memories.py to load agent distill configuration and handle missing model gracefully.
- Improved stage 2 output file management in make_dataset_stage2.py, ensuring proper validation against manifest files.
- Added new web assets including HTML, CSS, and JS files for profile review interface.
- Set up default static directory for serving web assets in profile_review.py.
2026-09-27 17:16:37 +08:00
pre-commit-ci[bot] 86b7651f79 🎈 auto fixes by pre-commit hooks 2026-09-27 08:34:55 +00:00
xming521 583cba0b5b Add authentication and authorization features for the web UI
- Implemented a new authentication system in `auth.py` to manage user sessions and password security.
- Created an `AuthStore` class to handle password hashing, session management, and login/logout functionality.
- Added API endpoints for login, logout, and session validation in the FastAPI application.
- Developed a React context for authentication in `Auth.tsx`, including a login form and logout button.
- Introduced tests for private routes and session management in `test_web_auth.py`.
- Added CSS styles for the authentication page in `index-DOIy3AqG.css`.
- Included a new JavaScript file for the web application in `index-DwlWkAhO.js`.
2026-09-27 16:00:03 +08:00
xming521 6a7a9577ad feat: add isolated_default_config fixture for default settings in tests 2026-09-27 11:28:02 +08:00
xming521 7837d35030 Add unit tests for profile review and server CLI functionality
- Implement comprehensive tests for profile review endpoints, including approval, editing, and exporting facts.
- Validate snapshot import behavior and ensure proper error handling for invalid data.
- Introduce tests for server CLI options and ensure correct application of configurations.
- Test unified routes with lifespan management, authentication, and streaming responses.
2026-09-27 10:51:19 +08:00
xming521 49aeff3592 feat: Add test cases for distill_cli and distill_windows to verify model invocation and input/output processing. 2026-09-27 10:49:08 +08:00
xming521 5c95829e5f feat: enhance CodexExecClient with MCP discovery and minimal context configuration 2026-09-15 16:32:02 +08:00
xming521 b0a4b18733 feat: enhance capacity handling and request pacing in Codex client 2026-09-15 15:59:44 +08:00
xming521 8e337d7eea refactor: unify inference kernel and migrate main callers 2026-09-13 13:58:39 +08:00
Iams4kura 5d66d32522 fix: validate configured dataset mapping 2026-08-26 12:40:05 +08:00
Iams4kura 23e47c63bd fix: save generated data to configured dataset 2026-08-26 12:35:21 +08:00
xming521 1107500c41 chore: update dependencies and configuration
- Updated Python version requirement in `pyproject.toml` to 3.12.
- Adjusted dependency specifications for `llamafactory`, `torch`, `torchvision`, and `torchaudio` to include version constraints.
2026-01-04 16:15:15 +08:00
xming521 bbe08f0c77 feat(data): enhance chat member relationship handling in QA generation
- Updated the `qa_generator.py` to include a new mechanism for managing chat member relationships, allowing the addition of contextual information about the relationship between users in conversations.
- Refactored the CSV loading function to support loading user relationship data from a `users.json` file, improving the context provided during QA generation.
- Added a new configuration option `add_relation` to the dataset settings, enabling users to toggle this feature.
- Updated the `.gitignore` to exclude additional data directories and cache files for better repository hygiene.
- Bumped version to 0.3.03 in `pyproject.toml` to reflect these changes.
2025-11-01 17:11:26 +08:00
xming521 41167f6d29 fix(dataset): fix #192 standardize dataset naming and cleaning workflow
Refactors dataset management to consistently append '-vl' for vision-language
datasets and dynamically name cleaned datasets (e.g., 'dataset-cleaned').

Enforces vLLM as a strict dependency for LLM-based data cleaning, exiting if
unavailable. Integrates 'enable_thinking' option for LLM cleaning and enables
cleaning by default in relevant test configurations.

Adds torchvision dependency for vision models and streamlines the cleaning call
in training to centralize decision-making.
2025-08-16 22:51:29 +08:00
xming521 c89dc8100c test(tests): update qwen3 test model to 0.6B
Updates the Qwen3 model in test configurations from 4B to the smaller 0.6B version.
This change helps improve test performance and reduce resource requirements.
2025-08-15 15:34:46 +08:00
xming521 a0f6d0e9b0 chore(tests): include PII test data file
Ensures a specific PII test data file is tracked by Git, enabling its use in related tests.
2025-08-10 10:29:16 +08:00
xming521 a24b7f2714 test(dataset): add image tag count check for Qwen2.5-VL
Ensures the data generation pipeline correctly includes the expected number of <image> tags for multimodal configurations, specifically for Qwen2.5-VL datasets.
2025-07-16 15:30:56 +08:00
xming521 67232998fa chore: ignore WC-exp directory 2025-07-16 14:16:47 +08:00
xming521 785ad1472a remove wechat 2025-07-16 11:21:11 +08:00
xming521 66e7fdbeb4 test(PII) add tests for PII 2025-07-14 17:26:44 +08:00
xming521 7ce7fcd47a fix(dataset): fix blocked words filter
Adds a blocked words filter to the dataset creation process.
This prevents undesirable words from appearing in the generated
datasets, enhancing data quality and safety.
A test case is added to verify the filter is working correctly.

Also, skips "text" type in qa_generator to avoid duplicated filter.
2025-07-14 13:38:19 +08:00
xming521 bbe4a26197 Rename test_full_pipe.py 2025-07-06 10:51:52 +08:00
xming521 0bcf6bdd18 chore(tests): improve test environment setup
Refactors the test environment setup to use pytest fixtures
for session-level setup and teardown. This ensures a clean
and consistent environment before and after each test session.
It also improves the robustness by backing up and restoring
existing 'model_output' and 'dataset/csv' directories.
2025-07-06 10:50:50 +08:00
xming521 43b22ce903 调整settings.template.jsonc以添加Telegram参数,更新数据处理逻辑以支持Telegram平台,优化数据模型以处理不同消息类型。 2025-06-23 18:06:55 +08:00
xming521 bc121f030b 0.2.23 运行 weclone-cli server 报错
Fixes #158
2025-06-19 16:38:09 +08:00
xming521 55fdfe2520 新增vllm_args配置,更新测试模型参数配置,允许自定义测试数据路径。 2025-06-13 18:27:24 +08:00
xming521 e239893884 feat(tests): 添加测试模型参数配置,允许自定义测试集文件 2025-06-13 15:49:02 +08:00
xming521 53fc844403 refactor(config): 移除configV2,整合配置加载逻辑至config.py,更新相关引用 2025-06-13 12:02:27 +08:00
xming521 c77cdee043 修改full_pipe.jsonc和full_pipeV2.jsonc中的数据处理配置,调整train_sft.py和qa_generatorV2.py中的配置加载方式,更新config_models.py和configV2.py以支持新的配置结构。 2025-06-11 11:57:55 +08:00
xming521 94b6e4ac92 更新测试文件以支持图像数据集的创建和复制功能。 2025-06-10 22:12:39 +08:00
pre-commit-ci[bot] cf30258091 🎈 auto fixes by pre-commit hooks 2025-06-09 03:27:38 +00:00
xming521 3842932cab 更新README.md;调整mllm模板中的训练参数。 2025-06-06 20:52:44 +08:00
xming521 47ce8833cc fix tests 2025-05-22 21:05:49 +08:00
xming521 bbe16fda73 Add full-flow test pipeline 2025-05-22 20:05:22 +08:00
xming521 631c1731ed 重构vllm_infer.py以支持新的输入类型和引入引导解码参数,新增QaPairScore模型以支持评分功能,实现大模型对数据评分。 2025-05-04 22:52:09 +08:00
xming521 810ca43f45 将配置文件名从settings.json更改为settings.jsonc 2025-05-03 14:50:22 +08:00