GithubStar/WeClone
1
0
Fork 0
mirror of https://github.com/xming521/WeClone.git synced 2026-10-18 14:36:48 +08:00
Code Issues Packages Projects Releases Wiki Activity

Commit Graph

  • 34fa44f357 Merge pull request #132 from xming521/xming521-patch-2 小铭 2025-05-29 15:53:33 +08:00
  • f5d0976423 Update README.md 小铭 2025-05-29 15:53:19 +08:00
  • b09edad868 improve qag V2 xming521 2025-05-28 23:21:53 +08:00
  • d9067920b6 Merge pull request #131 from xming521/xming521-patch-5 小铭 2025-05-28 20:40:16 +08:00
  • b57888bbbe Update README.md 小铭 2025-05-28 20:11:43 +08:00
  • babba7233a Merge remote-tracking branch 'origin/master' into dev xming521 2025-05-28 19:55:36 +08:00
  • 5c404c1709 Merge pull request #130 from BAIKEMARK/update-clean-strategy 小铭 2025-05-28 19:54:12 +08:00
  • 1898e48107 Merge branch 'master' into update-clean-strategy Mark 2025-05-28 18:56:44 +08:00
  • c19805d44a Merge branch 'update-clean-strategy' of https://github.com/BAIKEMARK/WeClone into update-clean-strategy BAIKEMARK 2025-05-28 18:54:19 +08:00
  • f2e79ad411 Fixed the bug where vllm uninstallation caused the inability to use online cleaning. BAIKEMARK 2025-05-28 18:54:15 +08:00
  • 7e4b0433f1 add QaPairV2 xming521 2025-05-27 23:07:17 +08:00
  • 5c28bf084e 更新数据模型以支持ShareGPT格式，新增QaPairV2类和Message类，修改数据处理逻辑以支持图片消息，优化图片文件检查功能。 xming521 2025-05-25 23:23:58 +08:00
  • 54cc16d29d Merge remote-tracking branch 'origin/master' into dev xming521 2025-05-24 23:27:29 +08:00
  • 07662a3dc1 更新pyproject.toml以使用llamafactory的GitHub链接，修改settings.template.jsonc以更新模型路径和添加平台配置，增强数据处理逻辑以支持多语言切割类型，并在qa_generator.py中整合切割类型列表。 xming521 2025-05-24 23:24:26 +08:00
  • 39ae0c791a Merge pull request #126 from BAIKEMARK/update-clean-strategy 小铭 2025-05-24 22:53:14 +08:00
  • a0b6863b15 Update README.md 小铭 2025-05-24 22:30:25 +08:00
  • ddd6b49af3 Changed commentjson back to json BAIKEMARK 2025-05-24 22:15:27 +08:00
  • b5c34ea521 update readme BAIKEMARK 2025-05-24 22:11:37 +08:00
  • b8b566f89b Debug and Update README BAIKEMARK 2025-05-24 22:10:21 +08:00
  • 9547392c5b Merge remote-tracking branch 'origin/master' into dev xming521 2025-05-24 15:33:23 +08:00
  • 9f59ec4261 更新版本号至0.2.22，添加微信图片复制功能，更新.gitignore以包含新目录 xming521 2025-05-24 15:32:37 +08:00
  • 6d3971fc44 Merge pull request #124 from BAIKEMARK/update-clean-strategy 小铭 2025-05-24 15:30:00 +08:00
  • 7cd7740d5e 完善clean_by_score逻辑并将其整合至strategies BAIKEMARK 2025-05-24 10:50:52 +08:00
  • 3db44677f0 修复了'qa' referenced before assignment的bug BAIKEMARK 2025-05-23 20:42:48 +08:00
  • 5fa391f55e 修改clean策略使llm打分结果保留 BAIKEMARK 2025-05-23 20:16:34 +08:00
  • 65233afb64 Merge pull request #123 from xming521:dev v0.2.21 小铭 2025-05-23 15:51:12 +08:00
  • 906b724524 change settings.template.jsonc xming521 2025-05-23 15:50:43 +08:00
  • 17b7c82e0f Merge pull request #119 from niulinbiao/nb-dev 小铭 2025-05-23 10:37:51 +08:00
  • 2bf7bcc9ce 编写配置文件更新日志 niuben 2025-05-22 21:43:25 +08:00
  • 20ac3caaf1 Merge pull request #121 from xming521:dev 小铭 2025-05-22 21:07:07 +08:00
  • 12dc50ac0e Merge remote-tracking branch 'origin/master' into dev xming521 2025-05-22 21:06:40 +08:00
  • 47ce8833cc fix tests xming521 2025-05-22 21:05:49 +08:00
  • 1de1373a19 删除 .idea 文件夹，清理项目配置文件 niuben 2025-05-22 20:29:34 +08:00
  • 84f2577837 Merge branch 'master' into nb-dev niulinbiao 2025-05-22 20:13:05 +08:00
  • d1e3d2c8a3 Merge pull request #118 from xming521/dev 小铭 2025-05-22 20:09:44 +08:00
  • bbe16fda73 Add full-flow test pipeline xming521 2025-05-22 20:05:22 +08:00
  • 5085a34fe3 实现在线LLM清洗数据 niuben 2025-05-22 20:04:45 +08:00
  • bcf89d22e9 实现在线LLM清洗数据 niuben 2025-05-22 19:50:57 +08:00
  • d794053dea Merge pull request #116 from BAIKEMARK/update-readme 小铭 2025-05-22 18:51:04 +08:00
  • 12d928c76a Merge pull request #117 from xming521/xming521-patch-4 小铭 2025-05-22 18:49:29 +08:00
  • 72afd4c554 Update issue-labeler.yml 小铭 2025-05-22 18:49:11 +08:00
  • ffd70062e1 添加了Linux教程【保姆级】 BAIKEMARK 2025-05-22 18:28:08 +08:00
  • bb0e4c4ece Merge remote-tracking branch 'origin/master' into dev xming521 2025-05-21 21:57:42 +08:00
  • ed9d8e320e Adjust the default values of lora_dropout and num_train_epochs, while applying a unified decorator in the CLI to support log capturing. Use an inelegant approach to print all terminal content to a file, enabled via full_log. xming521 2025-05-21 21:57:12 +08:00
  • 4b213145f3 Merge pull request #113 from xming521/xming521-patch-3 小铭 2025-05-21 16:05:34 +08:00
  • 71f71c8a73 Update issue-labeler.yml 小铭 2025-05-21 16:04:52 +08:00
  • cbf55df835 Merge pull request #112 from xming521/xming521-patch-2 小铭 2025-05-21 14:36:29 +08:00
  • 8b8d6d045a Update issue-labeler.yml 小铭 2025-05-21 14:36:18 +08:00
  • 4fd73939b0 Merge pull request #110 from xming521/xming521-patch-2 小铭 2025-05-21 14:32:05 +08:00
  • 13be9510fb Update issue-labeler.yml 小铭 2025-05-21 14:31:50 +08:00
  • 5d02822c90 Create issue-labeler.yml 小铭 2025-05-21 14:27:57 +08:00
  • 736136c2ee Create issue-labeler.yml 小铭 2025-05-21 11:37:17 +08:00
  • a81564cf8e Merge pull request #107 from xming521/xming521-patch-1 小铭 2025-05-20 20:41:51 +08:00
  • 218d9a0796 Update README.md 小铭 2025-05-20 20:41:39 +08:00
  • 4071ae3c82 Merge pull request #95 from xming521/dev 小铭 2025-05-18 22:31:57 +08:00
  • 42585eb663 更新版本号至 0.2.21，并在离线推理中添加 GPU 内存利用率参数。 xming521 2025-05-18 22:28:51 +08:00
  • 2db76b0968 在 qa_generator.py 中添加 vLLM 可用性检查并调整数据清洗逻辑；在 log.py 中设置日志级别为 INFO。 xming521 2025-05-17 23:04:56 +08:00
  • f4fa2df82e Merge remote-tracking branch 'origin/master' into dev xming521 2025-05-17 20:18:58 +08:00
  • 550eb3189e Merge pull request #87 from Mundi-Xu/csv-optimize 小铭 2025-05-17 20:08:19 +08:00
  • 33fb27811a 优化CSV文件读取 Hanyu 2025-05-17 14:11:55 +08:00
  • 4478cfbaa0 在 strategies.py 中添加对 vLLM 可用性的检查，并根据可用性选择推理引擎。暂不可用 xming521 2025-05-16 13:19:44 +08:00
  • 9604f352c9 更新 pyproject.toml 文件，添加了 vllm 依赖的 Linux 平台限制 xming521 2025-05-16 12:54:02 +08:00
  • 4c383d3c95 Merge pull request #74 from BAIKEMARK/update-readme 小铭 2025-05-15 14:17:14 +08:00
  • 5179c2db1b Update README BAIKEMARK 2025-05-15 12:07:22 +08:00
  • 3aa28128ba Merge pull request #68 from songhahaha66/master 小铭 2025-05-15 09:12:27 +08:00
  • f4d0d05ac8 Update README.md songhahaha66 2025-05-15 09:10:18 +08:00
  • 1da26350b7 Merge pull request #65 from RockChinQ/master 小铭 2025-05-15 08:54:35 +08:00
  • 5bb483df10 doc(README): add LangBot Junyan Qin (Chin) 2025-05-14 17:26:21 +08:00
  • baf63a0195 doc: add LangBot integration Junyan Qin 2025-05-14 17:23:04 +08:00
  • 4e12e225c1 Merge pull request #57 from xming521/xming521-patch-2 小铭 2025-05-14 09:50:14 +08:00
  • 9f2865f74d Update README.md 小铭 2025-05-14 09:50:00 +08:00
  • 537a221324 Merge pull request #52 from xming521/xming521-patch-1 v0.2.2 小铭 2025-05-11 19:57:08 +08:00
  • d76ddf0348 Create LICENSE 小铭 2025-05-11 19:56:53 +08:00
  • 4dfe42f8e7 Merge pull request #50 from xming521/dev 小铭 2025-05-11 19:54:00 +08:00
  • b16bd303e3 更新.gitignore以忽略输出文件，更新README.md以优化文本描述，修改vllm_infer.py和strategies.py以支持自定义禁用词配置。 xming521 2025-05-11 19:37:08 +08:00
  • 3246361c97 更新README.md。 xming521 2025-05-11 16:06:22 +08:00
  • 25c45b9103 修改dataset/test_data.json，增加对话内容并删除部分冗余问题。 xming521 2025-05-10 22:44:05 +08:00
  • 6500b3f8a1 更新README.md，添加DeepWiki链接并修改测试结果描述；在train_sft.py中增加日志记录微调配置。 xming521 2025-05-10 15:30:28 +08:00
  • bc8c9b392b 更新README.md。 xming521 2025-05-09 21:53:36 +08:00
  • c86173209b 更新README.md。 xming521 2025-05-09 20:41:06 +08:00
  • c4caaa27f9 更新README.md。 xming521 2025-05-09 20:39:00 +08:00
  • 592fd8711d 更新README.md。 v0.2.2-beta1 xming521 2025-05-08 22:11:04 +08:00
  • a18e474720 禁用词配置改到settings.json。 xming521 2025-05-08 20:44:32 +08:00
  • 03636d4a37 更新README.md。 xming521 2025-05-07 22:33:37 +08:00
  • d966e8490b 更新README.md。 xming521 2025-05-07 21:49:46 +08:00
  • 08b1640698 更新README.md和cli.py，简化命令行调用，修正文档中的命令格式。 xming521 2025-05-07 21:44:31 +08:00
  • 3bdcb3b8ac fixi 多卡length_cdf。 xming521 2025-05-06 22:35:10 +08:00
  • 528de9970b 固定length_cdf 单卡运行；去掉vllm_infer.py并行推理。 xming521 2025-05-05 22:48:45 +08:00
  • 91286029e1 fix vllm xming521 2025-05-05 21:29:28 +08:00
  • 5a48ab4ceb 修改vllm_infer.py以支持禁用\n，调整strategies.py以引入重复惩罚参数。 xming521 2025-05-05 20:35:24 +08:00
  • d7b77483a5 update llm judge xming521 2025-05-05 19:52:29 +08:00
  • 631c1731ed 重构vllm_infer.py以支持新的输入类型和引入引导解码参数，新增QaPairScore模型以支持评分功能，实现大模型对数据评分。 xming521 2025-05-04 22:52:09 +08:00
  • fec538fea6 在pyproject.toml中添加langchain依赖，并在strategies.py中引入PromptTemplate以支持数据清洗功能的实现。 xming521 2025-05-03 22:57:36 +08:00
  • 810ca43f45 将配置文件名从settings.json更改为settings.jsonc xming521 2025-05-03 14:50:22 +08:00
  • 319aaa1814 修改qa_generator.py以支持QaPair对象 xming521 2025-05-02 22:34:41 +08:00
  • 035d138d8d 更新pyproject.toml版本至0.2.2，优化vllm_infer函数以支持前缀缓存，调整qa_generator.py以处理结果并保存为JSON格式。 xming521 2025-05-01 23:04:37 +08:00
  • 5fea291883 更新版本至0.2.2，增加llm清洗数据配置，更新README.md以添加HelloGitHub推荐链接，调整settings.template.json以支持数据清洗功能，增强cli.py以打印配置文件更新日志。 xming521 2025-05-01 20:51:27 +08:00
  • aec5f4cfa2 注释掉evaluate命令的帮助信息 v0.2.1-beta1 xming521 2025-05-01 19:52:27 +08:00
  • 20904b0202 更新vllm依赖版本至0.8.2 xming521 2025-05-01 19:39:15 +08:00
  • 9827a24929 更新依赖项版本，提升torch和torchaudio至2.6.0，更新openai至1.52.0 相应更新test_model，调整pytorch源为cu124。 xming521 2025-04-29 22:47:02 +08:00