52 Commits

Author SHA1 Message Date
torzdf e0eda1cf80 bugfix: Extract memory leak 2023-06-16 13:31:45 +01:00
torzdf 1d19aea281 bugfix - numexpr. ignore OMP_NUM_THREADS EnvVar 2023-01-18 15:39:05 +00:00
torzdf 80f63280ca Bugfixes
- plugin.train.unbalanced - decoder b
  - gui.stats - crash when log folder deleted
  - setup.py - install gcc=12.1.0 on linux (scipy dep)
  - setup.py - don't test ROCm unless valid
  - typofix - cli_args extract singleprocess help
typing
  - lib.gui.analysis.event_reader
  - lib.gpu_stats.directml - make platform specific
unit tests
  - lib.gui.analysis.event_reader
2023-01-13 23:29:51 +00:00
torzdf 28cb2fcffb Unhide options for ROCm 2023-01-11 14:34:30 +00:00
torzdf dc94ed9cad DirectML Support (#1291)
* DML Support
  - Add requirements. Update setup.py. Update Windows Installer
  - Add 'directml' backend. Update backend supporting logic
  - GPUStats - Add GPU querying from Win SDK for DML backend
  - Windows PlaidML - Deprecation Warning
  - Remove deprecated '-d' training flag
  - Logging in DML functions
- Unittest
  - Windows Unit Test
  - lib.utils
2023-01-06 14:45:57 +00:00
torzdf d750af1e92 bump TF to version 2.10 2022-12-20 15:30:26 +00:00
torzdf 84ae6006bb Bugfix - TF Version check 2022-12-19 11:26:32 +00:00
torzdf 9e2026f6fe Extract: Implement re-align/2nd pass
- implement configurable re-align function in extract
  - update locales + documentation
  - re-factor align._base and split to separate modules
  - move normalization method to plugin parent
  - bugfix: FAN use zeros for pre-processing crop
  - lint AlignedFilter
2022-11-22 00:37:29 +00:00
torzdf e3b457693e bugfix - extract filters: spaces in folder names 2022-11-10 01:41:40 +00:00
torzdf 0f5d2e887c Face-filter updates:
- Allow selecting folder as well as multiple images
  - Lower default threshold and update helptext
  - bugfix: detector error when using all aligned faces
  - Standardize output folder name
2022-11-04 18:43:55 +00:00
torzdf 1d1face00d Update Face Filter
- Remove old face filter
  - plugins.extract.pipeline: Expose plugins directly
  - Change `is_aligned` from plugin level to ExtractMedia level
  - Allow extract pipeline to take faceswap aligned images
  - Add ability for recognition plugins to accept aligned faces as input
  - Add face filter to recognition plugin
  - Move extractor pipeline IO ops to own class
2022-10-31 18:25:32 +00:00
torzdf f1e3339fbe Add vggface2 to extraction pipeline 2022-10-10 13:09:02 +01:00
torzdf a8f22cc019 Extract updates:
- Default CPU detector to MTCNN
  - add basic Aligner false positive filters
  - Typing: align + plugins
  - Use specific AlignerBatch class for alignment
  -
2022-09-18 19:44:41 +01:00
torzdf 8a803e24c4 bugfix: Suppress OMP error on CPU Windows version 2022-09-18 18:25:33 +01:00
torzdf 13cfb3f39e extract: Add batch processing mode 2022-08-31 19:48:47 +01:00
torzdf 2beceffad9 Data Augmentation update (#1263)
- lib.detected_face
    - Subclass Masks for Landmark based masks
    - Add training mask propery + methods to DetectedFace
  - lib.training_training
    - subclass TrainingDataGenerator for training and preview data
    - Split cache into own module
    - Reduce thread count to 1 to prevent image corruption + data re-use
    - Process on largest model input/output size rather than stored image size
    - Size and crop masks during caching stage
    - Implement ring buffer for data flow
    - Fix preview reload bug
  - augmentation
    - typing
    - switch color aug order
    - better initialization
    - Fix warp + landmark warp to correctly apply at different image scales
    - Slightly improved warp caching
    - Don't store whether image is_preview. Handle all data as training images implicitly
  - plugins.trainer: Typing and fixes to work with trainingdata refactor
2022-08-21 18:59:45 +01:00
torzdf 629c02a61e Add "custom" mask 2022-08-05 14:09:05 +01:00
torzdf 26e26c6288 Update TF to 2.9
- Update TF to 2.7 to 2.9
    - Bump dependencies
    - Remove decode from pynvml calls
    - force keras predict functions to non-verbose
    - update tests
    - update Tensorboard logging
    - Update docs
2022-08-02 17:12:26 +01:00
torzdf 54398559ff Set default distribution strategy to 'default' 2022-07-20 12:29:14 +01:00
torzdf 2ea05623bd Update Distibution Strategies:
- Add Central Storage Stategy
  - Deprecate 'distributed' cli argument
2022-07-19 18:23:16 +01:00
torzdf 7b9fc0454d Live Preview - Replace cv2 with matplotlib viewer 2022-07-01 13:07:01 +01:00
torzdf 6e0360150d bugfix: Correctly show installed TF Version on AMD error 2022-06-29 18:10:47 +01:00
torzdf 20a657d6cf Bump minimum TF Version 2022-06-07 10:54:51 +01:00
torzdf bdbbad4d31 Refactor lib.gpu_stats (#1218)
* inital gpu_stats refactor

* Add dummy CPU Backend

* Update Sphinx documentation
2022-05-11 17:36:07 +01:00
torzdf aa39234538 Update all Keras Imports to be conditional (#1214)
* Remove custom keras importer

* first round keras imports fix

* launcher.py: Remove KerasFinder references

* 2nd round keras imports update (lib and extract)

* 3rd round keras imports update (train)

* remove KerasFinder from tests

* 4th round keras imports update (tests)
2022-05-03 20:18:39 +01:00
torzdf c1512fd41d Update code to support Tensorflow versions up to 2.8 (#1213)
* Update maximum tf version in setup + requirements

* - bump max version of tf version in launcher
- standardise tf version check

* update keras get_custom_objects  for tf>2.6

* bugfix: force black text in GUI file dialogs (linux)

* dssim loss - Move to stock tf.ssim function

* Update optimizer imports for compatibility

* fix logging for tf2.8

* Fix GUI graphing for TF2.8

* update tests

* bump requirements.txt versions

* Remove limit on nvidia-ml-py

* Graphing bugfixes
  - Prevent live graph from displaying if data not yet available

* bugfix: Live graph. Collect loss labels correctly

* fix: live graph - swallow inconsistent loss errors

* Bugfix: Prevent live graph from clearing during training

* Fix graphing for AMD
2022-05-02 14:30:43 +01:00
torzdf cf4b567cc6 Tensorflow 2.6 Support (#1182)
* lib.cli.launcher - Bump max tf version to 2.6

* Remove pathlib requirement

* Update requirements files

* Update setup.py

* bugfix - GUI: Supress errors when attempting to load previews in extract

* GUI: Suppress ptxas error messages for Windows
2021-09-05 02:25:12 +01:00
torzdf 6c439944cd Masks updates
- Add head centering support to training
  - Update helptext/tooltips
2021-05-17 18:20:08 +01:00
torzdf 29cfdaad46 Training startup updates
- Remove alignments path option from cli
  - Restructure training_data.py
  - Load alignment data from PNG header during first epoch
  - lib.image.read_image_batch - Add option to return metadata
  - lib.utils.get_image_paths - Add option for explicit extension
  - plugins.train.trainer._base - remove pre-cache alignments code
  - scripts.train - Check first image in training folders for metadata
  - Documentation
2021-03-21 23:29:31 +00:00
torzdf caa6e541c6 Training - Load Weights
- Enable the loading of previously trained encoders for new models.
2021-03-12 14:48:59 +00:00
torzdf a79b8c6905 Fixups
- Documentation - Update sphinx_requirements.txt
  - plugins.train.model - Add recursive keras model scanning
  - Change "freeze" option to "freeze-weights"
2021-03-12 09:48:00 +00:00
torzdf 52c10dcc35 scripts.train - Add freeze weights option
lib.config:
  - Better formatting for .ini files
  - Support multi-select options
gui - Add support for multi-select config items
2021-03-11 16:42:36 +00:00
torzdf 1d07dbaa5d train - Add option to output model summary and exit 2021-03-11 01:35:00 +00:00
torzdf 6ce09dbac6 Help text - Add "predicted" help tesk to convert mask option 2021-02-20 02:14:00 +00:00
torzdf 1f3e1b0656 Add Locale support for cli arguments 2021-02-18 23:53:01 +00:00
torzdf b1cfbe458c Update extracted faces to use PNG EXIF data (#1123)
Documentation
  - Update Usage.md, align.rst and image.rst
lib.image.py
  - read_image - Remove hash return, add metadata return
  - Remove read_image_hash functions
  - Add read_image_meta functios
  - Replace encode_image_with_hash with encode_image (to store metadata)
  - Add png meta reading and writing functions
  - Update Image Loaders/Savers to handle metadata rather than hashes
lib.training_data
  - Naming updates to remove references to hashes
lib.align.Alignments
  - Add versioning notes
  - Increment alignments version to 2.1
  - Deprecate hashing lookup functions
  - Replace filter_hashes with filter_faces
lib.align.detected_face
  - DetectedFace
    - Remove hash property
    - Add png header data serializing/deserializing functions
  - Mask
    - Add png header data serializing/deserializing functions
  - add update_legacy_png_header function to update png meta data
lib.cli.args - Deprecate alignments files for training
- plugins.train.trainer
  - Update alignments/mask code to read png header data
- scripts.convert
  - Aligned images folder - read data from png headers
- scripts.extract
  - Write png header information and no longer store hash of face
- tools.alignments
  - remove leftover-faces, merge and update-hashes jobs
  - Update jobs to use png meta data rather than hashes
- tools.manual
  - Update extract code to output png meta data and don't store hashes
  - Perform check on launch that tool is not pointing at a faces folder
tools.mask
  - Update to use png meta data
tools.sort
  - Update to use png meta data
2021-02-14 16:49:55 +00:00
torzdf a84462f4fa Training - Move disable warp option out of config 2021-02-11 10:24:03 +00:00
torzdf 3438198ac2 Merge branch 'master' into dev-tf2.4 2021-01-04 15:13:33 +00:00
torzdf 5a79a6f5e9 Bugfix - Warp to landmarks 2020-12-21 18:39:50 +00:00
torzdf 9182c56921 Re-instate tf2.2 minimum version (2.3 not on Conda for Linux) 2020-12-20 13:37:59 +00:00
torzdf 3b7535c732 Update to TF2.3 (#1100)
* Minimum requirements to tf2.3.
  - Handle upgrades for Windows users with tf2.2 installed by Pip
  - Handle windows upgrade from pip tf2.2
  - Explicitly install Cuda for Conda installs

* Update tensorflow errors api reference
* Suppress AutoGraph warning messages
* Update GUI Stats to work with tf2.3
* Fix live graph for tf2.3
* DSSIMObjective - autoGraph bugfix
* Update Travis test
2020-12-18 19:58:30 +00:00
Dominik Miszkiewicz 6279f85787 Update tensorflow to 2.4.0rc4 (#1086)
* Update tensorflow to 2.4.0rc1

* Update requirements_nvidia.txt

* Update requirements_nvidia.txt
2020-12-08 12:49:49 +00:00
torzdf 05018f6119 Extract - Increase area and move centering (#1095)
* Extract
  - Implement aligner re-feeding
  - Add extract type to pipeline.ExtractMedia
  - Add pose annotation to debug
* Convert
  - implement centering
  - remove usage of feed and reference face properties
  - Remove distributed option from convert
  - Force update of alignments file on legacy receive
* Train
  - Resize preview image to model output size
  - Force legacy centering if centering does not exist in model's state file
  - Enable training on legacy face sets

* Alignments Tool
  - Update draw to include head/pose
  - Remove DFL drop + linting
  - Remove remove-frames job
  - remove align-eyes option
  - Update legacy masks to new extract type
  - Exit if attempting to merge version 1.0 alignments files with version 2.0 alignments files
  - Re-generate thumbnails on legacy upgrade
* Mask Tool
  - Update for new extract + bugfix full frame
* Manual Tool
  - Update to new extraction method
   - Disable legacy alignments,
   - extract box bugfix
   - extract faces - size to 512 and center on head
* Preview Tool
  - Display based on model centering
* Sort Tool
  - Use alignments for sort by face

* lib.aligner
  - Add Pose Class
  - Add AlignedFace Class
  - center _MEAN_FACE on x
  - Add meta information with versioning to alignments file
  - lib.aligner.get_align_matrix to use landmarks not face
  - Refactor aligned faces in lib.faces_detect
* lib.logger
  - larger file log padding
* lib.config
  - Fix global changeable_items
* lib.face_filter
  - Use new extracted face images
* lib.image
  - bump thumbnail default size to 96px
2020-12-08 01:31:56 +00:00
torzdf 3dbbac5a98 Convert - Move scaling option exclusively to config 2020-10-20 14:36:43 +01:00
torzdf 0532c22dbf Bugfix: Extract - Fix Skip Saving Faces 2020-10-19 18:30:56 +01:00
torzdf b46c02d104 extract: S3FD - Convert to re-enable CPU 2020-10-18 23:35:50 +01:00
torzdf 42495e71bb Hold error messages open when launching GUI in Windows 2020-09-24 12:43:03 +01:00
torzdf 725d864975 Training - Move Mixed Precision to model config 2020-08-17 09:24:37 +01:00
torzdf 6b2aac65b1 Enable MTCNN for CPU extraction 2020-08-12 11:19:03 +01:00
torzdf d8557c1970 Faceswap 2.0 (#1045)
* Core Updates
    - Remove lib.utils.keras_backend_quiet and replace with get_backend() where relevant
    - Document lib.gpu_stats and lib.sys_info
    - Remove call to GPUStats.is_plaidml from convert and replace with get_backend()
    - lib.gui.menu - typofix

* Update Dependencies
Bump Tensorflow Version Check

* Port extraction to tf2

* Add custom import finder for loading Keras or tf.keras depending on backend

* Add `tensorflow` to KerasFinder search path

* Basic TF2 training running

* model.initializers - docstring fix

* Fix and pass tests for tf2

* Replace Keras backend tests with faceswap backend tests

* Initial optimizers update

* Monkey patch tf.keras optimizer

* Remove custom Adam Optimizers and Memory Saving Gradients

* Remove multi-gpu option. Add Distribution to cli

* plugins.train.model._base: Add Mirror, Central and Default distribution strategies

* Update tensorboard kwargs for tf2

* Penalized Loss - Fix for TF2 and AMD

* Fix syntax for tf2.1

* requirements typo fix

* Explicit None for clipnorm if using a distribution strategy

* Fix penalized loss for distribution strategies

* Update Dlight

* typo fix

* Pin to TF2.2

* setup.py - Install tensorflow from pip if not available in Conda

* Add reduction options and set default for mirrored distribution strategy

* Explicitly use default strategy rather than nullcontext

* lib.model.backup_restore documentation

* Remove mirrored strategy reduction method and default based on OS

* Initial restructure - training

* Remove PingPong
Start model.base refactor

* Model saving and resuming enabled

* More tidying up of model.base

* Enable backup and snapshotting

* Re-enable state file
Remove loss names from state file
Fix print loss function
Set snapshot iterations correctly

* Revert original model to Keras Model structure rather than custom layer
Output full model and sub model summary
Change NNBlocks to callables rather than custom keras layers

* Apply custom Conv2D layer

* Finalize NNBlock restructure
Update Dfaker blocks

* Fix reloading model under a different distribution strategy

* Pass command line arguments through to trainer

* Remove training_opts from model and reference params directly

* Tidy up model __init__

* Re-enable tensorboard logging
Suppress "Model Not Compiled" warning

* Fix timelapse

* lib.model.nnblocks - Bugfix residual block
Port dfaker
bugfix original

* dfl-h128 ported

* DFL SAE ported

* IAE Ported

* dlight ported

* port lightweight

* realface ported

* unbalanced ported

* villain ported

* lib.cli.args - Update Batchsize + move allow_growth to config

* Remove output shape definition
Get image sizes per side rather than globally

* Strip mask input from encoder

* Fix learn mask and output learned mask to preview

* Trigger Allow Growth prior to setting strategy

* Fix GUI Graphing

* GUI - Display batchsize correctly + fix training graphs

* Fix penalized loss

* Enable mixed precision training

* Update analysis displayed batch to match input

* Penalized Loss - Multi-GPU Fix

* Fix all losses for TF2

* Fix Reflect Padding

* Allow different input size for each side of the model

* Fix conv-aware initialization on reload

* Switch allow_growth order

* Move mixed_precision to cli

* Remove distrubution strategies

* Compile penalized loss sub-function into LossContainer

* Bump default save interval to 250
Generate preview on first iteration but don't save
Fix iterations to start at 1 instead of 0
Remove training deprecation warnings
Bump some scripts.train loglevels

* Add ability to refresh preview on demand on pop-up window

* Enable refresh of training preview from GUI

* Fix Convert
Debug logging in Initializers

* Fix Preview Tool

* Update Legacy TF1 weights to TF2
Catch stats error on loading stats with missing logs

* lib.gui.popup_configure - Make more responsive + document

* Multiple Outputs supported in trainer
Original Model - Mask output bugfix

* Make universal inference model for convert
Remove scaling from penalized mask loss (now handled at input to y_true)

* Fix inference model to work properly with all models

* Fix multi-scale output for convert

* Fix clipnorm issue with distribution strategies
Edit error message on OOM

* Update plaidml losses

* Add missing file

* Disable gmsd loss for plaidnl

* PlaidML - Basic training working

* clipnorm rewriting for mixed-precision

* Inference model creation bugfixes

* Remove debug code

* Bugfix: Default clipnorm to 1.0

* Remove all mask inputs from training code

* Remove mask inputs from convert

* GUI - Analysis Tab - Docstrings

* Fix rate in totals row

* lib.gui - Only update display pages if they have focus

* Save the model on first iteration

* plaidml - Fix SSIM loss with penalized loss

* tools.alignments - Remove manual and fix jobs

* GUI - Remove case formatting on help text

* gui MultiSelect custom widget - Set default values on init

* vgg_face2 - Move to plugins.extract.recognition and use plugins._base base class
cli - Add global GPU Exclude Option
tools.sort - Use global GPU Exlude option for backend
lib.model.session - Exclude all GPUs when running in CPU mode
lib.cli.launcher - Set backend to CPU mode when all GPUs excluded

* Cascade excluded devices to GPU Stats

* Explicit GPU selection for Train and Convert

* Reduce Tensorflow Min GPU Multiprocessor Count to 4

* remove compat.v1 code from extract

* Force TF to skip mixed precision compatibility check if GPUs have been filtered

* Add notes to config for non-working AMD losses

* Rasie error if forcing extract to CPU mode

* Fix loading of legace dfl-sae weights + dfl-sae typo fix

* Remove unused requirements
Update sphinx requirements
Fix broken rst file locations

* docs: lib.gui.display

* clipnorm amd condition check

* documentation - gui.display_analysis

* Documentation - gui.popup_configure

* Documentation - lib.logger

* Documentation - lib.model.initializers

* Documentation - lib.model.layers

* Documentation - lib.model.losses

* Documentation - lib.model.nn_blocks

* Documetation - lib.model.normalization

* Documentation - lib.model.session

* Documentation - lib.plaidml_stats

* Documentation: lib.training_data

* Documentation: lib.utils

* Documentation: plugins.train.model._base

* GUI Stats: prevent stats from using GPU

* Documentation - Original Model

* Documentation: plugins.model.trainer._base

* linting

* unit tests: initializers + losses

* unit tests: nn_blocks

* bugfix - Exclude gpu devices in train, not include

* Enable Exclude-Gpus in Extract

* Enable exclude gpus in tools

* Disallow multiple plugin types in a single model folder

* Automatically add exclude_gpus argument in for cpu backends

* Cpu backend fixes

* Relax optimizer test threshold

* Default Train settings - Set mask to Extended

* Update Extractor cli help text
Update to Python 3.8

* Fix FAN to run on CPU

* lib.plaidml_tools - typofix

* Linux installer - check for curl

* linux installer - typo fix
2020-08-12 10:36:41 +01:00