Summary: This PR adds support for configuration files (e.g., `.appiumrc.json`) and schema definitions. Command-line options _and_ configuration is now derived from the schema(s), for Appium _and_ its extensions (both plugins and drivers).
## Dependencies
### Development
- Added `through2` for a custom gulp task to convert `appium/lib/appium-config-schema.js` to a JSON file
- Added `json-schema-to-typescript` for converting aforementioned JSON file to a `.d.ts` file (which gets published)
### Production (applies to `appium` only)
- Added `libconfig` for multi-format config file loading, which is a smaller drop-in replacement for `cosmiconfig`
- Added `ajv` for schema support and validation
- Added `ajv-formats` to understand optional value formats (e.g., `ipv4`)
- Added `@sidvind/better-ajv-errors` which provides nice error output on the CLI if the config file is invalid
- Added `@oclif/errors` for better error output on the CLI if _arguments_ are invalid
## New Modules of Note
- `lib/schema.js` consumes `ajv` to create and modify a schema
- It exposes a handful of public API functions, so that consumers can query the schemas without actually running Appium.
- The flow is basically this: read the base schema (`appium-config-schema.js`). When extensions load and have schemas, add those to a pot. When `finalizeSchema()` is called, throw the base schema into the pot as well, stir, and add to singleton `ajv` instance. Now we can validate against the stew.
- The schema for an extension is read and registered via `readExtensionSchema()`
- Provides a way to get at default (for showing those "non-default" options) and `flattenSchema()` which squishes the finalized schema into a single `Record` key/value object, for use by the CLI. This may make sense to move into `schema-args.js`
- Provides access to ajv-and-JSON-schema-defined _formatters_ (which are basically just validation functions). For example, if we want any string, it's just a `string`. But if we want the string to match a regexp, that's a formatter. (e.g., the `hostname` type)
- lib/config-file.js` consumes `lilconfig` to read config files, validate them against the schema, and provide the result as CLI args
- Handles conversion between kebab-case and camelCase
- Contains `formatErrors()` which will be called after validation of a config file fails, which shows _exactly_ where the config file is wrong (usually).
- `lib/cli/schema-args.js` which serves as an adapter between the schema and `argparse`
- Translates between JSON schema and `argparse`, which...well, I did what I could. It's pretty close.
- Wraps the formatters as mentioned above, and provides custom ones.
- We cannot, unfortunately, wholly use schema validation against CLI args. The flag names are different, the structure is different, there's no actual _file_ to compare against, and `argparse` does not help. We _can_ use anything `argparse` natively provides, in addition to `ajv` formatters, and any custom ones (e.g., allowable number ranges).
- Will likely end up adding more formatters corresponding to various JSON schema features later. We _may_ be able to use _more_ of Ajv to do this (e.g., reuse its internal validators, if they are public APIs).
- `lib/appium-config-schema.js` which is the root schema. Extensions add to this before it is registered & compiled by `ajv`.
- The basic shape is that there are three top-level props, `server`, `driver`, and `plugin`. `server` contains all of the server options as you'd expect, and `driver` and `plugin` will be filled in by extensions. Each will have properties corresponding to the extension name. You can find examples in `sample-code` and the test fixtures
- There's a gulp task that takes this file, rewrites it as JSON, then pipes it into another tool which outputs a `.d.ts` file, nicely described using all of the metadata from the schema. This will be useful for mainly contributors, extension authors, and consumers of Appium (though not end-users).
- `lib/ext-config-io.js`
- A new module which handles reading/writing of YAML extension config files only.
- This class is a singleton.
- Some logic pulled out of `extension-config.js` and into here
- Previously, we would read/write this file multiple times (once per subclass of `ExtensionConfig`)
- This module uses a naive dirty-tracker to persist the extension data automatically.
- Subclasses of `ExtensionConfig` only have access to their "portion" of the config file; drivers can't see plugins and vice-versa.
- `DRIVER_TYPE` and `PLUGIN_TYPE` originate here, mainly to avoid a circular dependency
## Modifications
- The single source of truth for the CLI _and_ config file options is the schema, with the caveats that:
- Command-line arguments (for `appium server` are effectively a flat list, but the schema/config file is not.
- Extension-specific configuration lives in an extension-type-specific property (`driver` or `plugin`) in the config file
- Under each (`driver` or `plugin`), each extension may have a configuration section corresponding to the extension name
- The shape of the extension-specific options _can_ be defined by a schema (and referenced in the `appium.schema` prop of the extension's `package.json`)
- If an extension does not provide a schema, no validation will occur on any options found here -- TODO needs tests
- This does _not_ affect _CLI commands_ added by extensions; only options for `server`.
- The `--appium-home` flag is now removed, as we can't find extensions without `APPIUM_HOME`, which means we can't parse arguments (because extensions may have added some).
- `--plugin-args` and `--driver-args` are removed. These are replaced by individual flags for each argument, defined by the extension in its schema, e.g., `--driver-fake-host=127.0.0.1`
- Extensions can still define argument "constraints" using `argConstraints`, but this is considered deprecated
- `lib/main.js`
- Now exports a few public APIs
- split into functions `init` and `main`. `init` is intended to be run via a consumer, which does what the old `main` did up to starting the server. This bisection is not perfect, as `init` still does some CLI-related things; it needs to be further broken up.
- `driverConfig.read()` and `pluginConfig.read()` are now called in `lib/cli/parser.js`.
- Merges the CLI args, config file args, and defaults, in that order of precedence.
- `lib/grid-register.js`
- Supports embedded configuration instead of just a path to a JSON file.
- Not clear to me if `nodeconfig` is a thing we need to keep?
- `lib/extension-config.js`
- Given `APPIUM_HOME` is where we keep the extensions, this file now defines `APPIUM_HOME`. It is exported from the `appium` package.
- Add `getSchemaProblems()` method, which will read/validate/register the schema for an extension, if one exists.
- Added many type annotations
- `read()` and `write()` now delegate to the singleton `ExtConfigIO` instnace
- Re-exports `DRIVER_TYPE` and `PLUGIN_TYPE`
- `lib/driver-config.js`
- **Fix** the case where two drivers attempt to use the same `automationName`. It appears this was intended, but did not work correctly. This involved overloading `read()`.
- `lib/plugin-config.js`
- Literally nothing, which is neat.
- `lib/config.js`
- Remove unused validations (now occur via schema validation or are extension-specific)
- Rename `getNonDefaultArgs` to `getNonDefaultServerArgs` to be more specific
- Rewrote `getNonDefaultServerArgs`, which became much more complicated due to adding a config file and schema defaults to the mix. Flexed my lodash-fu, which is still kind of crappy because I'm better with `lodas/fp`.
- `lib/cli/parser.js`
- Created a wrapper for `argparse`: class `ArgParser`, which replaces the monkeypatching we were doing here.
- This class also stores extension-specific args in a namespace. For example, `--driver-foo-bar=baz` would normally be stored in prop `driverFooBar`, but this moves it into `driver: {foo: {bar: 'baz'}}`, which will be provided to its associated extension
- `addExtensionsToParser` renamed to `addExtensionCommandsToParser` which is more specific.
- Removed `getDefaultServerArgs` since those now come from the schema.
- Removed "shared args" stuff
- `getParser` is now async, and constructs an `ArgParser`. By necessity, the extension config YAML is read here (which is still wonky because we have to call `read()` twice; it only actually ends up reading once)
- `getParser` also _finalizes_ the schema, which means "katamari-up the base schema and all extension schemas, then tell ajv about the result"
- `lib/cli/parser-helpers.js`
- Allow security features to just be an array (as allowed in schema)
- TODO: needs test
- `lib/cli/extension.js`
- Use `APPIUM_HOME` exported from `lib/extension-config` instead of `args.appiumHome`
- Avoids re-instantiating `ExtensionConfig` subclasses if not needed
- `lib/cli/args.js`
- Remove all server argument definitions in lieu of schema; leave a few that do not make sense in a config file (like `--config`, which determines the location of the config file!)
- Server args are now pulled from the schema via a call to `toParserArgs()` with overrides for various validation functions (which are confusingly "types" in `argparse` parlance). Overrides can probably be removed or relocated to `schema-args`
- Instantiates the `DriverConfig` and `PluginConfig` instances, which are exported. These might want to move somewhere else
- Tests
- Added unit tests for `extension-config`, `grid-register` which were missing
- Added unit tests for new `schema` and `schema-args` and `config-file` modules
- Added e2e test for config file handling
- Other necessary modifications
- Fix path to `fake-driver` in `driver-e2e-specs`
- Removed stuff from `parser-specs` which no longer applies
## Other
- Added `scripts/generate-schema-declarations.js` which uses `json-schema-to-typescript` to create `appium-config.d.ts`
- Added config file samples in `.json` and `.yaml` formats
- Added custom gulp task to `appium/gulpfile.js` to generate schema in JSON format, since its "single source of truth" is actually just a big JS object
- Moved `appium` CLI-related tests into `test/cli`
- Added some declarations into `packages/appium/types`; the schema-related one is generated by the script mentioend above, but `types.d.ts` is just some stuff that was not expressible in a docstring
FOSSA was giving us a bunch of warnings about copyleft licenses, but these licenses are only present in our examples. This configuration file will tell FOSSA to ignore that stuff.
FOSSA recognizes `examples/` as something it should skip, but not `sample-code`. It's easier to rename the dir than it is to negotiate a PR with FOSSA. Why they don't allow the config file to just ignore directories is beyond me...
* add initial python sample
* add all of ios and android tests
* apply formatter
* add >=0.28 to avoid version error
* tweak versions of default OS
* convert some tests to unittest based ones
* use single quote mainly
* use {} format style for string
* use format syntax
* apply format and tweak assertions
* use is None
* use assert raise
* speficy a number of elements
* tweak exception
* move app path to helper
* tweak assertion conditions
* remove .close since with open() close file automatically
Christopher Hiller