This change makes Lerna execute E2E tests across two cores. Some of the slower E2E suites (`appium`, `support`, `opencv` and `base-driver`) also run in parallel via Mocha's `--parallel` flag.
In Appium, the main culprit was `cli.e2e.spec.js`, which I split into four files: `args` (argument handling), `cli-driver` (`driver` command), `cli-plugin` (`plugin` command) and `local` (behavior when Appium is a dependency). This should run faster (in theory) since Mocha runs tests in parallel _per file_ and will expand to the available cores.
CI has limited cores, so this won't really help there--but it will speed up local development. When running tests locally, you may see various delays, as Mocha buffers the output of a single test file.
This is a continuation of the work I did in #17252.
- `@appium/gulp-plugins` does not use `tsc` and continues to use...whatever it uses.
- `@appium/doctor` now has generated declarations, though I didn't bother to do anything with the types
- rewrote swaths of build scripts:
- removed the `fix` and `lint` scripts from each workspace (package) since I don't think they get used and they are of limited value
- for tests and autoinstallation of extensions, added `ts-node` for on-the-fly compilation
- workspaces _can_ define their own `build` and `clean` scripts (both should be required if either is used). these do not currently get run when running `npm run dev`, but they do occur on a bare `npm install` from the monorepo root or a `npm run reinstall`.
- `npm run rebuild` does a fresh rebuild, but does not clean any `node_modules` dirs nor does it reinstall anything.
- removed `prepublishOnly` since `preversion` does the same thing
- `npm test` now runs `build`/`lint` in parallel
- postinstall script of `appium` ("autoinstall") script now must call `npm run build` if in a fresh dev environment. this is because a) lifecycle scripts of packages run before lifecycle scripts in the monorepo root, and b) `postinstall` runs before `prepare`. so there's really no way around it; even using `ts-node` fails because other modules depend on `@appium/support/index.js` which expects the pkg to be built.
Closes#17746Closes#17807
Run `npm run typedoc` to generate docs in `typedoc-docs` dir.
I thought I'd take a few mins and set this up. It was _mostly_ straightforward, except:
1. Figuring out the "entry point" for each package was difficult until I realized `.js` entry points are assumed to be compiled, and thus require a `sourceMappingUrl` reference. IMO this is probably a bug in TypeDoc
2. The `sourceMappingUrl` must reference an external sourcemap file; not an inline source map. If it's an inline source map, the system tries to `fs.open` a large data-url.
3. The babel config does not control whether external sourcemaps are created; `@babel/cli` does. So I needed to change those.
4. Practically speaking, I don't see any drawback to external sourcemaps _yet_
5. TypeDoc has poor support for cross-package type references in monorepos, and I had to add a plugin which implements a workaround
6. Per some suggestions in the docs, I modified the order of references in the root `tsconfig.json` to start with "the biggest one first"
I didn't modify any types or docstrings anywhere.
Jimp will not get upgraded so we need to do this to avoid the vuln in older versions of jpeg-js.
I also removed the relative deps from the monorepo root since they seemed to befoul the overrides. They are no longer needed due to workspaces (I think)
When any package here is consumed as a library, the production deps installed alongside it _must_ also include depenencies related to type information. If these packages _aren't_ included, consumers will not have the requisite declarations installed and will get incomplete type information--at best--or worse; compilation errors.
Putting the onus on the consumer to install these requisite types is both user-hostile _and_ can result in the incorrect versions of type declarations being installed.
To that end, I've moved the various type declaration deps to production deps of the packages in which they are used. Sometimes this is multiple packages (in the case of, say, `@types/teen_process`). This results in many types being removed from the monorepo root. Those that remain are for _dev dependencies only_. If a packages uses, say, `ws`, but only in its tests, then the types can remain in the monorepo root along with the rest of the common dev deps.
I also saw an unused `supports-color` in `appium` and removed it.
This uses the [boneskull/nodejs-production-test-action](https://github.com/boneskull/nodejs-production-test-action) GH action to run a smoke test across all packages.
The action does the following:
1. Runs `npm pack` on each package, writing the tarball to a temp dir. This tarball contains the package as a user would `npm install` it.
2. Changes into the temp dir and runs `npm install <tarball>` on each
3. Runs a specified script (in our case, `test:smoke`) in each installed package's dir in `<tmp_dir>/node_modules`
Because `devDependencies` will not be present, this should avoid a class of issues where production deps are miscategorized as dev deps.
The action should probably use a separate temp dir for each--in the case of a dependency being present but only because some other package depends on it--but it's not smart enough to figure out peer dependencies yet.
Resolves#16924
Because the `postinstall` script of `appium` needs functions in `@appium/support`, that module must be compiled _before_ its `postinstall` script runs. Prior to this PR, only the monorepo root would run a `postinstall` script to build the subpackages. Now, each package has a `prepare` script which runs its own build; this happens prior to any `postinstall`. This way, the `postinstall` script in `appium` can run successfully.
- Some types in `appium` were renamed
- Others avoid errors
- Some are minor tweaks to `tsconfig.json` files
# Conflicts:
# packages/appium/lib/extension/extension-config.js
# packages/appium/types/extension-manifest.ts
# packages/appium/types/extension.ts
# packages/base-driver/lib/jsonwp-proxy/proxy.js
# packages/support/lib/logging.js
# packages/support/test/unit/logger/logger-force.spec.js
Moved the stuff in `test/setup-babel` into `test/setup` and then removed some cruft in the `test:*` scripts of each package referencing `test/setup-babel`
This change removes `gulp` and replaces it with plain ol' `babel` and `mocha`.
## `appium`
- No gulp.
- `test/` has been re-organized. All test files now end in the extension `.spec.js`, which is a recognized convention and understood by editors/IDEs. The tests are further split into `unit` and `e2e` subdirs. This makes working with both `babel` and `mocha` easier.
- The tests are _not_ run against the transpiled code; code is now transpiled on-the-fly via `@babel/register`. This only affects `appium`.
- `commands-yml/validator.js` moved to `scripts/parse-yml-commands.js`. It has been rewritten as a CJS module. `commands-yml/` should not contain `.js` files.
- `test/setup.js` is a new test harness specific to Appium, which loads & configures `@babel/register`. Eventually, the contents of this file can be moved into the root monorepo's `test/setup.js`, but the rest of the packages do not need `@babel/register`, so we avoid the overhead.
## `@appium/gulp-plugins`
- Unfortunately, due to the new `npm test` strategy (see below), we cannot use the `nyan` reporter. Or rather, we _can_, but will not see the animation. So the reporter is now set to default to `spec`.
- Modified tests so they didn't overwrite the actual `build` dir with transpiled fixtures, which is what was happening. This was causing failures depending on the transpile/test order.
## All Packages
- All packages now have their own `build`, `test`, `test:e2e` and `dev` (just build + watch mode) npm scripts. This makes running a complete build easier, since each package can now provide its own appropriate command. This is _mostly_ just `gulp once` (where `gulp` was used before), except for `appium`, where `@babel/cli` is used directly.
- This is what happens when `npm test` is run from the monorepo (in order):
1. All packages are built (in parallel) by running `npm run build` in each folder (via `lerna`). See `pretest`
2. All code is linted directly via `eslint`.
3. All _unit tests_ are run w/ `npm test` in each folder, having assumed transpilation already occurred (if necessary).
- `npm run e2e-test` runs the `npm run test:e2e` in each folder. No transpilation occurs! Careful.
- All scripts are expected to live in `packages/*/scripts`, and this directory enforces CJS via ESLint. `postinstall.js` moved, as well as `check-npm-pack-files.js`. `generate-schema-declarations.mjs` moved from the root `scripts/` dir into `packages/appium/scripts/`; it's now CJS instead of ESM.
- I had profiled running various things with `lerna run --parallel`, but it turns out this is not faster. I'm not sure why, but I think it may have something to do with streaming STDOUT/STDERR, or potentially the overhead of running child processes outweighs the time it takes to actually run tests and builds.
- The behavior of `lerna run` now mimicks `lerna exec`, which was "run stuff in serial, streaming the output". The default behavior of `lerna run` is "run stuff in parallel, buffering the output". Buffered output sucks, and streaming the output from all tasks at once creates chaos. To override this behavior, you'll see `--parallel --concurrency=8 --prefix` for tasks that can truly be parallelized, like transpilation. The number 8 is the "maximum", and CI usually won't have more than 2 cores available anyway.
- When run in CI, Mocha runs with `--forbid-only`, which will fail if there's an `.only()` somewhere in the tests. Further, it forces color output.
## Etc.
- Updated Wallaby config, `.gitignore`, `.eslintignore`.
- Sorted `package.json` files
- Removed `watch` and `coverage` scripts which were broken anyway.
chore(appium): fix schema json path issues
- the path to the JSON schema is now correct: `lib/appium-config-schema.json`
- add `log-symbols` to devdeps for niceness. this is already a transitive dep
Jonathan Lipps