mirror of
https://github.com/colbymchenry/codegraph.git
synced 2026-10-21 08:47:04 +08:00
fix(swift): a call written after try! links what it calls on Windows (#2683)
The Windows kernels read the `!` of every `try!` as a logical not, so `try! Message2(data: d)` became a call of `! Message2` that never resolves, and a call through a receiver lost it: `try! Settings.Defaults.load()` was a bare `load`. The cause is the grammar's C, not the walker. tree-sitter-swift 0.7.3's external scanner steps aside for the `!` of `try!` through a 64-bit mask whose entry is built as `1UL << FAKE_TRY_BANG`, bit 32. `unsigned long` is 32 bits on Windows and MSVC folds the overflowing shift to 0 (warning C4293), so the scanner never stepped aside: `try` parsed alone, then a prefix `!` on the callee. macOS and Linux have 64-bit longs, and clang's wasm32 build folds the shift to bit 31 (`_directive_endif`, valid at every expression start), so the wasm and the other kernels have always read `try!` right. The kernel now compiles the crate tarball's src/ itself, from codegraph-kernel/grammars/swift, with upstream 0.7.4's one-line fix (`1ULL`), as it already does for the Kotlin, Lua, Scala and Dart grammars. The patch and its provenance are in docs/grammars/tree-sitter-swift.patch and .md. parser.c is unchanged, so the node-kind and field tables still match the wasm, which is untouched. TortureTry.swift pins calls after try!, try? and try on both engines. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
Colby Mchenry
@@ -198,6 +198,7 @@ and adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html).
|
||||
- In JavaScript and TypeScript, an import written from a jsconfig or tsconfig `baseUrl`, like `import MainLayout from 'layout/MainLayout'` under `"baseUrl": "src"`, or through a catch-all `"paths": { "*": ["./src/*"] }`, now links to the file it names, read with the config of the app the importing file belongs to, and so does an import of a single name like `'config'` or `'themes'`. Create React App and Vite admin templates such as Berry, Mantis and MatX write nearly every import this way. Before, each one was taken for an npm package, so it linked nothing, its names were matched by name alone, which could reach a same-named component in another app of the same repository, and a React Router route whose page loads through `lazy(() => import('views/dashboard/Default'))` pointed at that loader instead of the page and its layout. A package the app's `package.json` declares, or a Node built-in, never links to a project module of the same name, only to type declarations the project keeps for it, and only a file the index holds counts, never a folder or something in `node_modules`. `codegraph sync` and the file watcher link such an import too when the file it names is added later or gains what it imports. Re-index JavaScript and TypeScript projects after upgrading.
|
||||
- `codegraph sync` and the file watcher now relink an import that reaches a module through a barrel file passing it on under another name, like `import { MatxLoading } from './components'` when `components/index.js` says `export { default as MatxLoading } from './MatxLoading'`, once an edit gives that module what is imported. Before, the import and every call through it stayed unlinked, or stayed on something else in the project with the same name, until the importing file itself changed or the project was indexed again.
|
||||
- In JavaScript and TypeScript projects, a folder named `build` inside `src/` is now indexed for its TypeScript, JSX, Vue, Svelte and Astro files, so a feature kept there, like the `src/build/` pages of luci-go's milo, shows up in `codegraph_explore` and the routes that lazily load its pages link to them. Before, every folder named `build` was skipped as build output. The JavaScript, declaration and other files such a folder holds are still left out, since a build may have written them, and so is every `build` folder outside `src/`.
|
||||
- On Windows, a Swift call written after `try!`, like `try! Message2(data: data)` or `try! Settings.Defaults.load()`, now links to what it calls, as it already did on macOS and Linux. Before, the `!` was read as a logical not, so the call linked to nothing, and a call made through a type lost the type it was made on. Re-index Swift projects on Windows after upgrading.
|
||||
|
||||
## [1.6.2] - 2026-10-03
|
||||
|
||||
|
||||
@@ -0,0 +1,56 @@
|
||||
// Calls written after `try!`, `try?` and `try`. `try!` is the shape the
|
||||
// Windows (MSVC) build of the kernel's grammar used to misparse: its scanner
|
||||
// read the `!` as a logical not, so `try! Message2(data: data)` came out as a
|
||||
// call of `!Message2` (a reference named `! Message2`, at the `!`).
|
||||
import Foundation
|
||||
|
||||
let globalStore = try! Store(path: "global")
|
||||
let globalLoader = try? Loader.make()
|
||||
|
||||
final class Repository {
|
||||
static let shared = try! Repository(path: "shared")
|
||||
let decoder = try! JSONDecoder.make(strategy: .iso8601)
|
||||
lazy var cache: Cache = try! Cache.open(named: "repo")
|
||||
var current: Message2? = try? Message2(data: Data())
|
||||
|
||||
init(path: String) throws {}
|
||||
|
||||
func load(data: Data?) throws -> Message2 {
|
||||
let msg2 = try! Message2(data: data!)
|
||||
let msg3 = try? Message2(data: data!)
|
||||
let msg4 = try Message2(data: data!)
|
||||
let user = try! JSONDecoder().decode(User.self, from: data!)
|
||||
let parsed = try! parser.parse(data!)
|
||||
let settings = try! Settings.Defaults.load(from: "defaults")
|
||||
let realm = try! Realm()
|
||||
_ = (msg3, msg4, user, parsed, settings, realm)
|
||||
try! FileManager.default.removeItem(atPath: "/tmp/x")
|
||||
try! self.flush()
|
||||
try! validate(msg2)
|
||||
try !isReady()
|
||||
use(try! Foo(), try? Bar.make(), try Baz.build())
|
||||
if let cached = try? Cache.open(named: "x") {
|
||||
cached.touch()
|
||||
}
|
||||
guard let fresh = try? decodeFresh(data) else {
|
||||
return try! Message2(data: Data())
|
||||
}
|
||||
_ = fresh
|
||||
return try! Message2(data: data!)
|
||||
}
|
||||
|
||||
func refresh() async throws {
|
||||
let page = try! await client.fetch(page: 1)
|
||||
let next = try? await Client.shared.fetch(page: 2)
|
||||
try await store.save(page, next)
|
||||
let items = try! items(in: page).map { try! Item(raw: $0) }
|
||||
_ = items
|
||||
}
|
||||
|
||||
func flush() throws {}
|
||||
func validate(_ m: Message2) throws {}
|
||||
func isReady() throws -> Bool { true }
|
||||
func use(_ a: Any?, _ b: Any?, _ c: Any?) {}
|
||||
func decodeFresh(_ d: Data?) throws -> Message2 { try Message2(data: d!) }
|
||||
func items(in page: Page) throws -> [String] { [] }
|
||||
}
|
||||
@@ -17,7 +17,8 @@
|
||||
* breaking, value-ref targets incl. the declared-then-assigned
|
||||
* assignment-prune case the swift-nio sweep caught, SWIFT_SPEC fn-refs with
|
||||
* the label-forward skip and #selector shapes) and its CRLF variant (derived
|
||||
* in-memory — #1329).
|
||||
* in-memory — #1329); TortureTry.swift: calls written after `try!`, `try?`
|
||||
* and `try`, which the Windows kernels' grammar build misparsed.
|
||||
*
|
||||
* The full-repo sweep lives in scripts/kernel-parity.mjs (Alamofire/vapor/
|
||||
* swift-nio, --max-deferral 0.3 — swift error incidence is structurally
|
||||
@@ -134,6 +135,43 @@ describe.skipIf(!kernelBuilt)('kernel Swift extraction parity', () => {
|
||||
assertParity('src/EditorStore.swift (crlf)', source.replace(/\n/g, '\r\n'), 5);
|
||||
});
|
||||
|
||||
// `try!` before a call. The grammar's scanner keeps the `!` for `try!` with
|
||||
// a bit mask built as `1UL << 32`, which overflows where `unsigned long` is
|
||||
// 32 bits: the MSVC-built Windows kernels read the `!` as a logical not, so
|
||||
// `try! Message2(data: d)` was a call of `! Message2` (at the `!`) and
|
||||
// `try! Settings.Defaults.load()` lost its receiver. The kernel now compiles
|
||||
// the grammar with `1ULL`. Fails on the old Windows kernel only — macOS and
|
||||
// Linux have 64-bit longs.
|
||||
it('a call after try!, try? or try: same refs on both arms, LF and CRLF', () => {
|
||||
const source = fs.readFileSync(path.join(FIXTURE_DIR, 'TortureTry.swift'), 'utf8');
|
||||
assertParity('fixtures/TortureTry.swift', source, 10);
|
||||
assertParity('fixtures/TortureTry.swift (crlf)', source.replace(/(?<!\r)\n/g, '\r\n'), 10);
|
||||
|
||||
process.env.CODEGRAPH_KERNEL_LANGS = 'all';
|
||||
const refs = tryKernelExtract('fixtures/TortureTry.swift', source, 'swift')!.unresolvedReferences;
|
||||
const lines = source.split(/\r?\n/);
|
||||
// Where `token` starts on the first line containing `text`.
|
||||
const at = (text: string, token: string): { line: number; column: number } => {
|
||||
const i = lines.findIndex((l) => l.includes(text));
|
||||
expect(i, `fixture line "${text}"`).toBeGreaterThanOrEqual(0);
|
||||
return { line: i + 1, column: lines[i]!.indexOf(token, lines[i]!.indexOf(text)) };
|
||||
};
|
||||
const expectRef = (referenceName: string, referenceKind: string, where: { line: number; column: number }) =>
|
||||
expect(refs).toContainEqual(expect.objectContaining({ referenceName, referenceKind, ...where }));
|
||||
|
||||
expectRef('Message2', 'calls', at('let msg2 = try! Message2(data: data!)', 'Message2'));
|
||||
expectRef('Realm', 'calls', at('try! Realm()', 'Realm'));
|
||||
expectRef('Repository', 'calls', at('static let shared = try! Repository', 'Repository'));
|
||||
expectRef('JSONDecoder.make', 'calls', at('try! JSONDecoder.make(', 'JSONDecoder'));
|
||||
expectRef('Settings.Defaults.load', 'calls', at('try! Settings.Defaults.load', 'Settings'));
|
||||
expectRef('Settings', 'references', at('try! Settings.Defaults.load', 'Settings'));
|
||||
expectRef('FileManager', 'references', at('try! FileManager.default', 'FileManager'));
|
||||
expectRef('Item', 'calls', at('try! Item(raw: $0)', 'Item'));
|
||||
// No callee on a `try!` line keeps the `!`.
|
||||
const tryBangLines = new Set(lines.flatMap((l, i) => (l.includes('try!') ? [i + 1] : [])));
|
||||
expect(refs.filter((r) => tryBangLines.has(r.line) && r.referenceName.includes('!'))).toEqual([]);
|
||||
});
|
||||
|
||||
it('files with parse errors defer to the wasm extractor (recovery is encoding-dependent)', () => {
|
||||
// A NEW-only regression construct (`#if` between enum cases — the swift
|
||||
// checklist's grammar-bump delta 5) — errors on the 0.7.3 grammar.
|
||||
|
||||
Generated
-11
@@ -67,7 +67,6 @@ dependencies = [
|
||||
"tree-sitter-r",
|
||||
"tree-sitter-ruby",
|
||||
"tree-sitter-rust",
|
||||
"tree-sitter-swift",
|
||||
"tree-sitter-typescript",
|
||||
]
|
||||
|
||||
@@ -620,16 +619,6 @@ dependencies = [
|
||||
"tree-sitter-language",
|
||||
]
|
||||
|
||||
[[package]]
|
||||
name = "tree-sitter-swift"
|
||||
version = "0.7.3"
|
||||
source = "registry+https://github.com/rust-lang/crates.io-index"
|
||||
checksum = "fe36052155b9dd69ca82b3b8f1b4ccfb2d867125ac1a4db1dd7331829242668c"
|
||||
dependencies = [
|
||||
"cc",
|
||||
"tree-sitter-language",
|
||||
]
|
||||
|
||||
[[package]]
|
||||
name = "tree-sitter-typescript"
|
||||
version = "0.23.2"
|
||||
|
||||
@@ -41,11 +41,10 @@ tree-sitter-ruby = "=0.23.1"
|
||||
# php: the walker calls LANGUAGE_PHP (the full HTML-interleaving variant the
|
||||
# wasm ships) — NEVER LANGUAGE_PHP_ONLY, which errors on leading HTML.
|
||||
tree-sitter-php = "=0.24.2"
|
||||
# swift: the vendored wasm is built from THIS crate's tarball src/ (the tag's
|
||||
# checked-in parser.c is an older ABI-14 generation that can never sha-match;
|
||||
# grammar.json rules are JSON-equal — swift checklist header). parser.c is
|
||||
# ~20MB generated — expect slow compiles.
|
||||
tree-sitter-swift = "=0.7.3"
|
||||
# swift: NOT a crate dep — the kernel compiles tree-sitter-swift 0.7.3's
|
||||
# tarball src/ itself (grammars/swift, build.rs), with a one-line scanner fix
|
||||
# the crate's own build lacks on Windows (`try!`; docs/grammars/
|
||||
# tree-sitter-swift.md). The vendored wasm is built from the same tarball.
|
||||
# r: the vendored wasm IS r-lib v1.2.0 and this crate's tarball ships both
|
||||
# generated artifacts sha-identical to the tag (parser.c 6221657347…,
|
||||
# scanner.c 1209d11076… — r checklist §Grammar prep). ABI 14 (the tag
|
||||
@@ -59,7 +58,7 @@ tree-sitter-r = "=1.2.0"
|
||||
tree-sitter-luau = "=1.2.0"
|
||||
|
||||
# tree-sitter-language: the version-agnostic LanguageFn shim for the vendored
|
||||
# kotlin grammar C (see build.rs — no kotlin crate dep is possible).
|
||||
# grammar C (kotlin, lua, scala, dart, swift — see build.rs).
|
||||
tree-sitter-language = "0.1"
|
||||
|
||||
# Stack-bounds queries for the walker stack guard (src/stack.rs, #1581):
|
||||
|
||||
@@ -86,4 +86,30 @@ fn main() {
|
||||
dart.flag_if_supported("-utf-8"); // msvc
|
||||
dart.compile("tree-sitter-dart");
|
||||
println!("cargo:rerun-if-changed=grammars/dart");
|
||||
|
||||
// Swift grammar — vendored C (fifth vendored-grammar-C language): the
|
||||
// crates.io tree-sitter-swift 0.7.3 tarball's src/, the same sources the
|
||||
// vendored wasm is built from (swift checklist §Grammar prep), with
|
||||
// docs/grammars/tree-sitter-swift.patch applied to scanner.c. The crate's
|
||||
// scanner builds `1UL << FAKE_TRY_BANG` (bit 32) into a constant table;
|
||||
// where `unsigned long` is 32 bits that overflows — MSVC folds it to 0,
|
||||
// so the Windows kernels read the `!` of every `try!` as a logical not
|
||||
// and `try! Foo()` called `!Foo`. Upstream 0.7.4 writes `1ULL`; a bump to
|
||||
// a fixed release can return to the crate (and the wasm moves with it).
|
||||
// parser.c d3edff6effe31b9a507f496577407987343b101b23eb7bee7a9b050e8ab5d27a
|
||||
// scanner.c f3d6271d64f58c39eed544104a70ca2cf9ecbf80c5d900620f1afd38836542cb
|
||||
// scanner.c 8acafef098ea4a52ebfc4cdf2fe26706113a22ac8ba2faf7d952755ff350cad6 (patched)
|
||||
// Flags crib the tarball's own bindings/rust/build.rs (c11). parser.c is
|
||||
// ~20 MB.
|
||||
let mut swift = cc::Build::new();
|
||||
swift.std("c11");
|
||||
swift.include("grammars/swift");
|
||||
swift.file("grammars/swift/parser.c");
|
||||
swift.file("grammars/swift/scanner.c");
|
||||
swift.flag_if_supported("-Wno-unused-parameter");
|
||||
swift.flag_if_supported("-Wno-unused-but-set-variable");
|
||||
swift.flag_if_supported("-Wno-trigraphs");
|
||||
swift.flag_if_supported("-utf-8"); // msvc
|
||||
swift.compile("tree-sitter-swift");
|
||||
println!("cargo:rerun-if-changed=grammars/swift");
|
||||
}
|
||||
|
||||
@@ -0,0 +1,21 @@
|
||||
MIT License
|
||||
|
||||
Copyright (c) 2021 alex-pinkus
|
||||
|
||||
Permission is hereby granted, free of charge, to any person obtaining a copy
|
||||
of this software and associated documentation files (the "Software"), to deal
|
||||
in the Software without restriction, including without limitation the rights
|
||||
to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
|
||||
copies of the Software, and to permit persons to whom the Software is
|
||||
furnished to do so, subject to the following conditions:
|
||||
|
||||
The above copyright notice and this permission notice shall be included in all
|
||||
copies or substantial portions of the Software.
|
||||
|
||||
THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
|
||||
IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
|
||||
FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
|
||||
AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
|
||||
LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
|
||||
OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
|
||||
SOFTWARE.
|
||||
File diff suppressed because it is too large
Load Diff
@@ -0,0 +1,931 @@
|
||||
#include "tree_sitter/parser.h"
|
||||
#include <string.h>
|
||||
#include <wctype.h>
|
||||
|
||||
#define TOKEN_COUNT 33
|
||||
|
||||
enum TokenType {
|
||||
BLOCK_COMMENT,
|
||||
RAW_STR_PART,
|
||||
RAW_STR_CONTINUING_INDICATOR,
|
||||
RAW_STR_END_PART,
|
||||
IMPLICIT_SEMI,
|
||||
EXPLICIT_SEMI,
|
||||
ARROW_OPERATOR,
|
||||
DOT_OPERATOR,
|
||||
CONJUNCTION_OPERATOR,
|
||||
DISJUNCTION_OPERATOR,
|
||||
NIL_COALESCING_OPERATOR,
|
||||
EQUAL_SIGN,
|
||||
EQ_EQ,
|
||||
PLUS_THEN_WS,
|
||||
MINUS_THEN_WS,
|
||||
BANG,
|
||||
THROWS_KEYWORD,
|
||||
RETHROWS_KEYWORD,
|
||||
DEFAULT_KEYWORD,
|
||||
WHERE_KEYWORD,
|
||||
ELSE_KEYWORD,
|
||||
CATCH_KEYWORD,
|
||||
AS_KEYWORD,
|
||||
AS_QUEST,
|
||||
AS_BANG,
|
||||
ASYNC_KEYWORD,
|
||||
CUSTOM_OPERATOR,
|
||||
HASH_SYMBOL,
|
||||
DIRECTIVE_IF,
|
||||
DIRECTIVE_ELSEIF,
|
||||
DIRECTIVE_ELSE,
|
||||
DIRECTIVE_ENDIF,
|
||||
FAKE_TRY_BANG
|
||||
};
|
||||
|
||||
#define OPERATOR_COUNT 20
|
||||
|
||||
const char* OPERATORS[OPERATOR_COUNT] = {
|
||||
"->",
|
||||
".",
|
||||
"&&",
|
||||
"||",
|
||||
"??",
|
||||
"=",
|
||||
"==",
|
||||
"+",
|
||||
"-",
|
||||
"!",
|
||||
"throws",
|
||||
"rethrows",
|
||||
"default",
|
||||
"where",
|
||||
"else",
|
||||
"catch",
|
||||
"as",
|
||||
"as?",
|
||||
"as!",
|
||||
"async"
|
||||
};
|
||||
|
||||
enum IllegalTerminatorGroup {
|
||||
ALPHANUMERIC,
|
||||
OPERATOR_SYMBOLS,
|
||||
OPERATOR_OR_DOT,
|
||||
NON_WHITESPACE
|
||||
};
|
||||
|
||||
const enum IllegalTerminatorGroup OP_ILLEGAL_TERMINATORS[OPERATOR_COUNT] = {
|
||||
OPERATOR_SYMBOLS, // ->
|
||||
OPERATOR_OR_DOT, // .
|
||||
OPERATOR_SYMBOLS, // &&
|
||||
OPERATOR_SYMBOLS, // ||
|
||||
OPERATOR_SYMBOLS, // ??
|
||||
OPERATOR_SYMBOLS, // =
|
||||
OPERATOR_SYMBOLS, // ==
|
||||
NON_WHITESPACE, // +
|
||||
NON_WHITESPACE, // -
|
||||
OPERATOR_SYMBOLS, // !
|
||||
ALPHANUMERIC, // throws
|
||||
ALPHANUMERIC, // rethrows
|
||||
ALPHANUMERIC, // default
|
||||
ALPHANUMERIC, // where
|
||||
ALPHANUMERIC, // else
|
||||
ALPHANUMERIC, // catch
|
||||
ALPHANUMERIC, // as
|
||||
OPERATOR_SYMBOLS, // as?
|
||||
OPERATOR_SYMBOLS, // as!
|
||||
ALPHANUMERIC // async
|
||||
};
|
||||
|
||||
const enum TokenType OP_SYMBOLS[OPERATOR_COUNT] = {
|
||||
ARROW_OPERATOR,
|
||||
DOT_OPERATOR,
|
||||
CONJUNCTION_OPERATOR,
|
||||
DISJUNCTION_OPERATOR,
|
||||
NIL_COALESCING_OPERATOR,
|
||||
EQUAL_SIGN,
|
||||
EQ_EQ,
|
||||
PLUS_THEN_WS,
|
||||
MINUS_THEN_WS,
|
||||
BANG,
|
||||
THROWS_KEYWORD,
|
||||
RETHROWS_KEYWORD,
|
||||
DEFAULT_KEYWORD,
|
||||
WHERE_KEYWORD,
|
||||
ELSE_KEYWORD,
|
||||
CATCH_KEYWORD,
|
||||
AS_KEYWORD,
|
||||
AS_QUEST,
|
||||
AS_BANG,
|
||||
ASYNC_KEYWORD
|
||||
};
|
||||
|
||||
const uint64_t OP_SYMBOL_SUPPRESSOR[OPERATOR_COUNT] = {
|
||||
0, // ARROW_OPERATOR,
|
||||
0, // DOT_OPERATOR,
|
||||
0, // CONJUNCTION_OPERATOR,
|
||||
0, // DISJUNCTION_OPERATOR,
|
||||
0, // NIL_COALESCING_OPERATOR,
|
||||
0, // EQUAL_SIGN,
|
||||
0, // EQ_EQ,
|
||||
0, // PLUS_THEN_WS,
|
||||
0, // MINUS_THEN_WS,
|
||||
// 1ULL, as upstream has it from 0.7.4: 1UL << 32 overflows where unsigned
|
||||
// long is 32 bits (MSVC), and `try!` lost its `!`. (codegraph patch)
|
||||
1ULL << FAKE_TRY_BANG, // BANG,
|
||||
0, // THROWS_KEYWORD,
|
||||
0, // RETHROWS_KEYWORD,
|
||||
0, // DEFAULT_KEYWORD,
|
||||
0, // WHERE_KEYWORD,
|
||||
0, // ELSE_KEYWORD,
|
||||
0, // CATCH_KEYWORD,
|
||||
0, // AS_KEYWORD,
|
||||
0, // AS_QUEST,
|
||||
0, // AS_BANG,
|
||||
0, // ASYNC_KEYWORD
|
||||
};
|
||||
|
||||
#define RESERVED_OP_COUNT 31
|
||||
|
||||
const char* RESERVED_OPS[RESERVED_OP_COUNT] = {
|
||||
"/",
|
||||
"=",
|
||||
"-",
|
||||
"+",
|
||||
"!",
|
||||
"*",
|
||||
"%",
|
||||
"<",
|
||||
">",
|
||||
"&",
|
||||
"|",
|
||||
"^",
|
||||
"?",
|
||||
"~",
|
||||
".",
|
||||
"..",
|
||||
"->",
|
||||
"/*",
|
||||
"*/",
|
||||
"+=",
|
||||
"-=",
|
||||
"*=",
|
||||
"/=",
|
||||
"%=",
|
||||
">>",
|
||||
"<<",
|
||||
"++",
|
||||
"--",
|
||||
"===",
|
||||
"...",
|
||||
"..<"
|
||||
};
|
||||
|
||||
static bool is_cross_semi_token(enum TokenType op) {
|
||||
switch(op) {
|
||||
case ARROW_OPERATOR:
|
||||
case DOT_OPERATOR:
|
||||
case CONJUNCTION_OPERATOR:
|
||||
case DISJUNCTION_OPERATOR:
|
||||
case NIL_COALESCING_OPERATOR:
|
||||
case EQUAL_SIGN:
|
||||
case EQ_EQ:
|
||||
case PLUS_THEN_WS:
|
||||
case MINUS_THEN_WS:
|
||||
case THROWS_KEYWORD:
|
||||
case RETHROWS_KEYWORD:
|
||||
case DEFAULT_KEYWORD:
|
||||
case WHERE_KEYWORD:
|
||||
case ELSE_KEYWORD:
|
||||
case CATCH_KEYWORD:
|
||||
case AS_KEYWORD:
|
||||
case AS_QUEST:
|
||||
case AS_BANG:
|
||||
case ASYNC_KEYWORD:
|
||||
case CUSTOM_OPERATOR:
|
||||
return true;
|
||||
case BANG:
|
||||
default:
|
||||
return false;
|
||||
}
|
||||
}
|
||||
|
||||
#define NON_CONSUMING_CROSS_SEMI_CHAR_COUNT 3
|
||||
const uint32_t NON_CONSUMING_CROSS_SEMI_CHARS[NON_CONSUMING_CROSS_SEMI_CHAR_COUNT] = { '?', ':', '{' };
|
||||
|
||||
/**
|
||||
* All possible results of having performed some sort of parsing.
|
||||
*
|
||||
* A parser can return a result along two dimensions:
|
||||
* 1. Should the scanner continue trying to find another result?
|
||||
* 2. Was some result produced by this parsing attempt?
|
||||
*
|
||||
* These are flattened into a single enum together. When the function returns one of the `TOKEN_FOUND` cases, it
|
||||
* will always populate its `symbol_result` field. When it returns one of the `STOP_PARSING` cases, callers should
|
||||
* immediately return (with the value, if there is one).
|
||||
*/
|
||||
enum ParseDirective {
|
||||
CONTINUE_PARSING_NOTHING_FOUND,
|
||||
CONTINUE_PARSING_TOKEN_FOUND,
|
||||
CONTINUE_PARSING_SLASH_CONSUMED,
|
||||
STOP_PARSING_NOTHING_FOUND,
|
||||
STOP_PARSING_TOKEN_FOUND,
|
||||
STOP_PARSING_END_OF_FILE
|
||||
};
|
||||
|
||||
struct ScannerState {
|
||||
uint32_t ongoing_raw_str_hash_count;
|
||||
};
|
||||
|
||||
void *tree_sitter_swift_external_scanner_create() {
|
||||
return calloc(1, sizeof(struct ScannerState));
|
||||
}
|
||||
|
||||
void tree_sitter_swift_external_scanner_destroy(void *payload) {
|
||||
free(payload);
|
||||
}
|
||||
|
||||
void tree_sitter_swift_external_scanner_reset(void *payload) {
|
||||
struct ScannerState *state = (struct ScannerState *)payload;
|
||||
state->ongoing_raw_str_hash_count = 0;
|
||||
}
|
||||
|
||||
unsigned tree_sitter_swift_external_scanner_serialize(void *payload, char *buffer) {
|
||||
struct ScannerState *state = (struct ScannerState *)payload;
|
||||
uint32_t hash_count = state->ongoing_raw_str_hash_count;
|
||||
buffer[0] = (hash_count >> 24) & 0xff;
|
||||
buffer[1] = (hash_count >> 16) & 0xff;
|
||||
buffer[2] = (hash_count >> 8) & 0xff;
|
||||
buffer[3] = (hash_count) & 0xff;
|
||||
return 4;
|
||||
}
|
||||
|
||||
void tree_sitter_swift_external_scanner_deserialize(
|
||||
void *payload,
|
||||
const char *buffer,
|
||||
unsigned length
|
||||
) {
|
||||
if (length < 4) {
|
||||
return;
|
||||
}
|
||||
|
||||
uint32_t hash_count = (
|
||||
(((uint32_t) buffer[0]) << 24) |
|
||||
(((uint32_t) buffer[1]) << 16) |
|
||||
(((uint32_t) buffer[2]) << 8) |
|
||||
(((uint32_t) buffer[3]))
|
||||
);
|
||||
struct ScannerState *state = (struct ScannerState *)payload;
|
||||
state->ongoing_raw_str_hash_count = hash_count;
|
||||
}
|
||||
|
||||
static void advance(TSLexer *lexer) {
|
||||
lexer->advance(lexer, false);
|
||||
}
|
||||
|
||||
static bool should_treat_as_wspace(int32_t character) {
|
||||
return iswspace(character) || (((int32_t) ';') == character);
|
||||
}
|
||||
|
||||
static int32_t encountered_op_count(bool *encountered_operator) {
|
||||
int32_t encountered = 0;
|
||||
for (int op_idx = 0; op_idx < OPERATOR_COUNT; op_idx++) {
|
||||
if (encountered_operator[op_idx]) {
|
||||
encountered++;
|
||||
}
|
||||
}
|
||||
|
||||
return encountered;
|
||||
}
|
||||
|
||||
static bool any_reserved_ops(uint8_t *encountered_reserved_ops) {
|
||||
for (int op_idx = 0; op_idx < RESERVED_OP_COUNT; op_idx++) {
|
||||
if (encountered_reserved_ops[op_idx] == 2) {
|
||||
return true;
|
||||
}
|
||||
}
|
||||
|
||||
return false;
|
||||
}
|
||||
|
||||
static bool is_legal_custom_operator(
|
||||
int32_t char_idx,
|
||||
int32_t first_char,
|
||||
int32_t cur_char
|
||||
) {
|
||||
bool is_first_char = !char_idx;
|
||||
switch (cur_char) {
|
||||
case '=':
|
||||
case '-':
|
||||
case '+':
|
||||
case '!':
|
||||
case '%':
|
||||
case '<':
|
||||
case '>':
|
||||
case '&':
|
||||
case '|':
|
||||
case '^':
|
||||
case '?':
|
||||
case '~':
|
||||
return true;
|
||||
case '.':
|
||||
// Grammar allows `.` for any operator that starts with `.`
|
||||
return is_first_char || first_char == '.';
|
||||
case '*':
|
||||
case '/':
|
||||
// Not listed in the grammar, but `/*` and `//` can't be the start of an operator since they start comments
|
||||
return char_idx != 1 || first_char != '/';
|
||||
default:
|
||||
if (
|
||||
(cur_char >= 0x00A1 && cur_char <= 0x00A7) ||
|
||||
(cur_char == 0x00A9) ||
|
||||
(cur_char == 0x00AB) ||
|
||||
(cur_char == 0x00AC) ||
|
||||
(cur_char == 0x00AE) ||
|
||||
(cur_char >= 0x00B0 && cur_char <= 0x00B1) ||
|
||||
(cur_char == 0x00B6) ||
|
||||
(cur_char == 0x00BB) ||
|
||||
(cur_char == 0x00BF) ||
|
||||
(cur_char == 0x00D7) ||
|
||||
(cur_char == 0x00F7) ||
|
||||
(cur_char >= 0x2016 && cur_char <= 0x2017) ||
|
||||
(cur_char >= 0x2020 && cur_char <= 0x2027) ||
|
||||
(cur_char >= 0x2030 && cur_char <= 0x203E) ||
|
||||
(cur_char >= 0x2041 && cur_char <= 0x2053) ||
|
||||
(cur_char >= 0x2055 && cur_char <= 0x205E) ||
|
||||
(cur_char >= 0x2190 && cur_char <= 0x23FF) ||
|
||||
(cur_char >= 0x2500 && cur_char <= 0x2775) ||
|
||||
(cur_char >= 0x2794 && cur_char <= 0x2BFF) ||
|
||||
(cur_char >= 0x2E00 && cur_char <= 0x2E7F) ||
|
||||
(cur_char >= 0x3001 && cur_char <= 0x3003) ||
|
||||
(cur_char >= 0x3008 && cur_char <= 0x3020) ||
|
||||
(cur_char == 0x3030)
|
||||
) {
|
||||
return true;
|
||||
} else if (
|
||||
(cur_char >= 0x0300 && cur_char <= 0x036f) ||
|
||||
(cur_char >= 0x1DC0 && cur_char <= 0x1DFF) ||
|
||||
(cur_char >= 0x20D0 && cur_char <= 0x20FF) ||
|
||||
(cur_char >= 0xFE00 && cur_char <= 0xFE0F) ||
|
||||
(cur_char >= 0xFE20 && cur_char <= 0xFE2F) ||
|
||||
(cur_char >= 0xE0100 && cur_char <= 0xE01EF)
|
||||
) {
|
||||
return !is_first_char;
|
||||
} else {
|
||||
return false;
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
static bool eat_operators(
|
||||
TSLexer *lexer,
|
||||
const bool *valid_symbols,
|
||||
bool mark_end,
|
||||
const int32_t prior_char,
|
||||
enum TokenType *symbol_result
|
||||
) {
|
||||
bool possible_operators[OPERATOR_COUNT];
|
||||
uint8_t reserved_operators[RESERVED_OP_COUNT];
|
||||
for (int op_idx = 0; op_idx < OPERATOR_COUNT; op_idx++) {
|
||||
possible_operators[op_idx] = valid_symbols[OP_SYMBOLS[op_idx]] && (!prior_char || OPERATORS[op_idx][0] == prior_char);
|
||||
}
|
||||
for (int op_idx = 0; op_idx < RESERVED_OP_COUNT; op_idx++) {
|
||||
reserved_operators[op_idx] = !prior_char || RESERVED_OPS[op_idx][0] == prior_char;
|
||||
}
|
||||
|
||||
bool possible_custom_operator = valid_symbols[CUSTOM_OPERATOR];
|
||||
int32_t first_char = prior_char ? prior_char : lexer->lookahead;
|
||||
int32_t last_examined_char = first_char;
|
||||
|
||||
int32_t str_idx = prior_char ? 1 : 0;
|
||||
int32_t full_match = -1;
|
||||
while(true) {
|
||||
for (int op_idx = 0; op_idx < OPERATOR_COUNT; op_idx++) {
|
||||
if (!possible_operators[op_idx]) {
|
||||
continue;
|
||||
}
|
||||
|
||||
if (OPERATORS[op_idx][str_idx] == '\0') {
|
||||
// Make sure that the operator is allowed to have the next character as its lookahead.
|
||||
enum IllegalTerminatorGroup illegal_terminators = OP_ILLEGAL_TERMINATORS[op_idx];
|
||||
switch (lexer->lookahead) {
|
||||
// See "Operators":
|
||||
// https://docs.swift.org/swift-book/ReferenceManual/LexicalStructure.html#ID418
|
||||
case '/':
|
||||
case '=':
|
||||
case '-':
|
||||
case '+':
|
||||
case '!':
|
||||
case '*':
|
||||
case '%':
|
||||
case '<':
|
||||
case '>':
|
||||
case '&':
|
||||
case '|':
|
||||
case '^':
|
||||
case '?':
|
||||
case '~':
|
||||
if (illegal_terminators == OPERATOR_SYMBOLS) {
|
||||
break;
|
||||
} // Otherwise, intentionally fall through to the OPERATOR_OR_DOT case
|
||||
// fall through
|
||||
case '.':
|
||||
if (illegal_terminators == OPERATOR_OR_DOT) {
|
||||
break;
|
||||
} // Otherwise, fall through to DEFAULT which checks its groups directly
|
||||
// fall through
|
||||
default:
|
||||
if (iswalnum(lexer->lookahead) && illegal_terminators == ALPHANUMERIC) {
|
||||
break;
|
||||
}
|
||||
|
||||
if (!iswspace(lexer->lookahead) && illegal_terminators == NON_WHITESPACE) {
|
||||
break;
|
||||
}
|
||||
|
||||
full_match = op_idx;
|
||||
if (mark_end) {
|
||||
lexer->mark_end(lexer);
|
||||
}
|
||||
}
|
||||
|
||||
possible_operators[op_idx] = false;
|
||||
continue;
|
||||
}
|
||||
|
||||
if (OPERATORS[op_idx][str_idx] != lexer->lookahead) {
|
||||
possible_operators[op_idx] = false;
|
||||
continue;
|
||||
}
|
||||
}
|
||||
|
||||
for (int op_idx = 0; op_idx < RESERVED_OP_COUNT; op_idx++) {
|
||||
if (!reserved_operators[op_idx]) {
|
||||
continue;
|
||||
}
|
||||
|
||||
if (RESERVED_OPS[op_idx][str_idx] == '\0') {
|
||||
reserved_operators[op_idx] = 0;
|
||||
continue;
|
||||
}
|
||||
|
||||
if (RESERVED_OPS[op_idx][str_idx] != lexer->lookahead) {
|
||||
reserved_operators[op_idx] = 0;
|
||||
continue;
|
||||
}
|
||||
|
||||
if (RESERVED_OPS[op_idx][str_idx + 1] == '\0') {
|
||||
reserved_operators[op_idx] = 2;
|
||||
continue;
|
||||
}
|
||||
}
|
||||
|
||||
possible_custom_operator = possible_custom_operator && is_legal_custom_operator(
|
||||
str_idx,
|
||||
first_char,
|
||||
lexer->lookahead
|
||||
);
|
||||
|
||||
uint32_t encountered_ops = encountered_op_count(possible_operators);
|
||||
if (encountered_ops == 0) {
|
||||
if (!possible_custom_operator) {
|
||||
break;
|
||||
} else if (mark_end && full_match == -1) {
|
||||
lexer->mark_end(lexer);
|
||||
}
|
||||
}
|
||||
|
||||
last_examined_char = lexer->lookahead;
|
||||
lexer->advance(lexer, false);
|
||||
str_idx += 1;
|
||||
|
||||
if (encountered_ops == 0 && !is_legal_custom_operator(
|
||||
str_idx,
|
||||
first_char,
|
||||
lexer->lookahead
|
||||
)) {
|
||||
break;
|
||||
}
|
||||
}
|
||||
|
||||
if (full_match != -1) {
|
||||
// We have a match -- first see if that match has a symbol that suppresses it. For example, in `try!`, we do not
|
||||
// want to emit the `!` as a symbol in our scanner, because we want the parser to have the chance to parse it as
|
||||
// an immediate token.
|
||||
uint64_t suppressing_symbols = OP_SYMBOL_SUPPRESSOR[full_match];
|
||||
if (suppressing_symbols) {
|
||||
for (uint64_t suppressor = 0; suppressor < TOKEN_COUNT; suppressor++) {
|
||||
if (!(suppressing_symbols & 1ULL << suppressor)) {
|
||||
continue;
|
||||
}
|
||||
|
||||
// The suppressing symbol is valid in this position, so skip it.
|
||||
if (valid_symbols[suppressor]) {
|
||||
return false;
|
||||
}
|
||||
}
|
||||
}
|
||||
*symbol_result = OP_SYMBOLS[full_match];
|
||||
return true;
|
||||
}
|
||||
|
||||
if (possible_custom_operator && !any_reserved_ops(reserved_operators)) {
|
||||
if ((last_examined_char != '<' || iswspace(lexer->lookahead)) && mark_end) {
|
||||
lexer->mark_end(lexer);
|
||||
}
|
||||
*symbol_result = CUSTOM_OPERATOR;
|
||||
return true;
|
||||
}
|
||||
|
||||
return false;
|
||||
}
|
||||
|
||||
static enum ParseDirective eat_comment(
|
||||
TSLexer *lexer,
|
||||
const bool *valid_symbols,
|
||||
bool mark_end,
|
||||
enum TokenType *symbol_result
|
||||
) {
|
||||
if (lexer->lookahead != '/') {
|
||||
return CONTINUE_PARSING_NOTHING_FOUND;
|
||||
}
|
||||
|
||||
advance(lexer);
|
||||
|
||||
if (lexer->lookahead != '*') {
|
||||
return CONTINUE_PARSING_SLASH_CONSUMED;
|
||||
}
|
||||
|
||||
advance(lexer);
|
||||
|
||||
bool after_star = false;
|
||||
unsigned nesting_depth = 1;
|
||||
for (;;) {
|
||||
switch (lexer->lookahead) {
|
||||
case '\0':
|
||||
return STOP_PARSING_END_OF_FILE;
|
||||
case '*':
|
||||
advance(lexer);
|
||||
after_star = true;
|
||||
break;
|
||||
case '/':
|
||||
if (after_star) {
|
||||
advance(lexer);
|
||||
after_star = false;
|
||||
nesting_depth--;
|
||||
if (nesting_depth == 0) {
|
||||
if (mark_end) {
|
||||
lexer->mark_end(lexer);
|
||||
}
|
||||
*symbol_result = BLOCK_COMMENT;
|
||||
return STOP_PARSING_TOKEN_FOUND;
|
||||
}
|
||||
} else {
|
||||
advance(lexer);
|
||||
after_star = false;
|
||||
if (lexer->lookahead == '*') {
|
||||
nesting_depth++;
|
||||
advance(lexer);
|
||||
}
|
||||
}
|
||||
break;
|
||||
default:
|
||||
advance(lexer);
|
||||
after_star = false;
|
||||
break;
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
static enum ParseDirective eat_whitespace(
|
||||
TSLexer *lexer,
|
||||
const bool *valid_symbols,
|
||||
enum TokenType *symbol_result
|
||||
) {
|
||||
enum ParseDirective ws_directive = CONTINUE_PARSING_NOTHING_FOUND;
|
||||
bool semi_is_valid = valid_symbols[IMPLICIT_SEMI] && valid_symbols[EXPLICIT_SEMI];
|
||||
uint32_t lookahead;
|
||||
while (should_treat_as_wspace(lookahead = lexer->lookahead)) {
|
||||
if (lookahead == ';') {
|
||||
if (semi_is_valid) {
|
||||
ws_directive = STOP_PARSING_TOKEN_FOUND;
|
||||
lexer->advance(lexer, false);
|
||||
}
|
||||
|
||||
break;
|
||||
}
|
||||
|
||||
lexer->advance(lexer, true);
|
||||
|
||||
lexer->mark_end(lexer);
|
||||
|
||||
if (ws_directive == CONTINUE_PARSING_NOTHING_FOUND && (lookahead == '\n' || lookahead == '\r')) {
|
||||
ws_directive = CONTINUE_PARSING_TOKEN_FOUND;
|
||||
}
|
||||
}
|
||||
|
||||
enum ParseDirective any_comment = CONTINUE_PARSING_NOTHING_FOUND;
|
||||
if (ws_directive == CONTINUE_PARSING_TOKEN_FOUND && lookahead == '/') {
|
||||
bool has_seen_single_comment = false;
|
||||
while (lexer->lookahead == '/') {
|
||||
// It's possible that this is a comment - start an exploratory mission to find out, and if it is, look for what
|
||||
// comes after it. We care about what comes after it for the purpose of suppressing the newline.
|
||||
|
||||
enum TokenType multiline_comment_result;
|
||||
any_comment = eat_comment(lexer, valid_symbols, /* mark_end */ false, &multiline_comment_result);
|
||||
if (any_comment == STOP_PARSING_TOKEN_FOUND) {
|
||||
// This is a multiline comment. This scanner should be parsing those, so we might want to bail out and
|
||||
// emit it instead. However, we only want to do that if we haven't advanced through a _single_ line
|
||||
// comment on the way - otherwise that will get lumped into this.
|
||||
if (!has_seen_single_comment) {
|
||||
lexer->mark_end(lexer);
|
||||
*symbol_result = multiline_comment_result;
|
||||
return STOP_PARSING_TOKEN_FOUND;
|
||||
}
|
||||
} else if (any_comment == STOP_PARSING_END_OF_FILE) {
|
||||
return STOP_PARSING_END_OF_FILE;
|
||||
} else if (any_comment == CONTINUE_PARSING_SLASH_CONSUMED) {
|
||||
// We accidentally ate a slash -- we should actually bail out, say we saw nothing, and let the next pass
|
||||
// take it from after the newline.
|
||||
return CONTINUE_PARSING_SLASH_CONSUMED;
|
||||
} else if (lexer->lookahead == '/') {
|
||||
// There wasn't a multiline comment, which we know means that the comment parser ate its `/` and then
|
||||
// bailed out. If it had seen anything comment-like after that first `/` it would have continued going
|
||||
// and eventually had a well-formed comment or an EOF. Thus, if we're currently looking at a `/`, it's
|
||||
// the second one of those and it means we have a single-line comment.
|
||||
has_seen_single_comment = true;
|
||||
while (lexer->lookahead != '\n' && lexer->lookahead != '\0') {
|
||||
lexer->advance(lexer, true);
|
||||
}
|
||||
} else if (iswspace(lexer->lookahead)) {
|
||||
// We didn't see any type of comment - in fact, we saw an operator that we don't normally treat as an
|
||||
// operator. Still, this is a reason to stop parsing.
|
||||
return STOP_PARSING_NOTHING_FOUND;
|
||||
}
|
||||
|
||||
// If we skipped through some comment, we're at whitespace now, so advance.
|
||||
while(iswspace(lexer->lookahead)) {
|
||||
any_comment = CONTINUE_PARSING_NOTHING_FOUND; // We're advancing, so clear out the comment
|
||||
lexer->advance(lexer, true);
|
||||
}
|
||||
}
|
||||
|
||||
enum TokenType operator_result;
|
||||
bool saw_operator = eat_operators(
|
||||
lexer,
|
||||
valid_symbols,
|
||||
/* mark_end */ false,
|
||||
'\0',
|
||||
&operator_result
|
||||
);
|
||||
if (saw_operator) {
|
||||
// The operator we saw should suppress the newline, so bail out.
|
||||
return STOP_PARSING_NOTHING_FOUND;
|
||||
} else {
|
||||
// Promote the implicit newline to an explicit one so we don't check for operators again.
|
||||
*symbol_result = IMPLICIT_SEMI;
|
||||
ws_directive = STOP_PARSING_TOKEN_FOUND;
|
||||
}
|
||||
}
|
||||
|
||||
// Let's consume operators that can live after a "semicolon" style newline. Before we do that, though, we want to
|
||||
// check for a set of characters that we do not consume, but that still suppress the semi.
|
||||
if (ws_directive == CONTINUE_PARSING_TOKEN_FOUND) {
|
||||
for (int i = 0; i < NON_CONSUMING_CROSS_SEMI_CHAR_COUNT; i++) {
|
||||
if (NON_CONSUMING_CROSS_SEMI_CHARS[i] == lookahead) {
|
||||
return CONTINUE_PARSING_NOTHING_FOUND;
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
if (semi_is_valid && ws_directive != CONTINUE_PARSING_NOTHING_FOUND) {
|
||||
*symbol_result = lookahead == ';' ? EXPLICIT_SEMI : IMPLICIT_SEMI;
|
||||
return ws_directive;
|
||||
}
|
||||
|
||||
return CONTINUE_PARSING_NOTHING_FOUND;
|
||||
}
|
||||
|
||||
#define DIRECTIVE_COUNT 4
|
||||
const char* DIRECTIVES[OPERATOR_COUNT] = {
|
||||
"if",
|
||||
"elseif",
|
||||
"else",
|
||||
"endif"
|
||||
};
|
||||
|
||||
const enum TokenType DIRECTIVE_SYMBOLS[DIRECTIVE_COUNT] = {
|
||||
DIRECTIVE_IF,
|
||||
DIRECTIVE_ELSEIF,
|
||||
DIRECTIVE_ELSE,
|
||||
DIRECTIVE_ENDIF
|
||||
};
|
||||
|
||||
static enum TokenType find_possible_compiler_directive(TSLexer *lexer) {
|
||||
bool possible_directives[DIRECTIVE_COUNT];
|
||||
for (int dir_idx = 0; dir_idx < DIRECTIVE_COUNT; dir_idx++) {
|
||||
possible_directives[dir_idx] = true;
|
||||
}
|
||||
|
||||
int32_t str_idx = 0;
|
||||
int32_t full_match = -1;
|
||||
while(true) {
|
||||
for (int dir_idx = 0; dir_idx < DIRECTIVE_COUNT; dir_idx++) {
|
||||
if (!possible_directives[dir_idx]) {
|
||||
continue;
|
||||
}
|
||||
|
||||
uint8_t expected_char = DIRECTIVES[dir_idx][str_idx];
|
||||
if (expected_char == '\0') {
|
||||
full_match = dir_idx;
|
||||
lexer->mark_end(lexer);
|
||||
}
|
||||
|
||||
if (expected_char != lexer->lookahead) {
|
||||
possible_directives[dir_idx] = false;
|
||||
continue;
|
||||
}
|
||||
}
|
||||
|
||||
uint8_t match_count = 0;
|
||||
for (int dir_idx = 0; dir_idx < DIRECTIVE_COUNT; dir_idx += 1) {
|
||||
if (possible_directives[dir_idx]) {
|
||||
match_count += 1;
|
||||
}
|
||||
}
|
||||
|
||||
if (match_count == 0) {
|
||||
break;
|
||||
}
|
||||
|
||||
lexer->advance(lexer, false);
|
||||
str_idx += 1;
|
||||
}
|
||||
|
||||
if (full_match == -1) {
|
||||
// No compiler directive found, so just match the starting symbol
|
||||
return HASH_SYMBOL;
|
||||
}
|
||||
|
||||
return DIRECTIVE_SYMBOLS[full_match];
|
||||
}
|
||||
|
||||
static bool eat_raw_str_part(
|
||||
struct ScannerState *state,
|
||||
TSLexer *lexer,
|
||||
const bool *valid_symbols,
|
||||
enum TokenType *symbol_result
|
||||
) {
|
||||
uint32_t hash_count = state->ongoing_raw_str_hash_count;
|
||||
if (!valid_symbols[RAW_STR_PART]) {
|
||||
return false;
|
||||
} else if (hash_count == 0) {
|
||||
// If this is a raw_str_part, it's the first one - look for hashes
|
||||
while (lexer->lookahead == '#') {
|
||||
hash_count += 1;
|
||||
advance(lexer);
|
||||
}
|
||||
|
||||
if (hash_count == 0) {
|
||||
return false;
|
||||
}
|
||||
|
||||
if (lexer->lookahead == '"') {
|
||||
advance(lexer);
|
||||
} else if (hash_count == 1) {
|
||||
lexer->mark_end(lexer);
|
||||
*symbol_result = find_possible_compiler_directive(lexer);
|
||||
return true;
|
||||
} else {
|
||||
return false;
|
||||
}
|
||||
|
||||
} else if (valid_symbols[RAW_STR_CONTINUING_INDICATOR]) {
|
||||
// This is the end of an interpolation - now it's another raw_str_part. This is a synthetic
|
||||
// marker to tell us that the grammar just consumed a `(` symbol to close a raw
|
||||
// interpolation (since we don't want to fire on every `(` in existence). We don't have
|
||||
// anything to do except continue.
|
||||
} else {
|
||||
return false;
|
||||
}
|
||||
|
||||
// We're in a state where anything other than `hash_count` hash symbols in a row should be eaten
|
||||
// and is part of a string.
|
||||
// The last character _before_ the hashes will tell us what happens next.
|
||||
// Matters are also complicated by the fact that we don't want to consume every character we
|
||||
// visit; if we see a `\#(`, for instance, with the appropriate number of hash symbols, we want
|
||||
// to end our parsing _before_ that sequence. This allows highlighting tools to treat that as a
|
||||
// separate token.
|
||||
while (lexer->lookahead != '\0') {
|
||||
uint8_t last_char = '\0';
|
||||
lexer->mark_end(lexer); // We always want to parse thru the start of the string so far
|
||||
// Advance through anything that isn't a hash symbol, because we want to count those.
|
||||
while (lexer->lookahead != '#' && lexer->lookahead != '\0') {
|
||||
last_char = lexer->lookahead;
|
||||
advance(lexer);
|
||||
if (last_char != '\\' || lexer->lookahead == '\\') {
|
||||
// Mark a new end, but only if we didn't just advance past a `\` symbol, since we
|
||||
// don't want to consume that. Exception: if this is a `\` that happens _right
|
||||
// after_ another `\`, we for some reason _do_ want to consume that, because
|
||||
// apparently that is parsed as a literal `\` followed by something escaped.
|
||||
lexer->mark_end(lexer);
|
||||
}
|
||||
}
|
||||
|
||||
// We hit at least one hash - count them and see if they match.
|
||||
uint32_t current_hash_count = 0;
|
||||
while (lexer->lookahead == '#' && current_hash_count < hash_count) {
|
||||
current_hash_count += 1;
|
||||
advance(lexer);
|
||||
}
|
||||
|
||||
// If we saw exactly the right number of hashes, one of three things is true:
|
||||
// 1. We're trying to interpolate into this string.
|
||||
// 2. The string just ended.
|
||||
// 3. This was just some hash characters doing nothing important.
|
||||
if (current_hash_count == hash_count) {
|
||||
if (last_char == '\\' && lexer->lookahead == '(') {
|
||||
// Interpolation case! Don't consume those chars; they get saved for grammar.js.
|
||||
*symbol_result = RAW_STR_PART;
|
||||
state->ongoing_raw_str_hash_count = hash_count;
|
||||
return true;
|
||||
} else if (last_char == '"') {
|
||||
// The string is finished! Mark the end here, on the very last hash symbol.
|
||||
lexer->mark_end(lexer);
|
||||
*symbol_result = RAW_STR_END_PART;
|
||||
state->ongoing_raw_str_hash_count = 0;
|
||||
return true;
|
||||
}
|
||||
// Nothing special happened - let the string continue.
|
||||
}
|
||||
}
|
||||
|
||||
return false;
|
||||
}
|
||||
|
||||
bool tree_sitter_swift_external_scanner_scan(
|
||||
void *payload,
|
||||
TSLexer *lexer,
|
||||
const bool *valid_symbols
|
||||
) {
|
||||
// Figure out our scanner state
|
||||
struct ScannerState *state = (struct ScannerState *)payload;
|
||||
|
||||
// Consume any whitespace at the start.
|
||||
enum TokenType ws_result;
|
||||
enum ParseDirective ws_directive = eat_whitespace(lexer, valid_symbols, &ws_result);
|
||||
if (ws_directive == STOP_PARSING_TOKEN_FOUND) {
|
||||
lexer->result_symbol = ws_result;
|
||||
return true;
|
||||
}
|
||||
|
||||
if (ws_directive == STOP_PARSING_NOTHING_FOUND || ws_directive == STOP_PARSING_END_OF_FILE) {
|
||||
return false;
|
||||
}
|
||||
|
||||
bool has_ws_result = (ws_directive == CONTINUE_PARSING_TOKEN_FOUND);
|
||||
|
||||
// Now consume comments (before custom operators so that those aren't treated as comments)
|
||||
enum TokenType comment_result;
|
||||
enum ParseDirective comment = ws_directive == CONTINUE_PARSING_SLASH_CONSUMED ? ws_directive : eat_comment(lexer, valid_symbols, /* mark_end */ true, &comment_result);
|
||||
if (comment == STOP_PARSING_TOKEN_FOUND) {
|
||||
lexer->mark_end(lexer);
|
||||
lexer->result_symbol = comment_result;
|
||||
return true;
|
||||
}
|
||||
|
||||
if (comment == STOP_PARSING_END_OF_FILE) {
|
||||
return false;
|
||||
}
|
||||
// Now consume any operators that might cause our whitespace to be suppressed.
|
||||
enum TokenType operator_result;
|
||||
bool saw_operator = eat_operators(
|
||||
lexer,
|
||||
valid_symbols,
|
||||
/* mark_end */ !has_ws_result,
|
||||
comment == CONTINUE_PARSING_SLASH_CONSUMED ? '/' : '\0',
|
||||
&operator_result
|
||||
);
|
||||
|
||||
if (saw_operator && (!has_ws_result || is_cross_semi_token(operator_result))) {
|
||||
lexer->result_symbol = operator_result;
|
||||
if (has_ws_result) lexer->mark_end(lexer);
|
||||
return true;
|
||||
}
|
||||
|
||||
if (has_ws_result) {
|
||||
// Don't `mark_end`, since we may have advanced through some operators.
|
||||
lexer->result_symbol = ws_result;
|
||||
return true;
|
||||
}
|
||||
|
||||
// NOTE: this will consume any `#` characters it sees, even if it does not find a result. Keep
|
||||
// it at the end so that it doesn't interfere with special literals or selectors!
|
||||
enum TokenType raw_str_result;
|
||||
bool saw_raw_str_part = eat_raw_str_part(state, lexer, valid_symbols, &raw_str_result);
|
||||
if (saw_raw_str_part) {
|
||||
lexer->result_symbol = raw_str_result;
|
||||
return true;
|
||||
}
|
||||
|
||||
return false;
|
||||
}
|
||||
|
||||
@@ -0,0 +1,54 @@
|
||||
#ifndef TREE_SITTER_ALLOC_H_
|
||||
#define TREE_SITTER_ALLOC_H_
|
||||
|
||||
#ifdef __cplusplus
|
||||
extern "C" {
|
||||
#endif
|
||||
|
||||
#include <stdbool.h>
|
||||
#include <stdio.h>
|
||||
#include <stdlib.h>
|
||||
|
||||
// Allow clients to override allocation functions
|
||||
#ifdef TREE_SITTER_REUSE_ALLOCATOR
|
||||
|
||||
extern void *(*ts_current_malloc)(size_t size);
|
||||
extern void *(*ts_current_calloc)(size_t count, size_t size);
|
||||
extern void *(*ts_current_realloc)(void *ptr, size_t size);
|
||||
extern void (*ts_current_free)(void *ptr);
|
||||
|
||||
#ifndef ts_malloc
|
||||
#define ts_malloc ts_current_malloc
|
||||
#endif
|
||||
#ifndef ts_calloc
|
||||
#define ts_calloc ts_current_calloc
|
||||
#endif
|
||||
#ifndef ts_realloc
|
||||
#define ts_realloc ts_current_realloc
|
||||
#endif
|
||||
#ifndef ts_free
|
||||
#define ts_free ts_current_free
|
||||
#endif
|
||||
|
||||
#else
|
||||
|
||||
#ifndef ts_malloc
|
||||
#define ts_malloc malloc
|
||||
#endif
|
||||
#ifndef ts_calloc
|
||||
#define ts_calloc calloc
|
||||
#endif
|
||||
#ifndef ts_realloc
|
||||
#define ts_realloc realloc
|
||||
#endif
|
||||
#ifndef ts_free
|
||||
#define ts_free free
|
||||
#endif
|
||||
|
||||
#endif
|
||||
|
||||
#ifdef __cplusplus
|
||||
}
|
||||
#endif
|
||||
|
||||
#endif // TREE_SITTER_ALLOC_H_
|
||||
@@ -0,0 +1,330 @@
|
||||
#ifndef TREE_SITTER_ARRAY_H_
|
||||
#define TREE_SITTER_ARRAY_H_
|
||||
|
||||
#ifdef __cplusplus
|
||||
extern "C" {
|
||||
#endif
|
||||
|
||||
#include "./alloc.h"
|
||||
|
||||
#include <assert.h>
|
||||
#include <stdbool.h>
|
||||
#include <stdint.h>
|
||||
#include <stdlib.h>
|
||||
#include <string.h>
|
||||
|
||||
#ifdef _MSC_VER
|
||||
#pragma warning(push)
|
||||
#pragma warning(disable : 4101)
|
||||
#elif defined(__GNUC__) || defined(__clang__)
|
||||
#pragma GCC diagnostic push
|
||||
#pragma GCC diagnostic ignored "-Wunused-variable"
|
||||
#endif
|
||||
|
||||
#define Array(T) \
|
||||
struct { \
|
||||
T *contents; \
|
||||
uint32_t size; \
|
||||
uint32_t capacity; \
|
||||
}
|
||||
|
||||
/// Initialize an array.
|
||||
#define array_init(self) \
|
||||
((self)->size = 0, (self)->capacity = 0, (self)->contents = NULL)
|
||||
|
||||
/// Create an empty array.
|
||||
#define array_new() \
|
||||
{ NULL, 0, 0 }
|
||||
|
||||
/// Get a pointer to the element at a given `index` in the array.
|
||||
#define array_get(self, _index) \
|
||||
(assert((uint32_t)(_index) < (self)->size), &(self)->contents[_index])
|
||||
|
||||
/// Get a pointer to the first element in the array.
|
||||
#define array_front(self) array_get(self, 0)
|
||||
|
||||
/// Get a pointer to the last element in the array.
|
||||
#define array_back(self) array_get(self, (self)->size - 1)
|
||||
|
||||
/// Clear the array, setting its size to zero. Note that this does not free any
|
||||
/// memory allocated for the array's contents.
|
||||
#define array_clear(self) ((self)->size = 0)
|
||||
|
||||
/// Reserve `new_capacity` elements of space in the array. If `new_capacity` is
|
||||
/// less than the array's current capacity, this function has no effect.
|
||||
#define array_reserve(self, new_capacity) \
|
||||
((self)->contents = _array__reserve( \
|
||||
(void *)(self)->contents, &(self)->capacity, \
|
||||
array_elem_size(self), new_capacity) \
|
||||
)
|
||||
|
||||
/// Free any memory allocated for this array. Note that this does not free any
|
||||
/// memory allocated for the array's contents.
|
||||
#define array_delete(self) \
|
||||
do { \
|
||||
if ((self)->contents) ts_free((self)->contents); \
|
||||
(self)->contents = NULL; \
|
||||
(self)->size = 0; \
|
||||
(self)->capacity = 0; \
|
||||
} while (0)
|
||||
|
||||
/// Push a new `element` onto the end of the array.
|
||||
#define array_push(self, element) \
|
||||
do { \
|
||||
(self)->contents = _array__grow( \
|
||||
(void *)(self)->contents, (self)->size, &(self)->capacity, \
|
||||
1, array_elem_size(self) \
|
||||
); \
|
||||
(self)->contents[(self)->size++] = (element); \
|
||||
} while(0)
|
||||
|
||||
/// Increase the array's size by `count` elements.
|
||||
/// New elements are zero-initialized.
|
||||
#define array_grow_by(self, count) \
|
||||
do { \
|
||||
if ((count) == 0) break; \
|
||||
(self)->contents = _array__grow( \
|
||||
(self)->contents, (self)->size, &(self)->capacity, \
|
||||
count, array_elem_size(self) \
|
||||
); \
|
||||
memset((self)->contents + (self)->size, 0, (count) * array_elem_size(self)); \
|
||||
(self)->size += (count); \
|
||||
} while (0)
|
||||
|
||||
/// Append all elements from one array to the end of another.
|
||||
#define array_push_all(self, other) \
|
||||
array_extend((self), (other)->size, (other)->contents)
|
||||
|
||||
/// Append `count` elements to the end of the array, reading their values from the
|
||||
/// `contents` pointer.
|
||||
#define array_extend(self, count, other_contents) \
|
||||
(self)->contents = _array__splice( \
|
||||
(void*)(self)->contents, &(self)->size, &(self)->capacity, \
|
||||
array_elem_size(self), (self)->size, 0, count, other_contents \
|
||||
)
|
||||
|
||||
/// Remove `old_count` elements from the array starting at the given `index`. At
|
||||
/// the same index, insert `new_count` new elements, reading their values from the
|
||||
/// `new_contents` pointer.
|
||||
#define array_splice(self, _index, old_count, new_count, new_contents) \
|
||||
(self)->contents = _array__splice( \
|
||||
(void *)(self)->contents, &(self)->size, &(self)->capacity, \
|
||||
array_elem_size(self), _index, old_count, new_count, new_contents \
|
||||
)
|
||||
|
||||
/// Insert one `element` into the array at the given `index`.
|
||||
#define array_insert(self, _index, element) \
|
||||
(self)->contents = _array__splice( \
|
||||
(void *)(self)->contents, &(self)->size, &(self)->capacity, \
|
||||
array_elem_size(self), _index, 0, 1, &(element) \
|
||||
)
|
||||
|
||||
/// Remove one element from the array at the given `index`.
|
||||
#define array_erase(self, _index) \
|
||||
_array__erase((void *)(self)->contents, &(self)->size, array_elem_size(self), _index)
|
||||
|
||||
/// Pop the last element off the array, returning the element by value.
|
||||
#define array_pop(self) ((self)->contents[--(self)->size])
|
||||
|
||||
/// Assign the contents of one array to another, reallocating if necessary.
|
||||
#define array_assign(self, other) \
|
||||
(self)->contents = _array__assign( \
|
||||
(void *)(self)->contents, &(self)->size, &(self)->capacity, \
|
||||
(const void *)(other)->contents, (other)->size, array_elem_size(self) \
|
||||
)
|
||||
|
||||
/// Swap one array with another
|
||||
#define array_swap(self, other) \
|
||||
do { \
|
||||
void *_array_swap_tmp = (void *)(self)->contents; \
|
||||
(self)->contents = (other)->contents; \
|
||||
(other)->contents = _array_swap_tmp; \
|
||||
_array__swap(&(self)->size, &(self)->capacity, \
|
||||
&(other)->size, &(other)->capacity); \
|
||||
} while (0)
|
||||
|
||||
/// Get the size of the array contents
|
||||
#define array_elem_size(self) (sizeof *(self)->contents)
|
||||
|
||||
/// Search a sorted array for a given `needle` value, using the given `compare`
|
||||
/// callback to determine the order.
|
||||
///
|
||||
/// If an existing element is found to be equal to `needle`, then the `index`
|
||||
/// out-parameter is set to the existing value's index, and the `exists`
|
||||
/// out-parameter is set to true. Otherwise, `index` is set to an index where
|
||||
/// `needle` should be inserted in order to preserve the sorting, and `exists`
|
||||
/// is set to false.
|
||||
#define array_search_sorted_with(self, compare, needle, _index, _exists) \
|
||||
_array__search_sorted(self, 0, compare, , needle, _index, _exists)
|
||||
|
||||
/// Search a sorted array for a given `needle` value, using integer comparisons
|
||||
/// of a given struct field (specified with a leading dot) to determine the order.
|
||||
///
|
||||
/// See also `array_search_sorted_with`.
|
||||
#define array_search_sorted_by(self, field, needle, _index, _exists) \
|
||||
_array__search_sorted(self, 0, _compare_int, field, needle, _index, _exists)
|
||||
|
||||
/// Insert a given `value` into a sorted array, using the given `compare`
|
||||
/// callback to determine the order.
|
||||
#define array_insert_sorted_with(self, compare, value) \
|
||||
do { \
|
||||
unsigned _index, _exists; \
|
||||
array_search_sorted_with(self, compare, &(value), &_index, &_exists); \
|
||||
if (!_exists) array_insert(self, _index, value); \
|
||||
} while (0)
|
||||
|
||||
/// Insert a given `value` into a sorted array, using integer comparisons of
|
||||
/// a given struct field (specified with a leading dot) to determine the order.
|
||||
///
|
||||
/// See also `array_search_sorted_by`.
|
||||
#define array_insert_sorted_by(self, field, value) \
|
||||
do { \
|
||||
unsigned _index, _exists; \
|
||||
array_search_sorted_by(self, field, (value) field, &_index, &_exists); \
|
||||
if (!_exists) array_insert(self, _index, value); \
|
||||
} while (0)
|
||||
|
||||
// Private
|
||||
|
||||
// Pointers to individual `Array` fields (rather than the entire `Array` itself)
|
||||
// are passed to the various `_array__*` functions below to address strict aliasing
|
||||
// violations that arises when the _entire_ `Array` struct is passed as `Array(void)*`.
|
||||
//
|
||||
// The `Array` type itself was not altered as a solution in order to avoid breakage
|
||||
// with existing consumers (in particular, parsers with external scanners).
|
||||
|
||||
/// This is not what you're looking for, see `array_erase`.
|
||||
static inline void _array__erase(void* self_contents, uint32_t *size,
|
||||
size_t element_size, uint32_t index) {
|
||||
assert(index < *size);
|
||||
char *contents = (char *)self_contents;
|
||||
memmove(contents + index * element_size, contents + (index + 1) * element_size,
|
||||
(*size - index - 1) * element_size);
|
||||
(*size)--;
|
||||
}
|
||||
|
||||
/// This is not what you're looking for, see `array_reserve`.
|
||||
static inline void *_array__reserve(void *contents, uint32_t *capacity,
|
||||
size_t element_size, uint32_t new_capacity) {
|
||||
void *new_contents = contents;
|
||||
if (new_capacity > *capacity) {
|
||||
if (contents) {
|
||||
new_contents = ts_realloc(contents, new_capacity * element_size);
|
||||
} else {
|
||||
new_contents = ts_malloc(new_capacity * element_size);
|
||||
}
|
||||
*capacity = new_capacity;
|
||||
}
|
||||
return new_contents;
|
||||
}
|
||||
|
||||
/// This is not what you're looking for, see `array_assign`.
|
||||
static inline void *_array__assign(void* self_contents, uint32_t *self_size, uint32_t *self_capacity,
|
||||
const void *other_contents, uint32_t other_size, size_t element_size) {
|
||||
void *new_contents = _array__reserve(self_contents, self_capacity, element_size, other_size);
|
||||
*self_size = other_size;
|
||||
memcpy(new_contents, other_contents, *self_size * element_size);
|
||||
return new_contents;
|
||||
}
|
||||
|
||||
/// This is not what you're looking for, see `array_swap`.
|
||||
static inline void _array__swap(uint32_t *self_size, uint32_t *self_capacity,
|
||||
uint32_t *other_size, uint32_t *other_capacity) {
|
||||
uint32_t tmp_size = *self_size;
|
||||
uint32_t tmp_capacity = *self_capacity;
|
||||
*self_size = *other_size;
|
||||
*self_capacity = *other_capacity;
|
||||
*other_size = tmp_size;
|
||||
*other_capacity = tmp_capacity;
|
||||
}
|
||||
|
||||
/// This is not what you're looking for, see `array_push` or `array_grow_by`.
|
||||
static inline void *_array__grow(void *contents, uint32_t size, uint32_t *capacity,
|
||||
uint32_t count, size_t element_size) {
|
||||
void *new_contents = contents;
|
||||
uint32_t new_size = size + count;
|
||||
if (new_size > *capacity) {
|
||||
uint32_t new_capacity = *capacity * 2;
|
||||
if (new_capacity < 8) new_capacity = 8;
|
||||
if (new_capacity < new_size) new_capacity = new_size;
|
||||
new_contents = _array__reserve(contents, capacity, element_size, new_capacity);
|
||||
}
|
||||
return new_contents;
|
||||
}
|
||||
|
||||
/// This is not what you're looking for, see `array_splice`.
|
||||
static inline void *_array__splice(void *self_contents, uint32_t *size, uint32_t *capacity,
|
||||
size_t element_size,
|
||||
uint32_t index, uint32_t old_count,
|
||||
uint32_t new_count, const void *elements) {
|
||||
uint32_t new_size = *size + new_count - old_count;
|
||||
uint32_t old_end = index + old_count;
|
||||
uint32_t new_end = index + new_count;
|
||||
assert(old_end <= *size);
|
||||
|
||||
void *new_contents = _array__reserve(self_contents, capacity, element_size, new_size);
|
||||
|
||||
char *contents = (char *)new_contents;
|
||||
if (*size > old_end) {
|
||||
memmove(
|
||||
contents + new_end * element_size,
|
||||
contents + old_end * element_size,
|
||||
(*size - old_end) * element_size
|
||||
);
|
||||
}
|
||||
if (new_count > 0) {
|
||||
if (elements) {
|
||||
memcpy(
|
||||
(contents + index * element_size),
|
||||
elements,
|
||||
new_count * element_size
|
||||
);
|
||||
} else {
|
||||
memset(
|
||||
(contents + index * element_size),
|
||||
0,
|
||||
new_count * element_size
|
||||
);
|
||||
}
|
||||
}
|
||||
*size += new_count - old_count;
|
||||
|
||||
return new_contents;
|
||||
}
|
||||
|
||||
/// A binary search routine, based on Rust's `std::slice::binary_search_by`.
|
||||
/// This is not what you're looking for, see `array_search_sorted_with` or `array_search_sorted_by`.
|
||||
#define _array__search_sorted(self, start, compare, suffix, needle, _index, _exists) \
|
||||
do { \
|
||||
*(_index) = start; \
|
||||
*(_exists) = false; \
|
||||
uint32_t size = (self)->size - *(_index); \
|
||||
if (size == 0) break; \
|
||||
int comparison; \
|
||||
while (size > 1) { \
|
||||
uint32_t half_size = size / 2; \
|
||||
uint32_t mid_index = *(_index) + half_size; \
|
||||
comparison = compare(&((self)->contents[mid_index] suffix), (needle)); \
|
||||
if (comparison <= 0) *(_index) = mid_index; \
|
||||
size -= half_size; \
|
||||
} \
|
||||
comparison = compare(&((self)->contents[*(_index)] suffix), (needle)); \
|
||||
if (comparison == 0) *(_exists) = true; \
|
||||
else if (comparison < 0) *(_index) += 1; \
|
||||
} while (0)
|
||||
|
||||
/// Helper macro for the `_sorted_by` routines below. This takes the left (existing)
|
||||
/// parameter by reference in order to work with the generic sorting function above.
|
||||
#define _compare_int(a, b) ((int)*(a) - (int)(b))
|
||||
|
||||
#ifdef _MSC_VER
|
||||
#pragma warning(pop)
|
||||
#elif defined(__GNUC__) || defined(__clang__)
|
||||
#pragma GCC diagnostic pop
|
||||
#endif
|
||||
|
||||
#ifdef __cplusplus
|
||||
}
|
||||
#endif
|
||||
|
||||
#endif // TREE_SITTER_ARRAY_H_
|
||||
@@ -0,0 +1,286 @@
|
||||
#ifndef TREE_SITTER_PARSER_H_
|
||||
#define TREE_SITTER_PARSER_H_
|
||||
|
||||
#ifdef __cplusplus
|
||||
extern "C" {
|
||||
#endif
|
||||
|
||||
#include <stdbool.h>
|
||||
#include <stdint.h>
|
||||
#include <stdlib.h>
|
||||
|
||||
#define ts_builtin_sym_error ((TSSymbol)-1)
|
||||
#define ts_builtin_sym_end 0
|
||||
#define TREE_SITTER_SERIALIZATION_BUFFER_SIZE 1024
|
||||
|
||||
#ifndef TREE_SITTER_API_H_
|
||||
typedef uint16_t TSStateId;
|
||||
typedef uint16_t TSSymbol;
|
||||
typedef uint16_t TSFieldId;
|
||||
typedef struct TSLanguage TSLanguage;
|
||||
typedef struct TSLanguageMetadata {
|
||||
uint8_t major_version;
|
||||
uint8_t minor_version;
|
||||
uint8_t patch_version;
|
||||
} TSLanguageMetadata;
|
||||
#endif
|
||||
|
||||
typedef struct {
|
||||
TSFieldId field_id;
|
||||
uint8_t child_index;
|
||||
bool inherited;
|
||||
} TSFieldMapEntry;
|
||||
|
||||
// Used to index the field and supertype maps.
|
||||
typedef struct {
|
||||
uint16_t index;
|
||||
uint16_t length;
|
||||
} TSMapSlice;
|
||||
|
||||
typedef struct {
|
||||
bool visible;
|
||||
bool named;
|
||||
bool supertype;
|
||||
} TSSymbolMetadata;
|
||||
|
||||
typedef struct TSLexer TSLexer;
|
||||
|
||||
struct TSLexer {
|
||||
int32_t lookahead;
|
||||
TSSymbol result_symbol;
|
||||
void (*advance)(TSLexer *, bool);
|
||||
void (*mark_end)(TSLexer *);
|
||||
uint32_t (*get_column)(TSLexer *);
|
||||
bool (*is_at_included_range_start)(const TSLexer *);
|
||||
bool (*eof)(const TSLexer *);
|
||||
void (*log)(const TSLexer *, const char *, ...);
|
||||
};
|
||||
|
||||
typedef enum {
|
||||
TSParseActionTypeShift,
|
||||
TSParseActionTypeReduce,
|
||||
TSParseActionTypeAccept,
|
||||
TSParseActionTypeRecover,
|
||||
} TSParseActionType;
|
||||
|
||||
typedef union {
|
||||
struct {
|
||||
uint8_t type;
|
||||
TSStateId state;
|
||||
bool extra;
|
||||
bool repetition;
|
||||
} shift;
|
||||
struct {
|
||||
uint8_t type;
|
||||
uint8_t child_count;
|
||||
TSSymbol symbol;
|
||||
int16_t dynamic_precedence;
|
||||
uint16_t production_id;
|
||||
} reduce;
|
||||
uint8_t type;
|
||||
} TSParseAction;
|
||||
|
||||
typedef struct {
|
||||
uint16_t lex_state;
|
||||
uint16_t external_lex_state;
|
||||
} TSLexMode;
|
||||
|
||||
typedef struct {
|
||||
uint16_t lex_state;
|
||||
uint16_t external_lex_state;
|
||||
uint16_t reserved_word_set_id;
|
||||
} TSLexerMode;
|
||||
|
||||
typedef union {
|
||||
TSParseAction action;
|
||||
struct {
|
||||
uint8_t count;
|
||||
bool reusable;
|
||||
} entry;
|
||||
} TSParseActionEntry;
|
||||
|
||||
typedef struct {
|
||||
int32_t start;
|
||||
int32_t end;
|
||||
} TSCharacterRange;
|
||||
|
||||
struct TSLanguage {
|
||||
uint32_t abi_version;
|
||||
uint32_t symbol_count;
|
||||
uint32_t alias_count;
|
||||
uint32_t token_count;
|
||||
uint32_t external_token_count;
|
||||
uint32_t state_count;
|
||||
uint32_t large_state_count;
|
||||
uint32_t production_id_count;
|
||||
uint32_t field_count;
|
||||
uint16_t max_alias_sequence_length;
|
||||
const uint16_t *parse_table;
|
||||
const uint16_t *small_parse_table;
|
||||
const uint32_t *small_parse_table_map;
|
||||
const TSParseActionEntry *parse_actions;
|
||||
const char * const *symbol_names;
|
||||
const char * const *field_names;
|
||||
const TSMapSlice *field_map_slices;
|
||||
const TSFieldMapEntry *field_map_entries;
|
||||
const TSSymbolMetadata *symbol_metadata;
|
||||
const TSSymbol *public_symbol_map;
|
||||
const uint16_t *alias_map;
|
||||
const TSSymbol *alias_sequences;
|
||||
const TSLexerMode *lex_modes;
|
||||
bool (*lex_fn)(TSLexer *, TSStateId);
|
||||
bool (*keyword_lex_fn)(TSLexer *, TSStateId);
|
||||
TSSymbol keyword_capture_token;
|
||||
struct {
|
||||
const bool *states;
|
||||
const TSSymbol *symbol_map;
|
||||
void *(*create)(void);
|
||||
void (*destroy)(void *);
|
||||
bool (*scan)(void *, TSLexer *, const bool *symbol_whitelist);
|
||||
unsigned (*serialize)(void *, char *);
|
||||
void (*deserialize)(void *, const char *, unsigned);
|
||||
} external_scanner;
|
||||
const TSStateId *primary_state_ids;
|
||||
const char *name;
|
||||
const TSSymbol *reserved_words;
|
||||
uint16_t max_reserved_word_set_size;
|
||||
uint32_t supertype_count;
|
||||
const TSSymbol *supertype_symbols;
|
||||
const TSMapSlice *supertype_map_slices;
|
||||
const TSSymbol *supertype_map_entries;
|
||||
TSLanguageMetadata metadata;
|
||||
};
|
||||
|
||||
static inline bool set_contains(const TSCharacterRange *ranges, uint32_t len, int32_t lookahead) {
|
||||
uint32_t index = 0;
|
||||
uint32_t size = len - index;
|
||||
while (size > 1) {
|
||||
uint32_t half_size = size / 2;
|
||||
uint32_t mid_index = index + half_size;
|
||||
const TSCharacterRange *range = &ranges[mid_index];
|
||||
if (lookahead >= range->start && lookahead <= range->end) {
|
||||
return true;
|
||||
} else if (lookahead > range->end) {
|
||||
index = mid_index;
|
||||
}
|
||||
size -= half_size;
|
||||
}
|
||||
const TSCharacterRange *range = &ranges[index];
|
||||
return (lookahead >= range->start && lookahead <= range->end);
|
||||
}
|
||||
|
||||
/*
|
||||
* Lexer Macros
|
||||
*/
|
||||
|
||||
#ifdef _MSC_VER
|
||||
#define UNUSED __pragma(warning(suppress : 4101))
|
||||
#else
|
||||
#define UNUSED __attribute__((unused))
|
||||
#endif
|
||||
|
||||
#define START_LEXER() \
|
||||
bool result = false; \
|
||||
bool skip = false; \
|
||||
UNUSED \
|
||||
bool eof = false; \
|
||||
int32_t lookahead; \
|
||||
goto start; \
|
||||
next_state: \
|
||||
lexer->advance(lexer, skip); \
|
||||
start: \
|
||||
skip = false; \
|
||||
lookahead = lexer->lookahead;
|
||||
|
||||
#define ADVANCE(state_value) \
|
||||
{ \
|
||||
state = state_value; \
|
||||
goto next_state; \
|
||||
}
|
||||
|
||||
#define ADVANCE_MAP(...) \
|
||||
{ \
|
||||
static const uint16_t map[] = { __VA_ARGS__ }; \
|
||||
for (uint32_t i = 0; i < sizeof(map) / sizeof(map[0]); i += 2) { \
|
||||
if (map[i] == lookahead) { \
|
||||
state = map[i + 1]; \
|
||||
goto next_state; \
|
||||
} \
|
||||
} \
|
||||
}
|
||||
|
||||
#define SKIP(state_value) \
|
||||
{ \
|
||||
skip = true; \
|
||||
state = state_value; \
|
||||
goto next_state; \
|
||||
}
|
||||
|
||||
#define ACCEPT_TOKEN(symbol_value) \
|
||||
result = true; \
|
||||
lexer->result_symbol = symbol_value; \
|
||||
lexer->mark_end(lexer);
|
||||
|
||||
#define END_STATE() return result;
|
||||
|
||||
/*
|
||||
* Parse Table Macros
|
||||
*/
|
||||
|
||||
#define SMALL_STATE(id) ((id) - LARGE_STATE_COUNT)
|
||||
|
||||
#define STATE(id) id
|
||||
|
||||
#define ACTIONS(id) id
|
||||
|
||||
#define SHIFT(state_value) \
|
||||
{{ \
|
||||
.shift = { \
|
||||
.type = TSParseActionTypeShift, \
|
||||
.state = (state_value) \
|
||||
} \
|
||||
}}
|
||||
|
||||
#define SHIFT_REPEAT(state_value) \
|
||||
{{ \
|
||||
.shift = { \
|
||||
.type = TSParseActionTypeShift, \
|
||||
.state = (state_value), \
|
||||
.repetition = true \
|
||||
} \
|
||||
}}
|
||||
|
||||
#define SHIFT_EXTRA() \
|
||||
{{ \
|
||||
.shift = { \
|
||||
.type = TSParseActionTypeShift, \
|
||||
.extra = true \
|
||||
} \
|
||||
}}
|
||||
|
||||
#define REDUCE(symbol_name, children, precedence, prod_id) \
|
||||
{{ \
|
||||
.reduce = { \
|
||||
.type = TSParseActionTypeReduce, \
|
||||
.symbol = symbol_name, \
|
||||
.child_count = children, \
|
||||
.dynamic_precedence = precedence, \
|
||||
.production_id = prod_id \
|
||||
}, \
|
||||
}}
|
||||
|
||||
#define RECOVER() \
|
||||
{{ \
|
||||
.type = TSParseActionTypeRecover \
|
||||
}}
|
||||
|
||||
#define ACCEPT_INPUT() \
|
||||
{{ \
|
||||
.type = TSParseActionTypeAccept \
|
||||
}}
|
||||
|
||||
#ifdef __cplusplus
|
||||
}
|
||||
#endif
|
||||
|
||||
#endif // TREE_SITTER_PARSER_H_
|
||||
@@ -33,6 +33,12 @@ extern "C" {
|
||||
extern "C" {
|
||||
fn tree_sitter_dart() -> *const ();
|
||||
}
|
||||
// Vendored swift grammar (build.rs-compiled C — crate 0.7.3's src/ with a
|
||||
// one-line scanner fix the crate's own build lacks on Windows; see
|
||||
// grammars/swift and docs/grammars/tree-sitter-swift.md).
|
||||
extern "C" {
|
||||
fn tree_sitter_swift() -> *const ();
|
||||
}
|
||||
|
||||
/// Languages this kernel binary can extract (reported by contractInfo;
|
||||
/// TS-side routing policy decides what actually routes).
|
||||
@@ -65,9 +71,12 @@ pub fn grammar_for(language: &str) -> Option<Language> {
|
||||
// R7b: v0.24.2, the full HTML-interleaving variant — LANGUAGE_PHP,
|
||||
// NEVER LANGUAGE_PHP_ONLY (which errors on leading HTML).
|
||||
"php" => Some(tree_sitter_php::LANGUAGE_PHP.into()),
|
||||
// R7b: crate 0.7.3 — the vendored wasm is built from this crate's own
|
||||
// tarball src/ (table identity by construction; see grammars.ts).
|
||||
"swift" => Some(tree_sitter_swift::LANGUAGE.into()),
|
||||
// R7b: crate 0.7.3's src/, the sources the vendored wasm is built
|
||||
// from (table identity by construction; see grammars.ts), compiled in
|
||||
// build.rs with the scanner's `try!` fix (`1ULL << FAKE_TRY_BANG`).
|
||||
"swift" => {
|
||||
Some(unsafe { tree_sitter_language::LanguageFn::from_raw(tree_sitter_swift) }.into())
|
||||
}
|
||||
// R7b: fwcd 0.3.8, vendored C compiled in build.rs (crate unusable).
|
||||
"kotlin" => {
|
||||
Some(unsafe { tree_sitter_language::LanguageFn::from_raw(tree_sitter_kotlin) }.into())
|
||||
|
||||
@@ -43,6 +43,12 @@ at survey time: exactly 348) ride it (§Dedicated property branch).
|
||||
|
||||
## Grammar prep (NOT staged — land FIRST, before any walker exists)
|
||||
|
||||
> **Update 2026-10-19:** the kernel no longer depends on the crate. It
|
||||
> compiles the same tarball's `src/` from `codegraph-kernel/grammars/swift/`
|
||||
> with a one-line scanner patch, because the crate's MSVC build misread every
|
||||
> `try!` on Windows (`docs/grammars/tree-sitter-swift.md`). The wasm below is
|
||||
> unchanged.
|
||||
|
||||
swift is **not** in `VENDORED_WASM_LANGS` (grammars.ts:291) — production loads
|
||||
`require.resolve('tree-sitter-wasms/out/tree-sitter-swift.wasm')` (mapping
|
||||
`swift: 'tree-sitter-swift.wasm'` at grammars.ts:34; `.swift` → swift at
|
||||
|
||||
@@ -0,0 +1,60 @@
|
||||
# tree-sitter-swift — provenance & the kernel's scanner patch
|
||||
|
||||
The kernel's vendored C (`codegraph-kernel/grammars/swift/`) is the `src/` of
|
||||
the [alex-pinkus/tree-sitter-swift](https://github.com/alex-pinkus/tree-sitter-swift)
|
||||
(MIT) **0.7.3 crates.io tarball** — the same files
|
||||
`src/extraction/wasm/tree-sitter-swift.wasm` is built from (see
|
||||
`docs/design/swift-kernel-port-checklist.md` §Grammar prep for why the
|
||||
tarball and not the tag) — with the patch in `tree-sitter-swift.patch`
|
||||
applied to `src/scanner.c`. The wasm is built from the unpatched tarball.
|
||||
|
||||
| file | upstream sha256 | patched sha256 |
|
||||
|---|---|---|
|
||||
| `src/parser.c` | `d3edff6effe31b9a507f496577407987343b101b23eb7bee7a9b050e8ab5d27a` | unchanged |
|
||||
| `src/scanner.c` | `f3d6271d64f58c39eed544104a70ca2cf9ecbf80c5d900620f1afd38836542cb` | `8acafef098ea4a52ebfc4cdf2fe26706113a22ac8ba2faf7d952755ff350cad6` |
|
||||
|
||||
## What the patch fixes
|
||||
|
||||
The external scanner produces its own `!` token, and steps aside for a `!`
|
||||
written straight after `try` so the parser can read `try!`. It finds those
|
||||
positions through a 64-bit mask whose entry for `!` is built as
|
||||
`1UL << FAKE_TRY_BANG`, bit 32. Where `unsigned long` is 32 bits that shift
|
||||
overflows, and each compiler folds it differently:
|
||||
|
||||
- **MSVC** — both Windows kernels (`x86_64`/`aarch64-pc-windows-msvc`) — folds
|
||||
it to 0 (warning C4293). The scanner never steps aside, so
|
||||
`try! Message2(data: d)` parsed as `try` followed by a call of `!Message2`:
|
||||
the call was named `! Message2` and linked nothing, and a call through a
|
||||
type, `try! Settings.Defaults.load()`, lost its receiver.
|
||||
- **clang for wasm32** (the wasm) folds it to bit 31, `_directive_endif`,
|
||||
which is valid at every expression start — right after `try` too — so the
|
||||
wasm reads `try!` correctly. Its one other effect: a prefix `!` at an
|
||||
expression start is lexed by the parser's own lexer, which leaves an
|
||||
anonymous `!` token under `bang` where the scanner's token is hidden. No
|
||||
walker reads it.
|
||||
- **macOS and Linux** kernels (64-bit `unsigned long`) were always correct.
|
||||
|
||||
The patch is upstream 0.7.4's one-line fix, `1ULL`. Measured on 2026-10-09
|
||||
over 1,172 Swift files (Alamofire, SDWebImage, protobuf, realm-swift,
|
||||
swift-nio, vapor and six smaller projects): the patched native build parses
|
||||
every error-free file to the same tree as the wasm, apart from that anonymous
|
||||
`!`. The 8 files that differ have parse errors on both, and the kernel defers
|
||||
a file with parse errors to the wasm. The unpatched MSVC build misparsed 157
|
||||
error-free files, every one at a `try!`.
|
||||
|
||||
## Rebuild
|
||||
|
||||
```
|
||||
curl -sL https://crates.io/api/v1/crates/tree-sitter-swift/0.7.3/download -o ts-swift-0.7.3.crate
|
||||
tar xzf ts-swift-0.7.3.crate && cd tree-sitter-swift-0.7.3
|
||||
git apply /path/to/codegraph/docs/grammars/tree-sitter-swift.patch
|
||||
cp src/parser.c src/scanner.c /path/to/codegraph/codegraph-kernel/grammars/swift/
|
||||
cp src/tree_sitter/*.h /path/to/codegraph/codegraph-kernel/grammars/swift/tree_sitter/
|
||||
```
|
||||
|
||||
Then rebuild the kernel (`npm run build:kernel`) and run
|
||||
`__tests__/kernel-grammar-parity.test.ts` and `scripts/kernel-parity.mjs
|
||||
--lang swift` on a Swift repo: the wasm and the kernel must parse the same.
|
||||
A bump to a release that carries the fix (0.7.4 or later) can drop the patch
|
||||
and the vendored copy: pin the crate in `codegraph-kernel/Cargo.toml` again
|
||||
and rebuild the wasm from the same tarball, together.
|
||||
@@ -0,0 +1,13 @@
|
||||
--- a/src/scanner.c 2026-10-09 00:00:00
|
||||
+++ b/src/scanner.c 2026-10-09 00:00:00
|
||||
@@ -128,7 +128,9 @@
|
||||
0, // EQ_EQ,
|
||||
0, // PLUS_THEN_WS,
|
||||
0, // MINUS_THEN_WS,
|
||||
- 1UL << FAKE_TRY_BANG, // BANG,
|
||||
+ // 1ULL, as upstream has it from 0.7.4: 1UL << 32 overflows where unsigned
|
||||
+ // long is 32 bits (MSVC), and `try!` lost its `!`. (codegraph patch)
|
||||
+ 1ULL << FAKE_TRY_BANG, // BANG,
|
||||
0, // THROWS_KEYWORD,
|
||||
0, // RETHROWS_KEYWORD,
|
||||
0, // DEFAULT_KEYWORD,
|
||||
@@ -422,11 +422,13 @@ const VENDORED_WASM_LANGS: ReadonlySet<GrammarLanguage> = new Set([
|
||||
// R7b (Swift kernel port prep): tree-sitter-swift crate 0.7.3. Built from
|
||||
// the CRATE TARBALL's src/ (NOT a tag sha-match: alex-pinkus keeps
|
||||
// generated files off main and the 0.7.3-with-generated-files tag ships an
|
||||
// older ABI-14 generation; grammar.json rules are JSON-equal, and the crate
|
||||
// tarball is byte-for-byte what the kernel's cargo build compiles — table
|
||||
// identity by construction). Replaces the ^0.4.0 tree-sitter-wasms build
|
||||
// (ABI 13 → 15). NOT graph-neutral — delta is error-set membership only;
|
||||
// classified list in the swift checklist doc.
|
||||
// older ABI-14 generation; grammar.json rules are JSON-equal, and the
|
||||
// kernel compiles the same tarball's src/ (codegraph-kernel/grammars/swift)
|
||||
// — table identity by construction). Replaces the ^0.4.0 tree-sitter-wasms
|
||||
// build (ABI 13 → 15). NOT graph-neutral — delta is error-set membership
|
||||
// only; classified list in the swift checklist doc. The kernel's copy
|
||||
// carries docs/grammars/tree-sitter-swift.patch (`try!` on Windows); this
|
||||
// wasm doesn't need it — see that doc's .md.
|
||||
'swift',
|
||||
// R7b (Kotlin kernel port prep): fwcd tree-sitter-kotlin 0.3.8 (tag
|
||||
// e1a2d5a), parser.c/scanner.c sha-matched crate↔tag; behavior-IDENTICAL
|
||||
|
||||
Reference in New Issue
Block a user